Flatpak is a sandboxed Linux application package manager, allowing individual applications to maintain a degree of isolation from the host system and libraries. This document describes Flatpak’s core architecture and trade-offs, with a focus on the embedded use case.

Core Concepts

Applications packaged via Flatpak are run in a sandboxed environment that is independent from the host system, i.e. the rootfs that they see is not the host’s and will have an independent set of libraries. This means that the applications are portable across host systems and largely self-contained, with their dependencies inside.

However, manually bundling all of an application’s dependencies with it would put a large maintenance burden on the individual packagers. Thus, Flatpak applications use shared runtimes, where a runtime is a common set of libraries and development tools that multiple installed Flatpaks can all share and reuse. Runtimes will include the majority of security-critical libraries, resulting in the burden of maintenance being placed primarily on the runtime maintainers, not the application maintainers. More particularly, this means that any security or bug fixes in a runtime only need to be changed once, compared to having to update each application. That being said, any application dependencies that are not in the base runtimes will still be bundled in manually into the individual app.

It is worth noting that, in the case where dependencies need to be bundled inside the application itself, there are tools to assist in automatically keeping the dependencies up-to-date.

In general, every runtime will have two variants: the “platform” (used to run the application inside) and the “SDK” (used to build the application). This ensures that the runtimes used by the actual applications to function do not need to contain the development tools needed to build said applications, meaning that any licensing constraints or large tools will not be present on end-user devices.

The duality between runtimes and apps results in the sandboxed Flatpak rootfs having the following filesystem layout:

  • /app contains the application’s files.
  • /usr contains the runtime’s files, and it may not be added to by an individual application.

This split does occassionally result in some difficulties in compiling applications or libraries, as many build tools tend to assume that /usr is the default installation directory and that its contents are always mutable.

There is a special type of runtime known as an extension, which can be used to add new files to the filesystem of an application or runtime. An application or runtime can declare the extension names that it supports, as well as where the extension’s files should be placed. At runtime, Flatpak will look for any extensions matching the names declared, and its files will be mounted at the appropriate locations. Extensions are generally useful to add optional functionality to a Flatpak app, such as features that take up too much disk space to bundle by default, or parts of an application that need to function differently on different devices. SDKs also have their own special type of extension, “SDK extensions”, which are simply extensions that are mounted at build time, usually containing some extra tools or libraries that do not fit in the standard SDK.

Note that both Flatpak applications and runtimes must use a “reverse DNS” naming scheme, similar in style to Java. An application’s desktop files, icons, and D-Bus services must all start with the rDNS name of the application. However, no other files inside need to follow that restriction, and flatpak-builder (described below in “Building”) can automatically rename the desktop files and icons to follow the rDNS names.

As a concrete example, consider the Chromium Flatpak, org.chromium.Chromium. It uses the runtime org.freedesktop.Platform, and org.freedesktop.Sdk is used for building. org.freedesktop.Platform contains the majority of the runtime dependencies Chromium needs, and org.freedesktop.Sdk includes the compilers and headers needed to build Chromium. As a result, we only need to bundle a small handful of extra dependencies that Chromium needs.

It additionally uses three different SDK extensions, for LLVM 12, Node, and OpenJDK. These all get mounted onto /usr/lib/sdk, where they can then be used at build time.

Selecting and Creating Runtimes

Two reference runtimes for Apertis are in development:

  • org.apertis.headless.Platform and org.apertis.headless.Sdk, for headless applications.
  • org.apertis.hmi.Platform and org.apertis.hmi.Sdk, for HMI applications.

These runtimes will include a standard set of libraries which are expected to be commonly used with reference Apertis systems for the two targeted use-cases (hmi and headless). There are a few circumstances where the reference runtimes may not be suitable, however:

  • A large amount of applications may be sharing a significant amount of libraries, thus it would be beneificial to have a runtime available with the needed libraries already added.
  • The reference runtimes may be too large for certain runtime environments, due to containing functionality that the target applications do not need.

In these cases, custom runtimes can be created. It is important to note that creating a runtime, even based on the existing Apertis ones, has some overhead. Because a runtime is acting as the “root filesystem” for applications using it, the same type of maintenance done for the host system needs to be done for the runtimes:

  • New updates to the libraries inside need to be pushed out promptly, as there may be critical security fixes inside the updates. As a result, having some form of automated builds would be important.
  • Certain packages may need various forms of custom configuration to work well inside a Flatpak environment; for instance, fontconfig needs custom configuration files to adjust the cache directory and read the host’s fonts.

On the flip side, as the runtimes are seperated from host system, it should be easier to update them in the field, as they do not have a direct impact on the host.

As a result, it is generally recommended to err on the side of fewer custom runtimes, and only create new ones when there are substantial benefits to be gained. In particular, when a large set of application share common libraries (such as the UI toolit), it may make sense to create a custom runtime to match.

Sandboxing

Flatpak applications have two separate permission models: “static” and “dynamic” permissions. Dynamic permissions are added at runtime, but the details of that are tied to portals, which will be discussed below. Static permissions are added at build time, and an application can utilize all of its static permissions immediately at runtime, without any user interaction. These permissions tend to be relatively coarse-grained for simplicity. They are usually identified by the CLI arguments passed to flatpak’s build commands, e.g. --share=ipc.

By default, Flatpaks operate under the following restrictions:

  • Many dangerous system calls, such as ptrace or unshare, are blocked.
  • The application and runtime filesystems are both read-only.
  • /proc is private to the application, i.e. it cannot view any other processes on the system.
  • /sys is read-only.
  • All other filesystems are either tmpfs mounts or read-only views into a few minor pieces of host information (DNS servers and the active timezone, primary), with the exception of a few private, writable directories to store application data.
  • No network access is available.

Some of these can be lifted via static permissions:

  • Access to the network, via --share=network. Note that there is currently no way to restrict the type of network access, such as limiting what ports can be listened on or the type of traffic that is allowed.
  • Access to various paths on the host filesystem, via --filesystem=PATH. Note that the dynamic permissions described above can, at runtime, expose various files or directories at the user’s request, rather than using a static permission.
  • Access to the sockets needed to display graphics (whether Xorg or Wayland) or play audio, via --socket=SOCKET-NAME (e.g. --socket=wayland).
  • Talking to D-Bus services, via --talk-name=SERVICE.NAME for session services or --system-talk-name=SERVICE.NAME for system services.
  • Permission to run 32-bit binaries on a 64-bit system, via --allow=multiarch.
  • Permission to use system calls that can be dangerous but needed for debugging, such as ptrace, is guarded by --allow=devel.

Many of these will be discussed in further detail below.

Note that some restrictions, such as blocking non-debugging-related system calls and the private /proc, cannot be lifted via static permissions.

Portals

There are certain scenarios where static permissions do not work very well, and a more dynamic or fine-grained approach is needed. For instance, some permissions may be temporary in nature, and shouldn’t be available to a Flatpak during its entire runtime. Others may require some host service running to properly check permissions or send out live information. This is where portals come in: fundamentally, a portal is simply a D-Bus API that grants some sort of restricted host access, available to all Flatpaks without needing any specific static permissions. These APIs may perform tasks such as monitoring for low memory, sending notifications, printing, and even spawning nested sandboxes. Some of these require additional dynamic permissions (such as location access and sending notifications), but others require no extra permissions at all (such as monitoring for low memory events). Any dynamic permissions that applications hold are stored in the permission store, which can be modified by the host system via the flatpak CLI at any point. (Full documentation on all the portals is available here.)

Portals that require dynamic permissions will often ask the user explicitly, or they might start out with a permission granted that can later be revoked. Regardless of which scenario this is, however, all dynamic permissions can be managed via the flatpak CLI as mentioned above, removing the need for interactive permission requests on headless systems.

Note that portals often do not need to be called directly. Libraries such as GLib and GTK will automatically use them when possible, and a separate utility library is available for calling most portal APIs not covered by those. In addition, Flatpak runtimes contain custom versions of CLI tools such as xdg-open that will use the portal APIs.

Backends

The default portals need some desktop-specific way to perform tasks such as showing the file chooser or showing interactive permissions dialogs. Portal backends exist for this purpose; they implement generic D-Bus APIs that the main portals can talk to in order to perform these tasks.

As one would expect given the above description, in an embedded world, the importance of these backends is downplayed significantly. In general, applications in this environment will not be performing tasks such as using file chooser dialogs at all, thus those APIs would not need to be implemented. However, other backend APIs may still be applicable to this use case:

  • Taking screenshots.
  • Showing notifications.
  • Exposing UI-related settings from the host system.

Portal Restrictions

It is important to note that most portals provide a more restricted subset of the full functionality associated with their task; for instance, the proxy portal only allows an application to query the proxies for a specific URL, not the full proxy settings that the host may have. In addition, some useful portals, such as for USB or microphone access, simply do not exist yet, and other tasks may be far too specific to have a generic portal shipping with Flatpak for them.

Custom Portals

There are undoubtedly various host resources that an application might need to access, such as various device sensors, that existing portals do not already provide for adequately. In that case, a developer can develop their own D-Bus services running on the host system that exposes APIs under the name org.freedesktop.portal.*, and all Flatpaks will be able to access them. However, it will also be the portal service’s responsibility for determining if a Flatpak should have permission to call it.

If a more restricted approach is desired, a proper portal featuring permission checks may not be needed. A Flatpak application can still talk to any D-Bus service via --talk-name, so it is possible to create generic D-Bus services on the host systems that Flatpaks can also interact with.

Filesystem Access

An application generally has two ways to request filesystem access:

  • Static permissions can grant access to certain paths on the host filesystem. However, it is not possible to see the entire host filesystem with identical directory structure. The “widest” filesystem permission is --filesystem=host, but that only grants access to most of the host filesystem, and not with a consistent filesystem tree. In particular:
    • Many directories right on the rootfs, such as /etc, /usr, /opt, and /var, are all mounted inside the sandbox under /run/host.
    • Access to the data files of other Flatpak applications is not granted.
    • Access to the locations where Flatpak stores the applications themselves is not granted. Both of the latter can be explicitly added via additional --filesystem= permissions.
  • Dynamic permissions can be used to grant access to specific files as needed, though the default APIs for this are desktop-oriented, relying on file open / save dialogs.

Graphics

Applications using Wayland undoubtedly need to talk to the compositor, which requires the static permission --socket=wayland.

Access to the standard Linux DRI/DRM interfaces is guarded by --device=dri, which is thus required to use OpenGL, Vulkan, and DRM APIs. If a proprietary driver uses its own, separate device nodes, then --device=all must be used, since the list of DRI device paths is hardcoded inside Flatpak. A better long-term option would be to add support to Flatpak itself as needed for these non-standard graphics stacks, which has already been done for Mali devices (ARM’s own GPU).

Flatpak has built-in support for being able to download the appropriate graphics drivers for the host system, including proprietary drivers. That being said, the runtime must also be set up explicitly to support this.

Audio/Video

PipeWire

PipeWire plays a key role in video access from within Flatpaks. Portal APIs are available for requesting permission to use the camera or capture the screen, which then return a PipeWire remote that has access to the corresponding nodes.

PulseAudio

For audio, --socket=pulseaudio will grant permission to talk to either ALSA or PulseAudio (the permission name only mentioning the latter is a historical artifact). Note that, access to PulseAudio results in full access to all audio streams, including microphone access. If this should be constrained, PipeWire’s PulseAudio compatibility and WirePlumber can be used together to manage the audio permissions that Flatpaks are granted by default.

JACK

PipeWire provides support for running JACK applications under it via specialized libraries, thus ensuring these are in the runtime and using the --filesystem=xdg-run/pipewire-0 permission is all that would be needed for these to work.

Devices

Flatpak does not have support for exposing individual devices outside of a few select categories (DRI devices as mentioned above is one of those, another one is KVM). If any devices not in these categories are needed, --device=all must be used, which grants access to all host devices. Note that, as root access cannot be gained from inside a Flatpak, the user itself needs to have read or write access to the device node for a Flatpak to be able to read or write to it.

Unfortunately, this does not grant access to udev device events. udev does not have a stable ABI, meaning that the format of event messages is not consistent across different versions. Thus, it is not possible for a Flatpak to talk to udev unless you can guarantee that the udev version inside the runtime and outside it are the exact same. If that can be guaranteed, --filesystem=/run/udev/control will make libudev watch device events, but note that opening the socket itself cannot be done due to it only being writable by root.

Bluetooth is a special case because there are two primary ways of accessing it:

  • In most cases, the BlueZ D-Bus service is used to access Bluetooth devices. This only requires a single permission: --system-talk-name=org.bluez.
  • Some applications may directly use AF_BLUETOOTH sockets instead. In this case, two permissions are needed: --allow=bluetooth and --share=network.

Nested Sandboxing

The only way to have additional sandboxes inside the outermost Flatpak sandbox is by using the portals. Applications that try to set up their own sandboxes using traditional methods will often not work, because most of the system calls required for this (user namespaces in particular) are blocked. (Some common use cases like Chromium / Electron apps have wrapper tools available to automatically redirect their sandboxing attempts to use Flatpak’s portal APIs.)

Restricting Application Resources

Flatpak can theoretically provide the ability to restrict the resource usage, such as CPU and RAM, of individual applications. However, the necessary hooks for this are not currently in-place, thus it is not possible to actually enforce the restrictions until an application is already running.

Interaction with LSMs

Flatpak, by default, does not integrate with LSM systems such as AppArmor, meaning that one cannot trivially apply LSM rules such as AppArmor profiles to Flatpak applications. The primary reason for this is that Flatpak’s static permissions use alternative Linux kernel systems (in particular, namespaces and seccomp) that can provide a similar degree of protection, just via different means.

As an example, a common use case for AppArmor profiles is to restrict where applications can read and write on the filesystem. When a Flatpak application, the filesystem access is limited so that it can only see parts of the filesystem it was explicitly granted access to, via kernel mount namespaces. Thus, the areas of the filesystem that the application should not be able to interact with simply do not exist for it in the first place.

It would be possible to add support for applying custom AppArmor profiles to Flatpak applications, but this integration does not exist yet. If this theoretical support existed, however, it would primarily apply to the static permissions model. For instance, if an application has --filesystem=home, a custom profile could be used to block access to ~/.gpg. Flatpak’s dynamic permissions, on the other hand, can not be trivially integrated with an LSM. The primary reason is that these dynamic permissions, as discussed above, generally take place via an intermediate service. For instance, if an application asks a custom portal for access to sensors, the portal is technically the one reading the sensor data. Attempting to use AppArmor to block off the application’s access some individual sensors would not work, because, as far as the LSM is concerned, the application is not accessing any sensors; all it is doing is talking to a service.

With the above in mind, adding support for LSMs to Flatpak would primarily be done for defense in depth, rather than as the first line of defense. This is similar to how LSMs are integrated in other modern container systems, as well as Kubernetes

One is still free to apply AppArmor profiles to non-Flatpak components on the same system; Flatpak will not cause any form of conflicts in that scenario.

IPC

As mentioned previously, Flatpak has built-in support for sandboxing D-Bus access. However, this does not extend to any other IPC mechanisms.

In general, most non-D-Bus IPC is performed over Unix domain sockets. Applications can simply include --filesystem=/the/socket/path in order to gain full read/write access to the socket. If the service being communicated with requires any special permission checks, it must do so itself.

Granting access to the socket file, however, does not extend to the ability to create the socket, only to read/write to an already existing socket. For this, access to the full directory containing the socket is required. Note that each application is automatically granted read/write access to its own well-known temporary directory shared with the host, $XDG_RUNTIME_DIR/app/$APP_ID. If an application needs to expose a socket for another applications to connect to, this would be the ideal location for the socket file. Then, any other application could just add --filesystem= permission for that directory.

As for other IPC systems, anything listed on this page (including POSIX message queues and any System V IPC such as semaphore sets) can be accessed via --share=ipc. Access to /dev/shm can be granted as well, via --device=shm.

Any IPC systems relying on scanning running processes manually will not work, as all Flatpak applications have a private /proc. In addition, any services relying on saving socket files into odd or inconsistent locations may not work without modifying the service or trying to forward sockets to each other from within the app.

If a Flatpak needs to talk to or start another Flatpak, then the application being started should be D-Bus activatable. This essentially means that the application can be started as a D-Bus service, even if it is a graphical application. (One can also distribute services as Flatpaks using this mechanism.) However, this is the only key way for Flatpaks to communicate with each other; if a custom IPC mechanism is desired, D-Bus would still need to be used to actually start the target application.

Building

There are two primary ways of building a Flatpak application:

  • The contents of /app can be manually set up via the flatpak build* CLI commands.
  • flatpak-builder can be used to interpret a JSON or YAML manifest file describing the steps needed to build a Flatpak. This is the most common method used.

A simple example flatpak-builder manifest is below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Defines the ID of the application.
app-id: com.foo.Bar
# Defines the runtime used, as well as the SDK that flatpak-builder will run the
# build commands inside of.
runtime: org.freedesktop.Platform
runtime-version: '21.08'
sdk: org.freedesktop.Sdk
# Defines the main entry point for the application.
command: my-app
finish-args:
  # Permissions go here:
  - --share=network
  - --socket=pulseaudio
  - --socket=wayland
  - --system-talk-name=org.bluez
# Use some SDK extensions: Node 12 and OpenJDK 11.
sdk-extensions:
  - org.freedesktop.Sdk.node12
  - org.freedesktop.Sdk.openjdk11
# The build "phases" go here:
modules:
  # Add a dependency we need.
  - name: my-library
    buildsystem: cmake-ninja  # use CMake+Ninja to build
    sources:
      # The source below will be extracted into the build directory, then
      # CMake will be run on it.
      - type: archive
        url: https://some-url.com/my-library.tar.gz
        sha256: aaaaaaaabbbbbbbbccccccccddddddddeeeeeeeeffffffff0000000011111111
  # Now all the future modules have access to the above dependency, like the
  # main app below:
  - name: my-application
    buildsystem: meson
    # Pass some configure options.
    config-opts:
      - -Dthing=enabled
    sources:
      # Use the current directory as the sources to build.
      - type: dir
        path: .

If this were in com.foo.Bar.yaml, then it would be built and installed via:

$ flatpak-builder --force-clean --install --user build com.foo.Bar.yaml

using build as the directory to store the contents of /app before installation.

Resource Usage

Disk Space

As Flatpaks rely on shared runtimes and bundled libraries, they do tend to use more disk space than traditional applications. This is primarily an upfront cost: a single installed application has to pull in an entire runtime, likely containing libraries it does not use. However, as more Flatpaks are installed, many of them will be using the same runtime, so the cost does not increase for every installation. Practically, this means that, for storage-constrained devices, a customized runtime can be built that includes just the common libraries needed by its typical applications. This will avoid much of the unneeded “bloat” resulting from using a more generic runtime.

However, there is a trick to minimizing disk usage, which is that Flatpak will unconditionally de-duplicate all files from installed runtimes and applications. This means that, if two different runtimes or applications have the exact same files inside, only one copy of the files is stored on disk.

RAM Overhead

Shared Libraries

Linux generally tries to share multiple in-memory instances of the same shared library. For instance, if applications foo and bar both link to libbaz.so, only one copy of the library is loaded into memory. However, because Flatpaks use a different version of their libraries than the host system, the library memory will not be shared with the host. Despite this, Flatpaks still share the libraries in RAM with each other, as the kernel only ever considers if the same library is being loaded twice, regardless of which application is loading it. In other words, multiple Flatpak applications using the exact same library version will have it shared in memory, but they will not share it with any applications using that library on the host system.

Extra Processes

Flatpaks require a few services to run:

  • The portal services (there are generally a small handful of these). The exact resources used would depend on the portals running, which can differ between different setups. However, in a current typical setup, they use around 30 MB of RAM or less. (This typical setup does include desktop-specific portal implementations, which would not apply to any embedded usage.)
  • The D-Bus proxy, used to implement the static D-Bus permissions. One of these must be running for every individual Flatpak, but they each use only around 0.5-1 MB of RAM.
  • The bwrap process, which is the outermost sandboxing layer. One of these must again be running for every individual Flatpak, but they each use only around a quarter of a megabyte of RAM.

Thus, this totals out to a ~30 MB fixed upfront cost, but only an extra 1MB or so of per-Flatpak overhead.

Note that the memory estimates use PSS, which attempts to only count a portion of memory used by shared libraries, due to the kernel sharing only one instance as mentioned above.

Runtime Overhead

Flatpaks should maintain negligable runtime overhead. The use of seccomp sandboxing results in a minor performance hit, but this is generally on the order of <1% and is usually unmeasurable outside of intensive benchmarks.

Practical Guidance

With all of the above restrictions in mind, it is worth adding some advice regarding practical decision making for Flatpak’s purposes.

Poor Fits

There are some system-level components that would not be a good fit for a Flatpak:

  • Anything that requires full-root access will not work.
  • Components that require a large amount of access or control of the host system will not translate well to Flatpak’s permissions model. For example, a system service that controls the resources of all other running processes would be a poor fit. In that case, it tends to be better to separate out the low-level controls into a system service, which is exposed to Flatpak applications via either a portal or standard D-Bus API.
  • Background daemons or services that multiple parts of a system (whether they be other daemons or Flatpak applications) depend on would likely benefit from being part of the host system, rather than in a Flatpak.

As a very general guideline, from a system architecture perspective: Components that are meant to be tightly coupled to, and/or manage, the host system (such as network connectivity management, lifecycle management, and FOTA) are best kept on the host. On the other hand, components that have a looser coupling to the host system (such as navigation, audio/video players, and web browsers) are more suitable for Flatpaks. For compoments that straddle the line between these categories, it’s worth evaluating if they can be split up into a service bundled with the host system and a Flatpak application that simply uses that service when required.

Degree of Separation

Here are some points to consider when deciding what components should go into a single Flatpak, compared to multiple:

  • Ideally, components are separated into separate Flatpaks when they are visibly separate. Media and Maps applications do not appear to be the same, thus they could easily be separated. However, Maps and Navigation appear visibly tied together, thus they should likely be bundled together. This is, however, a very theoretical guideline, as real-world situations can alter this drastically, as will shown in the next several points.
  • Multiple points of entry into the same component should be bundled together. If a Media application has to methods of entry, Music and Videos, but the actual application is the same core, then they should be bundled.
  • Services that a component requires but does not need to share with other applications can be bundled. For instance, a Messaging application that has a background service to manage sending and receiving messages can be bundled, as there is no use for the same service in other components.
  • Similar permissions is not a valid reason to bundle together. A voice assistant and a phone application both need access to the microphone feed, but the tasks they serve otherwise are logically separate. The assistant may start a phone call, but it may also send messages and manage todos as well; there is no inherent connection between the voice assistant and the phone.

Final Notes

In summary, this document went over the core concepts of Flatpak and some of the intricacies around its usage. For an application to be successfully migrated into a Flatpak environment, the primary challenge will be determining how exactly it interacts with the host system and how that can be translated to a Flatpak-based setup.

For the various system interactions, several mechanisms to achieve this in have been discussed:

  • Direct access to host resources via static permissions, such as:
    • Filesystem access
    • D-Bus service access
    • Audio/GPU access
    • Direct access to host devices
  • Indirect/dynamic access via services running on the host system, such as
    • Standard or custom portals
    • Generic D-Bus services (allowed via static permissions)
    • Services implemented using other IPC mechanisms

Apart from the application specifics, several more system level aspects need to be considered:

  • If an IPC system other than D-Bus is required to talk to a service, adaptions to it or Flatpak may be required.
  • For indirect access, services running on the host may check the dynamic permissions of the requesting application, as the portals do. If so, these permissions may be controlled automatically, or they may require explicit user interaction. The best method of granting access depends both on the product, resource, and requesting application.
  • Depending on the system, a custom runtime and/or runtime extension may need to be created if the generic Apertis runtime isn’t suitable, due to:
    • Being too large.
    • Not containing typical libraries.
    • Missing support for the target’s graphics stack.