Testing on a immutable rootfs
Tests should be self-contained and not require changes to the rootfs: any change to the rootfs causes the tested system to diverge from the one used in production, reducing the value of testing.
Changing the rootfs is also impossible or very cumbersome to do with
some deployment systems such as
or OSTree. Other systems may simply not ship
package management systems like
dpkg due to size constraints,
making package dependencies not viable.
Tests meant to be run on targets should then be self-sufficient and should not require dependencies to be installed on the rootfs.
Writing self-contained tests
The approach used by Apertis to create self contained tests is to ship all the required test code via dedicated git repositories under the tests GitLab group which get deployed by LAVA on the device before executing the tests.
Those repositories can either contain interpreted code that does not need to be
built, for instance by using the
dash POSIX shell, or they include binaries
built on OBS and synchronized with a GitLab CI/CD pipeline.
For instance, the
The testcase definition
Both manual and automated testcases are defined via YAML files in the
The YAML files are used to generate the qa.apertis.org website and the automated ones are executed by LAVA.
Usually testcase need supporting files and scripts and these are shipped via git repositories with the below directive:
The git repository
The easiest way to deploy supporting files, scripts and binaries on the device under test is by storing them in standalone git repositories that then get unpacked on the device before the test is executed.
In the simplest case the support git repository only ships a shell script and
maybe some static test samples. Keep in mind that the Apertis reference images
do not ship
bash on fixedfunction/HMI images but use
dash, so stick to POSIX
features and avoid bashisms.
If instead compiled binaries are needed, they need to be built on OBS and regularly synchronized in the git repository. Apertis provides a reusable GitLab pipeline to retrieve those external binaries and commit them in the git repository. To set it up:
Create a new GitLab project under the
Add a subtree of the
commonfolder. For convenience, use the
common-subtree.shscript to add a git-subtree into your repository:
See the README for further details
external-binaries.cfgfile which describes the binaries that need to be retrieved and committed. All the files that would otherwise need to be installed via
aptshould be listed here: check what the dependencies of the test are and inspect their contents, filtering out the files which are not needed to execute the test. The format is a simple list of
$PACKAGE $FILEPATHon each line. The retrieved binaries will be committed under the appropriate
$ARCH/binsubdirectory in the repository. An optional
prefix=attribute is supported to control where the retrieved file should be installed, for instance from the
apertis-tests-apparmor-ofono /usr/lib/apertis-tests/apparmor/ofono/libofonod-malicious-override.so.0 prefix=lib apertis-tests-apparmor-ofono /usr/lib/apertis-tests/apparmor/ofono/libofonod-malicious-override.so prefix=lib
.gitlab-ci.ymlwhich defines the GitLab CI/CD pipeline that is responsible of keeping the binaries updated by pulling them from the built packages and committing them to the repository:
1 2 3 4 5 6 7 8 9 10 11
variables: osname: apertis release: v2021 include: - project: tests/helper-tools ref: apertis/v2021 file: /gitlab-ci/update-test-binaries.yaml update-test-binaries: stage: build extends: - .update-test-binaries
Configure a CI/CD schedule to run the pipeline daily.
The deb package
In some cases the needed binaries can be extracted from existing packages.
For instance the
testcases sources them from the upstream
In other cases a new, dedicated package needs to be created to host the test
utilities. No specific requirements are needed on this package, but
conventionally they are named