Automatic testing of packages is essential to ensure high quality software. This document describe how to define tests and run them as part of the Apertis ci-package-builder pipeline. A job called package-centric-testing will install freshly built packages in an Apertis image and will run a test command to ensure the package works as expected.

How to enable testing of packages

The first step is to create a file debian/apertis/package-centric-testing.yaml in the source package. This file is used to define the test environment (image ID, image type, architecture, board type, installed packages) and the test itself (it can be a simple command or a test script for a more complex test). The presence of this file will add a testing job in the ci-package-builder pipeline.

package-centric-testing.yaml is a YAML file with a syntax similar to the one used for Apertis Test Cases. Here is a simple example for package liblc3:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


tests:
  image-types:
    fixedfunction: [ armhf, arm64 ]
    hmi:           [ armhf, arm64 ]
    basesdk:       [ amd64 ]
    sdk:           [ amd64 ]
  bin_pkg_list: >
     liblc3-0
     liblc3-tools     
  pkg_test_cmd: "echo Hello World"

In this example, we will test our package in 6 different environments which are:

on armhf with a fixedfunction image;
on arm64 with a fixedfunction image;
on armhf with a hmi image;
on arm64 with a hmi image;
on amd64 with a basesdk image;
on amd64 with a sdk image;

For each environment, freshly built packages listed in bin_pkg_list will be installed then the command defined in pkg_test_cmd will be run.

In other words, this test will trigger 6 different LAVA jobs.

It’s possible to refine the test environment by defining the board type and the image ID. By default, the image ID used is always the latest daily successfully built, but it is possible to override it with the image_buildid field. Regarding the board type, by default uboot is used as board type for both architectures arm64 and armhf, so based on the devices.yaml arm64 tests will run on an Renesas R-Car board, this can be overridden by selecting another board type, like in the following test definition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


tests:
  image-types:
    fixedfunction: [ armhf, arm64: [uboot, rpi64] ]
    hmi:           [ armhf, arm64: [rpi64] ]
    basesdk:       [ amd64 ]
    sdk:           [ amd64 ]
  image_buildid: "20231211.0015"
  bin_pkg_list: >
     liblc3-0
     liblc3-tools     
  pkg_test_cmd: "echo Hello World"

In this second example, the arm64 tests will be run on both an Renesas R-Car board and a Raspberry Pi 4B board with a fixedfunction image. And the arm64 tests with an hmi image will only run on a Raspberry Pi 4B board.

We can also see that the image_buildid field is used, that means the image used won’t be the latest daily but the daily built on 20231211.0015.

Alternately of using a daily image for testing, it is possible to build a custom image by setting build-custom-image: yes instead of image_buildid: "xxxx". This option will add jobs in the ci-package-builder pipeline building a new custom Apertis image that will then be used in LAVA to test the package.

The command defined in pkg_test_cmd will be run in the debian/tests/ folder (if the folder is not available, it will be created). Running the pkg_test_cmd in debian/tests/ allow us to easily re-use tests written by Debian community as part of the Debian’s autopkgtest (DEP8).

By default, generated LAVA jobs are based on the configurations and boards defined in tests/apertis-test-cases. Since projects may use a different instance of test-cases, for instance to use specific boards or configurations, it is possible to define the path using the variable test_cases_path. The default path is tests/apertis-test-cases as the git repository is available at https://gitlab.apertis.org/tests/apertis-test-cases/. If the desired test-cases instance is located at https://gitlab.apertis.org/my_project/tests/apertis-test-cases/, then it can be defined as test_cases_path: my_project/tests/apertis-test-cases. This variable doesn’t contain the full URL of test-cases because the GitLab CI job token is injected into the URL before cloning the repository in order to get access to non-public repositories.

How to write complex tests

The first step is to define a test with debian/apertis/package-centric-testing.yaml:

1
2
3


tests:
  ...
  pkg_test_cmd: "./run_complexe_test.sh"

For this test, instead of running a simple command, we will call a script called run_complexe_test.sh which allow us to do a more complex testing. This script run_complexe_test.sh must be located in debian/tests/ and can contain several commands or call other scripts like in this example:

1
2
3
4
5
6


sudo apt update
sudo apt install some-required-packages -y
my-binary --version
my-binary --help
./run-an-autopkgtest
./run-another-autopkgtest

This fake example will update package information before installing a package some-required-packages. Then, it will run my-binary with the --version argument and then with --help. Finally, our script will call other scripts run-an-autopkgtest and run-another-autopkgtest which are also located in debian/tests/ in our source package.

Behind the scene

When a debian/apertis/package-centric-testing.yaml is available in a source package, a new job is added to the ci-package-builder pipeline. This job package-centric-testing will clone apertis-test-cases and will use its logic to generate LAVA jobs. More precisely, it will use generate-pkg-centric-jobs.py which reads the job definition from package-centric-testing.yaml and from there it will call generate-jobs.py to generate LAVA jobs. Then, based on the LAVA jobs created, generate-test-pipeline.py will generate a child pipeline package-centric-testing-monitor composed of the generated LAVA jobs.

Search Apertis

Apertis packages testing with LAVA

Table of Contents:

How to enable testing of packages

How to write complex tests

Behind the scene