LAVA is a testing system allowing the deployment of operating systems to physical and virtual devices, sharing access to devices between developers. As a rule tests are started in non-interactive unattended mode and LAVA provides logs and results in a human-readable form for analysis.

As a common part of the development cycle we need to do some integration testing of the application and validate it's behavior on different hardware and software platforms.

Integration testing example

Let's take the systemd service and systemctl CLI tool as example to illustrate how to test the application with D-Bus interface.

The goal could be defined as follows:

as a developer of systemctl CLI tool, I want to ensure what the systemctl is able to provide correct information about the system state.

Local testing

To simplify the guide we are testing only the correct status of systemd with command below:

$ systemctl is-system-running
running

To ensure that the systemctl tool is really provides the correct information we may check the system state additionally via systemd D-Bus interface:

$ gdbus call --system --dest=org.freedesktop.systemd1 --object-path "/org/freedesktop/systemd1" --method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager SystemState
(<'running'>,)

So, for the local testing during development we are able to create a simple script validating that the systemctl works well in our development environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
#!/bin/sh

status=$(systemctl is-system-running)

gdbus call --system --dest=org.freedesktop.systemd1 \
  --object-path "/org/freedesktop/systemd1" \
  --method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager SystemState | \
  grep "${status}"

if [ $? -eq 0 ]; then
  echo "systemctl is working"
else
  echo "systemctl is not working"
fi

Testing in LAVA

As soon as we are done with development, we push all changes to GitLab and CI will prepare a new version of the package and OS images. But we do not know if the updated version of systemctl is working well for all supported devices and OS variants, so we want to have the integration test to be run by LAVA.

Since the LAVA is a part of CI and works in non-interactive unattended mode we can't use the test script above as is.

To start the test with LAVA automation we need to:

  1. Adopt the script for LAVA
  2. Integrate the testing script into Apertis LAVA CI

Changes in testing script

The script above is not suitable for unattended testing in LAVA due some issues:

  • LAVA relies on exit code to determine if test a passed or not. The example above always return the success code, only human-readable string allows to validate the status of systemctl
  • if systemctl is-system-running call is failed for some reason (segfault for instance), the script will proceed further without that error detection and LAVA will set the test as passed, so we will have a false positive result
  • LAVA is able to report separately for any part of the test suite – just need to use LAVA-friendly output pattern

So, more sophisticated script suitable both for local and unattended testing in LAVA could be following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#!/bin/sh

# Test if systemctl is not crashed
testname="test-systemctl-crash"
status=$(systemctl is-system-running)
if [ $? -le 4 ]; then
  echo "${testname}: pass"
else
  echo "${testname}: fail"
  exit 1
fi

# Test if systemctl return non-empty string
testname="test-systemctl-value"
if [ -n "$status" ]; then
  echo "${testname}: pass"
else
  echo "${testname}: fail"
  exit 1
fi

# Test if systemctl is reporting the same status as
# systemd exposing via D-Bus
testname="test-systemctl-dbus-status"
gdbus call --system --dest=org.freedesktop.systemd1 \
  --object-path "/org/freedesktop/systemd1" \
  --method org.freedesktop.DBus.Properties.Get \
  org.freedesktop.systemd1.Manager SystemState | \
  grep "${status}"
if [ $? -eq 0 ]; then
  echo "${testname}: pass"
else
  echo "${testname}: fail"
  exit 1
fi

Now the script is ready for adding into LAVA testing. Pay attention to output format which will be used by LAVA to detect separate tests from our single script. The exit code from the testing script must be non-zero to indicate the test suite failure.

Create GIT repository for the test suite

The test script must be accessible by LAVA for downloading. LAVA has support for several methods for downloading but for Apertis the GIT fetch is preferable since we are using separate versions of test scripts for each release.

It is strongly recommended to create a separate repository with test scripts and tools for each single test suite.

As a first step we need a fresh and empty GIT repository anywhere, for example in personal space of GitLab instance and it's local copy:

git clone git@gitlab.apertis.org:d4s/test-systemctl.git
cd test-systemctl

By default the branch name is set to main but Apertis automation require to use the branch name aimed to selected release (for instance apertis/v2022dev1), so need to create it:

git checkout HEAD -b apertis/v2022dev1

Copy your script into GIT repository, commit and push it into GitLab:

chmod a+x test-systemctl.sh
git add test-systemctl.sh
git commit -s -m "Add test script" test-systemctl.sh
git push -u origin apertis/v2022dev1

Add the test into Apertis LAVA CI

Apertis test automation could be found in GIT repository with Apertis test cases, so we need to fetch the local copy and create a work branch wip/example for our changes:

git clone git@gitlab.apertis.org:tests/apertis-test-cases.git
cd apertis-test-cases
git checkout HEAD -b wip/example
  1. Create test case description

    First of all we need to create the instruction for LAVA with following information:

    • where to get the test
    • how to run the test

    Create the test case file test-cases/test-systemctl.yaml with your favorite editor:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    
    metadata:
      name: test-systemctl
      format: "Apertis Test Definition 1.0"
      image-types:
        minimal:  [ armhf, arm64, amd64 ]
      image-deployment:
        - OSTree
      type: functional
      exec-type: automated
      priority: medium
      maintainer: "Apertis Project"
      description: "Test the systemctl."
    
      expected:
        - "The output should show pass."
    
    install:
      git-repos:
        - url: https://gitlab.apertis.org/d4s/test-systemctl.git
          branch: apertis/v2022dev1
    
    run:
      steps:
        - "# Enter test directory:"
        - cd test-systemctl
        - "# Execute the following command:"
        - lava-test-case test-systemctl --shell ./test-systemctl.sh
    
    parse:
      pattern: "(?P<test_case_id>.*):\\s+(?P<result>(pass|fail))"
    
    

    This test is aimed to be run for an ostree-based minimal Apertis image for all supported architectures. However the metadata is mostly needed for documentation purposes.

    Action “install” points to the GIT repository as a source for the test, so LAVA will fetch and deploy this repository for us.

    Action “run” provides the step-by-step instructions on how to execute the test. Please note that it is recommended to use wrapper for the test for integration with LAVA.

    Action “parse” provides its own detection for the status of test results printed by script.

  2. Push the test case to the GIT repository. This step is mandatory since the test case would be checked out by LAVA internally during the test preparation.

    git add test-cases/test-systemctl.yaml
    git commit -s -m "add test case for systemctl" test-cases/test-systemctl.yaml
    git push --set-upstream origin wip/example
    
  3. Add a job template to be run in lava. Job template contains all needed information for LAVA how to boot the target device and deploy the OS image onto it.

    Create the simple template lava/test-systemctl-tpl.yaml with your lovely editor:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    
     job_name: systemctl test on {{release_version}} {{pretty}} {{image_date}}
     {% if device_type == 'qemu' %}
     {% include 'common-qemu-boot-tpl.yaml' %}
     {% else %}
     {% include 'common-boot-tpl.yaml' %}
     {% endif %}
    
       - test:
           timeout:
             minutes: 15
           namespace: system
           name: common-tests
           definitions:
             - repository: https://gitlab.apertis.org/tests/apertis-test-cases.git
               revision: 'wip/example'
               from: git
               path: test-cases/test-systemctl.yaml
               name: test-systemctl
    

    Hopefully you don't need to deal with the HW-related part, since we already have those instructions for all supported boards and Apertis OS images.

    Please pay attention to revision – it must point to your development branch while you are working on your test.

    Instead of creating a new template, you may want to extend the appropriate existing template with additional test definition. In this case the next step could be omitted.

  4. Add the template into a profile.

    Profile file is mapping test jobs to devices under the test. So you need to add your job template into the proper list. For example we may extend the templates list named templates-minimal-ostree in file lava/profiles.yaml:

    1
    2
    3
    
    .templates:
      - &templates-minimal-ostree
        - test-systemctl-tpl.yaml
    

    It is highly recommended to temporarily remove or comment out the rest of templates from the list to avoid unnecessary workload on LAVA while you're developing the test.

  5. Check the profile and template locally first.

    As a first step you have to define the proper profile name to use for the test in LAVA.

    NB: you need to have lqa tool installed and configured as described in personal LAVA tests tutorial.

    Since the LAVA is a part of Apertis OS CI – it requires some variables to be provided for using Apertis profiles and templates. Let's define the board we will use for testing, as well as the image release and variant:

    release=v2022dev1
    version=v2022dev1.0rc2
    variant=minimal
    arch=armhf
    board=uboot
    baseurl="https://images.apertis.org"
    imgpath="release/$release"
    profile_name=apertis_ostree-${variant}-${arch}-${board}
    image_name=apertis_ostree_${release}-${variant}-${arch}-${board}_${version}
    

    And now we are able to submit the test in a dry-run mode:

    lqa submit -g lava/profiles.yaml -p ${profile_name} \
      -t visibility:"{'group': ['Apertis']}" -t priority:"high" \
      -t imgpath:${imgpath} -t release:${release} -t image_date:${version} \
      -t image_name:${image_name} -n
    

    There should not be any error or warning from lqa. You may want to add -v argument to see the generated LAVA job.

    NB: it is recommended to set visibility variable to “Apertis” group during development to avoid any credentials/passwords leak by occasion. Set the additional variable priority to high allows you to bypass the jobs common queue and do not wait for your job results for ages.

  6. Submit your first job to LAVA.

    Just repeat the lqa call above without the -n option. After the job submission you will see the job ID:

    lqa submit -g lava/profiles.yaml -p "${profile_name}" -t visibility:"{'group': ['Apertis']}" -t priority:"high" -t imgpath:${imgpath} -t release:${release} -t image_date:${version} -t image_name:${image_name} 
    Submitted job test-systemctl-tpl.yaml with id 3463731
    

    So it is possible to check the job status by URL with that ID: https://lava.collabora.co.uk/scheduler/job/3463731

  7. Push your changes from template and profile.

    As soon as your test case works as expected you may restore all commented templates for profile, change the revision key in file lava/test-systemctl-tpl.yaml to target branch and submit your changes:

    git add lava/test-systemctl-tpl.yaml lava/profiles.yaml
    git commit -a -m "hello world template added"
    git push
    

As a last step you need to create a merge request in GitLab. As soon as it gets accepted your test becomes part of Apertis testing CI.