Testing on a immutable rootfs

Tests should be self-contained and not require changes to the rootfs: any change to the rootfs causes the tested system to diverge from the one used in production, reducing the value of testing.

Changing the rootfs is also impossible or very cumbersome to do with some deployment systems such as dm-verity or OSTree. Other systems may simply not ship package management systems like apt/dpkg due to size constraints, making package dependencies not viable.

Tests meant to be run on targets should then be self-sufficient and should not require dependencies to be installed on the rootfs.

Writing self-contained tests

  • Each tests is shipped in a separate git repository under the tests GitLab group
  • Check existing tests like the D-Bus reply time ones or the Canterbury ones
  • Add a subtree of the common folder. For convenience, use the common-subtree.sh script to add a git-subtree into your repository: ../common/common-subtree.sh add See the README for further details
  • For tests requiring additional binaries not shipped on the base image:
    • Add an external-binaries.cfg file
    • Run the update_test_binaries.sh script: ./helper-tools/update_test_binaries.sh –repository –release 18.06 –dry-run      This should add a commit with the binaries required by your test repository
  • Tests should use compiled executables included with update_test_binaries.sh and/or POSIX shell scripts (avoid bashisms and assume that /bin/sh points to dash)
  • Test the scripts in a minimal or ostree image
  • Add your repository to the test-repositories.txt list and Jenkins will keep the binaries automatically updated