Apertis has a slowly growing number of test cases that have been written over the years that the Apertis project has been running. Effort has been made to automate many of these tests, however many still remain as manually run testing scenarios.

Due to limited testing bandwidth, many of these manual tests are not frequently performed. Additionally there may be automated tests that are being run that, due to changes in the focus and direction of the Apertis project since it’s inception, are no longer particularly useful. When performed, such tests, both manual and automated, are consuming valuable resources that could be better utilised elsewhere.

A review of the test cases has been carried out to identify tests that may now be redundant and should therefore be removed. The following table shows the analysis performed for this review along with any suggestions for change:

Test case Manual/Auto RBEI Estimated effort (min) Status Using Apt Using Git Value Effort Suggestion Notes
ade-commands automated Critical Yes high low Used by RBEI
apertis-update-manager-api manual 10 Low low high Remove Automated as aum-api
apertis-update-manager-automount manual 10 Medium Yes high high Used by PT
apertis-update-manager-diskfull manual 10 Low low high Remove Automated as aum-out-of-space
apertis-update-manager-ota-api manual 10 Medium high high
apertis-update-manager-ota-auto manual 10 Medium low high Remove Automated as aum-ota-auto
apertis-update-manager-ota-diskfull manual 10 Medium low high Remove Automated as aum-ota-out-of-space
apertis-update-manager-ota-rollback manual 10 Medium high high
apertis-update-manager-ota-signed manual 10 Medium low high Automated as aum-ota-signed. Keep whilst rolling out update management over all supported platforms
apertis-update-manager-powercut manual 10 Low high high Although automated as aum-power-cut, that only simulates a power cut, so this test is still good confirmation
apertis-update-manager-rollback manual 12 Low low high Automated as aum-rollback-blacklist. Keep whilst rolling out update management over all supported platforms
apertis-update-manager-usb-unplug manual 12 Medium high high
apertis-update-manager manual 10 Low low high Remove Automated as aum-offline-upgrade
apparmor-basic-profiles automated Critical Yes high low Basic security functionality (Still using git - needs updating)
apparmor-bluez-avrcp-volume manual 05 Medium low high Merge Performs “volume up” and “volume down” on paired device, merge with bluez-avrcp-volume
apparmor-bluez-setup manual 05 High low high Merge Merge with bluez-setup
apparmor-bluez manual 05 High high high Basic checks
apparmor-chaiwala-system automated High Yes high low Checks behaviour of chaiwala-base, inherited by some of the custom AppArmor profiles
apparmor-dbus automated Critical Yes high low Testing ability to contain dbus calls
apparmor-functional-demo automated Medium Yes Yes high high Review High effort as test is not easily understandable and thus harder to maintain. Contents needs full review
apparmor-geoclue automated Medium Yes Yes medium low Injects library to cause geoclue to try accessing something it shouldn’t.
apparmor-gstreamer1-0 automated Medium Yes high low
apparmor-ofono manual 08 Medium medium medium Injects library to cause ofono to try accessing something it shouldn’t.
apparmor-pulseaudio automated Medium Yes medium low Injects library to cause pulseaudio to try accessing something it shouldn’t.
apparmor-session-lockdown-no-deny automated Medium Yes Yes high low Checks things we want are running and apparmor enforcement in place.
apparmor-tumbler automated Medium Yes medium low Injects library to cause tumbler to try accessing something it shouldn’t.
apparmor-utils automated Medium Yes Yes high low Testing functionality of AppArmor utilities
apparmor automated Critical Yes high low Basic apparmor functionality tests
aum-api automated Critical Yes high low
aum-offline-upgrade-collection_id automated Critical Yes high low
aum-offline-upgrade-signed automated Critical Yes high low
aum-offline-upgrade automated Critical Yes high low
aum-ota-api automated Critical Yes high low
aum-ota-auto automated Critical Yes high low
aum-ota-out-of-space automated Critical Yes high low
aum-ota-rollback-blacklist automated Critical Yes high low
aum-ota-signed automated Critical Yes high low
aum-out-of-space automated Critical Yes high low
aum-power-cut automated Critical Yes high low
aum-rollback-blacklist automated Critical Yes high low
bluez-avrcp-volume manual 08 Medium low high Merge We have to do all this in apparmor-bluez-avrcp-volume. Merge the tests.
bluez-hfp manual 08 Medium low high Keep separate as merging this with bluez-phone has been known to cause timeout issues with some phones.
bluez-phone manual 08 Critical high high Tests lots of bluetooth functionality paired with a phone.
bluez-setup manual 08 Critical low high Merge We have to do all this in apparmor-bluez-setup. Merge the tests.
boot-no-crashes automated High Yes high low Checks journal for auditd messages suggesting app crashes
boot-performance automated High Yes high low Dump boot performance (no fail state, but useful information)
canterbury automated Critical Yes low low
cgroups-resource-control automated Medium Yes low medium Remove Cgroups are now a widely used part of the Linux infrastructure and not the new technology it was when this test was added. This cursory testing is unlikely to catch any real bugs. The test is broken due to the reliance on a package no longer shipped by Debian and would need to be ported to something else.
connman-new-supplicant automated Medium high low Merge Just checks that wpa_supplicant is on dbus (merge with connman test)
connman-pan-network-access manual 08 Medium Yes high high
connman-pan-tethering manual 08 Critical Yes high high
connman-services manual 05 Medium high high
connman-usb-tethering manual 08 Critical Yes high high
connman automated Medium Yes high low Tests bits are accessible on dbus
dbus-dos-reply-time automated High Yes low low Runs dbus under various call load and time it. Understood to be useful as a point of comparison.
dbus-installed-tests automated High Yes Yes high low Running dbus’ own regression tests
didcot automated Medium Yes Yes low low
disk-rootfs-fsck automated High Yes high low
eclipse-plugins-install-target manual 10 Medium low high Remove Eclipse is no longer in Debian and survives in Apertis as a quite stale package which is likely to break in the next rebase. These tests are broken as the custom integrations broke on the last rebase. It is unlikely that Eclipse will stay in Apertis much longer and even less likely that the plugins will be fixed, so remove the tests.
eclipse-plugins-remote-debugging manual 10 Medium low high Remove As eclipse-plugins-install-target
eclipse-sysroot-management manual 10 Medium Yes low high Remove As eclipse-plugins-install-target
evolution-sync-bluetooth manual 05 Medium Yes low high Remove The technologies behind this functionality have become unsupported over time. Implementing such functionality now may require a substantially different approach and thus a different testing strategy.
frome automated Low Yes Yes low low
gettext-i18n automated Medium Yes low low Fix Test broken, appears to have been for a while, needs fixing.
glib-gio-fs automated Medium Yes high low Test basic GIO functionality
grilo automated Medium Yes Yes low low Improve Still installing packages, needs improving (or removing)
gstreamer1-0-decode automated High Yes high low
gupnp-services automated Medium Yes low low
hmi-audio-detail-view manual Medium high high Focus shifting away from current HMI, but principal will remain relevant
hmi-audio-play-pause manual High high high Focus shifting away from current HMI, but principal will remain relevant
hmi-video-load manual High high high Focus shifting away from current HMI, but principal will remain relevant
hmi-video-play-pause manual High high high Focus shifting away from current HMI, but principal will remain relevant
image-bootable manual 02 Critical high high
image-gui-start manual 02 Critical high high
iptables-basic automated Critical Yes high low Test expected rules exist
iptables-nmap automated 02 High Yes high low
newport automated Medium Yes Yes low low Improve Tests installed via apt, needs improving
nfsroot-simple-boot automated High high low
ofono-sms-receive manual 05 Medium low high
ofono-sms-send manual 05 Medium low high
ostree-collection-id automated Critical Yes high low
polkit-parsing automated High Yes Yes high low
rfkill-toggle manual 02 Medium low high
rhosydd automated Medium Yes Yes high low Improve Installs lots via apt, including rhosydd. Not installed by default.
ribchester automated Critical Yes Yes low low
sanity-check-manual manual 02 Critical Yes high high Sanity check when performing manual tests
sanity-check automated Critical Yes high low
sdk-ade-remote-debugging manual 08 High Yes high high
sdk-code-analysis-tools-gcov automated Medium Yes Yes low low
sdk-code-analysis-tools-splint automated Medium Yes Yes low low
sdk-cross-compilation automated Medium Yes high low
sdk-dbus-tools-bustle manual 02 Medium Yes low high Fix
sdk-dbus-tools-d-feet manual 02 Medium Yes low high Fix
sdk-debos-image-building automated Critical Yes high low
sdk-debug-tools-gdb automated Medium Yes Yes low low
sdk-debug-tools-strace automated Medium Yes Yes low low
sdk-debug-tools-valgrind automated Medium Yes Yes low low
sdk-flatpak-build-helloworld-app manual 20 Medium Yes Yes high high
sdk-flatpak-demo automated Medium Yes Yes high low
sdk-flatpak-helloworld-app manual 10 Medium high high
sdk-ide-build-run-program manual 02 Medium low high Improve Needs updating to reflect subtle changes in new versions of Eclipse
sdk-performance-tools-gprof automated Medium Yes Yes low low
sdk-performance-tools-sysprof automated Medium Yes Yes low low
sdk-persistent-disk-etc manual 15 Critical Yes high high Remove if automated by T7325
sdk-persistent-disk-home-user manual 15 Critical Yes high high Remove if automated by T7325
sdk-persistent-disk-sysroot manual 15 Critical Yes high high Remove if automated by T7325
sdk-vb-fullscreen manual 05 Critical Yes high high
secure-boot-imx6 manual 20 Medium high high
tiny-container-system-aa-enforcement automated Medium Yes high low
tiny-container-system-basic automated Medium Yes high low
tiny-container-system-connectivity-profile automated Medium Yes high low
tiny-container-user-device-sharing automated Medium Yes high low
tiny-container-user-folder-sharing automated Medium Yes high low
tiny-container-user-seccomp automated Medium Yes high low
traprain automated Medium Yes Yes high low
tumbler-thumbnailing automated Medium Yes high low
video-animation-on-boot manual 03 High low high Update Expecting old animation - update
webkit2gtk-ac-3d-rendering manual 05 Low Yes high high Test excercises a big chunk of stack, so keep
webkit2gtk-ac-animations manual 05 Medium Yes low high Remove No longer working on webkit
webkit2gtk-ac-scrolling manual 05 Medium Yes low high Remove No longer working on webkit
webkit2gtk-drag-and-drop manual 05 Low Yes low high Remove No longer working on webkit
webkit2gtk-event-handling-redesign manual 05 Medium Yes low high Remove No longer working on webkit
webkit2gtk-gstreamer1.0 manual 05 Medium Yes high high Test excercises a big chunk of stack, so keep
webkit2gtk-mt-touch-events manual 05 Low Yes high high Test excercises a big chunk of stack, so keep
x-hw-accelerated manual 05 Medium Yes low high Automate and remove if T7325 proves viable

General status of the test framework

Whilst reviewing the test cases a number of additional A number of additional points have been noted that are mostly out of scope for this review, though may be worth considering for future work:

  • A lot of the tests seem to have needless levels of abstraction: A number of strategies have been employed over successive attempts to automate more of the test cases. In order to avoid extra overhead, generally infrastructure used to perform the previously automated tests has been kept and driven from the latest mechanism. In parallel to this, work has been done to provide a testing framework that makes steps carried out by the tests better visibly documented (leading to the test descriptions on qa.apertis.org. Due to may of these tests calling previous mechanisms, these tests are not as transparent as they could be and the (in some cases, multiple) layers of abstraction make them harder to read and maintain. It is suggested that efforts are taken to remove these layers of abstraction where possible.
  • It should be possible to merge some of the AppArmor tests with their non-AppArmor counterparts: Quite a few of the AppArmor tests run up the same image and perform the same steps as the non-AppArmor test for the same functionality, only differing in what they look at to deem the test a success or failure. This adds overhead as the infrastructure is setting up and performing the tests twice. This can be reduced by checking for both the outcome of the test steps and the AppArmor status in a single test run without loss of testing coverage.
  • Most tests should not be installing packages via apt: A considerable number of the tests are installing packages via apt to for use in the testing. These additions are fundamentally changing the image under test and may affect the results. We have had a mechanism to combat this for a while which needs to be universally rolled out.
  • We should update tests using git clone to use wget: Rather than requiring git to be install on the test image, we have recently been downloading archives of the test harnesses with are provided by GitLab. This only requires the ability to download a resource via HTTP, which can be reasonably assumed to be possible in all the images rather than the ability to perform a git clone. Tests should be updated to use this mechanism to avoid the need for git.
  • Merge some tests together: In some places we seem to have multiple tests, each testing a facet of a larger set of functionality. Whilst not viable or advisable for all the tests, merging some may be viable which would reduce the overhead of setting up the environment individually each time.