Introduction
The aim of this document is to provide a suitable solution for integrating Robot Framework on automated test infrastructure setup on LAVA. LAVA infrastructure doesn’t support triggering or executing Robot Framework test suites as it is intended only for continuous integration. Thanks to this integration the coverage test can be extended to cover different test areas by adding additional customized libraries and toolchains.
Integrating Robot Framework on LAVA infrastructure adds additional benefits of Robotic Process Automation (RPA), ATDD (Acceptance test–driven development) and also allows to use a wide range of open source libraries developed for automation testing.
Robot Framework is simple, yet powerful and easily extensible tool which utilizes the keyword driven testing approach. It uses a tabular syntax which enables creating test cases in a uniform way. All these features ensure that Robot Framework can be quickly used to automate test cases. The best benefit with Robot Framework for the users is that there is no need for using any sort of programming language for implementing and running tests.
LAVA (Linaro Automation and Validation Architecture) is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing, although extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis.
Architecture overview
Test Data
The Robot framework has a layered architecture. The top layer is the simple, powerful, and extensible keyword-driven descriptive language for testing and automation. This language resembles a natural language, is quick to develop, is easy to reuse, and is easy to extend.
Test data, the first layer of the Robot framework is in a tabular format. Since the data is in a tabular format, maintaining the data is very easy. This test data is the input to Robot Framework, once it is received, it is processed and on execution reports and logs are generated. The report is in HTML and XML format and of offer detailed information about every line that is executed as a part of the test case.
Robot Framework
The primary advantage of the Robot framework is that it is agnostic of the target under test. The interaction with the layers below the framework can be done using the libraries built-in or user-created that make use of application interfaces.
Test Libraries & Test Tools
A library in a Robot Framework terminology, extends the Robot Framework language with new keywords, and provides the implementation for these new keywords. Each Robot Framework library acts as glue between the high level language and low level details of the item being tested, or of the environment in which the item to be tested is present.
System Under Test
This is the actual target on which the testing activity is performed. It could either be a library or a app. Libraries act as an interface between the Robot Framework and the system under test. Hence, there is no way through which the framework can directly talk to the system under test. The Robot Framework supports various file formats namely HTML, TSV (Tab Separated Values), reST (Restructured Text), and Plain text. As per the official documentation of Robot framework, the plain text format is recommended.
Integration of Robot Framework on LAVA
Apertis does the complete test automation setup on LAVA infrastructure for all it’s reference hardware. System integration and boot level testing is done as part of automated on successfully generated image from CI/CD image generation pipeline. Test report are displayed on QA report app and bug task is created for each failure test cases on Phabricator.
There are two main constraints on automated tests setup on LAVA, the asynchronous way of updating results and user not having control over the job once it is submitted. Developers and CI pipeline can both submit jobs to LAVA, but they cannot interact with a job while it is running. The LAVA workflow define the process of submitting a job, wait for the job to be selected for execution, wait for the job to complete it’s execution, and download test results.
Considering the above constraints and covering the wide range of test areas including HMI tests, integrating Robot Framework on LAVA setup will provide more chances to automate complex tests.
A LAVA instance consists of two primary components masters and workers. The simplest possible configuration is to run the master and worker components on a single machine, but for larger instances it can also be configured to support multiple workers controlling a larger number of attached devices in a multi node mechanism.
There are two possible approaches available to integrate Robot Framework on LAVA:
- Creating a QEMU emulator which uses Apertis SDK image and starts to execute Robot Framework test suites
- Creating a Docker based container and start executing Robot Framework test suites
The first approach consists in creating a QEMU emulator with Apertis SDK image and installing Robot Framework. By using it all the SDK related test cases can be migrated to Robot Framework and run successfully. However, running tests on Fixed Function or HMI images is not feasible, making this approach not suitable for testing target relates tests, and therefore not meeting all the use cases.
The second approach consists of creating a Docker container which runs under the worker context, which provides the benefits of isolation and security. An additional advantage is that each test suite execution will be run on independent Docker container making parallel execution possible in some scenarios.
The worker is responsible for running the lava-worker daemon to start and monitor test jobs running
on the dispatcher. Each master has a worker installed by default and additional workers can be added on
separate machines, known as remote workers. The admin decides how many devices are assign to each worker.
In large instances, it is common for all devices to be assigned to remote workers to manage the load.
Workflow shows the stages of running jobs starting from triggering the test suite as a job and finally updating test report back to the server. A test suite consists of multiple test cases related which will be run on a specific release, image type and hardware as described in LAVA testing A Docker instance will be created for each job instance and deleted once the job execution is completed. Running each job on dispatcher will create a separate docker instance of Robot Framework and monitor the job completion. The test cases execution will be done inside the Robot Framework instance, and once the test execution is completed the generated report will be sent back to dispatcher to be parsed and send back to server.
Impact analysis on Apertis distribution
Infrastructure
Integrating Robot Framework on existing Apertis infrastructure will requires the following changes :
- Integrating Docker container on worker, so worker setup should improved to meet the Docker requirement.
- Storage area of worker should be increased to store all type of Robot Framework result.
- Extend the master web interface to show the Robot Framework result
Development environment
The current development environment integrates Robot Framework with all standard libraries along
with SSH library as part of SDK distribution. Using Apertis SDK a developer can write Robot
Framework testcases to run on SDK and target devices running Fixed Function or HMI images.
Test development
- Impact on Apertis development is that we have start developing new test suites for robot framework.
- Start developing new yaml files which helps in executing the robot test suites from containers
- Apertis tests needs to rewrite the existing LAVA test job to execute the robot test suites
Testing
- With approaches mentioned above we can keep the existing scripts as they are and start executing tests defined with the new Robot Framework test suites which will help to improve the test coverage.
Summary
- Stable version of Robot Framework 4.1.2 and core library of
SSHandlaxerare integrated on SDK distribution from V2022dev1 release onwards. - Robot Framework on LAVA setup is completed with QEMU emulator and successfully executed the jobs.
- Evaluation of RFS for Docker container setup on LAVA worker is ongoing
- Above implementation is planed to host on cloud setup.