This document provides and overview of the image build pipeline prior to the migration to GitLab CI/CD that has been completed during the v2021 development cycle. Refer to the documentation in the infrastructure/apertis-image-recipes project for information about the current pipeline.
The Apertis infrastructure supports continuous building of reference images, hwpacks and ospacks. This document explains the infrastructure setup, configuration and concepts.
To build the various packs (hardware, os) as well as images, Apertis uses
Debos, a flexible tool to configure the
build of Debian-based operating systems. Debos uses tools like
already present in the environment and relies on virtualisation to securely do
privileged operations without requiring root access.
For orchestrating Apertis uses the well-known Jenkins automation server. Following current best practices the Apertis image build jobs use Jenkins pipelines (introduced in Jenkins 2.0) to drive the build process as well as doing the actual build inside Docker images to allow for complete control of the job specific build-environment without relying on job-specific Jenkins slave configuration. As an extra bonus the Docker images used by Jenkins can be re-used by developers for local testing in the same environment.
For each Apertis release there are two relevant Jenkins jobs to build images; The first job builds a Docker image which defines the build environment and uploads the resulting image to the Apertis Docker registry. This is defined in the apertis-docker-images git repository. The second job defines the build steps for the various ospacks, hardware packs and images which are run in the Docker image build by the previous job; it also uploads the results to images.apertis.org.
Jenkins master setup
Instructions to install Jenkins can be can be found on the Jenkins download page. Using the Long-Term support version of Jenkins is recommended. For the Apertis infrastructure Jenkins master is being run on Debian 9.3 (stretch).
The plugins that are installed on the master can be found in the [plugins appendix][Appendix: List of plugins installed on the Jenkins master]
Jenkins slave setup
Each Jenkins slave should be installed on a separate machine (or VM) in line with the Jenkins best practices. As the image build environment is contained in a Docker image, the Jenkins slave requires only a few tools to be installed. Apart from running a Jenkins slave itself, the following requirements must be satisfied on slave machines:
- git client installed on the slave
- Docker installed on the slave and usable by the Jenkins slave user
- /dev/kvm accessible by the Jenkins slave user (for hw acceleration support in the image builder)
For the last requirement on Debian systems this can be achieved by dropping a file called /etc/udev/rules.d/99-kvm-perms.rules in place with the following content.
Documentation for installing Docker on Debian can be found as part of the
To allow Docker to be usable by Jenkins, the Jenkins slave user should be
configured as part of the
Documentation on how to setup Jenkins slaves can be found as part of the Jenkins documentation.
Docker registry setup
To avoid building Docker images for every image build round and to make it easier for Jenkins and developers to share the same Docker environment for build testing, it is recommended to run a Docker registry. The Docker registry documentation contains information on how to setup a registry.
Docker images for the build environment
The Docker images defining the environment for building the images can be found in the apertis-docker-images git repository.
The toplevel Jenkinsfile is setup to build a Docker image
based on the Dockerfile
defined in the Apertis-image-builder directory and upload the result to the
public Apertis Docker registry
docker-registry.apertis.org through the
authenticated upload channel
For Apertis derivatives this file should be adjusted to upload the Docker image to the Docker registry of the derivative.
Image building process
The image recipes and configuration can be found in the
apertis-image-recipes git repository.
As with the Docker images, the top-level
the Jenkins job. For each image type to be built a parallel job is started
which runs the image-building toolchain in the Docker-defined environment.
The various recipes provide the configuration for debos, documentation about the available actions can be found in the Debos documentation.
Jenkins jobs instantiation
Jenkins needs to be pointed to the repositories hosting the Jenkinsfiles by
creating matching jobs on the master instance. This can be done either manually
from the web UI or using the YAML templates supported by the
command-line tool from the
jenkins-job-builder package, version 2.0 or later
for the support of pipeline jobs.
For that purpose Apertis uses a set of job templates hosted in the
OSTree support (server side)
The image build jobs prepare OSTree repository to be installed server side. In
order to properly support OSTree server side,
ostree-push package must be
installed in the OSTree repository server.
Appendix: List of plugins installed on the Jenkins master
At the time of this writing the following plugins are installed on the Apertis Jenkins master:
To retrieve the list, access the script console and enter the following Groovy script: