Table of Contents:
This document aims as a single resource for all information related to Apertis Release Process. It covers the Major Release process as well as the Point Release process.
Apertis Infrastructure
The Apertis project is hosted on a couple of infrastructure services, which are tightly coupled to each other, providing end-to-end automation. This includes:
- Landing new packages into Apertis
- Landing new security and general updates into Apertis
- Landing downstream changes into Apertis
This document stands as a comprehensive guide to all package related processes as outlined above.
At a higher level, Apertis infrastructure includes:
- GitLab instance for source code hosting, CI, Automation, Image building etc
- OBS (Open Build Service) for package building
- aptly for APT repository management
- images.apertis.org for Apertis image hosting
- LAVA for Quality Assurance Testing
Development/Preview Releases
As described in Release flow and product lines Apertis ships three types of releases: development, preview and stable. This section describes the release process for development and preview releases.
Development/Preview Release Steps
As a high level overview, these are the steps to be executed during the release cycle, to perform a release:
- Keep release schedule wiki page updated
- Announce soft feature freeze
- Check available disk space on OBS Server
- Full archive rebuild to ensure all the packages are buildable from source
- Announce hard feature freeze
- Update the freshness status of all the design documents
- Write the release notes
- Confirm that no release blocker issue is present
- Publish Release Candidate (RC1) images and announce hard code freeze
- RC images validation (QA)
- Check available disk space on OBS Server
- Perform Release Branching
- Publish Release
- Update channels for Apertis packages Dashboard
Release Branching
Once the Release Preparation is complete, the next step in the Release Process is to run the Release Branching steps.
At a higher level, Release Branching includes the following steps which needs to be run in close co-ordination.
- Branching all package repositories with the new release
- Branching all image recipes
- Branching all test recipes
- Branching docker image recipes
- Branching all OBS repositories
Release Branching through GitLab CI
The release process has been semi-automated with the GitLab CI framework. The following screenshots/notes will give a run through of the steps, to make a release.
Some terms in regard to release branching:
RELEASE
: The current release codenameNEXT_RELEASE
: The new release codename that should be branched toPREV_STABLE_RELEASE
: The previous stable release codename. Only used when branching a new stable release (i.e. not adev
orpre
release)PURPOSE
: Use stringbranching
to instruct CI that it is a release procedure jobNOACT
: Set to0
to trigger the release
By default, the release procedure jobs will run in dry-run mode, i. e. with NOACT
set to 1
. After running the release process in dry-run mode, go through the job logs to uncover potential issues. When ready, re-run the CI pipeline jobs with NOACT
set to 0
to actually apply the changes.
This is the initial step to trigger release branching. Here:
- Specify the current
RELEASE
to be branched, i.e.v2025dev2
NEXT_RELEASE
is the name of the new release that is to be branched, i.e.v2025dev3
- If the new release to be branched is a stable release (i.e. not a
dev
orpre
release), thenPREV_STABLE_RELEASE
needs to be defined, i.e. when branchingv2025
, this variable will be defined onv2024
PURPOSE
is set tobranching
to tell the CI Pipeline that this is a release branching job.
Note: NOACT
has been deliberately left out in this example screenshot. So, in this case, the entire procedure will run in dry-run
mode. To run in effective mode, pass NOACT
with value 0
The above example complete the semi-automated aspect of the Release Branching steps. For more information, please check Release Branching examples.
The job branch-protection-tighten
may fail with error messages such as:
project pkg/foo: settings: save changes -> error: 400: {'base': ["Could not change HEAD: branch 'apertis/v2024dev3' does not exist"]}
This happens when a package has been dropped in a newer Apertis release (which
frequently happens after a rebase), but the
rulez.yaml
has not been updated accordingly. The rulez.yaml
file needs to be updated, then
the pipeline can be re-triggered.
Post CI Manual Steps
After the completion of the Release Branching CI Jobs, a certain set of manual steps need to be performed, which are outlined below.
- Add a directory for the release to the LAVA NFS path at:
/srv/images/public/lava/nfs/
on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release’s directory path. Refer to the README for details. - Instantiate a new timer to rotate images for the new release (eg. On host
aurora
, run commandsystemctl enable --now rotate-images@v2024.timer
). - Update the ci-package-builder to target the next release.
- Update the gitlab-rulez recipe to point to the new development release.
- Set the default development branch for all git projects.
- Set the protection settings onto all
apertis/*
branches, which were unprotected in the initial release steps.
- Update the dashboard’s channels to track the new release.
- Trigger a build pipeline on the newly created branch of base-files package, after the new docker images have been built.
- This step should automatically update the release specific attributes in the new package and auto propose the changes through a new merge request.
- Validate the newly created pipeline ensuring that it generates the right set of changes with respect to the new release.
- Validate the new package is built proper and pushed to OBS and the APT repositories.
Stable Point Release
Point Releases for the Stable Release happen on a timely cadence as defined in the Release Schedule. A Stable Point Release includes:
- Security Updates to software packages (funneled through $RELEASE-security). These include security vulnerability fixes.
- Normal updates to software packages (funneled through $RELEASE-updates). These include bug fixes to existing version of software packages.
Stable Point Release Steps
- Keep release schedule wiki page updated
- Announce soft feature freeze
- Announce hard feature freeze
- Check available disk space on OBS Server
- Perform Release folding
- Full archive rebuild to ensure all the packages are buildable from source
- Write the release notes
- Confirm that no release blocker issue is present
- Publish Release Candidate (RC1) images and announce hard code freeze
- RC images validation (QA)
- Publish Release
- Update channels for Apertis packages Dashboard
Several of the steps above were already described in the previous section, the following section will describe those steps specific for stable point releases.
Folding Stable Point Release
Once the Release Preparation is complete, the next step in the Release Process is to run the Release Folding steps.
At a higher level, Release Folding is the process of merging the updates from security
and updates
branches into the main ones, and includes the following steps which needs to be run in close co-ordination.
- Fold changes in
security
andupdates
branches into the main ones - Confirm that newer versions of the packages are available in the main repositories
- Drop
security
andupdates
branches for the release in GitLab - Drop
security
andupdates
repositories for the release in OBS
These changes are performed by the fold-security-update-branches
script from the apertis-infrastructure/release-scripts/gitlab/, which automates many of the steps in a Stable Point Release Preparation:
- List the packages in the
:security
,:updates
and other similar repositories - Fast Forward Merge
--ff-only
the respective security and updates branches on Git. For example,apertis/v2024-security
andapertis/v2024-updates
should be merged back toapertis/v2024
. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed. - Poke the pipeline and ensure that all the new packages (from the merged back changes to
apertis/v2024
) get built on OBS and land into the main repository. - Drop the updates, security branches (Eg.
apertis/v2024-updates
,apertis/v2024-security
) - Delete the packages from the updates, security etc repositories on OBS (Eg.
apertis:v2024:updates:target
,apertis:v2024:security:development
)
The script can be run in a semi-automatic mode through a GitLab CI pipeline.
Similarly as with release branching, the pipeline runs in the dry-run mode by default.
To actually perform the changes, set NOACT
variable to 0
when triggering the pipeline.
The pipeline is organised in the following stages:
- Fold. In this stage, the script submits merge requests to all packages with updates in
updates
andsecurity
. Packages where these merge requests can be automatically merged, are set to auto-merge once the pipelines succeed. - Manual review. Since the script cannot properly handle all updates yet, manual merges are often necessary. See below for more details.
- Cleanup. After performing the reviews and verifying that all packages have been built and published in the main repositories, run this step to clean-up the GitLab branches and public apt repositories.
To make the review process easier, each component is processed in a separate job in the “fold” child pipeline:
At the end of each job’s logs, there’s a summary with status of each package and links to the merge requests:
The meaning of each status:
CLEAN
: this package needs no further processing other than possibly a clean-upNEEDS-MERGE
: this package is out of sync and needs a merge, but the dry-run mode is in forceMERGE
: a merge request has been submitted for this packageSKIP
: a package was found in one of the OBS repositories, but no corresponding branch was found in GitLabERROR
: an error occurred during processing this package, check the detailed logs above
A known situation when the pipeline submits a merge request that GitLab cannot process is when a new package has been introduced in updates or security. In such cases, the merge request is created against a non-existent branch, and it cannot be edited or merged using the GitLab UI, and the branch needs to be manually pushed.
Common tasks during release
OBS disk space
Archive rebuild, branching and folding will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary. Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
OBS is the build server used by Apertis to build the packages before they are published. During the process of the release it is heavily used by different stages, like archive rebuild, branching and folding.
Archive Rebuilds
Archive rebuild should be done in a separate OBS repository to avoid disrupting the main one. The only purpose is to ensure packages can be built from sources, so after this has been checked the repository can be dropped.
Archive Rebuilds are done before a release to ensure that the entire repository of packages are in good shape, can build proper, and that all their inter-dependencies are satisfied. The recommended timeline for an archive-wide rebuild is right after the Feature Freeze. Once a freeze is in place, we don’t anticipate new changes to be introduced, and thus this is the optimal time to re-build all the packages slated for the release.
Note: An archive-wide rebuild is a time consuming task. Depending on the build infrastructure, it can take large amount of time. Thus, it is important to account for this time and plan a rebuild accordingly to ensure we have proper build images for the first Release Candidate, which is typically 2 weeks from the Feature Freeze date.
Some notes about the Archive Rebuild
task
- Rebuilds are usually not required to be triggered manually.
- Archive wide rebuild of packages should never be triggered on the main (i.e.
default
) repositories. Use separaterebuild
repositories and delete them once the rebuilds complete. This is done by editing the project’sMeta Configuration
:- Disable publishing of the
rebuild
repositories by adding the following snippet:
<publish> <enable repository="default"/> <disable repository="rebuild"/> </publish>
- Create the
rebuild
repositories with this snippet:
<repository name="rebuild" rebuild="local" block="never"> <path project="apertis:v2024:target" repository="default"/> <arch>x86_64</arch> <arch>armv7hl</arch> <arch>aarch64</arch> </repository>
- Each
rebuild
repository depends on itsdefault
variant enabling it to have binary packages to bootstrap the rebuild. Without this dependency, packages will be in anunresolvable
state on OBS. - Ensure that all rebuild repositories have
rebuild="local" block="never"
in the project meta configuration. This is an important step needed to ensure that packages don’t get into cyclic rebuilds everytime a relevant library is built. - The rebuild will start as soon as OBS has computed the repositories’ state. Otherwise, rebuilds can be triggered with:
osc rebuild apertis:v2024:target --all -r rebuild
- Once all packages have been successfully built, delete
rebuild
repositories.
- Disable publishing of the
- Never trigger an archive wide rebuild on the primary
default
repositories. - In case of last minute changes in a package which could trigger dependencies issues, it is possible
to trigger the rebuild of packages in the dependency chain by setting
rebuild="transitive"
. This will instruct OBS to compute the list of packages that depend on new version of packages and schedule a rebuild of them.
Publish Release Candidate
- Run the
copy-release
script on the images server (aurora
) to prepare the RC. copy-release -v 20201203.0116 v2021pre.0rc1
copy-release -v -i 20201203.0116 v2021pre.0rc1
- First for the public images and then second for Internal images (-i option).
- The source folder is picked from the daily image builds.
- Send release announcement for the release.
Build Suffix
For every release (Developer, Preview, Product), we add a build suffix string to the packages, which relates to the release name of Apertis. The build suffix gets added to every built .deb package. Having a build suffix helps determine which Apertis release the .deb package was built for.
The suffix string is constructed of: b + Release Name + b<B_CNT>
- Where, the initial
b
is for backward compatibility ensuring the new suffix string can work well together with older release packages - String
Release Name
will refer to the Apertis release’s name - String,
b<B_CNT>
refers to being a binary build along with the build count
For example, for an Apertis Developer Release of v2025dev0
, we add string bv2025dev0b<B_CNT>
to the project configuration on OBS.
Similarly, for an Apertis Product Release of v2024.0
, we add string bv2024.0b<B_CNT>
to the project configuration on OBS
Stable Build Suffix
The Build Suffix
is a crucial metadata for an Apertis Stable release. Setting it correct is important for smooth package upgrades across different repository components of a Stable Apertis Release.
In a typical Apertis Release, we have multiple repository components
- Main repository components like:
apertis:v2024:development
,apertis:v2024:target
- Security repository components like:
apertis:v2024:security:development
,apertis:v2024:security:target
- Updates repository components like:
apertis:v2024:updates:development
,apertis:v2024:updates:target
- Backports repository components like:
apertis:v2024:backports:development
,apertis:v2024:backports:target
For packages landing into an Apertis Stable Release, it would land through either of security
or updates
repository components.
Packages accumulate into either of the repositories and eventually get folded into the main respective repository, just like the stable release procedure followed in the Debian project.
The above structure allows Apertis users to have immediate access to updates through either of the repository channels, provided those repositories are enabled in their apt configuration. For users that only follow the main repository, the same changes land into main on every Stable Point Release.
Given the structure above, of the lifetime of a package, across different repositories, it is important to set the correct revision for the package builds; to ensure that users’ have a smooth package upgrade experience. In short, we need to ensure that stable package updates, when finally landing into the main
repository have a higher build revision than the rest of repository components.
To achieve the above goal, consider the below project config metadata:
bv2024.0db<B_CNT> in v2024
bv2024.0cb<B_CNT> in v2024-security
bv2024.0bb<B_CNT> in v2024-updates
bv2024.0ab<B_CNT> in v2024-backports
In the above example, the revisioning is carefully ordered to ensure that when Stable Update packages finally land into the main repository, they have the proper revision.
Consider the above with an example of package foo
and version 1.0-1
. Under Apertis, its typical reflection would be something like: foo-1.0-1+apertis0bv2024.0db1_amd64.deb
which would be the initial version in the main repository. Over the course, we may push a new security update reflecting as foo-1.0-3+apertis0bv2024.0cb1_amd64.deb
which would live for a defined time until the next Stable Point Release in the security repository, like apertis:v2024:security:development
. Eventually, as part of the Stable Point Release, the package will be folded into the main repository, at which point, its reflection in the main repository will become: foo-1.0-3+apertis0bv2024.0db1_amd64.deb
.
Over its lifecycle in the Apertis v2024 release so far, package foo
has had 3 versions: 1.0-1+apertis0bv2024.0db1
, 1.0-3+apertis0bv2024.0cb1
and 1.0-3+apertis0bv2024.0db1
From apt/dpkg point of view:
$ dpkg --compare-versions 1.0-1+apertis0bv2024.0db1 lt 1.0-3+apertis0bv2024.0cb1 && echo "Latter version is greater"
Latter version is greater
Similarly, now for the example security update folded into main:
$ dpkg --compare-versions 1.0-3+apertis0bv2024.0cb1 lt 1.0-3+apertis0bv2024.0db1 && echo "Latter version is greater"
Latter version is greater
$ dpkg --compare-versions 1.0-3+apertis0bv2024.0cb2 lt 1.0-3+apertis0bv2024.0db1 && echo "Latter version is greater"
Latter version is greater
The above repository versioning scheme is reliable across the entire lifespan of an Apertis Stable Release. No further tweaks to project config are required.
Appendix
Recommendations for downstream distributions
Downstreams should use the same workflow and tools described in this section to automate and ensure a smooth release process. To do so they are encouraged to extend Apertis release scripts and CI to include their own repositories and other required delta. As in any other piece of software in the Apertis ecosystem it is recommended that downstreams propose new changes to these tools to make the release process smoother and more reliable.
Helper scripts to ease Apertis development
release-scripts/
: contains helper scripts to ease release process.
update-*
scripts create branches in key Git repositories for the current release and commit changes necessary to build images for the next release.create-branches
is a script to create branches for the current release in all Git repositories. This script requirescurl
andjq
.add-new-repo
is a script which updates thereprepro
configuration at the OBS backend host, and it has to run there as root. It depends on OSC clone plugin.do-branching.sh
is the main script used to branch the next release off in preparation for the current release.- In addition to
RELEASE
andNEXT_RELEASE
it also reacts onNOACT
skipping the execution of most commands when set to any non-empty value.NOVERIFY
(currently hardcoded to ‘y’) skips the verification of the package copy process. - NOTE: The branching script expects root access on certain hosts to be configured
in
~/.ssh/config
- In addition to
Release Branching by CI examples
The previous sections showed the steps to launch the branching through GitLab CI. This section provides additional information to developers on the use of this particular CI.
This is an overview of all the release branching jobs queued in the pipeline
This is an example of the same jobs, with their dependencies chalked out. There are multiple jobs that depend on one another. This is to ensure that jobs, that depend on certain outputs or tasks to be completed, are run only after.
This example highlights the dependency in action. A job is only run after its parent dependency job is run successfully. Jobs that do not have a connecting line are independent jobs that are run in parallel. In this example, most jobs in the branch
stage are independent and run in parallel
This example highlights Manual Jobs and their dependencies. Manual Jobs are differentiated with the Play Button. A Manual Job is to be run manually by a user. In the context of this document, a manual job constitutes of certain commands that the user is expected to run by hand on respective servers. Most manual jobs, in this case, are to be run on the OBS Backend server. For each manual job, the exact commands will be displayed in the job’s console view.
This example highlights the exact set of commands that should be manually run. Most commands to be manually run, are to be run on specific servers, which will be mentioned in the console output for the particular job.
Most manual jobs are made dependent on other job. This is to ensure that steps are performed in a particular order. In the above example, the dependency is defined as: obs_prjconf => obs_reprepro => obs_clone_binary
, where obs_prjconf
runs a certain set of tasks, which are needed by obs_reprepro
. Similarly obs_reprepro
performs a certain set of tasks which are a pre-requisite for obs_clone_binary
. The job chain is strictly defined and users should ensure to successfully run the manual jobs on respective servers, before progressing to the next job.
Skipping jobs
There can be scenarios during the Release Branching wherein the user would be required to re-run the entire pipeline. Under such situation, it can be desired to have certain specific jobs to be skipped from the consecutive pipeline run. For example, in first run, branch-create-mass-branches
was run and it created all the branches in respect to the NEXT_RELEASE
. For such cases, on consecutive run, it is desired to skip the branch-create-mass-branches
job. Below is an example on how to invoke the pipeline while defining a set of jobs to be skipped.
In the above example screenshot, we specify that the branch-create-mass-branches
and branch-update-tests-projects
jobs be skipped. The important bit is to pass the FORCE_JOB_SKIP
variable with the required values.
As was specified in the previous job invocation, this example pipeline job run has the branch-create-mass-branches
and branch-update-tests-projects
skipped.
Release Branching manual steps
Given the number of steps involved in Release Branching, the following Branching Checklists should be executed in close co-ordination.
Branching Checklists
There are a couple of checks to be performed on individual host machines. The machine names mentioned are specific and should be adopted to users’ environment
- Host
niobium
: Hosts the OBS service - Host
images.apertis.org
: Hosts the Apertis images repository - Host
gitlab.apertis.org
: Hosts the Apertis GitLab instance
Branching Preparation
- set up
~/.ssh/config
such that theniobium
alias is available with appropriateroot
privilege.
Host niobium
User andrewshR
Host images.apertis.org
User andrewshR
- Check available disk space on OBS Server
- Release Branching is a task that will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary.
- Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
- Install the
jq
package. It is already packaged in Apertis - Install the osc-plugin-clone plugin for the OBS client
- Install the
osc-plugins-dput
. It is already packaged in Apertis - Create a GitLab personal access token and configure it in
$HOME/.gitlab-apertis-org-token
, e.g.GITLAB_TOKEN=XXXXxxxxxXXXXXX > $HOME/.gitlab-apertis-org-token
- Make sure your API token is active. When you create your API token, you have an option to set an expiry date. So make sure to verify the validity of your granted token
- Ensure that the
osc
tool is configured with theAdmin
user credentials forOBS
. This is essential for the creation of additional repositories - Check out the apertis-infrastructure repository
- Install
gitlab-rulez
tool and unprotect the current release branch for all repositories.- Verify that all relevant package groups, like
pkg/*
, havepush_access_level: DEVELOPER_ACCESS
inapertis-infrastructure/gitlab-scripts/rulez.yaml
- Apply the rule with
gitlab-rulez apply apertis-infrastructure/gitlab-scripts/rulez.yaml
- Verify that all relevant package groups, like
- Change the working directory to release-scripts/
- Launch
export RELEASE=v2019; export NEXT_RELEASE=v2020dev0; ./do-branching.sh
which will:- Set new release to string v2020dev0 and set current release to string v2019, for example.
- Create a new debootstrap script entry on the image builder host
- Update the debootstrap package on OBS
- Update the apertis-customizations, apertis-image-customization, apertis-tests repositories, both public and internal
- Create branches in every repository on
gitlab.apertis.org
- Add the next-release aptly entry to the OBS configuration files (
/etc/obs/BSConfig.*.pm
files). - Restart OBS
publisher
andsrcserver
services to pick up the new aptly entry. - Stop the OBS schedulers
- Fork the OBS projects, copying packages and metadata
- Clone and republish the OBS binary repositories
- Restart the OBS schedulers
- Update build suffix on OBS Project configuration (eg.
bv2021.0b<B_CNT>
,bv2020dev0b<B_CNT>
). Please refer to section Build Suffix and Stable Build Suffix for more details about it
Removing an old distribution - Reprepro
Perform the following steps to remove an old distributions. This would usually be the
obsolete Beta and Preview distribution releases, like: v2019dev0
, v2019pre
-
Remove the entry for the distribution from the
reprepro
configuration file. Example:/srv/obs/repos/shared/apertis/internal/apertis/conf/distributions
-
Run the following
reprepro
command to apply the changes# sudo -u obsrun reprepro --gnupghome /srv/obs/gnupg/ -Vb /srv/obs/repos/shared/apertis/internal/apertis --delete clearvanished Deleting vanished identifier 'u|v2019dev0|proprietary|amd64'. Deleting vanished identifier 'u|v2019dev0|proprietary|arm64'. Deleting vanished identifier 'u|v2019dev0|proprietary|armhf'. Deleting vanished identifier 'u|v2019pre|demo|amd64'. Deleting vanished identifier 'u|v2019pre|demo|armhf'. Deleting vanished identifier 'u|v2019pre|nothumb|amd64'. Deleting vanished identifier 'u|v2019pre|nothumb|armhf'. Deleting vanished identifier 'u|v2019pre|proprietary|amd64'. Deleting vanished identifier 'u|v2019pre|proprietary|armhf'.
Dashboard update
At the end of the release process, it’s necessary to update the channels tracked
by the Apertis packages Dashboard.
This ensures that we focus on current releases.
As soon as we publish a new release, it can be marked as closed (with status: closed
)
in the channels.yaml,
this will hide some reports which are no longer of interest.
Two weeks after a release, it can be completely removed from this channels file.
This period allows us to keep an eye on a release after its publication in case
of potential issues while downstream projects are synchronizing.