Table of Contents:
This document aims as a single resource for all information related to Apertis Release Process. It covers the Major Release process as well as the Point Release process.
Apertis Infrastructure
The Apertis project is hosted on a couple of infrastructure services, which are tightly coupled to each other, providing end-to-end automation. This includes:
- Landing new packages into Apertis
- Landing new security and general updates into Apertis
- Landing downstream changes into Apertis
This document stands as a comprehensive guide to all package related processes as outlined above.
At a higher level, Apertis infrastructure includes:
- Gitlab instance for source code hosting, CI, Automation, Image building etc
- OBS (Open Build Service) for package building
- aptly for APT repository management
- images.apertis.org for Apertis image hosting
- LAVA for Quality Assurance Testing
Development/Preview Releases
As described in Release flow and product lines Apertis ships three types of releases: development, preview and stable. This section describes the release process for development and preview releases.
Development/Preview Release Steps
As a high level overview, these are the steps to be executed during the release cycle, to perform a release:
- Keep release schedule wiki page updated
- Announce soft feature freeze
- Check available disk space on OBS Server
- Full archive rebuild to ensure all the packages are buildable from source
- Announce hard feature freeze
- Update the freshness status of all the design documents
- Write the release notes
- Confirm that no release blocker issue is present
- Publish Release Candidate (RC1) images and announce hard code freeze
- RC images validation (QA)
- Check available disk space on OBS Server
- Perform Release Branching
- Publish Release
Release Branching
Once the Release Preparation is complete, the next step in the Release Process is to run the Release Branching steps.
At a higher level, Release Branching includes the following steps which needs to be run in close co-ordination.
- Branching all package repositories with the new release
- Branching all image recipes
- Branching all test recipes
- Branching docker image recipes
- Branching all OBS repositories
Release Branching through Gitlab CI
The release process has been semi-automated with the Gitlab CI framework. The following screenshots/notes will give a run through of the steps, to make a release.
Some terms in regard to release branching:
RELEASE
: The current release codenameNEXT_RELEASE
: The new release codename that should be branched toPURPOSE
: Use stringbranching
to instruct CI that it is a release procedure jobNOACT
: Set to0
to trigger the release
By default, the release procedure job will run in dry-run
mode, i.e. with NOACT:1
. It is a good practice to first run the release process in dry-run
mode to uncover any possible issues. When ready, re-run the CI pipeline jobs with NOACT:0
to run in real
mode.
This is the inital step to trigger release branching. Here:
- We specify the current
RELEASE
to be branched, i.e.v2023dev0
NEXT_RELEASE
is the name of the new release that is to be branched, i.e.v2023dev1
PURPOSE
is set tobranching
to tell the CI Pipeline that this is a release branching job.
Note: NOACT
has been deliberately left out in this example screenshot. So, in this case, the entire procedure will run in dry-run
mode. To run in effective mode, pass NOACT
with value 0
The above example complete the semi-automated aspect of the Release Branching steps. For more information, please check Release Branching examples.
Post CI Manual Steps
After the completion of the Release Branching CI Jobs, a certain set of manual steps need to be performed, which are outlined below.
- Add a directory for the release to the LAVA NFS path at:
/srv/images/public/lava/nfs/
on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release’s directory path. Refer to the README for details - Instantiate a new timer to rotate images for the new release (eg. On host
aura
, run commandsystemctl enable --now rotate-images@v2019.timer
) - Update the ci-package-builder to target the next release
- Update the gitlab-rulez recipe to point to the new development release
- Set the default development branch for all git projects
- Set the protection settings onto all
apertis/*
branches, which were unprotected in the initial release steps
- Manually branch the base-files package, after the new docker images have been built.
- This step should automatically update the release specific attributes in the new package and auto propose the changes through a new merge request
- Validate the newly created pipeline ensuring that it generates the right set of changes with respect to the new release.
- Validate the new package is built proper and pushed to OBS and the APT repositories
Stable Point Release
Point Releases for the Stable Release happen on a timely cadence as defined in the Release Schedule. A Stable Point Release includes:
- Security Updates to software packages (funneled through $RELEASE-security). These include security vulnerability fixes.
- Normal updates to software packages (funneled through $RELEASE-updates). These include bug fixes to existing version of software packages.
- Backport of software packages (funneled through $RELEASE-backports). These include newer version of software packages backported to older OS releases
Stable Point Release Steps
- Keep release schedule wiki page updated
- Announce soft feature freeze
- Announce hard feature freeze
- Check available disk space on OBS Server
- Perform Release folding
- Full archive rebuild to ensure all the packages are buildable from source
- Write the release notes
- Confirm that no release blocker issue is present
- Publish Release Candidate (RC1) images and announce hard code freeze
- RC images validation (QA)
- Publish Release
Several of the steps above were already described in the previous section, the following section will describe those steps specific for stable point releases.
Folding Stable Point Release
Once the Release Preparation is complete, the next step in the Release Process is to run the Release Folding steps.
At a higher level, Release Folding is the process of merging the updates from security
, updates
and backports
branches into the main ones, and includes the following steps which needs to be run in close co-ordination.
- Update Build Suffix
- Fold changes in
security
,updates
andbackports
branches into the main ones - Confirm that newer versions of the packages are available in the main repositories
- Drop
security
,updates
andbackports
branches for the release in Gitlab - Drop
security
,updates
andbackports
repositories for the release in OBS
To perform these changes launch fold-security-update-branches
script from the apertis-infrastructure/release-scripts/gitlab/, which automates many of the steps in a Stable Point Release Preparation
- Increment the Build Suffix in the Project configuration for each repository relevant to the release
- List the packages in the
:security
,:updates
and other similar repositories - Fast Forward Merge
--ff-only
the respective security and updates branches on Git. For example,apertis/v2020-security
andapertis/v2020-updates
should be merged back toapertis/v2020
. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed. - Poke the pipeline and ensure that all the new packages (from the merged back changes to
apertis/v2020
) get built on OBS and land into the main repository. - Drop the updates, security branches (Eg.
apertis/v2020-updates
,apertis/v2020-security
) - Delete the packages from the updates, security etc repositories on OBS (Eg.
apertis:v2020:updates:target
,apertis:v2020:security:development
)
Note: The cleanup of the public apt repositories is to be run separately, after validation of the folded packages in the main repositories
See Folding examples for more information.
Common tasks during release
OBS disk space
Archive rebuild, branching and folding will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary. Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
OBS is the build server used by Apertis to build the packages before they are published. During the process of the release it is heavily used by different stages, like archive rebuild, branching and folding.
Archive Rebuilds
Archive rebuild should be done in a separate OBS repository to avoid disrupting the main one. The only purpose is to ensure packages can be built from sources, so after this has been checked the repository can be dropped.
Archive Rebuilds are done before a release to ensure that the entire repository of packages are in good shape, can build proper, and that all their inter-dependencies are satisfied. The recommended timeline for an archive-wide rebuild is right after the Feature Freeze. Once a freeze is in place, we don’t anticipate new changes to be introduced, and thus this is the optimal time to re-build all the packages slated for the release.
Note: An archive-wide rebuild is a time consuming task. Depending on the build infrastructure, it can take large amount of time. Thus, it is important to account for this time and plan a rebuild accordingly to ensure we have proper build images for the first Release Candidate, which is typicall 2 weeks from the Feature Freeze date.
Some notes about the Archive Rebuild
task
- Rebuilds are usually not required to be triggered manually.
- Archive wide rebuild of packages should never be triggered on the main repositories. Use separate rebuild repositories and delete them once the rebuilds complete:
- Create the rebuild repositories. Use automation script
clone-rebuilds.sh
from the apertis-infrastructure repository. - Ensure that all rebuild repositories have
rebuild="local" block="never"
in the project meta configuration. This is an important step needed to ensure that packages don’t get into cyclic rebuilds everytime a relevant library is built.<repository name="rebuild" rebuild="local" block="never"> <path project="apertis:v2021dev3:target" repository="default"/> <arch>x86_64</arch> <arch>armv7hl</arch> <arch>aarch64</arch> </repository>
- Trigger the rebuilds:
osc rebuild apertis:v2019:target --all -r rebuild
- Delete rebuild repositories
- Never trigger an archive wide rebuild on the primary repositories like target, sdk, development, hmi, helper-libs etc.
Build Suffix
For every release (Developer, Preview, Product), we add a build suffix string to the packages, which relates to the release name of Apertis. The build suffix gets added to every built .deb package. Having a build suffix helps determine which Apertis release the .deb package was built for.
The suffix string is constructed of: b + Release Name + b<B_CNT>
- Where, the initial
b
is for backward compatibility ensuring the new suffix string can work well together with older release packages - String
Release Name
will refer to the Apertis release’s name - String,
b<B_CNT>
refers to being a binary build along with the build count
For example, for an Apertis Developer Release of v2020dev0
, we add string bv2020dev0b<B_CNT>
to the project configuration on OBS.
Similarly, for an Apertis Product Release of v2019.0
, we add string bv2019.0b<B_CNT>
to the project configuration on OBS
Publish Release Candidate
- Run the
copy-release
script on the images server (aura) to prepare the RC. copy-release -v 20201203.0116 v2021pre.0rc1
copy-release -v -i 20201203.0116 v221pre.0rc1
- First for the public images and then second for Internal images (-i option).
- The source folder is picked from the daily image builds.
- Send release announcement for the release.
Publish Release
- Run the
copy-release
script on the images server (aura) to prepare the RC. copy-release -v -r v2021pre.0rc1 v2021pre.0
copy-release -v -i -r v2021pre.0rc1 v2021pre.0
- First for the public images and then second for Internal images (-i option).
- Send release announcement for the release.
Appendix
Recommendations for downstream distributions
Downstreams should use the same workflow and tools described in this section to automate and ensure a smooth release process. To do so they are encouraged to extend Apertis release scripts and CI to include their own repositories and other required delta. As in any other piece of software in the Apertis ecosystem it is recommended that downstreams propose new changes to these tools to make the release process smoother and more reliable.
Helper scripts to ease Apertis development
release-scripts/
: contains helper scripts to ease release process.
update-*
scripts create branches in key Git repositories for the current release and commit changes necessary to build images for the next release.create-branches
is a script to create branches for the current release in all Git repositories. This script requirescurl
andjq
.add-new-repo
is a script which updates thereprepro
configuration at the OBS backend host, and it has to run there as root. It depends on OSC clone plugin.do-branching.sh
is the main script used to branch the next release off in preparation for the current release.- In addition to
RELEASE
andNEXT_RELEASE
it also reacts onNOACT
skipping the execution of most commands when set to any non-empty value.NOVERIFY
(currently hardcoded to ‘y’) skips the verification of the package copy process. - NOTE: The branching script expects root access on certain hosts to be configured
in
~/.ssh/config
- In addition to
Release Branching by CI examples
The previous sections showed the steps to launch the branching through Gitlab CI. This section provides additional information to developers on the use of this particular CI.
This is an overview of all the release branching jobs queued in the pipeline
This is an example of the same jobs, with their dependencies chalked out. There are multiple jobs that depend on one another. This is to ensure that jobs, that depend on certain outputs or tasks to be completed, are run only after.
This example highlights the dependency in action. A job is only run after its parent dependency job is run successfully. Jobs that do not have a connecting line are independent jobs that are run in parallel. In this exmaple, most jobs in the branch
stage are independent and run in parallel
This example highlights Manual Jobs and their dependencies. Manual Jobs are differentiated with the Play Button. A Manual Job is to be run manually by a user. In the context of this document, a manual job constitutes of certain commands that the user is expected to run by hand on respective servers. Most manual jobs, in this case, are to be run on the OBS Backend server. For each manual job, the exact commands will be displayed in the job’s console view.
This example highlights the exact set of commands that should be manually run. Most commands to be manually run, are to be run on specific servers, which will be mentioned in the console output for the particular job.
Most manual jobs are made dependent on other job. This is to ensure that steps are performed in a particular order. In the above example, the dependency is defined as: obs_prjconf => obs_reprepro => obs_clone_binary
, where obs_prjconf
runs a certain set of tasks, which are needed by obs_reprepro
. Similarly obs_reprepro
performs a certain set of tasks which are a pre-requisite for obs_clone_binary
. The job chain is strictly defined and users should ensure to successfully run the manual jobs on respective servers, before progressing to the next job.
Skipping jobs
There can be scenarios during the Release Branching wherein the user would be required to re-run the entire pipeline. Under such situation, it can be desired to have certain specific jobs to be skipped from the consecutive pipeline run. For example, in first run, branch-create-mass-branches
was run and it created all the branches in respect to the NEXT_RELEASE
. For such cases, on consecutive run, it is desired to skip the branch-create-mass-branches
job. Below is an example on how to invoke the pipeline while defining a set of jobs to be skipped.
In the above example screenshot, we specify that the branch-create-mass-branches
and branch-update-tests-projects
jobs be skipped. The important bit is to pass the FORCE_JOB_SKIP
variable with the required values.
As was specified in the previous job invocaton, this example pipeline job run has the branch-create-mass-branches
and branch-update-tests-projects
skipped.
Release Branching manual steps
Given the number of steps involved in Release Branching, the following Branching Checklists should be executed in close co-ordination.
Branching Checklists
There are a couple of checks to be performed on individual host machines. The machine names mentioned are specific and should be adopted to users’ environment
- Host
niobium
: Hosts the OBS service - Host
images.apertis.org
: Hosts the Apertis images repository - Host
gitlab.apertis.org
: Hosts the Apertis Gitlab instance
Branching Preparation
- set up
~/.ssh/config
such that theniobium
alias is available with appropriateroot
privilege.
Host niobium
User andrewshR
Host images.apertis.org
User andrewshR
- Check available disk space on OBS Server
- Release Branching is a task that will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary.
- Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
- Install the
jq
package. It is already packaged in Apertis - Install the osc-plugin-clone plugin for the OBS client
- Install the
osc-plugins-dput
. It is already packaged in Apertis - Create a GitLab personal access token and configure it in
$HOME/.gitlab-apertis-org-token
, e.g.GITLAB_TOKEN=XXXXxxxxxXXXXXX > $HOME/.gitlab-apertis-org-token
- Make sure your API token is active. When you create your API token, you have an option to set an expiry date. So make sure to verify the validity of your granted token
- Ensure that the
osc
tool is configured with theAdmin
user credentials forOBS
. This is essential for the creation of additional repositories - Check out the apertis-infrastructure repository
- Install
gitlab-rulez
tool and unprotect the current release branch for all repositories.- Verify that all relevant package groups, like
pkg/*
, havepush_access_level: DEVELOPER_ACCESS
inapertis-infrastructure/gitlab-scripts/rulez.yaml
- Apply the rule with
gitlab-rulez apply apertis-infrastructure/gitlab-scripts/rulez.yaml
- Verify that all relevant package groups, like
- Change the working directory to release-scripts/
- Launch
export RELEASE=v2019; export NEXT_RELEASE=v2020dev0; ./do-branching.sh
which will:- Set new release to string v2020dev0 and set current release to string v2019, for example.
- Create a new debootstrap script entry on the image builder host
- Update the debootstrap package on OBS
- Update the apertis-customizations, apertis-image-customization, apertist-tests repositories, both public and internal
- Create branches in every repository on
gitlab.apertis.org
- Add the next-release aptly entry to the OBS configuration files (
/etc/obs/BSConfig.*.pm
files). - Restart OBS
publisher
andsrcserver
services to pick up the new aptly entry. - Stop the OBS schedulers
- Fork the OBS projects, copying packages and metadata
- Clone and republish the OBS binary repositories
- Restart the OBS schedulers
- Update build suffix on OBS Project configuration (eg.
bv2021.0b<B_CNT>
,bv2020dev0b<B_CNT>
). Please refer to section Build Suffix and Stable Build Suffix for more details about it
Folding examples
Following are the example runs for the automation script.
Example Run with help sub-command
$ ./fold-security-update-branches help
Usage: BASE_RELEASE='v2021' ./fold-security-update-branches [apply] [gen-pkg-list] [dry-run] [clean-obs] [clean-reprepro] [clean-obs-reprepro] [help]
Example: BASE_RELEASE='v2021' ./fold-security-update-branches apply 2>&1 | tee -a console.log
Example Run with gen-pkg-list sub-command
The gen-pkg-list
sub-command generates a bunch of data files, based on the Apertis Release and Apertis Repositories. These generated data files are used by the script to process changes (Security and Regular Updates) in Apertis point releases.
rrs@priyasi:~/rrs-home/devel/CCU/Apertis/apertis-infrastructure/release-scripts/gitlab (v2020-folding-work-data)$ ./fold-security-update-branches gen-pkg-list
Package lists extracted from OBS' repository v2020-security-target.txt
Package lists extracted from OBS' repository v2020-updates-target.txt
Package lists extracted from OBS' repository v2020-security-development.txt
Package lists extracted from OBS' repository v2020-updates-development.txt
Package lists extracted from OBS' repository v2020-security-sdk.txt
Package lists extracted from OBS' repository v2020-updates-sdk.txt
Package lists extracted from OBS' repository v2020-security-hmi.txt
Package lists extracted from OBS' repository v2020-updates-hmi.txt
Example Run with dry-run sub-command
The dry-run
sub-command will run the actual commands without any effect on the server-side. This is very handy to check and determine the actual actions that will be run.
$ ./fold-security-update-branches dry-run
Package apt is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package bluez is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package boost1.67 is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package cups is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Example Run with apply sub-command
The apply
sub-command does the actual job.
$ ./fold-security-update-branches apply
Running in apply mode
Package apt is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/apt/-/merge_requests/10
Package bluez is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/bluez/-/merge_requests/12
Package boost1.67 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/boost1.67/-/merge_requests/5
Package cups is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/cups/-/merge_requests/8
Package curl is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/curl/-/merge_requests/7
Package cyrus-sasl2 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
Removing an old distribution - Reprepro
Perform the following steps to remove an old distributions. This would usually be the
obsolete Beta and Preview distribution releases, like: v2019dev0
, v2019pre
-
Remove the entry for the distribution from the
reprepro
configuration file. Example:/srv/obs/repos/shared/apertis/internal/apertis/conf/distributions
-
Run the following
reprepro
command to apply the changes# sudo -u obsrun reprepro --gnupghome /srv/obs/gnupg/ -Vb /srv/obs/repos/shared/apertis/internal/apertis --delete clearvanished Deleting vanished identifier 'u|v2019dev0|proprietary|amd64'. Deleting vanished identifier 'u|v2019dev0|proprietary|arm64'. Deleting vanished identifier 'u|v2019dev0|proprietary|armhf'. Deleting vanished identifier 'u|v2019pre|demo|amd64'. Deleting vanished identifier 'u|v2019pre|demo|armhf'. Deleting vanished identifier 'u|v2019pre|nothumb|amd64'. Deleting vanished identifier 'u|v2019pre|nothumb|armhf'. Deleting vanished identifier 'u|v2019pre|proprietary|amd64'. Deleting vanished identifier 'u|v2019pre|proprietary|armhf'.