Table of Contents:

This document aims as a single resource for all information related to Apertis Release Process. It covers the Major Release process as well as the Point Release process.

Apertis Infrastructure

The Apertis project is hosted on a couple of infrastructure services, which are tightly coupled to each other, providing end-to-end automation. This includes:

• Landing new packages into Apertis
• Landing new security and general updates into Apertis
• Landing downstream changes into Apertis

This document stands as a comprehensive guide to all package related processes as outlined above.

At a higher level, Apertis infrastructure includes:

• Gitlab instance for source code hosting, CI, Automation, Image building etc
• OBS (Open Build Service) for package building
• aptly for APT repository management
• images.apertis.org for Apertis image hosting
• LAVA for Quality Assurance Testing

Development/Preview Releases

As described in Release flow and product lines Apertis ships three types of releases: development, preview and stable. This section describes the release process for development and preview releases.

Development/Preview Release Steps

As a high level overview, these are the steps to be executed during the release cycle, to perform a release:

• Keep release schedule wiki page updated
• Announce soft feature freeze
• Check available disk space on OBS Server
• Full archive rebuild to ensure all the packages are buildable from source
• Announce hard feature freeze
• Update the freshness status of all the design documents
• Write the release notes
• Confirm that no release blocker issue is present
• Publish Release Candidate (RC1) images and announce hard code freeze
• RC images validation (QA)
• Check available disk space on OBS Server
• Perform Release Branching
• Publish Release

Release Branching

Once the Release Preparation is complete, the next step in the Release Process is to run the Release Branching steps.

At a higher level, Release Branching includes the following steps which needs to be run in close co-ordination.

• Branching all package repositories with the new release
• Branching all image recipes
• Branching all test recipes
• Branching docker image recipes
• Branching all OBS repositories

Release Branching through Gitlab CI

The release process has been semi-automated with the Gitlab CI framework. The following screenshots/notes will give a run through of the steps, to make a release.

Some terms in regard to release branching:

• RELEASE: The current release codename
• NEXT_RELEASE: The new release codename that should be branched to
• PURPOSE: Use string branching to instruct CI that it is a release procedure job
• NOACT: Set to 0 to trigger the release

By default, the release procedure job will run in dry-run mode, i.e. with NOACT:1. It is a good practice to first run the release process in dry-run mode to uncover any possible issues. When ready, re-run the CI pipeline jobs with NOACT:0 to run in real mode.

This is the inital step to trigger release branching. Here:

• We specify the current RELEASE to be branched, i.e. v2023dev0
• NEXT_RELEASE is the name of the new release that is to be branched, i.e. v2023dev1
• PURPOSE is set to branching to tell the CI Pipeline that this is a release branching job.

Note: NOACT has been deliberately left out in this example screenshot. So, in this case, the entire procedure will run in dry-run mode. To run in effective mode, pass NOACT with value 0

The above example complete the semi-automated aspect of the Release Branching steps. For more information, please check Release Branching examples.

Post CI Manual Steps

After the completion of the Release Branching CI Jobs, a certain set of manual steps need to be performed, which are outlined below.

• Add a directory for the release to the LAVA NFS path at: /srv/images/public/lava/nfs/ on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release’s directory path. Refer to the README for details
• Instantiate a new timer to rotate images for the new release (eg. On host aura, run command systemctl enable --now rotate-images@v2019.timer)
• Update the ci-package-builder to target the next release
• Update the gitlab-rulez recipe to point to the new development release
  • Set the default development branch for all git projects
  • Set the protection settings onto all apertis/* branches, which were unprotected in the initial release steps
• Manually branch the base-files package, after the new docker images have been built.
  • This step should automatically update the release specific attributes in the new package and auto propose the changes through a new merge request
  • Validate the newly created pipeline ensuring that it generates the right set of changes with respect to the new release.
  • Validate the new package is built proper and pushed to OBS and the APT repositories

Stable Point Release

Point Releases for the Stable Release happen on a timely cadence as defined in the Release Schedule. A Stable Point Release includes:

• Security Updates to software packages (funneled through $RELEASE-security). These include security vulnerability fixes.
• Normal updates to software packages (funneled through $RELEASE-updates). These include bug fixes to existing version of software packages.
• Backport of software packages (funneled through $RELEASE-backports). These include newer version of software packages backported to older OS releases

Stable Point Release Steps

• Keep release schedule wiki page updated
• Announce soft feature freeze
• Announce hard feature freeze
• Check available disk space on OBS Server
• Perform Release folding
• Full archive rebuild to ensure all the packages are buildable from source
• Write the release notes
• Confirm that no release blocker issue is present
• Publish Release Candidate (RC1) images and announce hard code freeze
• RC images validation (QA)
• Publish Release

Several of the steps above were already described in the previous section, the following section will describe those steps specific for stable point releases.

Folding Stable Point Release

Once the Release Preparation is complete, the next step in the Release Process is to run the Release Folding steps.

At a higher level, Release Folding is the process of merging the updates from security, updates and backports branches into the main ones, and includes the following steps which needs to be run in close co-ordination.

• Update Build Suffix
• Fold changes in security, updates and backports branches into the main ones
• Confirm that newer versions of the packages are available in the main repositories
• Drop security, updates and backports branches for the release in Gitlab
• Drop security, updates and backports repositories for the release in OBS

To perform these changes launch fold-security-update-branches script from the apertis-infrastructure/release-scripts/gitlab/, which automates many of the steps in a Stable Point Release Preparation

• Increment the Build Suffix in the Project configuration for each repository relevant to the release
• List the packages in the :security, :updates and other similar repositories
• Fast Forward Merge --ff-only the respective security and updates branches on Git. For example, apertis/v2020-security and apertis/v2020-updates should be merged back to apertis/v2020. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed.
• Poke the pipeline and ensure that all the new packages (from the merged back changes to apertis/v2020) get built on OBS and land into the main repository.
• Drop the updates, security branches (Eg. apertis/v2020-updates, apertis/v2020-security)
• Delete the packages from the updates, security etc repositories on OBS (Eg. apertis:v2020:updates:target, apertis:v2020:security:development)

Note: The cleanup of the public apt repositories is to be run separately, after validation of the folded packages in the main repositories

See Folding examples for more information.

Common tasks during release

OBS disk space

OBS is the build server used by Apertis to build the packages before they are published. During the process of the release it is heavily used by different stages, like archive rebuild, branching and folding.

Archive Rebuilds

Archive Rebuilds are done before a release to ensure that the entire repository of packages are in good shape, can build proper, and that all their inter-dependencies are satisfied. The recommended timeline for an archive-wide rebuild is right after the Feature Freeze. Once a freeze is in place, we don’t anticipate new changes to be introduced, and thus this is the optimal time to re-build all the packages slated for the release.

Note: An archive-wide rebuild is a time consuming task. Depending on the build infrastructure, it can take large amount of time. Thus, it is important to account for this time and plan a rebuild accordingly to ensure we have proper build images for the first Release Candidate, which is typicall 2 weeks from the Feature Freeze date.

Some notes about the Archive Rebuild task

• Rebuilds are usually not required to be triggered manually.
• Archive wide rebuild of packages should never be triggered on the main repositories. Use separate rebuild repositories and delete them once the rebuilds complete:
• Create the rebuild repositories. Use automation script clone-rebuilds.sh from the apertis-infrastructure repository.
• Ensure that all rebuild repositories have rebuild="local" block="never" in the project meta configuration. This is an important step needed to ensure that packages don’t get into cyclic rebuilds everytime a relevant library is built.

     <repository name="rebuild" rebuild="local" block="never">
       <path project="apertis:v2021dev3:target" repository="default"/>
       <arch>x86_64</arch>
       <arch>armv7hl</arch>
       <arch>aarch64</arch>
     </repository>
  

• Trigger the rebuilds: osc rebuild apertis:v2019:target --all -r rebuild
• Delete rebuild repositories
• Never trigger an archive wide rebuild on the primary repositories like target, sdk, development, hmi, helper-libs etc.

Publish Release Candidate

• Run the copy-release script on the images server (aura) to prepare the RC.
• copy-release -v 20201203.0116 v2021pre.0rc1
• copy-release -v -i 20201203.0116 v221pre.0rc1
• First for the public images and then second for Internal images (-i option).
• The source folder is picked from the daily image builds.
• Send release announcement for the release.

Build Suffix

For every release (Developer, Preview, Product), we add a build suffix string to the packages, which relates to the release name of Apertis. The build suffix gets added to every built .deb package. Having a build suffix helps determine which Apertis release the .deb package was built for.

The suffix string is constructed of: b + Release Name + b<B_CNT>

• Where, the initial b is for backward compatibility ensuring the new suffix string can work well together with older release packages
• String Release Name will refer to the Apertis release’s name
• String, b<B_CNT> refers to being a binary build along with the build count

For example, for an Apertis Developer Release of v2020dev0, we add string bv2020dev0b<B_CNT> to the project configuration on OBS. Similarly, for an Apertis Product Release of v2019.0, we add string bv2019.0b<B_CNT> to the project configuration on OBS

Stable Build Suffix

The Build Suffix is a crucial metadata for an Apertis Stable release. Setting it correct is important for smooth package upgrades across different repository components of a Stable Apertis Release.

In a typical Apertis Release, we have multiple repository components

• Main repository components like: apertis:v2023:development, apertis:v2023:target
• Security repository components like: apertis:v2023:security:development, apertis:v2023:security:target
• Updates repository components like: apertis:v2023:updates:development, apertis:v2023:updates:target
• Backports repository components like: apertis:v2023:backports:development, apertis:v2023:backports:target

For packages landing into an Apertis Stable Release, it would land through either of security, updates, or backports repository components. Packages accumulate into either of the repositories and eventually get folded into the main respective repository, just like the stable release procedure followed in the Debian project.

The above structure allows Apertis users to have immediate access to updates through either of the repository channels, provided those repositories are enabled in their apt configuration. For users that only follow the main repository, the same changes land into main on every Stable Point Release.

Given the structure above, of the lifetime of a pacakge, across different repositories, it is important to set the correct revision for the pacakge builds; to ensure that users’ have a smooth package upgrade experience. In short, we need to ensure that stable package updates, when finally landing into the main repository have a higher build revision than the rest of repository components.

To achive the above goal, consider the below project config metadata:

bv2023.0db<B_CNT> in v2023
bv2023.0cb<B_CNT> in v2023-security
bv2023.0bb<B_CNT> in v2023-updates
bv2023.0ab<B_CNT> in v2023-backports

In the above example, the revisioning is carefully ordered to ensure that when Stable Update packages finally land into the main repository, they have the proper revision.

Consider the above with an example of package foo and version 1.0-1. Under Apertis, its typical reflection would be something like: foo-1.0-1+apertis0bv2023.0db1_amd64.deb which would be the initial version in the main repository. Over the course, we may push a new security update reflecting as foo-1.0-3+apertis0bv2023.0cb1_amd64.deb which would live for a defined time until the next Stable Point Release in the security repository, like apertis:v2023:security:development. Eventually, as part of the Stable Point Release, the package will be folded into the main repository, at which point, its reflection in the main repository will become: foo-1.0-3+apertis0bv2023.0db1_amd64.deb.

Over its lifecycle in the Apertis v2023 release so far, package foo has had 3 versions: 1.0-1+apertis0bv2023.0db1, 1.0-3+apertis0bv2023.0cb1 and 1.0-3+apertis0bv2023.0db1

From apt/dpkg point of view:

$ dpkg --compare-versions 1.0-1+apertis0bv2023.0db1 lt 1.0-3+apertis0bv2023.0cb1 && echo "Latter version is greater"
Latter version is greater

Similarly, now for the example security update folded into main:

$ dpkg --compare-versions 1.0-3+apertis0bv2023.0cb1 lt 1.0-3+apertis0bv2023.0db1 && echo "Latter version is greater"
Latter version is greater

$ dpkg --compare-versions 1.0-3+apertis0bv2023.0cb2 lt 1.0-3+apertis0bv2023.0db1 && echo "Latter version is greater"
Latter version is greater

The above repository versioning scheme is reliable across the entire lifespan of an Apertis Stable Release. No further tweaks to project config are required.

Appendix

Recommendations for downstream distributions

Downstreams should use the same workflow and tools described in this section to automate and ensure a smooth release process. To do so they are encouraged to extend Apertis release scripts and CI to include their own repositories and other required delta. As in any other piece of software in the Apertis ecosystem it is recommended that downstreams propose new changes to these tools to make the release process smoother and more reliable.

Helper scripts to ease Apertis development

release-scripts/: contains helper scripts to ease release process.

• update-* scripts create branches in key Git repositories for the current release and commit changes necessary to build images for the next release.
• create-branches is a script to create branches for the current release in all Git repositories. This script requires curl and jq.
• add-new-repo is a script which updates the reprepro configuration at the OBS backend host, and it has to run there as root. It depends on OSC clone plugin.
• do-branching.sh is the main script used to branch the next release off in preparation for the current release.
  • In addition to RELEASE and NEXT_RELEASE it also reacts on NOACT skipping the execution of most commands when set to any non-empty value. NOVERIFY (currently hardcoded to ‘y’) skips the verification of the package copy process.
  • NOTE: The branching script expects root access on certain hosts to be configured in ~/.ssh/config

Release Branching by CI examples

The previous sections showed the steps to launch the branching through Gitlab CI. This section provides additional information to developers on the use of this particular CI.

This is an overview of all the release branching jobs queued in the pipeline

This is an example of the same jobs, with their dependencies chalked out. There are multiple jobs that depend on one another. This is to ensure that jobs, that depend on certain outputs or tasks to be completed, are run only after.

This example highlights the dependency in action. A job is only run after its parent dependency job is run successfully. Jobs that do not have a connecting line are independent jobs that are run in parallel. In this exmaple, most jobs in the branch stage are independent and run in parallel

This example highlights Manual Jobs and their dependencies. Manual Jobs are differentiated with the Play Button. A Manual Job is to be run manually by a user. In the context of this document, a manual job constitutes of certain commands that the user is expected to run by hand on respective servers. Most manual jobs, in this case, are to be run on the OBS Backend server. For each manual job, the exact commands will be displayed in the job’s console view.

This example highlights the exact set of commands that should be manually run. Most commands to be manually run, are to be run on specific servers, which will be mentioned in the console output for the particular job.

Most manual jobs are made dependent on other job. This is to ensure that steps are performed in a particular order. In the above example, the dependency is defined as: obs_prjconf => obs_reprepro => obs_clone_binary, where obs_prjconf runs a certain set of tasks, which are needed by obs_reprepro. Similarly obs_reprepro performs a certain set of tasks which are a pre-requisite for obs_clone_binary. The job chain is strictly defined and users should ensure to successfully run the manual jobs on respective servers, before progressing to the next job.

Skipping jobs

There can be scenarios during the Release Branching wherein the user would be required to re-run the entire pipeline. Under such situation, it can be desired to have certain specific jobs to be skipped from the consecutive pipeline run. For example, in first run, branch-create-mass-branches was run and it created all the branches in respect to the NEXT_RELEASE. For such cases, on consecutive run, it is desired to skip the branch-create-mass-branches job. Below is an example on how to invoke the pipeline while defining a set of jobs to be skipped.

In the above example screenshot, we specify that the branch-create-mass-branches and branch-update-tests-projects jobs be skipped. The important bit is to pass the FORCE_JOB_SKIP variable with the required values.

As was specified in the previous job invocaton, this example pipeline job run has the branch-create-mass-branches and branch-update-tests-projects skipped.

Release Branching manual steps

Given the number of steps involved in Release Branching, the following Branching Checklists should be executed in close co-ordination.

Branching Checklists

There are a couple of checks to be performed on individual host machines. The machine names mentioned are specific and should be adopted to users’ environment

• Host niobium: Hosts the OBS service
• Host images.apertis.org: Hosts the Apertis images repository
• Host gitlab.apertis.org: Hosts the Apertis Gitlab instance

Branching Preparation

• set up ~/.ssh/config such that the niobium alias is available with appropriate root privilege.

Host niobium
    User andrewshR

Host images.apertis.org
    User andrewshR

• Check available disk space on OBS Server
  • Release Branching is a task that will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary.
  • Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
• Install the jq package. It is already packaged in Apertis
• Install the osc-plugin-clone plugin for the OBS client
• Install the osc-plugins-dput. It is already packaged in Apertis
• Create a GitLab personal access token and configure it in $HOME/.gitlab-apertis-org-token, e.g. GITLAB_TOKEN=XXXXxxxxxXXXXXX > $HOME/.gitlab-apertis-org-token
  • Make sure your API token is active. When you create your API token, you have an option to set an expiry date. So make sure to verify the validity of your granted token
• Ensure that the osc tool is configured with the Admin user credentials for OBS. This is essential for the creation of additional repositories
• Check out the apertis-infrastructure repository
• Install gitlab-rulez tool and unprotect the current release branch for all repositories.
  • Verify that all relevant package groups, like pkg/*, have push_access_level: DEVELOPER_ACCESS in apertis-infrastructure/gitlab-scripts/rulez.yaml
  • Apply the rule with gitlab-rulez apply apertis-infrastructure/gitlab-scripts/rulez.yaml
• Change the working directory to release-scripts/
• Launch export RELEASE=v2019; export NEXT_RELEASE=v2020dev0; ./do-branching.sh which will:
  • Set new release to string v2020dev0 and set current release to string v2019, for example.
  • Create a new debootstrap script entry on the image builder host
  • Update the debootstrap package on OBS
  • Update the apertis-customizations, apertis-image-customization, apertist-tests repositories, both public and internal
  • Create branches in every repository on gitlab.apertis.org
  • Add the next-release aptly entry to the OBS configuration files (/etc/obs/BSConfig.*.pm files).
  • Restart OBS publisher and srcserver services to pick up the new aptly entry.
  • Stop the OBS schedulers
  • Fork the OBS projects, copying packages and metadata
  • Clone and republish the OBS binary repositories
  • Restart the OBS schedulers
• Update build suffix on OBS Project configuration (eg. bv2021.0b<B_CNT>, bv2020dev0b<B_CNT>). Please refer to section Build Suffix and Stable Build Suffix for more details about it

Folding examples

Following are the example runs for the automation script.

Example Run with help sub-command

$ ./fold-security-update-branches help
Usage: BASE_RELEASE='v2021' ./fold-security-update-branches [apply] [gen-pkg-list] [dry-run] [clean-obs] [clean-reprepro] [clean-obs-reprepro] [help]
Example: BASE_RELEASE='v2021' ./fold-security-update-branches apply 2>&1 | tee -a console.log

Example Run with gen-pkg-list sub-command

The gen-pkg-list sub-command generates a bunch of data files, based on the Apertis Release and Apertis Repositories. These generated data files are used by the script to process changes (Security and Regular Updates) in Apertis point releases.

rrs@priyasi:~/rrs-home/devel/CCU/Apertis/apertis-infrastructure/release-scripts/gitlab (v2020-folding-work-data)$ ./fold-security-update-branches  gen-pkg-list
Package lists extracted from OBS' repository v2020-security-target.txt
Package lists extracted from OBS' repository v2020-updates-target.txt
Package lists extracted from OBS' repository v2020-security-development.txt
Package lists extracted from OBS' repository v2020-updates-development.txt
Package lists extracted from OBS' repository v2020-security-sdk.txt
Package lists extracted from OBS' repository v2020-updates-sdk.txt
Package lists extracted from OBS' repository v2020-security-hmi.txt
Package lists extracted from OBS' repository v2020-updates-hmi.txt

Example Run with dry-run sub-command

The dry-run sub-command will run the actual commands without any effect on the server-side. This is very handy to check and determine the actual actions that will be run.

$ ./fold-security-update-branches  dry-run
Package apt is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package bluez is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package boost1.67 is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package cups is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security

Example Run with apply sub-command

The apply sub-command does the actual job.

$ ./fold-security-update-branches apply
Running in apply mode
Package apt is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/apt/-/merge_requests/10
Package bluez is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/bluez/-/merge_requests/12
Package boost1.67 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/boost1.67/-/merge_requests/5
Package cups is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/cups/-/merge_requests/8
Package curl is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/curl/-/merge_requests/7
Package cyrus-sasl2 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020

Removing an old distribution - Reprepro

Perform the following steps to remove an old distributions. This would usually be the obsolete Beta and Preview distribution releases, like: v2019dev0, v2019pre

• Remove the entry for the distribution from the reprepro configuration file. Example: /srv/obs/repos/shared/apertis/internal/apertis/conf/distributions

• Run the following reprepro command to apply the changes

  # sudo -u obsrun reprepro --gnupghome /srv/obs/gnupg/ -Vb /srv/obs/repos/shared/apertis/internal/apertis --delete clearvanished
  Deleting vanished identifier 'u|v2019dev0|proprietary|amd64'.
  Deleting vanished identifier 'u|v2019dev0|proprietary|arm64'.
  Deleting vanished identifier 'u|v2019dev0|proprietary|armhf'.
  Deleting vanished identifier 'u|v2019pre|demo|amd64'.
  Deleting vanished identifier 'u|v2019pre|demo|armhf'.
  Deleting vanished identifier 'u|v2019pre|nothumb|amd64'.
  Deleting vanished identifier 'u|v2019pre|nothumb|armhf'.
  Deleting vanished identifier 'u|v2019pre|proprietary|amd64'.
  Deleting vanished identifier 'u|v2019pre|proprietary|armhf'.