Table of Contents:

Apertis stores the source of all the shipped packages in GitLab and uses a GitLab CI pipeline to manage the workflows and automate many of the tasks that are required to release a package, thus for many tasks such as adding or updating packages, interaction is only required with GitLab, with automation taking care of the rest of the process.

Please see the Apertis workflow for more information regarding the automation provided by Apertis.

The VirtualBox-based Apertis SDK virtual machine images ship with all the needed tools installed, providing a reliable, self-contained environment ready to be used. Alternatively, you can run the Apertis package-source-builder Docker container.

For v2021 and newer:

docker pull${APERTIS_RELEASE}-package-source-builder
docker run -it --rm --env HOME -w "$(pwd)" -v "$HOME:$HOME" -v "$HOME:/root" --security-opt label=disable${APERTIS_RELEASE}-package-source-builder bash

For v2020 and older:

docker pull${APERTIS_RELEASE}-package-source-builder
docker run -it --rm --env HOME -w "$(pwd)" -v "$HOME:$HOME" -v "$HOME:/root" --security-opt label=disable${APERTIS_RELEASE}-package-source-builder bash

These docker images provide a preconfigured environment, preinstalled with tools such as apertis-pkg-pull-updates and apertis-pkg-merge-updates`.

Adding New Components

The software components used in Apertis images are packaged in the deb packaging format, as used by Debian. It is preferred, where possible, to add additional packages from Debian as covered by the contribution policy.

Adding New Packages from Debian

As the adding of existing Debian packages is mostly scripted/automated it is expected that such actions will be carried out by maintainers after a request made in line with the contribution policy.

This is the process to import a new package from Debian to Apertis:

  • Create a folder, in the name of the package to import.

    • Eg. for package hello, run:

      mkdir hello
    • chdir to created folder:

      cd hello
  • Invoke import-debian-package from the packaging-tools repository. To fetch the latest version:

    import-debian-package --upstream buster --downstream apertis/v2021 --component target --package hello

    To fetch a specific version, add the --version option:

    import-debian-package --upstream buster --downstream apertis/v2021 --component target --package hello --version 2.10-2
    • The argument to --component reflects the repository component it is part of (for instance, target); it will be stored in debian/apertis/component.
    • Multiple downstream branches can be specified, in which case all of them will be updated to point to the newly imported package version.
    • The Apertis version of the package will have a local suffix (apertis0) appended.

    Don’t use import-debian-package on existing repositories, it does not attempt to merge apertis/* branches and instead it re-sets them to new branches based on the freshly imported Debian package.

  • Create an empty project on GitLab under the pkg namespace (for instance, pkg/hello).

  • Configure the origin remote on your local git:

    git remote add origin
  • Push your local git contents to the newly created GitLab project:

    git push --all --follow-tags origin
  • Set it up with gitlab-rulez from the gitlab-rulez repository:

    gitlab-rulez apply rulez.yaml --filter pkg/hello


    • sets the CI config path to ci-package-builder.yml@infrastructure/ci-package-builder
    • changes the merge request settings to:
      • only allow fast-forward merges
      • ensure merges are only allowed if pipelines succeed
    • marks the apertis/* and debian/* branches as protected
  • As the CI configuration was set by gitlab-rulez after the initial commit, it will be necessary to run the CI/CD pipeline on each apertis/* branch to push the package through to OBS.

    Ensure that one branch successfully completes it’s upload to OBS before starting other branches to ensure that the different branches don’t conflict over uploading the tarballs.

Adding New Packages from Upstream Sources

There are likely to be instances where it is desirable to import the latest version of a piece of software into Apertis directly from it’s original authors or maintainers. This may be as a result of the software in question not being packaged by Apertis' default upstream distribution, Debian, or their being a mismatch between the desired version in the upstream distribution and what is required for a specific goal.

For example, the Apertis release flow stipulates that each Apertis release should include the latest mainline kernel LTS release. Due to Debian’s release cadence being approximately half that of Apertis', there are 2 Apertis releases for each Debian stable release and no guarantees that the kernel’s packaged for Debian’s current stable or in progress releases will align with Apertis' requirements. As a result it is expected that Apertis will occasionally need to take its kernel from the mainline kernel LTS tree to satisfy it’s requirements.

Such packages will require special attention to assure that they remain up-to-date with any security releases made by the upstream and updated as and when apropriate.

Unless the imported package can be used as-is without any modification, there will be a patch series that potentially needs tweaking to apply after the update. The workflow set out below takes this into consideration.

Below we use the process of importing GNU Hello application as an example.

  • Using the Gitlab Web UI, create an empty project in your personal area, for example called hello.

  • Download the latest release tarball from upstream. At the time of writing this is hello-2.10.tar.gz for GNU Hello.

  • Extract the tarball and enter the extracted folder

    $ tar xf hello-2.10.tar.gz
    $ cd hello-2.10
  • Create a new git repository and commit the whole source tree; this will create the upstream/apertis branch containing the untouched upstream source code:

    $ git init
    $ git checkout -b upstream/apertis
    $ git add .
    $ git commit -s -m "Initial commit of GNU Hello 2.10"
  • Commit the original tarball to the pristine-lfs branch:

    $ pristine-lfs commit ../hello-2.10.tar.gz
  • Create the packaging branch for the Apertis version you’re targetting, for example if you are targetting Apertis v2021, create the apertis/v2021 branch:

    $ git checkout -b apertis/${APERTIS_RELEASE}
  • Add packaging files and commit them to the packaging branch.

    Debian packaging is not covered by this document, users interested in that matter should refer to the Debian Packaging Intro, Debian Policy Manual and Debian Developer’s Reference.

    If you are packaging a newer version of a component which is already available in Debian, starting from the existing Debian packaging may save you considerable time.

  • Ensure that the package builds for you locally:

    $ gbp buildpackage -uc -us --git-debian-branch=apertis/${APERTIS_RELEASE} --git-upstream-tree=upstream/apertis
  • Add the Gitlab project you created earlier as the origin git remote (replacing <username> with your GitLab username):

    $ GITLAB_USER=<username>
    $ git remote add origin${GITLAB_USER}/hello.git
  • Push your branches to the repository:

    $ git push --all --follow-tags origin

If you wish to submit this package for inclusion in the Apertis packages, please make yourself familiar with the contribution policy.

Updating Existing Components

Once we have components added to Apertis, it is important that we keep them up to date with any security updates that are made available.

Updating Components from Debian

Updates coming from Debian have been automated using a CI/CD pipeline. Such updates are usually triggered from the infrastructure dashboard but can be manually triggered from the GitLab web UI.

This functionality will only be available to users with sufficient GitLab privileges to enable them to run CI/CD pipelines.

To trigger the updates pipeline:

  • Navigate to the “Pipelines” page of the component in GitLab.
  • Click on the Run pipeline button in the top right-hand corner.

If the Run pipeline button doesn’t appear, append /new to the page URL, for example

  • Select the relevant debian/* branch that you would like to update (such as debian/bullseye) under Run for branch name or tag and click the Run pipeline button shown on that page.

The pipeline will check the Debian archive for updates, pull them in to the debian/$RELEASE-like branch (for instance, debian/bullseye or debian/bullseye-security), try to merge the new contents with the matching apertis/* branches and, if successful, will create a merge request for each apertis/* branch that requires updating.

Updating Components to a New Debian Release

The CI/CD pipeline can also be used to pull in a new release of Debian. If the needed debian/$RELEASE branch doesn’t exist (for example, there is a debian/buster branch but no debian/bullseye), the Gitlab Web UI can be used to create it:

  • Navigate to the Branches page and click on the New branch button.
  • Enter the new branch name as debian/$RELEASE, where $RELEASE is the new release name, for example debian/bullseye.
  • In the Create from field select the previous Debian release. For example, the release that preceeds Debian Bullseye is Buster, so pick debian/buster.
  • Click the Create branch button.

In some situations manual intervention may be required to tweak the packaging, patches or the automated merge machinery may ask to PLEASE SUMMARIZE remaining Apertis Changes. When this happens:

  • Check out the proposed update branch.
  • If asked to summarize the changes, edit the changelog to list all the downstream changes the package still ships compared to the newly merged upstream package and their reason, describing the purpose of each downstream patch and of any other change shown by git diff against the debian/* branch.
  • Where other issues are found, either modify the packaging data or add/modify the Apertis patch series to address the issue.
  • Amend the merge commit.
  • Force-push to the proposed update branch
  • If changes beyond summarizing the changelog are made, the update branch should be reviewed before it is merged.

Updating from Debian Development Repositories

This is another scenario, wherein the user may need updates which are not yet released into the Upstream Distributions' package repositories.

For example, for Apertis, we may need a very newer version of a package, which may not yet have been released into any of Debian development releases (Unstable, Testing). In such cases, where the changes may only be available in the packaging repositories. We need to take extra care when pulling in such updates as they will not have been through the same levels of testing as packages taken from stable or even testing releases.

  • Clone the remote git packaging repository from Debian.

    $ PACKAGE=hello
    $ DEBIAN_BRANCH=debian/master
    $ git clone -b ${DEBIAN_BRANCH} ${DEBIAN_REPO} ${PACKAGE}-debian
    $ cd ${PACKAGE}-debian
  • Generate a source package out of the packaging repository

    $ gbp buildpackage -S --no-sign --git-debian-branch=${DEBIAN_BRANCH}

    If successful, this will give us a proper source packages in the parent directory

  • Clone the Apertis git packaging repository for the same component

    $ cd ..
    $ GITLAB_GROUP=pkg
    $ git clone${GITLAB_GROUP}/${PACKAGE}
    $ cd ${PACKAGE}
  • If the required debian/ and upstream/ branches don’t yet exist in your repository, create them based on the version that you have for instance. For instance, if you currently have a version of Apertis based on Debian Buster and wish to use the updated package in Bullseye:

    $ DESIRED_UPSTREAM=unreleased
    $ git push origin origin/upstream/${CURRENT_UPSTREAM}:refs/heads/upstream/${DESIRED_UPSTREAM}
    $ git push origin origin/debian/${CURRENT_UPSTREAM}:refs/heads/debian/${DESIRED_UPSTREAM}
  • Ensure you have the required branches locally:

    $ git branch debian/${DESIRED_UPSTREAM} origin/debian/${DESIRED_UPSTREAM}
    $ git branch upstream/${DESIRED_UPSTREAM} origin/upstream/${DESIRED_UPSTREAM}
  • Run gbp import-dsc to update the debian/ and upstream/ branches.

    $ NEW_DSC=$(ls -t ../*.dsc | head -n 1)
    $ GBP_CONF_FILES=/dev/null gbp import-dsc --author-is-committer --author-date-is-committer-date \
      --upstream-branch=upstream/${DESIRED_UPSTREAM} --debian-branch=debian/${DESIRED_UPSTREAM} \
      --debian-tag='debian/%(version)s' --no-sign-tags --no-pristine-tar ${NEW_DSC}
  • Use the pristine-lfs tool to import the source package generated from the Debian repository into Apertis packaging repository.

    $ pristine-lfs import-dsc ${NEW_DSC}
  • Ensure that the changes to these branches are pushed to your remote origin. It is important that these branches are pushed and in sync with the default remote.

    $ git push origin --follow-tags pristine-lfs debian/${DESIRED_UPSTREAM} upstream/${DESIRED_UPSTREAM}
  • The upstream changes now need to be merged into the apertis branch. We will do this via a development branch that can be pushed for review.

    $ git checkout -b ${PROPOSED_BRANCH} apertis/${APERTIS_RELEASE}
    $ apertis-pkg-merge-updates --upstream=debian/${DESIRED_UPSTREAM} --downstream=${PROPOSED_BRANCH}
  • If the merge fails, fix the conflicts and continue

    $ git add ...
    $ git merge --continue
  • Create a new changelog entry for the new version

    $ DEBEMAIL="User Name <>" dch -D apertis --local +apertis --no-auto-nmu ""
  • Check debian/changelog and update it if needed by listing all the remaining Apertis changes then commit the changelog.

    $ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
    $ git commit -s -m "Release ${PACKAGE} version ${DEBIAN_VERSION}" debian/changelog
  • Push your changes for review

    $ git push origin ${PROPOSED_BRANCH}:${PROPOSED_BRANCH}

    Follow the instructions provided by the above command to create a merge request

Creating New Modifications

The standard Contribution Process applies unchanged to packaging repositories, the Apertis development team and trusted contributors push changes to wip/ branches and get them landed to the apertis/* branches via merge requests. Other developers can fork packaging repositories into their personal space and send merge requests from there as documented in the development process guide.

The only additional requirement imposed by the Debian packaging format is that changes outside of the debian/ folder are not allowed and would cause the source-building pipeline to fail. Check the Debian Packaging documentation to find how patches affecting code outside of the debian/ folder should be handled.

The Debian Changelog and Releases

Beyond informing developers and interested users of what has changed between sucessive releases of a component, the changelog controls the release process in a number of ways:

  • Package versioning: The changelog is the canonical location used by the Debian packaging process to determine the version number used for the binary packages.
  • Release control: The “distribution” field of a changelog entry controls whether the component is ready to be cut as a new release.

The changelog format is documented in the Debian Policy Manual

It is therefore important to correctly handle the changelog file when creating modifications to components.

The CI pipeline generates a source package locally for each commit pushed to the packaging repositories. You can download this package by browsing the pipeline artifacts. The generated sources are versioned to indicate that they are not yet suitable for release.

With the “distribution” field set to UNRELEASED, package sources get uploaded to the :snapshots OBS project with the matching branch (that is, when targetting the apertis/v2021 branch of a component from the target suite, the source package will be uploaded to the apertis:v2021:target:snaphots project).

It is advisable to update the debian/changelog file in a separate commit, as the very last step when you issue a release. Such changelog entries should be generated from the Git commit log. This can be achieved by using gbp dch. Writing good commit messages ensures that you don’t have to spend time editing the generated changelog entries.

It is best to submit a separate merge request on GitLab for each bug or task. As performing a separate release between each merge request may not be wanted and to avoid churn in the case of rebases (in particular to ease the review process), it is recommended to leave the editing of debian/changelog to a dedicated merge request once all the other MRs have been landed.

In order to follow a release early, release often philosophy it is also recommended to avoid delaying release commits to include additional features. This gives the possibility to receive early feedback on the downstream changes.

If you still wish to edit debian/changelog as you go along for some reason, make sure that the changelog entry you’re writing has the distribution field set to UNRELEASED and use gbp dch --auto --ignore-branch to ensure the formatting of the changelog is correct.

Landing Downstream Changes to the Main Archive

Once downstream changes to a package are deemed ready to be published in the Apertis main archive, a proper release needs to be issued. To achieve this create a merge request containing an update to the changelog. The description should contain details of all the changes since the last release. This can be easily achieved with gbp dch if the commits have been well written:

$GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/${APERTIS_RELEASE}

Ensure that the “distribution” field has been changed from UNRELEASED to apertis.

If making changes to more than one release, start with the most recent release branch where you want to land your changes. For instance, if you want to land changes to both the development and stable releases, submit a merge request for the development one first (for instance, apertis/v2020dev0) and then, once merged, create a merge request for the stable one (for instance, apertis/v2019-updates)

Once a merge request has been reviewed and landed, the CI pipeline will build-check the source package as usual. Since the distribution field is no longer UNRELEASED it will also:

  • add a Git tag for the release version to the repository
  • rebuild the release source package
  • store the release sources in the pristine-lfs-source branch
  • upload the release source package to the main project on OBS (for instance apertis:v2020dev0:target)

Published stable release branches (for instance apertis/v2019) should never be targeted directly. Updates to these should go through the -security or -updates branches (for example apertis/v2019-security or `apertis/v2019-updates).

If the apertis/$RELEASE-updates or apertis/$RELEASE-security branches for published stable releases do not exist yet, they should be created from the GitLab web UI since their protected status makes pushing forbidden.

For trivial changes it is also possible to combine the release commit in the same MR as the changes. Again, developers need to be careful to ensure the changelog entries are kept up-to-date when the commit messages get changed via rebase.

Backporting Updates or Security Fixes

Often downstream fixes, upstream updates or security fixes need to be applied to multiple active releases. Changes should be introduced in the most recent development release where they can be tested and regressions detected with little impact. Once the changes have been thoroughly tested, paying close attention to the QA test results, they can then be propagated to the more stable releases, where any mistake can impact the product teams using Apertis in the field.

For instance, once a fix is landed to apertis/v2021dev0 and no regressions are found in the subsequent QA test results, an MR should be create to land the changes to the relevant stable releases.

If there is no divergence between the packages in the different releases and the backport can be done with a fast-forward, an MR should be created to submit the changes from for instance apertis/v2021dev0 to apertis/v2020-updates or apertis/v2020-security. Choose the required destination depending on the nature and impact of the fix.

If package diverged across releases, a separate branch has to be created where the fixes are cherry-picked appropriately before creating the MR.

Care needs to be taken to give these diverging release branches suitable version numbers.

Diverging Release Branches

Sometimes different downstream patches need to be applied in the different Apertis release branches. A clear case of that is the base-files package which ships the release name in /etc/os-release.

In such situation it is crucial to use different version identifiers in each branch: the version for a given package needs to be globally unique across the whole archive since uploading different package sources with the same name/version would lead to difficult to diagnose errors.

When targeting a specific release, ~${RELEASE}.${COUNTER} needs to be appended to the version identifier after the local build suffix:

Previous Version New release specific version (for v2020) Description
0.42+apertis0 0.42+apertis0~v2020.0 Increment Apertis release, append release specific identifier
0.42+apertis0~v2020.0 0.42+apertis0~v2020.1 Increment the release specific counter

This uses the fact that ~ in Debian package numbers sorts before anything. Adding ~ is necessary so that if a new upstream version 0.42.1 or a new non-release-specific downstream version 0.42+apertis1 is introduced, they will replace the release-specific package.

Note that dpkg considers 2020.0 to be newer than 2020pre.0, so the Apertis release identifiers can be used with no modification. If in doubt, check with dpkg --compare-versions:

$ dpkg --compare-versions 0.42+apertis0~2020pre.0 '<<' 0.42+apertis0~2020.0 && echo true || echo false
$ dpkg --compare-versions 0.42+apertis1 '<<' 0.42+apertis0~2020.0 && echo true || echo false

Advanced Configuration

License Scans

As merge requests to components are submitted, the CI pipeline performs license scans on the package. The scans are performed on all files in the package, not just the new submission. The pipeline fails or emits a warning (depending on the configuration) if it finds files with unknown or unclear licensing terms, or files under licenses not allowed in the package. When such a situation arises, it is the responsibility of the submitter to perform the review of the license scan results and make updates to the package if necessary.

Some packages like gcc-10 require extra steps before the license scan: in this case, the package can provide a debian/apertis/pre-scan-hook executable script to run arbitrary preparation steps.

When the license scan mistakenly identifies a file as being under an incorrect license, or fails to process it correctly, there are three ways to fix this:

  1. Specify the correct copyright and the license in debian/apertis/copyright.yml. The format of the file is specified in the Dpkg::Copyright::Scanner manpage. In short, it’s a YAML file mapping paths to their licensing information:

      copyright: 2015, Marcel
      license: Expat
      copyright: 2016, Joe
      license: Expat
      skip: 1
      'override-copyright': 2016 Marcel MeXzigue

    Patterns follow the Perl regular expression rules.

    Please also verify debian/copyright specifies the correct license, and if it doesn’t, submit a patch to Debian.

    The license statement should be used when the license scan fails to determine a license (reporting it as as UNKNOWN), while override-license should be used when it reports an incorrect one.

    Since override-license will override the reported license it should be used with caution and only if it is needed.

  2. Add the file to the list of ignored files, debian/apertis/copyright.whitelist. This file is formatted the same way as gitignore, please refer to the gitignore manpage for more information.

    This approach is useful in cases where a file has a license unsuitable for Apertis but which won’t affect the license of the generated package. A common example of this situation are the tests that run at build time, which will not be part of the binary package.

    The whitelisting will allow license information to be stored in the report, which could potentially be not suitable for Apertis, while not raising an error on the CI pipeline.

  3. If the file is under a license not suitable for Apertis, it can be removed from the package by either repackaging the tarball or patching it out, in which case the scanner will not take it into account.

The license scanner will store the automatically generated copyright report file under debian/apertis/copyright, updating the merge request when necessary.

Custom Pipelines

When using the packaging pipeline, developers cannot put their CI/CD automation in .gitlab-ci.yml anymore, as the CI config path points to the ci-package-builder definition.

However, developers can put their jobs in the debian/apertis/local-gitlab-ci.yml file and have them executed in a child pipeline whenever the main packaging pipeline is executed. This is specially handy to run tests before the actual packaging process begins.