This document outlines the OBS aptly publisher backend that has been implemented on the Collabora instance of OBS. The upstreaming process of this feature is currently in-progress.

Background

Apertis relies on OBS for building and publishing binary packages. However, upstream OBS provides an APT publisher based on dpkg-scanpackages, which is not suitable for a project the scale of Apertis, where a single OBS project contains a lot of packages.

For a period of time, our OBS instance used a custom publisher based on reprepro, but it is still subject to some limitations that became more noticeable as the scale of Apertis increased considerably.

Aptly

Aptly is a complete solution for Debian repository management, including mirroring, snapshots and publication. It has been chosen as the target APT publisher and therefore it has been integrated to OBS.

For a quick introduction and in-depth comparison on aptly features check this article.

OBS aptly integration overview

OBS backend has been integrated to aptly, using a locally running aptly instance (no remote API).

The only requirement to start using the aptly publisher backend on a project/repository is to have it defined in the OBS configuration files (i.e. BSConfig.pm), check the OBS aptly documentation.

The following features are integrated to OBS:

• OBS repositories gets automatically created/removed on aptly.
• Each arch with the publish flag enabled in a repository gets published on aptly.
• Package binaries get added to the aptly repository right after they get built. Old versions are automatically removed.
• Every time a package is published, a snapshot is taken and published by calling the aptly publisher hook (if defined in the OBS configuration).
• Aptly database cleanup routine is called periodically running a cron job.

Testing

Configuration requirements

• Make sure the required environments variables are properly set as follows (default values are shown here):
$export GNUPGHOME=/srv/obs/gnupg$ export OBS_FRONTEND_HOST=obs-frontend

• Add the following configuration to /etc/obs/BSConfig.pm, setting the gpg-key configuration to a valid GPG key:
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62  my @reprepro_releases = ("v2022"); my @aptly_releases = ("v2023"); my @apertis_components = ("target", "development", "sdk"); my $apertis_prefix = "shared/apertis/public/apertis"; our$publishedhook_use_regex = 1; our $publishedhook = {}; our$aptly_defconfig = { "gpg-key" => "D4B4146191456112D814554C1DB3714017240E34", }; foreach $release (@reprepro_releases) {$publishedhook->{"apertis:$release:.*"} = "/usr/lib/obs/server/reprepro-snapshot"; }; foreach$release (@aptly_releases) { $publishedhook->{"apertis:$release:.*"} = "/usr/lib/obs/server/bs_published_hook_aptly_snapshot"; }; our $reprepository = {}; foreach$release (@reprepro_releases) { foreach $component (@apertis_components) {$reprepository->{"apertis:$release:$component/default"} = { "repository" => $apertis_prefix, "codename" =>$release, "component" => $component };$reprepository->{"apertis:$release:updates:$component/default"} = { "repository" => $apertis_prefix, "codename" => "$release-updates", "component" => $component };$reprepository->{"apertis:$release:security:$component/default"} = { "repository" => $apertis_prefix, "codename" => "$release-security", "component" => $component }; }; }; our$aptly_config = {}; foreach $release (@aptly_releases) { foreach$component (@apertis_components) { $aptly_config->{$apertis_prefix}{$release}{"components"}{$component} = { "project" => "apertis:$release:$component", "repository" => "default", }; $aptly_config->{$apertis_prefix}{$release."-updates"}{"components"}{$component} = { "project" => "apertis:$release:updates:$component", "repository" => "default", }; $aptly_config->{$apertis_prefix}{$release."-security"}{"components"}{$component} = { "project" => "apertis:$release:security:$component", "repository" => "default", }; }; }; 1; 

Storing the package files on Azure

In order to configure aptly to place its package pool in Azure, add the following to JSON key/value pair to /etc/obs/aptly.conf:

  1 2 3 4 5 6 7 8 9 10 11  { "packagePoolStorage": { "type": "azure", "azure": { "accountName": "ACCOUNT_NAME", "accountKey": "ACCOUNT_KEY", "container": "CONTAINER", "prefix": "PREFIX" } } } 

where:

• ACCOUNT_NAME is the name of your Azure storage account.
• ACCOUNT_KEY is one of your Azure storage account keys.
• CONTAINER is the blob storage container in your storage account where packages should be placed.
• PREFIX (optional) is a directory prefix to prepend to each path placed in the Azure container. If empty, no prefix will be used.

Before this can be used, the current package pool contents should then be uploaded onto the configured Azure container, after which aptly will be able to access its package pool from the configured cloud storage.

OBS aptly integration test

Simple test setting up aptly repositories to validate OBS aptly backend integration.

• Copy the following test files to a temporary location with write permissions to the obsrun user: tests/aptly/aptly.sh tests/aptly/common.sh

• Run the test script, which should show this message on success:

$./aptly.sh [...] PASSED: all tests successfully passed!  Reprepro to aptly migration test This test set a few reprepro repositories on OBS, then migrates them to aptly. • Copy the following test files to a temporary location with write permissions to the obsrun user: tests/aptly/common.sh tests/aptly/migration.sh tests/aptly/reprepro.sh • Run the tests scripts, which should show these messages on success: $ ./reprepro.sh
[...]
PASSED: all tests successfully passed!
$./migration.sh [...] PASSED: all tests successfully passed!  Upgrading downstream instances to aptly publisher This section describes how to upgrade a downstream OBS instance to use aptly publisher backend on project/repositories. Branch a new release A new Apertis release can be branched out of a previous release following the same release process as done previously with reprepro. Note that this branching process is supported by the Apertis infrastracture gitlab CI, which has been updated to switch from reprepro published backend to aptly. The only requirement before running the gitlab CI steps is to have the NEXT_RELEASE projects setup in the OBS configuration file so these use aptly as the publisher backend. Migrate a reprepro repository to aptly The script ./src/backend/bs_aptly_migration can be used to migrate reprepro repositories and snapshots to aptly. Note that this script needs to be run as obsrun user. Before running this script, OBS configuration must be updated moving the repositories from reprepro backend to aptly. As reference, check the OBS aptly documentation. OBS services (srcserver and publisher) should be stopped before running this script and re-enabled afterwards, e.g.: $ supervisorctl stop srcserver
$supervisorctl stop publisher [...] # Run migration$ supervisorctl start srcserver
$supervisorctl start publisher  Note that OBS projects are not modified by this script at all. Once the services are restarted it will start using the new aptly publisher instead of reprepro. All the previous packages and snapshots should be available and re-published transparently. On the other hand, reprepro repositories and snapshots are not modified by this script. These should be cleaned up manually from the filesystem afterwards. Cleanup migrated reprepro repository After a reprepro-to-aptly migration has been performed, it might be useful to remove the previous reprepro repository entirely. The following script reprepro_remove.sh is a reference snippet that can be used to clean up a reprepro repository, along with all its snapshots. Before running this script, make sure the configuration entry for the target release has been removed from the reprepro conf/distributions file.   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37  #!/bin/bash set -eu # Before running this script, make sure the configuration entry for the target # release has been removed from the reprepro conf/distributions file. ROOT_DIR="$(dirname -- "${BASH_SOURCE}")" ABS_ROOT_DIR="$(realpath $ROOT_DIR)" REPOS_DIR=/srv/obs/repos if [$(id -nu) != obsrun ]; then echo "ERROR: needs to be run as obsrun" >&2 exit 1 fi if [ $# -ne 2 ]; then echo "ERROR: wrong arguments" >&2 echo "Usage:$0 " >&2 exit 1 fi prj_osname="$1" prj_distro="$2" # Publish path is assumed to have the following format prj_public_prefix="shared/$prj_osname/public/$prj_osname" reprepro="reprepro --gnupghome $GNUPGHOME -Vb$REPOS_DIR/$prj_public_prefix"$reprepro --delete clearvanished for d in $REPOS_DIR/$prj_public_prefix/dists/$prj_distro/snapshots/*; do timestamp=$(basename $d)$reprepro _removereferences s=$prj_distro=$timestamp done $reprepro deleteunreferenced rm -rf$REPOS_DIR/$prj_public_prefix/dists/$prj_distro 

The above script receives two arguments <osname> <distro> pointing to the target reprepro release to remove. For example, to drop apertis:v2022 release, you should run:

./reprepro_remove.sh apertis v2022