Background

Apertis relies on OBS for building and publishing binary packages. However, upstream OBS provides an APT publisher based on dpkg-scanpackages, which is not suitable for a project the scale of Apertis, where a single OBS project contains a lot of packages.

For a period of time, our OBS instance used a custom publisher based on reprepro, but it is still subject to some limitations that became more noticeable as the scale of Apertis increased considerably.

Aptly

Aptly is a complete solution for Debian repository management, including mirroring, snapshots and publication. It has been chosen as the target APT publisher and therefore it has been integrated to OBS.

For a quick introduction and in-depth comparison on aptly features check this article.

OBS aptly integration overview

OBS backend has been integrated to aptly, using a locally running aptly instance (no remote API).

The only requirement to start using the aptly publisher backend on a project/repository is to have it defined in the OBS configuration files (i.e. BSConfig.pm), check the OBS aptly documentation.

The following features are integrated to OBS:

OBS repositories gets automatically created/removed on aptly.
Each arch with the publish flag enabled in a repository gets published on aptly.
Package binaries get added to the aptly repository right after they get built. Old versions are automatically removed.
Every time a package is published, a snapshot is taken and published by calling the aptly publisher hook (if defined in the OBS configuration).
Aptly database cleanup routine is called periodically running a cron job.

Testing

Configuration requirements

Make sure the required environments variables are properly set as follows (default values are shown here):

$ export GNUPGHOME=/srv/obs/gnupg
$ export OBS_FRONTEND_HOST=obs-frontend

Add the following configuration to /etc/obs/BSConfig.pm, setting the gpg-key configuration to a valid GPG key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


my @reprepro_releases = ("v2022");
my @aptly_releases = ("v2023");
my @apertis_components = ("target", "development", "sdk");
my $apertis_prefix = "shared/apertis/public/apertis";

our $publishedhook_use_regex = 1;
our $publishedhook = {};

our $aptly_defconfig = {
  "gpg-key" => "D4B4146191456112D814554C1DB3714017240E34",
};

foreach $release (@reprepro_releases) {
  $publishedhook->{"apertis:$release:.*"} = "/usr/lib/obs/server/reprepro-snapshot";
};

foreach $release (@aptly_releases) {
  $publishedhook->{"apertis:$release:.*"} = "/usr/lib/obs/server/bs_published_hook_aptly_snapshot";
};

our $reprepository = {};

foreach $release (@reprepro_releases) {
  foreach $component (@apertis_components) {
    $reprepository->{"apertis:$release:$component/default"} = {
      "repository" => $apertis_prefix,
      "codename" => $release,
      "component" => $component
    };
    $reprepository->{"apertis:$release:updates:$component/default"} = {
      "repository" => $apertis_prefix,
      "codename" => "$release-updates",
      "component" => $component
    };
    $reprepository->{"apertis:$release:security:$component/default"} = {
      "repository" => $apertis_prefix,
      "codename" => "$release-security",
      "component" => $component
    };
  };
};

our $aptly_config = {};

foreach $release (@aptly_releases) {
  foreach $component (@apertis_components) {
    $aptly_config->{$apertis_prefix}{$release}{"components"}{$component} = {
      "project" => "apertis:$release:$component",
      "repository" => "default",
    };
    $aptly_config->{$apertis_prefix}{$release."-updates"}{"components"}{$component} = {
      "project" => "apertis:$release:updates:$component",
      "repository" => "default",
    };
    $aptly_config->{$apertis_prefix}{$release."-security"}{"components"}{$component} = {
      "project" => "apertis:$release:security:$component",
      "repository" => "default",
    };
  };
};

1;

Storing the package files on Azure

This functionality requires changes in a custom fork of aptly that are currently not deployed onto any Apertis or downstream infrastructure.

In order to configure aptly to place its package pool in Azure, add the following to JSON key/value pair to /etc/obs/aptly.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
  "packagePoolStorage": {
    "type": "azure",
    "azure": {
      "accountName": "ACCOUNT_NAME",
      "accountKey": "ACCOUNT_KEY",
      "container": "CONTAINER",
      "prefix": "PREFIX"
    }
  }
}

where:

ACCOUNT_NAME is the name of your Azure storage account.
ACCOUNT_KEY is one of your Azure storage account keys.
CONTAINER is the blob storage container in your storage account where packages should be placed.
PREFIX (optional) is a directory prefix to prepend to each path placed in the Azure container. If empty, no prefix will be used.

Before this can be used, the current package pool contents should then be uploaded onto the configured Azure container, after which aptly will be able to access its package pool from the configured cloud storage.

OBS aptly integration test

Simple test setting up aptly repositories to validate OBS aptly backend integration.

Copy the following test files to a temporary location with write permissions to the obsrun user: tests/aptly/aptly.sh tests/aptly/common.sh
Run the test script, which should show this message on success:

$ ./aptly.sh
[...]
PASSED: all tests successfully passed!

Reprepro to aptly migration test

This test set a few reprepro repositories on OBS, then migrates them to aptly.

Copy the following test files to a temporary location with write permissions to the obsrun user: tests/aptly/common.sh tests/aptly/migration.sh tests/aptly/reprepro.sh
Run the tests scripts, which should show these messages on success:

$ ./reprepro.sh
[...]
PASSED: all tests successfully passed!
$ ./migration.sh
[...]
PASSED: all tests successfully passed!

Upgrading downstream instances to aptly publisher

This section describes how to upgrade a downstream OBS instance to use aptly publisher backend on project/repositories.

Branch a new release

A new Apertis release can be branched out of a previous release following the same release process as done previously with reprepro. Note that this branching process is supported by the Apertis infrastracture gitlab CI, which has been updated to switch from reprepro published backend to aptly.

The only requirement before running the gitlab CI steps is to have the NEXT_RELEASE projects setup in the OBS configuration file so these use aptly as the publisher backend.

Migrate a reprepro repository to aptly

The script ./src/backend/bs_aptly_migration can be used to migrate reprepro repositories and snapshots to aptly. Note that this script needs to be run as obsrun user.

Before running this script, OBS configuration must be updated moving the repositories from reprepro backend to aptly. As reference, check the OBS aptly documentation.

OBS services (srcserver and publisher) should be stopped before running this script and re-enabled afterwards, e.g.:

$ supervisorctl stop srcserver
$ supervisorctl stop publisher
[...] # Run migration
$ supervisorctl start srcserver
$ supervisorctl start publisher

Note that OBS projects are not modified by this script at all. Once the services are restarted it will start using the new aptly publisher instead of reprepro. All the previous packages and snapshots should be available and re-published transparently.

On the other hand, reprepro repositories and snapshots are not modified by this script. These should be cleaned up manually from the filesystem afterwards.

Cleanup migrated reprepro repository

After a reprepro-to-aptly migration has been performed, it might be useful to remove the previous reprepro repository entirely.

The following script reprepro_remove.sh is a reference snippet that can be used to clean up a reprepro repository, along with all its snapshots.

Before running this script, make sure the configuration entry for the target release has been removed from the reprepro conf/distributions file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


#!/bin/bash

set -eu

# Before running this script, make sure the configuration entry for the target
# release has been removed from the reprepro conf/distributions file.

ROOT_DIR="$(dirname -- "${BASH_SOURCE}")"
ABS_ROOT_DIR="$(realpath $ROOT_DIR)"
REPOS_DIR=/srv/obs/repos

if [ $(id -nu) != obsrun ]; then
	echo "ERROR: needs to be run as obsrun" >&2
	exit 1
fi

if [ $# -ne 2 ]; then
	echo "ERROR: wrong arguments" >&2
	echo "Usage: $0 <osname> <distro>" >&2
	exit 1
fi

prj_osname="$1"
prj_distro="$2"

# Publish path is assumed to have the following format
prj_public_prefix="shared/$prj_osname/public/$prj_osname"

reprepro="reprepro --gnupghome $GNUPGHOME -Vb $REPOS_DIR/$prj_public_prefix"

$reprepro --delete clearvanished
for d in $REPOS_DIR/$prj_public_prefix/dists/$prj_distro/snapshots/*; do
	timestamp=$(basename $d)
	$reprepro _removereferences s=$prj_distro=$timestamp
done
$reprepro deleteunreferenced
rm -rf $REPOS_DIR/$prj_public_prefix/dists/$prj_distro

The above script receives two arguments <osname> <distro> pointing to the target reprepro release to remove. For example, to drop apertis:v2022 release, you should run:

./reprepro_remove.sh apertis v2022

Search Apertis

OBS aptly publisher integration

Table of Contents:

Background

Aptly

OBS aptly integration overview

Testing

Configuration requirements

Storing the package files on Azure

OBS aptly integration test

Reprepro to aptly migration test

Upgrading downstream instances to aptly publisher

Branch a new release

Migrate a reprepro repository to aptly

Cleanup migrated reprepro repository