Table of Contents:

Each Apertis release is based on the current Debian stable release. This means that approximately bi-yearly Apertis makes the jump to be based on a new version of Debian. This process is known as “rebasing”.

Please see Release flow and product lines for more information about this policy.

The following terms will be used here:

  • OLD_DEV_RELEASE: The existing Apertis release on which the new release will be based.
  • NEW_DEV_RELEASE: The new Apertis development release that we are creating during the rebasing process.
  • OLD_DEBIAN_RELEASE: The Debian release on which OLD_DEV_RELEASE is based.
  • NEW_DEBIAN_RELEASE: The Debian release on which NEW_DEV_RELEASE will be based.

Early Tests

These tests should be carried out in Q4 ready for the rebase in the follwoing Q1.

A. Test Rebase Workflow

  1. Follow the rebase plan, but with a limited set of packages from target using NEW_DEV_RELEASE set to rebase-test.
  2. Check that pipelines run as expected
  3. Check that the scripts used to check status work as expected

It is not required to merge all the changes, focus on ensuring that the workflow is working as expected.

B. Test Debian Sources

If sufficient time is available early in Q1, it is advantageous to ensure that we have a relatively accurate seed list and have already imported new dependencies where possible. It is unlikely that the seed list will be perfect even after this, however any remaining missing dependencies, etc. will be flagged by OBS. The aim here is to reduce the time required when rebasing.

  1. Use germinate to generate seeds for NEW_DEBIAN_RELEASE.
  2. Compare “existing-packages” lists generated for NEW_DEBIAN_RELEASE with the list of packages in the current Apertis release. Review differences and update the new seed list as required.
  3. Review packages listed in “new-packages” lists for addition to Apertis

Preparing for Rebase

C. Prepare Development Workflow

During the rebase NEW_DEV_RELEASE will be unstable as many packages will be updated and images will not be generated. To allow development work during this time, OLD_DEV_RELEASE will be used as development branch. For that reason, after OLD_DEV_RELEASE freeze ends, it will be used for development.

For repositories that diverge during the rebase cycle, after the rebase is completed, changes will need forward porting to NEW_DEV_RELEASE.

D. Prepare Infrastructure

  1. Create NEW_DEV_RELEASE branch for apertis-docker-images
  2. Tweak the package-source-builder dockerfile in the NEW_DEV_RELEASE release to enable Debian NEW_DEBIAN_RELEASE
  3. Enable apertis OLD_DEV_RELEASE repositories in dockerfile, to allow the installation of Apertis specific packages, such as apertis-dev-tools
  4. Temporarily disable other docker images as they won’t be needed
  5. Create new OBS projects for NEW_DEV_RELEASE
  6. Disable publish of NEW_DEV_RELEASE on OBS projects
  7. Add NEW_DEBIAN_RELEASE DoD on NEW_DEV_RELEASE OBS repositories
  8. Create Apertis NEW_DEV_RELEASE branches for the repositories in the following groups based on the OLD_DEV_RELEASE branches in Gitlab. (This should create new branches but not trigger any pipelines):
  9. Continue to use OLD_DEV_RELEASE for development work.

Perform Rebase

E. Upgrade packages from NEW_DEBIAN_RELEASE

Extract per repository component package list generated from germinate. Work through the package lists for target, non-free, development and sdk repositories in this order:

These commands act on a single package rather than on lists of them and thus will need to be run for each package.

It is better to process the list of packages in batches, starting with a small number of packages from target, to make sure that no unexpected issues happen, before continuing with subsequent batches, working through target and then onto the other repositories in order.

  1. Create debian/NEW_DEBIAN_RELEASE branches in Gitlab for these packages in their git repositories. This can be done using the create-branches script:

    $ ./create-branches --filter pkg/${pkg} --branch-prefix debian --new-release ${NEW_DEBIAN_RELEASE} --base-release ${OLD_DEBIAN_RELEASE}

  2. Create upstream/NEW_DEBIAN_RELEASE branches in Gitlab for these packages in their git repositories. This can also be done using the create-branches script:

    $ ./create-branches --filter pkg/${pkg} --branch-prefix upstream --new-release ${NEW_DEBIAN_RELEASE} --base-release ${OLD_DEBIAN_RELEASE}

  3. Trigger the pipeline on the debian/NEW_DEBIAN_RELEASE branches using the trigger-pipeline script:

    $ ./trigger-pipeline ${pkg} debian/${NEW_DEBIAN_RELEASE}

    This will trigger a chain of pipelines that will sync with the branch with the required Debian release and attempt to merge with the NEW_DEV_RELEASE branch. This results in open MRs with failed pipelines where this merge can’t happen automatically.

    Since Debian DoD will be available no additional configuration should be required to resolve dependencies

F. Check status of each package

As there will be a large number of merge requests open, it is important to carefully prioritise the order in which packages are targeted. The packages should be checked in the following order:

  • Packages required for bootstrapping (get a list based on NEW_DEBIAN_RELEASE)
  • Packages in the target and non-free repositories
  • Other packages, such as those in development and sdk.

The analise-rebases.py script can be used to aid with this process.

There will be different scenarios depending on the package:

  1. Most of the packages will have an update available in NEW_DEBIAN_RELEASE and an MR should be available for it in one of the following states:
    • MR pipeline failure due to unable to merge: Manually merge the branch.
    • MR pipeline failure due to license scan failure: Manually fix license scan
    • MR pipeline failure due to OBS build error: Manually check and/or retry
    • MR pipeline success with not relevant changes from NEW_DEBIAN_RELEASE (only changelog or debian/apertis): Should be automerged
    • MR pipeline success with relevant changes (delta) to NEW_DEBIAN_RELEASE (changes other than in changelog or debian/apertis): Review and determine if delta is still valid
  2. Some packages will not have an update available in NEW_DEBIAN_RELEASE, run the pipeline for NEW_DEV_RELEASE in these repositories
  3. Some new packages will need to be imported

All successful pipelines will result in a package being uploaded to OBS proposed-updates/debian/NEW_DEBIAN_RELEASE branches. Merging all the changes is a time consuming task, apply common sense to avoid waiting time in things that can be addressed later.

Keep track of delta introduced as this will need to be documented, rectified or forwarded upstream.

Image generation and testing

G. Target image generation

  1. Run local bootstrap tests by downloading packages required for bootsrapping directly from OBS and fix any issue found.
  2. Once packages in target and non-free have been built it is possible to enable publishing of these repositories. It is important to do this to decouple the rebase work from the image generation/testing.
  3. During this stage we focus in target images (fixed function and hmi)
  4. Run local tests to building target images using NEW_DEV_RELEASE and fix any issue found.
  5. Run image pipeline only for fixed function and HMI (disable the rest):
    1. Fix any issue found by either proposing proper fixes or workarounds (disabling functionality)
    2. The goal of this stage is to have a minimal image that can be tested on the different reference boards
    3. In case of temporary workarounds create tasks to address them later
    4. Test image on LAVA by using UPLOAD_ARTIFACTS and SUBMIT_LAVA_JOBS

H. Check status of OBS projects

  1. Check that all the packages in germinate are in OBS using analyse-missing.py
  2. Check if all build statuses are green on OBS
  3. Check that OBS is NOT using Debian DoD to build packages using the check-for-dod script.
  4. Drop Bookworm DoD repositories from OBS
  5. Fix any leftover cracks:
    • New package imports
    • Build dependency trimming
    • Feature/plugin [dis|en]ablement
  6. Enable publishing on NEW_DEV_RELEASE OBS development and sdk repositories
  7. Validate the public apt repositories for NEW_DEV_RELEASE

I. Image generation

  1. Run image pipeline for all the images (reenable previously disabled images)
  2. Fix any issue found by either proposing proper fixes of workarounds (disabling functionality)
  3. The goal of this stage is to have a minimal image that can be tested on the different reference boards
  4. In case of temporary workarounds, create tasks to address them later
  5. Test image on LAVA by using UPLOAD_ARTIFACTS and SUBMIT_LAVA_JOBS

J. Cleanup infrastructure

  1. Tweak docker images for package-source-builder in NEW_DEV_RELEASE to only use NEW_DEV_RELEASE
  2. Re-enable the other docker images
  3. Build new docker images
  4. Rebuild whole archive
  5. Generate a list of packages with a NEW_DEV_RELEASE branch in GitLab but which are not present in OBS. Dashboard will report the following error for such packages and the list can be generated from this. 
    Missing on OBS apertis:NEW_DEV_RELEASE, found on Git apertis/NEW_DEV_RELEASE
  6. Drop these packages from the new release:
    • Remove the apertis/NEW_DEV_RELEASE branch from the package in Gitlab
    • Set the default branch as apertis/OLD_DEV_RELEASE using set-delete-branches.
    • Update gitlab-rulez to contain a list for OLD_DEV_RELEASE containing the packages for which it is the last release. This causes gitlab-rulez to leave the default branch at this release for these packages.

K. Forward port any changes to OLD_DEV_RELEASE

  1. Check which repositories have diverged between OLD_DEV_RELEASE and NEW_DEV_RELEASE. This means for repositories where OLD_DEV_RELEASE is not a descendant of NEW_DEV_RELEASE and thus further changes have been merged to OLD_DEV_RELEASE.
  2. Evaluate and prepare forward ports for changes where required.
  3. Fix any temporary workarounds that were added during rebasing.

L. Rejoice! Rebase party!