Getting from source code to an image suitable for loading into a target device is typically a two step process. The first step is creating binary packages and the second combining them to create an image. A more detailied explanation of this process is documented in the Apertis workflow.

The Apertis project automates the creation of the official images, which are built for the reference hardware and supported by the project. Whilst these images may prove enough for early exploration of the Apertis platform, they are unlikely to be suitable for direct use in a specific bespoke use case. The automation used to build these images is not currently available for direct use by unofficial projects on the Apertis infrastructure.

If you are in the fortunate position that your target hardware is supported by Apertis and are just looking to include extra packages already available in Apertis, then this can be achieved on the Apertis infrastructure and is documented in How to Build Your First Image on Apertis.

It is very likely that as a developer looking at Apertis you have your own bespoke hardware, a piece of hardware that isn’t one of the reference boards or additional hardware attached that isn’t currently supported in the reference images. In this guide we look at building an Apertis image locally with support to boot on such a board.

Whilst this process currently can’t be carried out on the Apertis infrastructure from a basic account. The Apertis project is open to the idea of creating dedicated project areas for projects on the Apertis infrastructure which might be able to take advantage of the Apertis’ automation.

The Target Hardware

We are going to use a low cost development board that is not currently supported by Apertis as an example. As the focus of this guide is on the development workflow, we will pick a board that has reasonable upstream support, but which will need the configuration of a few packages to be modified and these packages rebuilt to build an image that will boot. The board chosen for this is the Orange Pi Zero2.

The Orange Pi Zero2

If you are looking to target a less well supported or custom hardware platform where you need to add support, there is a guide for building, patching and maintaining the Apertis kernel which provides some guidance on how to approach this.

Picking a version of Apertis

It is important to consider which version of Apertis is going to be used. At any given time, Apertis has an old and new stable release and a version being actively developed. As this guide is acting as a demonstration and as there’ll be no product release the latest Apertis stable release will be used, v2023 at time of writing.

To make best use of the release support of the Apertis project, developers are advised to take the Apertis release flow and schedule into account when creating a product release roadmap and to pick the branch on which they perform their development carefully.

Summary of the Required Changes

We will base our initial image on the Fixed Function image type, as a nice small image with which to test booting our desired hardware. Our initial target will to be to have an image built that can be written to an SD card, which boots to a login prompt on the serial console. As the Orange Pi Zero2 is an ARM64 board, Apertis already provides basic support for the architecture, however there are a few key packages that we’ll need to check/modify to successfully boot:

  • Arm Trusted Firmware
  • U-Boot
  • Linux kernel

As Arm boards don’t generally provide a standardised way of booting, with each vendor of Arm based SoCs having their own requirements to boot, we will need to create a HWpack in the form of a hardware specific Debos recipe that can extend and massage the Apertis ARM64 OSpack into a form that can boot.

We have picked an 64-bit Arm based board due to it being a bit more complex to boot compared with an 64-bit Intel based board. With that said, it is likely that any custom board will require some steps to be taken to customise the boot firmware, kernel and/or image contents. It should also be noted that we have picked an architecture that is already supported in the Apertis project, targeting an as yet unsupported architecture would require significantly more effort and is out of scope for this guide. Please contact the Apertis developers if you have requirements that you need help with.

Download and Setup a Devroot

We are going to be downloading the git source trees for packages we need to modify and buildng them locally. To do this we need a suitable toolchain and environment to perform these builds. Whilst a lot of packages can be successfully built with a cross compiler tool chain, there are others that can’t be easily setup to do this. Apertis provides Devroots to act as a build environment, enabling packages to be built as though they are being built on a machine of the target architecture.

The process for setting up a devroot is documented in Development containers using devroot-enter. Download the latest v2023 ARM 64-bit devroot, install it and spawn a devroot container.

Building Arm Trusted Firmware

Looking at the documentation for building U-Boot for the Allwinner Arm SoCs, we can see that Arm Trusted Firmware (ATF) is a dependency for building U-Boot, so we’ll start with that.

The Apertis project already has a repository for the Arm trusted Firmware, we can clone that repsoitory locally:

$ git clone https://gitlab.apertis.org/pkg/arm-trusted-firmware.git
$ cd arm-trusted-firmware

In order to make it a little easier to track changes made here and updates made by the Apertis project to ATF over time (allowing easier updating in the future) we will create a branch under a distinct project name. For example if we use “orange” as our project name, we will create an orange/v2023 branch from the apertis/v2023 branch:

$ git checkout -b orange/v2023 apertis/v2023

We will also be using this project name as a suffix to the version number when creating our customised “downstream” packages.

Looking at the hardware documentation we can see that the device has the Allwinner H616 CPU. The U-Boot documentation helpfully tells us for this SoC we need to build ATF for the sun50i_h616 platform. Looking in debian/rules we find a list of platforms that the package is built for, so we add sun50i_h616 to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
--- a/debian/rules
+++ b/debian/rules
@@ -14,7 +14,7 @@
 VERBOSE=1
 endif
 
-platforms := g12a gxbb sun50i_a64 sun50i_h6 rk3328 rk3399 rpi3 rpi4 imx8mn k3
+platforms := g12a gxbb sun50i_a64 sun50i_h6 sun50i_h616 rk3328 rk3399 rpi3 rpi4 imx8mn k3
 platforms_nodebug := imx8mq
 
 # By default, iMX8MN uses UART2 console. However, other boards supported

Commit this change:

$ git commit -s -m "Add sun50i_h616 to build platforms" debian/rules

We should also create a changelog entry for a new release and commit it:

$ gbp dch --git-author --ignore-branch -l orange -R
$ DEBIAN_PACKAGE=$(dpkg-parsechangelog -S Source)
$ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
$ git commit -s -m "Release ${DEBIAN_PACKAGE} version ${DEBIAN_VERSION}" debian/changelog

Note that we pass -l orange when calling gbp dch. This uses orange as a suffix for a local “downstream” revision of the Apertis package. See our documentation on versioning for more details.

We will now build it locally, using the devroot that we setup earlier. First enter the devroot and find the source tree, the devroot_enter command will automatically bind the user’s home directory:

$ devroot-enter ${DEVROOT_PATH}
...
user@devroot:/tmp$ cd ${ATF_SOURCE_PATH}
user@devroot:/home/user/arm-trusted-firmware$ 

In order to build ATF successfully, we need to install it’s build dependencies. We will also install git-buildpackage which is a useful tool to help with building debian packages:

user@devroot:/home/user/arm-trusted-firmware$ sudo apt update
...
user@devroot:/home/user/arm-trusted-firmware$ sudo apt build-dep arm-trusted-firmware
...
user@devroot:/home/user/arm-trusted-firmware$ sudo apt install git-buildpackage
...
user@devroot:/home/user/arm-trusted-firmware$ 

We can then use git-buildpackage to build ATF:

user@devroot:/home/user/arm-trusted-firmware$ gbp buildpackage --git-ignore-branch -a arm64 -B -d -uc -us

In order for the U-Boot build to use the binaries in the package of ATF that was just built, we must install the new package in the devroot:

user@devroot:/home/user/arm-trusted-firmware$ cd ..
user@devroot:/home/user$ sudo dpkg -i arm-trusted-firmware_*_arm64.deb 
(Reading database ... 31313 files and directories currently installed.)
 Preparing to unpack arm-trusted-firmware_2.7.0+dfsg-2+apertis3orange1_arm64.deb ...
Unpacking arm-trusted-firmware (2.7.0+dfsg-2+apertis3orange1) over (2.7.0+dfsg-2+apertis3bv2023preb1) ...
Setting up arm-trusted-firmware (2.7.0+dfsg-2+apertis3orange1) ...
user@devroot:/home/user$

We will now have the required binary available:

user@devroot:/home/user$ ls /usr/lib/arm-trusted-firmware/sun50i_h616/
bl31.bin
user@devroot:/home/user$ 

Building U-Boot

Now that we have the required ATF binaries we can modify U-Boot to build for our board. Start by cloning the Apertis U-Boot repository (as we did with ATF) and creating a downstream project branch:

$ git clone https://gitlab.apertis.org/pkg/u-boot.git
$ cd u-boot
$ git checkout -b orange/v2023 apertis/v2023

Like with ATF, we need to update the package configuration to build for the Orange Pi Zero2. The U-Boot package builds for a lot of different boards already, with the configuration for these split out from debian/rules into debian/targets/mk. To add the Pi Zero, we add the following entries to this file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
--- a/debian/targets.mk
+++ b/debian/targets.mk
@@ -263,6 +263,11 @@
     spl/sunxi-spl.bin u-boot-nodtb.bin u-boot-sunxi-with-spl.fit.itb \
     u-boot.bin uboot.elf
 
+  # External Developer <external.developer@collabora.com>
+  u-boot-sunxi_platforms += orangepi_zero2
+  orangepi_zero2_assigns := BL31=/usr/lib/arm-trusted-firmware/sun50i_h616/bl31.bin
+  orangepi_zero2_targets := u-boot-sunxi-with-spl.bin uboot.elf
+
   # Frederic Danis <frederic.danis@collabora.com>
   u-boot-sunxi_platforms += orangepi_zero_plus2
   orangepi_zero_plus2_assigns := BL31=/usr/lib/arm-trusted-firmware/sun50i_a64/bl31.bin

Commit this change, create a changelog entry for a new release and commit that too:

$ git commit -s -m "Add Orange Pi Zero2 to built platforms" debian/targets.mk
$ gbp dch --git-author --ignore-branch -l orange -R
$ DEBIAN_PACKAGE=$(dpkg-parsechangelog -S Source)
$ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
$ git commit -s -m "Release ${DEBIAN_PACKAGE} version ${DEBIAN_VERSION}" debian/changelog

As with building ATF we need to ensure that all required dependencies are install in the devroot:

user@devroot:/home/user/u-boot$ sudo apt build-dep u-boot

The U-Boot package will build U-Boot for numerous target devices, which takes a long time. We can limit the number of devices it’s going to build for by setting DEB_BUILD_PROFILES in the environment prior to running gbp buildpackage. Here we limit building to just the target we are interested in:

user@devroot:/home/user/u-boot$ export DEB_BUILD_PROFILES=pkg.uboot.platform.orangepi_zero2
user@devroot:/home/user/u-boot$ gbp buildpackage --git-ignore-branch -a arm64 -B -d -uc -us
...

This generates a bunch of U-Boot binary packages in the parent directory, including one for sunxi based devices such as the Orange Pi Zero2:

user@devroot:/home/user/u-boot$ ls ../u-boot-sunxi*.deb
../u-boot-sunxi_2023.01+dfsg-1+apertis6orange1_arm64.deb

Checking kernel

Downloading the current Apertis kernel and extract it (it is easy to do this in the devroot), we find that it already ships the device tree for the OrangePi Zero2:

user@devroot:/home/user$ apt download "linux-image-*-arm64"
Get:1 https://repositories.apertis.org/apertis v2023/target arm64 linux-image-6.1.0-7-arm64 arm64 6.1.20-1+apertis4~v2023bv2023.0db1 [60.3 MB]
Fetched 60.3 MB in 13s (4580 kB/s)
user@devroot:/home/user$ dpkg -X linux-image-*.deb linux-deb
./
./boot/
...
./usr/share/lintian/overrides/
./usr/share/lintian/overrides/linux-image-6.1.0-7-arm64
user@devroot:/home/user$ ls linux-deb/boot/dtbs/6.1.0-7-arm64/allwinner/*zero2*
linux-deb/boot/dtbs/6.1.0-7-arm64/allwinner/sun50i-h616-orangepi-zero2.dtb

A quick look at the config used to build it also suggests it’s been built with support for the H616, so we’ll assume for now that the Apertis kernel is ready to boot on the OrangePi Zero2:

user@devroot:/home/user$ grep H616 linux-deb/boot/config-6.1.0-7-arm64 
CONFIG_PINCTRL_SUN50I_H616=y
CONFIG_PINCTRL_SUN50I_H616_R=y
CONFIG_SUN50I_H616_CCU=y

Image scripts

The Apertis images are assembled using Debos in a multi-stage process. Debos uses YAML based configuration files, each stage is configured by it’s own configuration file and these files are usually stored in the apertis-image-recipes repository.

To build an image, the first stage is to build an OSpack (the rootfs without the basic hardware-specific components like bootloader, kernel and hardware-specific support libraries) which tends to be a common artifact used between boards with the same architecture, this is then enhanced for a specific device or use case by a HWpack.

Clone the apertis-image-recipes repository as with previous repositories and create a downstream project branch:

$ git clone https://gitlab.apertis.org/infrastructure/apertis-image-recipes.git
$ cd apertis-image-recipes
$ git checkout -b orange/v2023 apertis/v2023

Building an OSpack

As previously mentioned, the aim is to build a fixedfunction image, to do this we start by building the appropriate OSpack (ospack-fixedfunction.yaml). It is best to build these images from the appropriate image-builder docker container, as documented in the apertis-image-recipes README.md.

$ RELEASE=v2023
$ docker run \
  -i -t --rm \
  -w /recipes \
  -v $(pwd):/recipes \
  --security-opt label=disable \
  -u $(id -u):$(id -g) \
  --group-add=$(getent group kvm | cut -d : -f 3) \
  --device /dev/kvm \
  --cap-add=SYS_PTRACE \
  --tmpfs /scratch:exec \
  registry.gitlab.apertis.org/infrastructure/apertis-docker-images/$RELEASE-image-builder \
  debos ospack-fixedfunction.yaml -t suite:$RELEASE -t architecture:arm64

Building an image with a HWpack

HWpacks are typically hardware specific and as Apertis doesn’t currently support the Orange Pi Zero2, a new HWpack will need to be created. The main differentiator which requires different HWpacks is the firmware that needs to be loaded to boot the device and where.

We create a recipe for debos to perform the steps required to make a bootable image. We can base this on an existing recipe. In this instance we have based it on image-rk3399.yaml and made the following changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
--- image-rk3399.yaml	2023-06-22 12:01:06.974489949 +0100
+++ image-h616.yaml	2024-02-23 14:23:43.067319370 +0000
@@ -2,7 +2,7 @@
 {{ $type := or .type "fixedfunction" }}
 {{ $suite := or .suite "v2023" }}
 {{ $ospack := or .ospack (printf "ospack_%s-%s-%s" $suite $architecture $type) }}
-{{ $sbc := or .sbc "rock-pi-4-rk3399" }}
+{{ $sbc := or .sbc "orangepi_zero2" }}
 {{ $image := or .image (printf "apertis-%s-%s-%s-%s" $suite $type $architecture $sbc) }}
 
 {{ $cmdline := or .cmdline "rootwait rw quiet splash plymouth.ignore-serial-consoles fsck.mode=auto fsck.repair=yes" }}
@@ -31,7 +31,7 @@
 {{ else }}
     imagesize: 15G
 {{ end }}
-    partitiontype: gpt
+    partitiontype: msdos
 
     mountpoints:
       - mountpoint: /
@@ -103,22 +103,25 @@
       - linux-image-{{$architecture}}
 {{ end }}
 
-  - action: apt
-    description: U-Boot package
-    packages:
-      - u-boot-rockchip
+  - action: overlay
+    description: Add local debs
+    source: overlays/local-debs
 
-  - action: raw
-    description: Install loader1 for {{ $sbc }}
-    origin: filesystem
-    source: /usr/lib/u-boot/{{ $sbc }}/idbloader.img
-    offset: {{ sector 64 }}
+  - action: run
+    description: Install local debs
+    chroot: true
+    command: dpkg -i /home/user/*.deb
+
+  - action: run
+    description: Remove local debs
+    chroot: true
+    command: rm /home/user/*.deb
 
   - action: raw
     description: Install U-Boot for {{ $sbc }}
     origin: filesystem
-    source: /usr/lib/u-boot/{{ $sbc }}/u-boot.itb
-    offset: {{ sector 16384 }}
+    source: /usr/lib/u-boot/{{ $sbc }}/u-boot-sunxi-with-spl.bin
+    offset: {{ sector 16 }}
 
   - action: run
     description: Switch to live APT repos

Going through these changes one at a time:

1
2
-{{ $sbc := or .sbc "rock-pi-4-rk3399" }}
+{{ $sbc := or .sbc "orangepi_zero2" }}

A fairly obvious change, we switch the name of the sbc from rock-pi-4-rk3399 to orangepi_zero2. Note however that the name chosen is the name used in the U-Boot packaging (it’s used later in the script as a directory name).

1
2
-    partitiontype: gpt
+    partitiontype: msdos

Looking at the documentation we need to copy the u-boot-sunxi-with-spl.bin binary we built to 8MB into the boot device. With 512B sectors, this is at sector 16.

Since sector 16 is within the gpt partition space, we will instead utilise a msdos partition table.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
-  - action: apt
-    description: U-Boot package
-    packages:
-      - u-boot-rockchip
+  - action: overlay
+    description: Add local debs
+    source: overlays/local-debs
 
-  - action: raw
-    description: Install loader1 for {{ $sbc }}
-    origin: filesystem
-    source: /usr/lib/u-boot/{{ $sbc }}/idbloader.img
-    offset: {{ sector 64 }}
+  - action: run
+    description: Install local debs
+    chroot: true
+    command: dpkg -i /home/user/*.deb
+
+  - action: run
+    description: Remove local debs
+    chroot: true
+    command: rm /home/user/*.deb

The rk3399 is an officially supported board and the offical Apertis U-Boot build is used. The Orange Pi Zero2 build has required us build a U-Boot package locally, so instead of installing a U-Boot package from the Apertis repositories, we need to :

  • Add an overlay holding the binary U-Boot package,
  • Install these binary packages using dpkg when building the image,
  • Delete the binary packages from the image.

In addition to the changes in the recipe, the U-Boot package u-boot-sunxi_2023.01+dfsg-1+apertis6orange1_arm64.deb has been copied to a new overlay in the local repository:

$ mkdir -p overlays/local-debs/home/user
$ cp ../u-boot-sunxi*.deb overlays/local-debs/home/user/

Additionally from the above patch segment (and the one below) we can see that rockchip installs 2 pieces of firmware at different offsets. With the Orange Pi Zero2 we only install one and thus one of these actions gets removed.

1
2
3
4
5
6
7
   - action: raw
     description: Install U-Boot for {{ $sbc }}
     origin: filesystem
-    source: /usr/lib/u-boot/{{ $sbc }}/u-boot.itb
-    offset: {{ sector 16384 }}
+    source: /usr/lib/u-boot/{{ $sbc }}/u-boot-sunxi-with-spl.bin
+    offset: {{ sector 16 }}

Lastly we have the step where the U-Boot binary is written to the required offset in the image. We are simply modifying this section to use the desired binary (/usr/lib/u-boot/orangepi_zero2/u-boot-sunxi-with-spl.bin, provided by the u-boot-sunxi package) and write it to the required offset (sector 16, which is an 8MB offset).

We can now build the final image, again using the appropriate image-builder docker container:

$ RELEASE=v2023
$ docker run \
  -i -t --rm \
  -w /recipes \
  -v $(pwd):/recipes \
  --security-opt label=disable \
  -u $(id -u):$(id -g) \
  --group-add=$(getent group kvm | cut -d : -f 3) \
  --device /dev/kvm \
  --cap-add=SYS_PTRACE \
  --tmpfs /scratch:exec \
  registry.gitlab.apertis.org/infrastructure/apertis-docker-images/$RELEASE-image-builder \
  debos image-h616.yaml -t suite:$RELEASE -t architecture:arm64

Installing the image on an SD card

The standard Debos recipes, on which we have based the recipe for H616 image creation, create ancillary files when generating the image to allow images to be quickly written on to SD cards and alike using bmaptool.

The image writing process is quite simple as the Debos scripts have already generated the image with the required partitions and firmware installed as required so that the image can just be written to the beginning of the SD card:

$ sudo bmaptool copy apertis-v2023-fixedfunction-arm64-orangepi_zero2.img.gz $sdcard

Make sure to set $sdcard to the device file of the SD card.

Insert the SD card into the board, attach the serial console cable and power supply and boot:

U-Boot SPL 2023.01+dfsg-1+apertis6orange1 (Jun 12 2023 - 17:51:06 +0000)
DRAM: 1024 MiB
Trying to boot from MMC1
NOTICE:  BL31: v2.7(debug):apertis/2.7.0+dfsg-2+apertis3-2-g6830268
NOTICE:  BL31: Built : 16:00:31, Jun 13 2023
NOTICE:  BL31: Detected Allwinner H616 SoC (1823)
NOTICE:  BL31: Found U-Boot DTB at 0x4a0923e8, model: OrangePi Zero2
INFO:    ARM GICv2 driver initialized
INFO:    Configuring SPC Controller
INFO:    PMIC: Probing AXP305 on RSB
INFO:    PMIC: aldo1 voltage: 3.300V
INFO:    PMIC: aldo2 voltage: 3.300V
INFO:    PMIC: aldo3 voltage: 3.300V
INFO:    PMIC: bldo1 voltage: 1.800V
INFO:    PMIC: dcdcd voltage: 1.500V
INFO:    PMIC: dcdce voltage: 3.300V
INFO:    BL31: Platform setup done
INFO:    BL31: Initializing runtime services
INFO:    BL31: cortex_a53: CPU workaround for 855873 was applied
INFO:    BL31: cortex_a53: CPU workaround for 1530924 was applied
INFO:    PSCI: Suspend is unavailable
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x4a000000
INFO:    SPSR = 0x3c9
INFO:    Changed devicetree.


U-Boot 2023.01+dfsg-1+apertis6orange1 (Jun 12 2023 - 17:51:06 +0000) Allwinner Technology

CPU:   Allwinner H616 (SUN50I)
Model: OrangePi Zero2
DRAM:  1 GiB
Core:  48 devices, 18 uclasses, devicetree: separate
WDT:   Not starting watchdog@30090a0
MMC:   mmc@4020000: 0
Loading Environment from FAT... Unable to use mmc 0:1...
In:    serial@5000000
Out:   serial@5000000
Err:   serial@5000000
Net:   eth0: ethernet@5020000
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
U-Boot menu
1:	Apertis v2023 6.1.0-0.deb11.13-arm64
2:	Apertis v2023 6.1.0-0.deb11.13-arm64 (rescue target)
Enter choice: 1:	Apertis v2023 6.1.0-0.deb11.13-arm64
Retrieving file: /initrd.img-6.1.0-0.deb11.13-arm64
Retrieving file: /vmlinuz-6.1.0-0.deb11.13-arm64
append: root=UUID=208a5921-308d-4b07-b953-ef916f2f6da9 rootwait rw quiet splash plymouth.ignore-serial-consoles fsck.mode=auto fsck.repair=yes
Retrieving file: /dtbs/6.1.0-0.deb11.13-arm64/allwinner/sun50i-h616-orangepi-zero2.dtb
Moving Image from 0x40080000 to 0x40200000, end=42120000
## Flattened Device Tree blob at 4fa00000
   Booting using the fdt blob at 0x4fa00000
Working FDT set to 4fa00000
   Loading Ramdisk to 48054000, end 49fff45c ... OK
   Loading Device Tree to 000000004804d000, end 00000000480530d0 ... OK
Working FDT set to 4804d000

Starting kernel ...

Apertis v2023 apertis ttyS0
apertis login: