Often target platforms are not supported by Apertis. This may be due to the way the Apertis components are built/configured or due to these components lacking support for the required hardware. Whilst they may lack support in Apertis, these boards frequently have reference images based on another Linux variant with the associated source code being available. This support is often provided in a Yocto-based deliverable. The support generally provides good coverage of the vendors hardware, however the quality of implementation can vary quite dramatically.
When evaluating a platform for use in a product, it is important to evaluate both the technical and business related factors to determine whether it represents a cost effective solution. This document aims to:
- highlight deficiencies commonly found in vendor hardware support,
- provide some ways in which the quality of vendor provided support can be quantified,
- suggestions to help with evaluating the changes to be integrated into Apertis, and
Hardware packages and OSpacks
Hardware packages (HWpacks) provide tailored Linux kernels, bootloaders and any specific drivers required to enable Apertis OSpacks to be run on target boards. Apertis provides hardware packages that enable it to be used on several reference boards across a number of architectures.
This document focuses on the Linux kernel as it is expected that a version of the Linux kernel will be required by all Apertis users. Whilst a Bootloader will be required for a device, these can vary far more and thus do not provide such a good example.
Common vendor kernel issues
Whilst the software provided by the vendor typically does a good job of showcasing their hardware it is typically not provided in a form that would be considered to follow best practice. Common issues are outlined below.
Limited enablement of hardware and core functionality in provided configuration
The configuration provided when building the kernel and other low-level components is generally tailored to just provide the support required to demo the vendors hardware. As a result this configuration is generally inappropriate for direct use in products as they will typically require drivers for hardware not provided on the vendors reference hardware and require additional functionality to be enabled to support application specific workloads.
Disregard for common code
The vendor is typically only interested in providing working support for their hardware and this is frequently done in the most expedient manner possible. As a result both the parameters for and logic of common code, which exist to support multiple existing pieces of hardware, are modified without regard for the continued operation of the other hardware. This is problematic when attempting to utilise a common source tree for multiple projects, utilizing different hardware platforms, where the changes made for one platform break support implemented for others.
This can sometimes result in build failures when attempting to enable hardware not found in the vendor provided configuration (see “Limited Enablement of Hardware and Core Functionality In Provided Configuration”).
No “clean” patch series implementing the vendors changes
Frequently the vendor will provide a git tree containing the kernel
source and any target specific changes. The commits from which these
changes are comprised are not usually provided as a clean set of changes on top
of the mainline version (such as would be achieved by performing
when shifting focus to a newer version of the underlying component), but are
scattered through the commit history as subsequent mainline versions have been
added through the use of
git merge. This can make it difficult to generate
a clean set of patches to apply the vendor BSP to the mainline kernel source
of the same version and to port support to a newer kernel version to take advantage of security and bug fixes.
The changes to the source code might not only be changes for the required processor/target board, but also other changes like:
- Realtime patches
- Some improvements from perspective of vendor not directly related to hardware support.
- Addition of hardware support for other hardware provided by the vendor but which is not of consequence for the projects target platform
These changes may be scattered throughout the commit history (see No “clean” patch series implementing the vendors changes and typically lack useful commit messages to enable the intention behind any given change from being understood. This makes resolving conflicting changes between the newer mainline and BSP specific changes difficult to rectify.
This leads to problems when upgrading with newer versions of the mainline kernel.
Compatibility with ospacks
The Apertis operating system packages (ospacks) rely on the kernel with which they are paired to provide certain functionality. This functionality is expected to be provided using the common APIs found in the mainline kernel. Vendors sometimes either modify/extend or utilize custom APIs to expose functionality supported by their product. This combined with limitations in functionality enabled in the vendors kernel config can lead to the kernel not providing the functionality expected by the ospack, either in a form it can use or at all.
Lack of Mainline Support for Vendors Hardware
The Apertis project recommends hardware support is submitted for inclusion in the mainline kernel. This results in:
- Greater testing of common code.
- Ability to easily generate a single binary, supporting multiple target boards with a common architecture.
- Best support for security updates due to being able to efficiently track the active upstream code changes.
- Best compatibility with Apertis.
Many vendors do not actively work with the mainline kernel developers to get support for their hardware into the mainline kernel. The use of a vendor specific kernel reduces the upfront time and effort required to build a solution, but results in a significant increase in long term maintenance effort. Whilst more effort is required to achieve mainline kernel support, this effort frequently results in a higher quality of support being achieved than what would be expected from a vendor kernel.
There are often a number of changes that need to be made before the mainline kernel supports a target board. Sometimes the mainline kernel can be used directly though typically the following additions are required (in increasing effort):
- The kernel configuration has to be adopted
- A new device tree (.DTB) has to be added
- Board specific drivers have to be added
- Addition of support for unsupported system on chip.
The board specific drivers are available in the vendor specific kernel. With some kernel knowledge, the drivers can usually be ported to the mainline kernel (though this can be difficult for low quality vendor kernels). If the platform is important to you and the vendor has no interest in upstreaming support themselves, taking on the challenge of upstreaming support yourself may be beneficial in the long term.
Evaluating the quality of the vendor BSP
With the increasing complexity and requirement for devices to be internet enabled, the old “embedded” approach of freezing and hardening a BSP no longer works. Any issues that are not detected during the hardening process may result in serious security vulnerabilities being left present in shipped devices. As a result there is an increasing requirement to perform “in the field” updates to shipped devices so that upgraded kernels and other system components to resolve security issues.
Some vendors do not yet provide on going support for newer kernels, nor work to include support in mainline kernels. Some will use development practices that hinder the ability of a product team to perform these updates themselves when necessary. It is therefore imperative that the quality of the vendors software support be evaluated when selecting hardware for a project as this can drastically effect the economic viability of a vendors offering over the lifetime of a product.
Understanding Kernel Versioning
Each mainline kernel is defined by its version number. This is typically in the form of 2 or 3 dot separated numbers, where these numbers are:
<major version>.<minor version>.<stable release>
This may have other parts appended to it after a dash with these extra parts being used to denote “release candidate” kernels or vendor specific kernels.
This description holds true back to the beginning of the
kernels prior to that used a different convention, though we would not
expect kernels of that vintage to be being used in active development
owing to them being over 10 years old and thus will not cover that here.
The mainline kernel is released as a 2-number version, such as
minor version is incremented with each official release. Every so often
the major version number is incremented and the minor is returned to zero.
There is no defined point at which this occurs and it is at Linus Torvalds'
discretion as to when this happens.
Once a release is made, this is considered stable. Bugfixes will be applied to this until the next kernel release. Each bugfix release is released with a 3 number version, with the “stable release” part incrementing for each release. Some kernels will be supported for an extended period and are know as longterm stable (LTS) releases. Theses are the kernels that are used by Apertis.
More information regarding kernel release can be found on kernel.org.
Determine what kernel upgrades can be expected by the vendor
The vendor support for upgrades to newer versions varies a lot. Some vendors will do a single code drop with no support provided beyond that. Others provide upgrades, but without offering an upgrade plan. Most of the time it will be months between updates, during which time the vendor kernel will lack security and bug fixes that may be found in the mainline and longterm stable kernels.
Age and maintenance state of the related mainline kernel
Each vendor kernel is based on a mainline kernel. It is important to find out which kernel the vendors support is based on and determine it’s age and it’s support status. Wikipedia maintains a full list of release kernel versions, noting both their age and anticipated support “end of life” (EOL) dates, whilst kernel.org lists the projected EOL dates for still supported LTS releases along with detailing the latest release for each supported version.
When shipping a product, we do not want to be following the mainline development as a source of bug and security fixes as this will result in significant changes being pulled in when updating the kernel, which is likely to result in both unwanted changes to the features and performance on the target product and is likely to result in difficulties porting any vendor or product specific kernel modifications.
As stable kernel releases aim to only include security and bug fixes, this results in significantly less changes to the code base, reducing the risk of feature or performance change and reducing or eliminating issues with porting modifications. From a long term maintenance perspective it is therefore beneficial for a product to use a vendor that bases their vendor kernel on the recent LTS kernel (one with sufficient time before it reaches it’s end of life, as determined by the project time frames), even more so if they actively track the stable point releases.
Is a rebase to a longterm stable kernel needed?
Since development of the mainline kernel is always going on, the quality of these parts varies dependant on maturity. Also security issues like vulnerabilties are handled during this development process with security fixes backported to the kernels supported stable releases.
Apertis tracks this development effort to ensure the bug fixes, security fixes and feature improvements made during the mainline development process are available in Apertis, with each Apertis release including the latest Longterm Stable kernel with point releases tracking changes on that branch. Any vendor BSP integrated into Apertis should also be able to update regularly to include these fixes and improvements, however many vendors do not provide kernel updates leaving it to their users to either perform these updates themselves or remain on older kernels versions with known security vulnerabilities.
Is the vendor working to integrate support into the mainline kernel?
Even vendors that provide good on going support for their hardware via vendor trees lag behind the fixes provided in the mainstream kernel, as they require these releases to be made before porting any of their changes. Additionally, projects with long expected lifetimes may exceed the period for which a vendor is willing to provide support.
If a vendor “upstreams” support for their device, integrating that support right into the mainline kernel, the vendor kernel is typically no longer required. A correctly configured mainline kernel can be used, removing any lag and typically enabling the device to be used long after the vendor stops actively supporting it without significant addition effort and cost.
Are there qualified test results for vendor BSP ?
Some vendors provide tests and results for their specific images to check hardware and kernel features. These tests usually cover specific hardware functions and less common features of the kernel. So additional tests have to be planned and organized, probably got from mainline kernel. Since the image is a new binary (compared to images of mainline kernel), these tests have to be redone best at automatic test systems like LAVA (see also Apertis integration testing with LAVA).
How is the responsibility to maintain the vendor BSP organized ?
Developing and purchasing a product includes product liability for the company over product lifetime. This includes aspects for security (loss of personal data) and safety (risk to customers physically). The Linux kernel potentially plays a very important role in both, thus ensuring bugs and security issues get fixed promptly and reliably is critical. These updates either have to be done by the product manufacturer or the supplier of the vendor providing the kernel support. As vendors will typically stop providing support after a defined period, the timing of this needs to be understood in relation to the products life time. If the vendors support period would end before that of the product, then it needs to be understood (and factored in) that at such a time it becomes the responsibility of the product team to organize such maintenance support.
Evaluate the Quality of Vendor Added Drivers
Any drivers added by a vendor can be found by analyzing a diff between the
vendor’s kernel sources against the related mainline kernel source. Drivers
should be fairly self contained under
drivers/, with minor changes in:
- Startup files
A deep review of implementation and testing of the relevant kernel driver is recommended. This is a time consuming task.
The following practical checks can be made to help determine the quality of the vendor supplied kernel source.
Check the Ability to Build Mainline Supported Devices
Well integrated vendor support will result in a kernel that can still be built
for other machines which share the same architecture and which are defined in
the mainline kernel. Attempting to build using the configs provided in the
configs/ folder for your target architecture (such as
32-bit ARM devices). Build problems may indicate incomplete or unintended
changes to common parts of the code.
Check the Ability to Build for Machines With Different Architectures
This is similar to testing configs for other machines from your target architecture, but can point to additional issues where changes have been made to very common code in architecture specific ways. This will require a toolchain supporting the relevant architecture. Build problems may indicate incomplete or unintended changes to common parts of the code.
Attempting to build a multi-target kernel binary
As long as sufficient similarities exist, it is possible to build a single kernel binary that can be used with multiple different machines with different peripherals and even processors.
The kernel provides configs that build a kernel binary which can run on
multiple different devices. An example is
arch/arm/configs/multi_v7_defconfig, which will build a kernel for many ARMv7
compatible devices. If such a configuration exists that may be compatible with
the device your interested in, it may be instructive to attempt to add the
target device to such a config and build it.
The need to build separate kernel binaries may indicate, that some driver interfaces are not implemented optimally. This will lead to reduced test depth and increased build efforts.
Determining quantity and quality of vendor kernel mondifications
It is typically not possible to utilise the git log to compare a vendor kernel to the mainline kernel from which it was built due to the previously mentioned development practices. As a result it is typical to need to utilise standard “diff” tools to make the comparison.
The differences highlighted by these tools need to be examined to identify the changes relating to target hardware. It would be typical to see:
- Hardware drivers and related header files
- Files to integrate hardware drivers like Kconfig(s) or Makefile(s)
- Addition of hooks to common files
- Expanded support in shared architecture files.
Thus the majority of the changes required to support a target device should be expected to be limited to those files providing drivers and support for the applicable architecture.
It is important to identify changes in other areas, including those not related to the target hardware, such as:
- Changes of significant complexity to common headers and source files
- Addition of unrelated support, such as those provided by the real-time patch set
- Unintended changes, e.g. changes of header files with contents of older kernel versions
- Unrelated improvements (at least from viewpoint of the supplier of the vendor kernel)
- Reformatting indenting and code style of entire files, masking subtle changes
The bigger the changes unrelated to the target hardware and the more spread the changes are, the greater the effort that will be required to upgrade to newer kernel versions.
Attempt to build a kernel using the Debian config with Debian patches applied
When adding kernel support for a device to Apertis, it is advised to base this on the Apertis kernel to enable you to easier take advantage of security updates. If the vendor kernel and the Apertis kernel are both based on a similar kernel, in order to test compatibility between the changes introduced by the vendor and those applied to the Apertis kernel it can be instructive to apply the Apertis patches to the vendor kernel and attempt to build with the Apertis kernel configuration (which is likely to have support for a broader number of features enabled).
customary with Debian packaging,
changes to an upstream source and any other files (such as the component
configuration file) are stored in the
.debian.tar.xz file. The relevant files
for Apertis can be found on the
Apertis package repository.
Extracting this archive will create a
debian/ folder. The patches applied by
Apertis can be found in
series file provides the order
and enables the series to be quickly applied with tools such as
Debian configuration is split in to a generic portion (in
debian/config/config), architecture configuration (in
debian/config/kernelarch-<arch>) and more fine grained architecture config
debian/config/<arch>), with a “full” configuration comprised of the
options found in one of each of these sets.
While Debian builds kernels for many different
architectures (and thus configuration files are present for many
architectures), Apertis limits it’s builds to
thus it is advised to limit testing to those architectures.
Applying these patches and building the kernel with this kernel configuration can highlight problems in common parts, which may make upgrades difficult without loss of functionality.