Set-up the R-Car board

To evaluate Pipewire, we will use an R-Car Starter Kit Premier board (RTP8J779M1ASKB0SK0SA003) with several Leagy USB Sound Cards (also available at www.conrad.de).

The first step is to install Apertis v2023 (previous Apertis releases don’t support R-Car audio output) on the board eMMC. Running tests from Apertis installed in an SD card is not recommended as it may reduce performance.

The support of R-Car audio output has recently been added to the Apertis v2023 linux kernel, thus we need at least linux 6.1.20-2~bpo11+1+apertis4.

To avoid having to update the kernel after installing Apertis, we will flash a daily image (i.e. built after 2023.10.10) of Apertis which includes the required kernel.

The easiest way to achieve that is first to flash Apertis on an SD card, then use this SD card to boot the board. From there, we can flash Apertis on the eMMC.

How to flash Apertis on SD card

First, insert your SD card in a computer then run fdisk -l to identify by which device it is referenced:

$ sudo fdisk -l

Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Device           Start     End Sectors   Size Type
/dev/mmcblk0p1      34  500000  499967 244.1M EFI System
/dev/mmcblk0p2  500001 5859375 5359375   2.6G Linux filesystem
/dev/mmcblk0p3 5859376 7812466 1953091 953.7M Linux filesystem

In our example, SD card is /dev/mmcblk0. Now, we can tell bmaptool to copy our Apertis image to the SD card:

$ sudo bmaptool copy https://images.apertis.org/daily/v2023/20231010.0115/arm64/fixedfunction/apertis_v2023-fixedfunction-arm64-uboot_20231010.0115.img.gz /dev/mmcblk0

For this evaluation, we flashed a daily FixedFunction (APT-based) image of Apertis.

Once copying is complete, the SD card is ready to go in our R-car board.

How to flash Apertis on eMMC

Before, booting your board, you will have to use a microUSB cable to connect the PC to R-Car board and then open a serial console with for instance picocom:

$ sudo dmesg | tail	# To help identifying the tty device name
$ sudo picocom -b 115200 /dev/ttyUSB0

Now, it’s time to boot the board, if u-boot is correctly configured, apertis will start otherwise, you will have to re-install a new one (please refer to the Apertis documentation to update u-boot, see rcar-gen3_setup).

Once, you have access to the Apertis shell, the first step is to enable the development repository in order to install bmap-tools. This can be done by adding:

deb https://repositories.apertis.org/apertis/ v2023 target development

to /etc/apt/sources.list.

$ sudo vi /etc/apt/sources.list
$ sudo apt update && sudo apt install bmap-tools

Once, bmap-tools is installed, you can flash Apertis on the eMMC. Please refer to steps above to identify the device name of your eMMC and how to use bmaptool copy.

When, Apertis is flashed on the eMMC, it’s time to remove the SD card from the board and to reboot the board.

How to check if sound works

After booting Apertis from the eMMC, we need to enable again the development repository on this new system because we will need tools only available in development for our tests (i.e. alsa-utils, sndfile-tools, graphviz and python3-matplotlib). At the time of writing this report, the updates repository was also required for sndfile-tools, but this is no longer relevant since the folding happened. sndfile-tools is now available in the main repository.

Now, we can install some tools that will be used in the next steps:

$ sudo apt update
$ sudo apt install alsa-utils pulseaudio-utils \
                   pipewire-jack sndfile-tools \
                   psmisc graphviz python3 python3-matplotlib \
                   stress-ng

alsa-utils will be used to test sound cards directly using ALSA.
pulseaudio-utils gives us pactl to check if pipewire is running.
pipewire-jack provides the PipeWire implementation of JACK, it’s the one we are going to evaluate using pw-jack.
sndfile-tools for its sndfile-jackplay allowing us to play wav files with JACK.
psmisc will install killall which is used during tests to kill some processes.
graphviz provides dot which can convert dot files generated by pw-dot into png files to represent the PipeWire graph.
python3 to execute python test scripts.
python3-matplotlib to generate graph from results.
stress-ng to simulate a high system load.

If the linux kernel is recent enough, you should have sound cards registrered:

$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: rcarsound [rcar-sound], device 0: rsnd-dai.0-ak4613-hifi ak4613-hifi-0 []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: rcarsound [rcar-sound], device 1: rsnd-dai.1-i2s-hifi i2s-hifi-1 []
  Subdevices: 1/1
  Subdevice #0: subdevice #0

The tool speaker-test can be used to verify if speakers can make sound, but before we need to bump the volume of DVC Out with:

alsamixer -D hw:0

Select the item DVC Out with the left and right arrows and then use the up and down arrows to adjust the volume. Now, speaker-test should make noise (be careful when using a headset because the volume can be quite loud).

Now that the sound works, it is time to connect all devices together.

Setup for pipewire evaluation

A USB sound card is connected to the board while the other two are connected to the laptop for capture. This is a picture of our setup:

Setup for testing

Speakers (or earphones in our example) can be plugged in the LINE OUT of board’s sound cards in order to confirm it works as expected:

# Identify playback device
$ aplay -l

# Test onboard sound card output
speaker-test -Dplughw:CARD=rcarsound -c2

# Test USB sound card output
speaker-test -Dplughw:CARD=ICUSBAUDIO7D -c2

Before running tests, we need to substitute earphones with jack cables plugged on the other side in the LINE IN of the two other capturing sound cards:

Setup for PipeWire evaluation

Test case 1: Switching audio streams onto running devices

This test case is meant to validate that adding or switching an audio stream in a running output neither impacts that output or any other system outputs. To do this both available outputs will receive an audio stream (from a jack based application) with a unique constant tone. A third audio stream (with a seperate tone) will be used for the switching test and every 5 second switches from the onboard to the usb audio output.

Test case 1

Three scripts are available to run this test test-case-capture.py, test-case-1-run.py and generate_wav.sh.

The first one is to capture sound coming from the sound cards of the board and has to be run on the laptop side. This script uses pw-record to record sound that means pipewire must be used on the computer doing the capture (for example by using Apertis SDK or Debian Bookworm). From this script, two wav files are generated.
The second script starts two different streams on both board sound cards (onboard and USB), and then it will start a third steam and will switch the output every each 5 seconds.
The last one generates wav files using GStreamer. These files will be played during tests with pw-jack sndfile-jackplay.

Results

During the test, two wav files are generated test-case-capture1.wav and test-case-capture2.wav (both are available in results/test-case-1/). The first one captures the sound coming from the board USB sound card while the second capture the sound coming from the board built-in sound card.

At the beginning of test-case-capture1.wav, we can hear a mixture of both tone test300.wav and test800.wav while for test-case-capture2.wav only the tone from test300.wav is audible. At ~ 7 sec, the tone from test800.wav is transferred from the USB card to the built-it card. In other words, we stop hearing it in test-case-capture1.wav and we start hearing it in the other file test-case-capture2.wav. The switch of the test800.wav tone appears every 5 secs, the first one is at ~ 7 secs, then 12, 17, etc until 37 secs then the capture stops.

When the switch occurs, we don’t hear any silence or any artefact like cracking. The sound coming from the different outputs remains smooth when a new stream is added or removed.

Test case 2: Audio input reconfiguration

As with the first test case both outputs will receive constant audio tones; On top of that the onboard microphone output will be capture via loopback and also output via the onboard line-out. The audio captured will be reconfigured from a rate of 8K to 16K and vice-versa every few second to validate that this doesn’t impact either the direct output or the unrelated output.

Test case 2

For this test case, we use the previous setup as basis then we plug a microphone (in our example, we use the microphone of the white earset) into the MIC INPUT of the board.

Setup for PipeWire evaluation with test case 2

NOTE: The built-in sound card is affected by a hardward limitation preventing to independently change the rate of input and output. Since the aim of this test is to evaluate pipewire and not the built-in sound card, the audio input reconfiguration is done on the USB sound card and not on the built-in sound card. In other words, we capture sound with the USB card LINE In and redirect it to the LINE Out of the same card (which differs from the diagram and picture above). The reason for this apparently limitation is that the onboard R-Car card only has one reference clock available for the input/outputs. Which means both input and output rates need to be compatible with the current clock rate, hence the requirement to reconfigure both whenever changing that clock rate. Other hardware (like the USB card in this case) often has seperate clocks for different inputs and outputs and hence is capable of switching rate independently.

The script test-case-2-run.py does the same setup as test-case-1-run.py, it only differs by not starting a third moving stream but instead capture the USB LINE In and redirect it to the USB LINE Out by changing every 10 secs the capture sample rate. Due to a hardward limitation, the capture sample rate is reconfigured between 44100 and 48000 instead of the unspported 8000 and 16000.

The script test-case-capture.py can be used to record sound generated by the board sound cards during the test (i.e. with changing capture sample rate of the stream redirected to 1 output) to evaluate how the sound is affected.

Results

Currently, it’s not possible to reconfigure a running device. That means, to reconfigure the sample rate input, we have to force the device input to be idle.

During the test, two wav files are generated test-case-capture1.wav and test-case-capture2.wav (both are available in results/test-case-2/). The first one captures the sound coming from the board built-in sound card while the second capture the sound coming from the board USB sound card.

In test-case-capture1.wav, we can hear the test300.wav tone without any artefact and without any interruption. This stream is therefore not affected by the reconfiguration of the input of the other sound card.

In the other file test-case-capture2.wav, the stream starts with the test300.wav tone, then at 0:10 we start hearing the test800.wav tone which is captured at 44100 Hz and mixed to the first tone. We stop hearing the test800.wav tone at 0:20 while the test300.wav tone is still audible. This pause of the test800.wav tone continues for 5 secs allowing the reconfigure the capture input rate at 48000 Hz by waiting the input device to be idle. Then, at 0:25 we hear again a mix of test300.wav and test800.wav captured at 48000 Hz for 10 secs. At 0:35, we stop hearing the test800.wav tone because of the reconfiguration at 44100 Hz, which restart 5 secs later and so on until 1:10. The sample rate reconfiguration is checked during the test in the FORMAT column of the pw-top output.

In this case, only a part of the direct output is affected by the reconfiguration. The stream which is not reconfigured (i.e. coming from pw-jack sndfile-jackplay) is not affected, whereas the captured stream which is reconfigured is affected. Indeed, to apply the reconfiguration we have to wait the input device to be idle to make pipewire apply the new configuration. The default timeout for switching a device to idle is defined in a wireplumber config file (i.e. /usr/share/wireplumber/main.lua.d/50-alsa-config.lua) at 5 secondes (see session.suspend-timeout-seconds). It should be possible to reduce the timeout to only 1 seconde (setting 0 disables suspend on idle, so not the desired behavior), but that will still impact the stream.

To summarize, the reconfiguration of audio capture only impacts the captured stream and not the direct output since we have to stop the stream to make pipewire apply the changes. Morevoer, the unrelated output is not affected. If feasible, adding a way to force pipewire to reconfigure a device without impacting (or at least by reducing its impact) the current stream is probably a way to go.

Test case 3: Capturing multiple inputs to multiple outputs

As in previous test case as a basis each output will get a unique tone again. Also audio from both inputs is captured via loopbacks and output via a unique specfic channel on each output, for example the onboard mic input will be output via the left channel of both outputs and the usb microphone via the right channel. Overall each output will get a mixed stream of its unique tone and and one microphone input in each of its channels. The goal here is to validate that starting, stopping inputs independently does not cause disruptions in the outputs.

Test case 3

This test case requires the setup used in the test case 2 to which we add a new source sound (here we use a music player) plugged into the LINE IN of the board USB sound card.

Setup for PipeWire evaluation with test case 3

The script test-case-3-run.py does the same setup as test-case-1-run.py, it only differs by not starting a third moving stream but instead captures one input and redirects it to a unique channel of both outputs and capture the other input to redirect it to a other channel of both outputs. Then, the capture is stopped and (re)started repeatedly.

The script test-case-capture.py can be used to record sound generated by the board sound cards during the test (i.e. repeatedly starting and stopping the capture) to evaluate how the sound is affected.

Results

Both output were recorded in test-case-capture1.wav and test-case-capture2.wav, respectively from the USB and built-in sound cards.

In test-case-capture1.wav, we hear on both channels (left and right sides) the test300.wav tone played with pw-jack sndfile-jackplay. Moreover, we can hear on the left channel the sound captured from the MIC Input of the board whereas on the right channel we can hear the test800.wav tone coming from the music player through the LINE IN input of the USB sound card. The capture of both inputs is stopped and restarted every 5 secondes, in other words we have 5 secondes of capture then 5 secondes without capture and so on.

In the other recorded file test-case-capture2.wav, we hear on both channels the test800.wav tone played with pw-jack sndfile-jackplay. And like for this previous file, the left side captured the MIC Input of the board whereas the left side captured the LINE IN of the USB sound card i.e. the test800.wav tone.

In both recordings, we cannot detect any disruptions in the outputs when starting and/or stopping the differents inputs. But, on these two recordings, we have noise during the capture. Both sides give different artefacts, the left (capturing the MIC Input) is mainly affected by hatching sound whereas the right (capturing the LINE IN) is mainly affected by cracking noise.

In order to identify the cause of these artefacts, the same test was done without playing tones with pw-jack sndfile-jackplay. The result was similar with same artefacts (results are available in results/test-case-3/without-jackplay). Redirecting only one stream on one side (so without mixing several streams) gives a good quality result without any artefact.

Adding an additional intermidiate node to couple streams before routing them to outputs doesn’t give better results. See test-case-3-add-intermediate-nodes.sh for the manual steps.

We are evaluating the pipewire version provided by apertis v2023 i.e. pipewire 0.3.59, however this use-case might have improved in more recent versions. A quick look into the changelog after 0.3.59 shows us improvements in pipewire-jack, audioresampler, etc that could fix issues we are facing.

Test case 4: Benchmarking load caused by a bigger number of clients

In this test case only the onboard audio output will be used. Up to 50 audio clients generating a constant tone will be running in parallel; Next to that a test application will connected to pipewire, output a short tone and exit; Measuring how long it takes to connect and play back the tone.

The goal here is to benchmark if more audio clients have an impact on initial time to playing back the audio.

Test case 4

The script test-case-4-run.py measures how long it takes to run a pw-jack sndfile-jackplay my_sound.wav on the board after having started an increasing instance number of other pw-jack sndfile-jackplay. In other words, it will measure the time required to run pw-jack sndfile-jackplay when only 1 other instance is running, then with 11 other instances, 21 other instances, etc until 51 instances.

The script test-case-capture.py can be used to record sound generated by the board sound cards during the test (i.e. with many instances of pw-jack sndfile-jackplay) to evaluate how the sound is affected.

Results

To eliminate random variability, each scenario is performed ten times then a mean is computed to provide the most accurate overview of pipewire capabilities.

In the graph below, the horizontal axis is the number of already running sndfile-jackplay instances (from 1 to 51) whereas the vertical axis is the time required to start a new pw-jack sndfile-jackplay and to play the file /usr/share/sounds/alsa/Front_Center.wav (lasting 1.428 secs). The red line is drawn from the means and every blue dot is a measure of required time.

Test case 4

As we can see, the time starts to increase when we already have 30 stream running. Indeed, the means is 1.499 secs for both for 1 and 11 running streams and 1.503 secs for 21 streams (+ 4ms). From here, the time will increase to 1.511 secs for 31 streams (+ 12ms) and it will reach 1.524 secs for 41 streams and 51 streams (+ 25ms).

To summarize, we can see a variability in the order of ~ 30 ms for each set of tests, for example with 1 instance the time ranges from 1.49 to 1.52 secs, and the same for 51 instances from 1.51 to 1.54 secs. By comparing the maximum of each set, 1.52 secs for 1 instance whereas it is 1.54 secs for 51 instances, we see a variability of in the order of ~ 20 ms. The same variability of ~ 20 ms is seen when comparing the minimum, 1.49 sec for 1 instance vs 1.51 for 51 instances.

This variability is likely only noise and the actual longer times could simply be due to higher system load as opposed to pipewire API. While there is an overall upward trend in the measurements, it’s both quite small. With only a 20 ms increase in end to end latency on average and 50 ms or so between absolute min and max, we can conclude there is no big impact with regards to the number of clients. For measuring more detailed impacts another approach would be needed.

All raw results are available in results/test-case-4/, including a list of means test-case-4-list-means.txt, a list of every measure test-case-4-list-results.txt and a capture of the output test-case-capture1.wav.

Test case 5: xrun behaviour under load

In this test the same setup as Test case 1 will be used. To test the xrun behavior the CPU availability for pipewire will be constrained such that it actually triggers xruns. The goal here is to evaluate how pipewire responds to that situation and whether/how it recovers properly when CPU becomes available again.

The script test-case-5-run.py does the same setup as test-case-1-run.py, it only differs by not starting a third moving stream but instead by using stress-ng to simulate a high CPU load in order to generate xruns. The number of xruns will be measured with pw-top and a profil is generated by pw-profiler.

The script test-case-capture.py can be used to record sound generated by the board sound cards during the test (i.e. under high system load) to evaluate how the sound is affected.

Results

stress-ng has completed its run in 92.91s (1 min, 32.91 secs) (for a requested timeout of 1 min). This run was performed with --class cpu --all 20, that means 20 instances of all tests from the cpu class were running simultaneously. The aim was to trigger xruns to check how pipewire responds and how it recovers properly after the load. Before starting to stress the CPU, pw-top reported 0 xrun all for objects which is expected, while after the stressful period, 548 xruns are reported for the node associated to the built-in sound card and 0 xrun for the node associated to the USB sound card.

In the captured streams, test-case-capture1.wav from the USB sound card and test-case-capture2.wav from the built-in sound card (both available in results/test-case-5/), we begin to hear the effects of the load at 0:34 until 1:40.

For the sound coming from the USB sound card (so without any xrun counted), we hear crackling sound during 1min00 which corresponds to the duration of the stress run. The effect is much more pronounced on the sound coming from the built-in sound card as there was mostly silence with spaced crackling for 1min06 during this time. Interestingly, the USB sound card is less affected than the built-in sound card.

Because the sound disturbances are expected and desired in this test since, we intentionally generated an intense CPU usage. The main point here is how pipewire recovers after the stress. From the USB sound card point of view, pipewire recovers almost immediately as the disturbances last for 1min00. Regarding the built-in sound card, it’s a little longer with 1min06 (+6secs) to fully recover.

Results summary

Adding or removing an audio stream in a running output neither impacts that output or any other system outputs. See Test case 1: Results for more details.
Reconfigurating of audio capture only impacts the reconfigured stream, as we have to make the input device idle to force pipewire to apply the changes. The direct or unrelated outputs are not affected themselves by the reconfiguration. See Test case 2: Results for more details.
Starting and/or stopping inputs independently does not cause disruptions in the outputs, but mixing streams from different sources gives sound arfects during our test. See Test case 3: Results for more details.
The results show a variability between the different sets of tests which is quite small and which is probably only noise and resulting from the difference of system load levels as opposed to pipewire API. There is no big impact with regards to the number of clients. For measuring more detailed impacts another approach would be needed. See Test case 4: Results for more details.
Interestingly, both sound cards (USB and built-in) are not equally affected by a CPU limitation. Pipewire manages to keep sound (although of poor quality because chopped) on the USB sound card but not for the built-in sound card which mostly emits only silence. Pipewire recovers almost instantly in the first case, and with a delay of a few seconds in the other case. See Test case 5: Results for more details.

Evaluation of PipeWire with Apertis v2024

A first evaluation of PipeWire 0.3.59 on R-Car was done with Apertis v2023. Because the results present some artefacts, the same evaluation has been repeated with Apertis v2024pre and PipeWire 0.3.84.

Please refer to Set-up the R-Car board for the set-up of the R-Car board. Here, the only difference is the Apertis image flashed on the board, we use the daily (20231110.0016) FixedFunction (APT-based) image of Apertis v2024pre.

$ sudo bmaptool copy https://images.apertis.org/daily/v2024pre/20231110.0016/arm64/fixedfunction/apertis_v2024pre-fixedfunction-arm64-uboot_20231110.0016.img.gz /dev/mmcblk0

As reminder, this is the setup used: A USB sound card is connected to the board while the other two are connected to the laptop for capture. This is a picture of our setup:

Setup for testing

Speakers (or earphones in our example) can be plugged in the LINE OUT of board’s sound cards in order to confirm it works as expected:

# Identify playback device
$ aplay -l

# Test onboard sound card output
speaker-test -Dplughw:CARD=rcarsound -c2

# Test USB sound card output
speaker-test -Dplughw:CARD=ICUSBAUDIO7D -c2

Before running tests, we need to substitute earphones with jack cables plugged on the other side in the LINE IN of the two other capturing sound cards:

Setup for PipeWire evaluation

Test case 1: Switching audio streams onto running devices

Test case 1

Results

During the test, two wav files are generated test-case-capture1.wav and test-case-capture2.wav (both are available in results_v2024/test-case-1/). The first one captures the sound coming from the board USB sound card while the second capture the sound coming from the board built-in sound card.

At the beginning of test-case-capture1.wav, we can hear the tone test300.wav while for test-case-capture2.wav we hear a mixture of both tone test300.wav and test800.wav. At 0:10, the switch happens, and the test800.wav tone is transferred to test-case-capture1.wav. A switch of test800.wav happens every 5 secs until the end of the captured stream.

Like for the previous evaluation in Apertis v2023, when a switch happens, we don’t hear any silence or any artefact like cracking. The sound coming from the different outputs remains smooth when a new stream is added or removed. There is no regression and/or improvement compared to the previous evaluation.

Test case 2: Audio input reconfiguration

Test case 2

For this test case, we use the previous setup as basis then we plug a microphone (in our example, we use the microphone of the white earset) into the MIC INPUT of the board.

Setup for PipeWire evaluation with test case 2

NOTE: The built-in sound card is affected by a hardward limitation preventing to independently change the rate of input and output. Since the aim of this test is to evaluate pipewire and not the built-in sound card, the audio input reconfiguration is done on the USB sound card and not on the built-in sound card. In other words, we capture sound with the USB card LINE In and redirect it to the LINE Out of the same card (which differs from the diagram and picture above). The reason for this apparently limitation is that the onboard R-Car card only has one reference clock available for the input/outputs. Which means both input and output rates need to be compatible with the current clock rate, hence the requirement to reconfigure both whenever changing that clock rate. Other hardware (like the USB card in this case) often has seperate clocks for different inputs and outputs and hence is capable of switching rate independently.

Results

Like for the previous evaluation , it’s not possible to reconfigure a running device. That means, to reconfigure the sample rate input, we have to force the device input to be idle.

The script for this test was adjusted for the new pipewire behavior. In the previous evaluation, pipewire assigned each JACK client to a unique node group based on their PID (i.e. “jack-PID”). With recent versions of pipewire, all JACK clients are assigned to the same node group called “group.dsp.0” (see commit bafa890a). The aim of this change was to schedule all JACK clients together, but this change affects our test since we want to reconfigure only one node without touching the other ones. However, it is possible to force our different JACK clients to be in different groups allowing us to restore the previous pipewire behavior and to reconfigure only specific JACK nodes. The node group is defined within the PIPEWIRE_PROPS variable when the client is started:

# Start a first instance of sndfile-jackplay which is assigned to the "group_A" node group
PIPEWIRE_PROPS='{ node.group = group_A }' pw-jack sndfile-jackplay test300.wav
# Start a second instance of sndfile-jackplay which is assigned to the "group_B" node group
PIPEWIRE_PROPS='{ node.group = group_B }' pw-jack sndfile-jackplay test300.wav

During the test, two wav files are generated test-case-capture1.wav and test-case-capture2.wav (both are available in results_v2024/test-case-2/). The first one captures the sound coming from the board built-in sound card while the second capture the sound coming from the board USB sound card.

In the file test-case-capture2.wav, the stream starts with the test300.wav tone, then at 0:10 we start hearing the test800.wav tone which is captured at 44100 Hz and mixed to the first tone. We stop hearing the test800.wav tone at 0:20 while the test300.wav tone is still audible. This pause of the test800.wav tone continues for 5 secs allowing the reconfigure the capture input rate at 48000 Hz by waiting the input device to be idle. Then, at 0:25 we hear again a mix of test300.wav and test800.wav captured at 48000 Hz for 10 secs. At 0:35, we stop hearing the test800.wav tone because of the reconfiguration at 44100 Hz, which restart 5 secs later and so. The sample rate reconfiguration is checked during the test in the FORMAT column of the pw-top output.

There is no change compared to the previous evaluation. The reconfiguration of audio capture only impacts the captured stream and not the direct output since we have to stop the stream to make pipewire apply the changes. Morevoer, the unrelated output is not affected.

Test case 3: Capturing multiple inputs to multiple outputs

Test case 3

This test case requires the setup used in the test case 2 to which we add a new source sound (here we use a music player) plugged into the LINE IN of the board USB sound card.

Setup for PipeWire evaluation with test case 3

Results

Both output were recorded in test-case-capture1.wav and test-case-capture2.wav, respectively from the USB and built-in sound cards.

In both recordings, we cannot detect any disruptions in the outputs when starting and/or stopping the differents inputs. In both recordings, we don’t hear sound artefacts like in the previous evaluation. Streams are smooth without hatching sound or loud cracking noise. We can hear few light cracking, but this probably only due to the microphone saturation because the microphone is not the best quality.

Test case 4: Benchmarking load caused by a bigger number of clients

The goal here is to benchmark if more audio clients have an impact on initial time to playing back the audio.

Test case 4

Results

To eliminate random variability, each scenario is performed ten times then a mean is computed to provide the most accurate overview of pipewire capabilities.

Test case 4

As we can see, the time starts to increase when we already have 30 stream running. Indeed, the means is ~ 1.48 secs for 1, 11 and 21 running streams. From here, the time will increase to 1.492 secs for 31 streams (+ 12ms), to 1.507 secs for 41 streams (+ 27ms) and it will reach 1.511 secs for 51 streams (+ 31ms).

To summarize, we can see a variability in the order of ~ 30 ms for each set of tests, for example with 1 instance the time ranges from 1.46 to 1.49 secs, and the same for 51 instances from 1.50 to 1.53 secs. By comparing the maximum of each set, 1.49 secs for 1 instance whereas it is 1.53 secs for 51 instances, we see a variability of in the order of ~ 40 ms. The same variability of ~ 20 ms is seen when comparing the minimum, 1.46 sec for 1 instance vs 1.50 for 51 instances.

This variability is likely only noise and the actual longer times could simply be due to higher system load as opposed to pipewire API. While there is an overall upward trend in the measurements, it’s both quite small. With only a 40 ms increase in end to end latency on average and 70 ms or so between absolute min and max, we can conclude there is no big impact with regards to the number of clients. For measuring more detailed impacts another approach would be needed.

All raw results are available in results_v2024/test-case-4/, including a list of means test-case-4-list-means.txt, a list of every measure test-case-4-list-results.txt and a capture of the output test-case-capture1.wav.

To facilitate comparison with the previous evaluation, the two graphs from both evaluation (Apertis v2023 & Pipewire 0.3.59 versus Apertis v2024 & Pipewire 0.3.84) are placed side by side.

Test case 4 with Apertis v2023 and Pipewire 0.3.59 \ Test case 4 with Apertis v2024 and Pipewire 0.3.84 \begin{figure}[!h] \captionsetup[subfigure]{labelformat=empty} \begin{subfigure}[t]{0.6\textwidth} \caption{Apertis v2023 & Pipewire 0.3.59} \end{subfigure} \hfill \begin{subfigure}[t]{0.4\textwidth} \caption{Apertis v2024 & Pipewire 0.3.84} \end{subfigure} \end{figure}

Between both graphs, we see the same increasing trend and the same order of magnitude for latency range ~ 30/40 ms. The maximum is same in both evaluation (1.54 with pipwire 0.3.59 and 1.52 with pipewire 0.3.84), and interestingly there seems to be an improvement in the lowest latency observed. In our previous evaluation, the lowest time to play the file was arround 1.49 secs whereas here we have several observations at 1.45 secs and 1.46 secs. To conclude, the new evaluation shows a small improvements regarding the lowest latency achieved in this test.

Test case 5: xrun behaviour under load

Results

stress-ng has completed its run in 120.55s (2 min, 0.55 secs) (for a requested timeout of 1 min). This run was performed with --class cpu --all 20, that means 20 instances of all tests from the cpu class were running simultaneously. The aim was to trigger xruns to check how pipewire responds and how it recovers properly after the load. Before starting to stress the CPU, pw-top reported few xruns for the different pipewire objects, while after the stressful period, we see a small increase of xruns accross our nodes:

+3 xruns for alsa_output.usb-0d8c_USB_Sound_Device-00.analog-stereo
+0 xrun for alsa_output.platform-sound.stereo-fallback
+1 xrun for my-sink-A
+2 xruns for my-sink-B
+3 xruns for jackplay
+2 xruns for jackplay (our second instance of sndfile-jackplay)

This is relatively small compared to the +548 xruns we had in the previous evaluation. Interestingly, in the previous evaluation, increasing the system load had the effect of generating xruns in only one node whereas here xruns are distributed accross nodes.

In the captured streams, test-case-capture1.wav from the USB sound card and test-case-capture2.wav from the built-in sound card (both available in results_v2024/test-case-5/), we begin to hear the effects of the load at 0:13 until 2:13. At the beginning of load increase (i.e. 0:13), in both captures we hear several cracking sounds, then the streams become smooth with rares and isolated cracking until end of system load. At the end of system load (i.e. 2:13), we hear again several cracking sounds for a few secondes, then streams become smooth again. The streams are mainly affected only at the beginning and at the end of the stressful period while during this period pipewire is able to maintain a constant stream with a certain quality.

Compared to the previous evaluation, the quality of the stream provided by pipewire is greatly improved during the simulation of a high system load. In the previous evaluation, a stream was stopped whereas here the stream continues with only small artefacts.

Results summary (comparing results to the previous evaluation)

Test case 1: Compared to the previous evaluation, there is no change in the results: streams are smoooth and no artefacts are audible when the switch happens.
Test case 2: After adjusting the test script to the new pipewire behavior, results are similar to the previous evaluation: only the reconfigured stream is affected as we have to make the input device idle to force pipewire to apply the changes. The direct or unrelated outputs are not affected themselves by the reconfiguration.
Test case 3: In this test, results are improved as we don’t hear sound artefacts due to mixing streams like in the previous evaluation. Streams are smooth without hatching sound or loud cracking noise. We can hear few light cracking, but this probably only due to the microphone saturation because the microphone is not the best quality.
Test case 4: Results are similar to the previous evaluation in terms of variability, latency ranges observed, etc. Tests seem to show a small improvement regarding the lowest time required to play the file.
Test case 5: Pipewire’s streams are less affected by a high system load than in our previous evaluation. Now, pipewire is able to maintain smooth streams with only small cracking sounds whereas in the previous evaluation streams were strongly affected to the point of stopping a stream.

From a subjective point of view, the sound quality was improved between both version. While doing these tests, the sound had less crackling and was smoother overall.

Search Apertis

Evaluation of PipeWire with JACK clients

Table of Contents:

Set-up the R-Car board

How to flash Apertis on SD card

How to flash Apertis on eMMC

How to check if sound works

Setup for pipewire evaluation

Test case 1: Switching audio streams onto running devices

Results

Test case 2: Audio input reconfiguration

Results

Test case 3: Capturing multiple inputs to multiple outputs

Results

Test case 4: Benchmarking load caused by a bigger number of clients

Results

Test case 5: xrun behaviour under load

Results

Results summary

Evaluation of PipeWire with Apertis v2024

Test case 1: Switching audio streams onto running devices

Results

Test case 2: Audio input reconfiguration

Results

Test case 3: Capturing multiple inputs to multiple outputs

Results

Test case 4: Benchmarking load caused by a bigger number of clients

Results

Test case 5: xrun behaviour under load

Results

Results summary (comparing results to the previous evaluation)

References