Platform

Real-Time Linux on a Raspberry Pi 5: What 24 Hours of Testing Showed

EmberNet white paper cover for real-time latency measured on real hardware

If you run a control loop on Linux, the number you care about is not how fast the kernel is on a good day. It is how late the kernel can be on its worst one. So we asked a plain question of our Raspberry Pi 5 running EmberRTOS: what is the worst it does in a full day?

The answer was 23 microseconds. Here is how we got it, why it took a day, and what it does not tell you.

What we measured, and how

We use a standard real-time latency test. It wakes a high-priority thread at a fixed interval and records how late the thread actually ran. The worst of those delays is the figure that matters, because a control loop does not fail on its average.

The board was a Raspberry Pi 5 running the EmberRTOS real-time kernel. We isolated three of the four cores for real-time work and left the first core for everything else: interrupts, the network, and the load. One top-priority thread ran on each isolated core, at intervals of 200, 700, and 1200 microseconds. The load was a mix of scheduler stress, two CPU-burn loops, and a memory streamer in RAM, all kept on the first core so the shared memory controller was working the whole time.

23 microseconds over 24 hours

The run lasted 24 hours, 86,423 seconds, and finished clean. Across the three threads it took about 627 million samples. The worst wake-up on each thread was 17, 20, and 23 microseconds, so the worst case for the day was 23 microseconds. The average was 1 microsecond.

The board started at 30 degrees Celsius and peaked near 60 under load. We saw no throttling.

Why we ran it for a day

We ran the same board and image for five minutes first. The worst wake-up was 12 microseconds. Over a day it was 23. The tail grew, and that is exactly what a real-time test should show you: rare delays live in the hours, not the minutes. A five minute result is a floor. A day is the first number we would put in front of an engineer, and a longer soak is still ahead of us.

What isolating cores does

Every result we have comes from the same pattern. A real-time kernel lets the high-priority thread run in place of nearly everything the kernel is doing. A few cores handle the housekeeping. The rest are kept for the real-time threads. Deep sleep states are limited, so waking a core takes a steady amount of time, and the CPU governor is set to performance.

The part we learned the hard way is that an isolated core only helps if nothing else is on it. On one deployed industrial PC we found unrelated work sitting on cores we had isolated, including the kernel workers that encrypt the node's network. We now keep everything else off those cores from code at boot, and we check it after every restart.

What we did not measure

This list is the reason to trust the rest of the post.

  • The numbers above come from an earlier kernel build. The current EmberRTOS release ships a newer real-time kernel, and we have not yet measured it on real hardware.
  • The load ran on the housekeeping core. That shows isolation working, but it is not the worst case. A memory-heavy load across every core would lean on the memory bus harder, and we have not run it.
  • Only one board has a 24 hour run. A four-core Intel machine has a five minute run, and no Raspberry Pi 4 has been measured at all.
  • The test measures wake-up latency. It does not measure your application's cycle time or the jitter of a fieldbus or network. Check your own loop on your own system.
  • No virtual machine figure is used here, because a virtual machine's worst case is set by its host.

Read the whole paper, and check it yourself

The full method, every thread, the Intel and industrial PC results, and the longer version of that list are in the white paper: Real-Time Latency, Measured on Real Hardware. The EmberRTOS manual shows how to check real-time on your own box, and the images, including the Raspberry Pi 4 and 5 images, are on the EmberRTOS download page. To see where it fits, start with EmberRTOS on embernet.ai.

If your numbers differ from ours, we want to hear about it.