Experience shows that dual kernels – and Xenomai in particular – have been consistently successful at delivering stringent real-time support to applications in the embedded space so far.
In order to keep pace with mainline Linux development, those systems need Linux to exhibit a simple, properly integrated, well-documented interface for hosting a co-kernel, which could be maintained with common kernel development knowledge, at a fraction of the engineering and maintenance cost native preemption requires.
The I-pipe – which stands for interrupt pipeline – is the option Xenomai’s Cobalt core currently uses for interfacing with the Linux kernel. Although the I-pipe has been successfully maintained for more than fifteen years over up to seven architectures with only very limited manpower, the implementation nevertheless shows its age and shortcomings:
- the incompatibility with power management strategies, like CPU frequency throttling (which extends to the Cobalt implementation too), or active switching of the CPU to idle state.
- the incompatibility with context tracking, which prevents from enabling full NO_HZ mode when interrupt pipelining is on.
- a redundant API for dealing with high-priority interrupts, which uselessly duplicates portions of the standard genirq feature set, but also increases the amount of code to be patched into the irqchip drivers for supporting interrupt pipelining.
- an interface for handing the timer hardware over the co-kernel which is too intrusive, and does not leverage the clockevent abstraction as much as it could do.
- an interface for exposing the current clocksource to user-space applications (esp. ARM) which uselessly duplicates vDSO-based features available with current Linux ports.
- the incompatibility with the standard IRQ latency tracer, which makes the implementation uselessly intrusive in the assembly code where some related tracepoints are defined. This also requires us to provide a dedicated tracing facility, instead of extending the existing one to cope with co-kernel contexts.
- the incompatibility with the LOCKDEP infrastructure, which has to be turned off. As a consequence, the co-kernel cannot benefit from this facility for detecting invalid locking sequences with pipeline-safe spinlocks.
- the lack of support for sharing all aspects of the task context switching code with a co-kernel, which requires the latter to provide its own implementation. Not only such a redundant code does not benefit from mainline fixes or improvements, but it also creates undesirable and bug-prone dependencies between the co-kernel implementation and other parts of the standard scheduler (e.g. FPU management code).
- the bit rotting support for legacy hardware which makes the process of rebasing the code over newer kernels uselessly difficult. For instance, support for armv4/v5 CPUs – including the tricky fast-context switch extension for dealing with VIVT caches – has neither been tested nor received any feedback whatsoever on the Xenomai mailing list for years.
- last but not least, the lack of proper documentation about the implementation, which is aggravated by the bulky and hairy organization of the existing code, which all turns a simple idea into a convoluted and opaque implementation.
The interrupt pipelining scheme proved to be a straightforward and efficient way of introducing a high-priority execution stage, for running out of band co-kernel activities within Linux. But to deliver on maintainability, it should be implemented as a native feature of the kernel, fitting in the existing interfaces, not sideways. Lessons learned.
With this in mind, I started laying the groundwork for a different implementation of interrupt pipelining called Dovetail, addressing the known issues with the I-pipe. As its name may suggest, the purpose of Dovetail is to introduce the logic required for joining together Linux and a specialized, high-priority co-kernel into the former.
Along with experimenting with new ideas in Dovetail about the best interface between Linux and co-kernels, I’m toying with a compact, lean and mean real-time core for ARM and ARM64 SoCs, derived from Xenomai’s Cobalt which I’m using to assess how good (or terrible) those ideas may be. The API support is purposedly limited to POSIX.
At some point, this co-kernel workhorse called Steely should showcase the integration effort of a modern dual kernel technology with mainline Linux, with good SMP scalability over a large number of CPUs and support for power management. It should provide a sufficient set of real-time POSIX services to applications, delivering dependable low latency performances out of the box, regardless of other components running in the Linux system.
#1 Have fun hacking.
#2 Upstream a lightweight dual kernel technology to the greater community so we can address specific but critical use cases at a small cost.
#3 Ensure that the best ideas brought by Dovetail and Steely which have survived a reality check, can help in improving Xenomai.
Both Dovetail and Steely are maintained in the same kernel tree at:
Philippe Gerum <firstname.lastname@example.org> December 17, 2017