IRQ's off issue

[Greg Gallagher]

On Mon, Mar 2, 2020 at 7:45 AM Jan Kiszka via Xenomai
<xenomai at xenomai.org> wrote:
>
> On 02.03.20 12:03, Bradley Valdenebro Peter (DC-AE/ESW52) via Xenomai wrote:
> > Hello Xenomai team,
> >
> > We need you help understanding and solving a possible IRQ's off issue.
> > We are running a Xenomai/Linux setup on a Zynq Z-7020 SoC (We run Linux on CPU0 and Xenomai on CPU1):
> >   - Linux version 4.4.0-xilinx (gcc version 8.3.0 (Buildroot 2019.02-00080-gc31d48e) ) #1 SMP PREEMPT
> >   - ipipe ARM patch #8
> >   - Xenomai 3.0.10
> >
> > Lately we have been experiencing that our highest priority real time Xenomai thread sort of halts for around 1ms every now and then.
> > After some investigation and tests we decided to enable ipipe trace to measure IRQs-off times. See below the output of /proc/ipipe/trace/max
> >
> > I-pipe worst-case tracing service on 4.4.0-xilinx/ipipe release #8
> > -------------------------------------------------------------
> > CPU: 0, Begin: 2944366216549 cycles, Trace Points: 2 (-10/+1), Length: 780 us
> > Calibrated minimum trace-point overhead: 0.288 us
> >
> >   +----- Hard IRQs ('|': locked)
> >   |+-- Xenomai
> >   ||+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
> >   |||                      +---------- Delay flag ('+': > 1 us, '!': > 10 us)
> >   |||                      |        +- NMI noise ('N')
> >   |||                      |        |
> >            Type    User Val.   Time    Delay  Function (Parent)
> >   | +begin   0x80000001   -12      0.414  ipipe_stall_root+0x54 (<00000000>)
> >   | #end     0x80000001   -11      0.822  ipipe_stall_root+0x8c (<00000000>)
> >   | #begin   0x80000001   -11      0.414  ipipe_test_and_stall_root+0x5c (<00000000>)
> >   | #end     0x80000001   -10      1.095  ipipe_test_and_stall_root+0x98 (<00000000>)
> >   | #begin   0x90000000    -9      0.665  __irq_svc+0x58 (arch_cpu_idle+0x0)
> >   | #begin   0x00000025    -8      2.883  __ipipe_grab_irq+0x38 (<00000000>)
> >   |#*[  558] SampleI 49    -5      2.619  xnthread_resume+0x88 (<00000000>)
> >   |#*[    0] -<?>-   -1    -3      2.052  ___xnsched_run+0xfc (<00000000>)
> >   | #end     0x00000025    -1      0.760  __ipipe_grab_irq+0x7c (<00000000>)
> >   | #end     0x90000000     0      0.530  __ipipe_fast_svc_irq_exit+0x1c (arch_cpu_idle+0x0)
> >> | #begin   0x80000000     0! 780.110  arch_cpu_idle+0x9c (<00000000>)
> > <| +end     0x80000000   780      0.570  ipipe_unstall_root+0x64 (<00000000>)
> >   | +begin   0x90000000   780      0.000  __irq_svc+0x58 (ipipe_unstall_root+0x68)
>
> Looks like the CPU was idle and received no IRQ during that time. Is
> some power management active? Is some timer misprogrammed? Or where
> should the next interrupt have from?
>
> >
> > We have trouble understanding the output but we can see a max length of 780us on CPU0. We find this value extremely high.
> > With our current requirements anything beyond 10us is not acceptable.
>
> 780 us is definitely off on that target, but 10 us will likely be too
> ambitious as well. Maybe, maybe, with well configured strict core
> isolation, practically no Linux load on the RT core and your critical RT
> code path always in cache... But I would consider that highly risky,
> given this low-end CPU on that target SoC.
>
> Jan
>
Just to add, this chip (like most cortex-A9's) uses the PL-310 cache
controller.  This controller has high latency, in the past it has
impacted the performance of  a number of SOC's.

This link shows the an issue found on the imx6:
https://www.xenomai.org/pipermail/xenomai/2018-July/039268.html

You may run into this issue with the Zynq as well.

-Greg
> >
> > Can someone with experience with the ipipe tracer please help us understand what is going on and how can we fix it?
> >
> > Thanks in advance for your support.
> >
> > Best regards.
> > Peter Bradley
> >
> >
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>