FW: Xenomai with isolcpus and workqueue task

Lange Norbert norbert.lange at andritz.com
Mon Jul 13 10:41:16 CEST 2020

(fwd to list)

> -----Original Message-----
> From: Lange Norbert
> Sent: Montag, 13. Juli 2020 10:34
> To: Alexander Frolov <frolov at nicevt.ru>
> Subject: RE: Xenomai with isolcpus and workqueue task
> > -----Original Message-----
> > From: Xenomai <xenomai-bounces at xenomai.org> On Behalf Of Alexander
> > Frolov via Xenomai
> > Sent: Samstag, 11. Juli 2020 16:26
> > To: xenomai at xenomai.org
> > Subject: Xenomai with isolcpus and workqueue task
> >
> >
> >
> > Hi all!
> >
> > I am using Xenomai 3.1 with 4.19.124 I-pipe patchon a smp motherboard.
> > For my RT task I allocate few CPU cores with isolcpus option. However,
> > large latency spikes are noticed due to igb watchdog activities (I am
> > using common igb driver, not rt_igb).
> >
> > Looking into igb sources, it was understood that workqueue is used for
> > some tasks (afaiu, it is used to link status
> > monitoring)
> >
> > from igb_main.c
> > ...
> >    INIT_WORK(&adapter->reset_task, igb_reset_task);
> >    INIT_WORK(&adapter->watchdog_task, igb_watchdog_task); ...
> >
> > The Linux kernel scheduler runs this igb activities on isolated CPUs
> > disregarding isolcpus option, ruining real-time system behavior.
> isolcpus does not mean the CPUs aren't used, it means they are excluded
> from the normal CPU scheduler. No process will automatically be moved
> from/to isolated CPUs, but you still need to make sure to free them of any
> tasks.
> Irq-handlers still run anywhere, and processes still can allow those CPUs to
> be used.
> > So the question, is it a correct way to use normal igb on Xenomai at
> > all or it is not recommended? What can be done to prohibit Linux
> > scheduler to allocate those tasks on isolated cores?
> I use the normal igb and rt_igb concurrently, I doubt it is recommended but
> possible ;)
> You should add irqaffinity=0 to the cmdline (CPU0 is apparently always used
> for irqs), then check 'cat /proc/irq/*/smp_affinity'. This keeps the other
> CPUs free from linux IRQs.
> You can use some measures to bind Linux tasks to CPU0 aswell. One of:
> -   isolcpus (sets default affinity mask aswell)
> -   set affinity early (like in Ramdisk)
> -   Use cgroups (cset-shield)
> Only cgroups actually prohibit processes ignoring your defaults and using
> other CPUs, I did not get around playing with this, and just use isolcpus.
> But the most important part is to dont run RT on cores dealing with Linux
> interrupts, some handlers/drivers don’t expects being preempted, had the
> MMC driver bail because of a timeout.
> I haven’t solved moving the rtnet-stack, rtnet-rpc off CPU0, and the rt_igb
> IRQs will use all CPUs.
> Norbert

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806

Thank You

More information about the Xenomai mailing list