[Xenomai] xenomomai-3 kernel freeze on RPI2

Henning Schild henning.schild at siemens.com
Thu Feb 2 10:09:45 CET 2017


Am Wed, 1 Feb 2017 13:42:07 -0500
schrieb John Gwynne <john.gwynne at matrixresearch.com>:

> I've been watching for a while for a solution, or even conformation,
> of a potential kernel freeze problem on the Raspberry PI 2/3 platform
> with the Xenomai patches.  Having not seen anything that appears
> related, I'm posting this description of what I've experienced.
> 
> I'm interesting in advice on how to solve this problem as well as how
> to best debug the issue.  I've enabled 'Detect Hard and Soft Lockups'
> and 'Panic on Soft Lockups', but have not been successful in
> recovering from the hung state (other than by a power cycle) nor
> retrieving information related to the cause.

Did you capture the console output via serial? That might give a clue
especially with increased loglevels and maybe additional printks on the
"new" or "suspicous" paths.

Your tests look like they put stress on the SD-card and CPU. Maybe try
stressing just the SD-card interrupt-path. (i.e. multiple dd
to /dev/null) Or just the CPU, or another interrupt.

Further debugging techniques would be JTAG and tracing to nvram. They
depend on your hardware and i am not sure they can be used on the
Raspberry Pi.

regards,
Henning

> The details are below.
> 
> Thanks,
> john
> 
> ()  Problem:
> Under heavy load, a kernel compiled with the Xenomai patches 
> ('CONFIG_XENOMAI=y' in .config) will freeze without warnings or error 
> messages.  The same kernel version compiled with xenomai patches off
> ('# CONFIG_XENOMAI is not set' in .config) does not experience such
> problems.
> 
> () Hardware:
> Tested on several Raspberry 2 and 3 boards with various sdcards and 
> power supplies.  Using the default system boot configuration and
> system clock (i.e. not overclocked).
> 
> () Software:
> kernel 4.1.21 from https://github.com/raspberrypi/linux/tree/rpi-4.1.y
> Mathieu Rondonneau RPI2 patch from this list, Apr 16 2016
> ipipe-core-4.1.18-arm-8.patch
> xenomai-3-3.0.3
> note: I have tested, with similar results, many other kernels and as
> far back as kernel 3.18 & xenomai-3-3.0.1
> 
> () Conditions to reproduce kernel freeze:
> Boot xenomai patched kernel
> Launch a kernel build process and unzip process.  I commonly do this 
> through 'screen' as follows:
> (I'm leaving out obvious setup steps such as 'mkdir <foobar>', 'make 
> defconfig', etc)
> []# screen
> []# cd linux-rpi-3.18.y/
> []# for i in `seq 1 5`; do make -j4 bzImage; make clean; done
> []# <ctl-a> c  (new screen & shell)
> []# cd tmp
> []# for i in `seq 1 30`; do unzip ../linux-rpi-3.18.y.zip; rm -rf 
> linux-rpi-3.18.y; done
> []# <ctl-a> c  (new screen & shell)
> []# top
> 
> With the 'CONFIG_XENOMAI=y' kernel, this will typically create a
> kernel freeze in 3 to 20 minutes.
> 
> Note that this load does not seem to be enough to use swap space and
> I have not launched any Xemonai threads.
> 
> () Comments
> I have tested this level of load for 18+ hours using the same kernel 
> compiled with  '# CONFIG_XENOMAI is not set' in the configuration
> file. There is no failure in this case.
> 
> The basic .config file differences are shown below. I can make
> available the actual configuration files if needed.
> 
> < '# CONFIG_XENOMAI is not set' .config file
>  > '# CONFIG_XENOMAI=y' .config file  
> 
> < CONFIG_PREEMPT_RCU=y
>  > CONFIG_TREE_RCU=y
>  > CONFIG_OPTPROBES=y  
> < CONFIG_HAVE_CONTEXT_TRACKING=y
> < CONFIG_UNINLINE_SPIN_UNLOCK=y
>  > CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
>  > CONFIG_INLINE_READ_UNLOCK=y
>  > CONFIG_INLINE_READ_UNLOCK_IRQ=y
>  > CONFIG_INLINE_WRITE_UNLOCK=y
>  > CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
>  > CONFIG_XENOMAI=y
>  > CONFIG_XENO_OPT_STATS=y
>  > CONFIG_XENO_OPT_RR_QUANTUM=1000
>  > CONFIG_XENO_OPT_AUTOTUNE=y
>  > CONFIG_XENO_OPT_TIMER_LIST=y
>  > CONFIG_XENO_OPT_HOSTRT=y
>  > CONFIG_XENO_OPT_VFILE=y
>  > CONFIG_XENO_OPT_REGISTRY_NRSLOTS=512
>  > CONFIG_XENO_OPT_SYS_HEAPSZ=1024
>  > CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
>  > CONFIG_XENO_OPT_SHARED_HEAPSZ=128
>  > CONFIG_XENO_OPT_NRTIMERS=128
>  > CONFIG_XENO_OPT_TIMING_SCHEDLAT=0
>  > CONFIG_XENO_OPT_TIMING_KSCHEDLAT=0
>  > CONFIG_XENO_OPT_TIMING_IRQLAT=0
>  > CONFIG_XENO_OPT_RTDM_COMPAT_DEVNODE=y
>  > CONFIG_XENO_DRIVERS_AUTOTUNE=y
>  > CONFIG_XENO_DRIVERS_TIMERBENCH=y
>  > CONFIG_XENO_DRIVERS_SWITCHTEST=y
>  > CONFIG_XENO_ARCH_UNLOCKED_SWITCH=y
>  > CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH=y
>  > CONFIG_IPIPE_WANT_ACTIVE_MM=y
>  > CONFIG_XENO_ARCH_WANT_TIP=y
>  > CONFIG_XENO_ARCH_FPU=y
>  > CONFIG_XENO_ARCH_OUTOFLINE_XNLOCK=y
>  > CONFIG_XENO_VERSION_MAJOR=3
>  > CONFIG_XENO_VERSION_MINOR=0
>  > CONFIG_XENO_REVISION_LEVEL=3
>  > CONFIG_XENO_VERSION_STRING="3.0.3"
>  > CONFIG_XENO_VERSION_NAME="Groovy Cosmic Halo"
>  > CONFIG_IPIPE_ARM_KUSER_TSC=y
>  > CONFIG_NEED_KUSER_HELPERS=y
>  > CONFIG_IPIPE=y
>  > CONFIG_IPIPE_CORE=y
>  > CONFIG_IPIPE_CORE_APIREV=2
>  > CONFIG_IPIPE_WANT_APIREV_2=y
>  > CONFIG_IPIPE_TARGET_APIREV=2
>  > CONFIG_IPIPE_HAVE_HOSTRT=y  
> < CONFIG_PREEMPT=y
> < CONFIG_PREEMPT_COUNT=y
>  > CONFIG_PREEMPT_VOLUNTARY=y  
> < CONFIG_SECCOMP=y
>  > CONFIG_IPV6_SUBTREES=y
>  > CONFIG_BT_HCIUART=m
>  > CONFIG_BT_HCIUART_H4=y
>  > CONFIG_BT_HCIUART_3WIRE=y
>  > CONFIG_BT_HCIUART_BCM=y
>  > CONFIG_QCA7000=m
>  > CONFIG_GPIO_BCM_VIRT=y
>  > CONFIG_DRM=m
>  > CONFIG_DRM_KMS_HELPER=m
>  > CONFIG_DRM_KMS_FB_HELPER=y
>  > CONFIG_DRM_GEM_CMA_HELPER=y
>  > CONFIG_DRM_KMS_CMA_HELPER=y
>  > CONFIG_DRM_VC4=m
>  > CONFIG_HDMI=y
>  > CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
>  > CONFIG_SND_BCM2708_SOC_BOOMBERRY_DAC=m
>  > CONFIG_SND_BCM2708_SOC_BOOMBERRY_DIGI=m
>  > CONFIG_HID_BETOP_FF=m
>  > CONFIG_MCP3422=m  
> < CONFIG_DEBUG_PREEMPT=y
> < CONFIG_DYNAMIC_FTRACE=y
> < CONFIG_FTRACE_MCOUNT_RECORD=y
> 
> 
> 
> 
> _______________________________________________
> Xenomai mailing list
> Xenomai at xenomai.org
> https://xenomai.org/mailman/listinfo/xenomai




More information about the Xenomai mailing list