Real time Linux
- 1. Title: RT Linux Seyed Navid Ashrafi Telecommunication Networks Group Technische Universität Berlin Summer Semester 2018 35-minute presentation NES SS2018 1
- 2. Over view >> Introduction to RTOS - Definitions and history - Usages & requirements NES SS2018 2 >> PREEMPT_RT patch - Overview - Features >> Latency measurement test >> Conclusion >> Different approaches - Xenomai - PREEMPT_RT - Cons & pros
- 3. What is real time? • It is about fast operation time • It is about performance • Its about DETERMINISM - You need to execute an operation in a specific time frame and you need a guarantee for that. - As fast as specified, not as fast as possible - RT-Linux: Deterministic timing behavior in Linux. - Depends on the system NES SS2018 3
- 4. Usage and requirements of RT-Linux • Usages: - Industrial applications - Multimedia systems - Aerospace system - Automative - Financial services • Requirements: - Deterministic timing behavior - Preemption - priority inheritance NES SS2018 4
- 5. Soft Realtime?? • It is in contrast with deterministic behavior • We concentrate on 100% and 95% hard RT. • 100% hard RT: Realtime requirements should be met 100% otherwise it will lead to an error condition (industry) • 95% hard RT: Realtime requirements should be met 95% (data collection) NES SS2018 5
- 6. Critical section • Concurrent access to a shared resource can lead to unexpected or error condition • There are over 10,000 individual critical section in a typical Linux 2.6 kernel • In the non-preemptible design of Linux 2.6 kernel, any critical section can block preemption • In the Preemptible design the effort was to reduce the total size and number of non-preemptible critical sections. NES SS2018 6
- 7. Approaches Dual-kernel / Co-kernel Single-kernel / In-kernel NES SS2018 7
- 8. Dual-kernel approach • First approaches which are still existing. • Linux is running on top of a microkernel as a low priority real time task. • The microkernel does the real time stuff. • You need to maintain the microkernel and a hardware abstraction layer on Linux to make it run on the microkernel: - Big, time consuming effort. - Reason why most of these approaches are always one step behind Linux fast development NES SS2018 8
- 9. Single-kernel approach • A way to make Linux it self real-time capable. • You would have to touch every single file in the kernel. • No microkernel or hardware abstraction layer (you can just work with Linux standard tools). NES SS2018 9
- 10. Xenomai Xenomai 1.0 • Announced in 2001 – as portability framework for RTOS applications • Required a real-time basis • Development of ADEOS layer for Linux and RTAI • Merged with RTAI Xenomai 2.0 • Departed from RTAI in 2005 – incompatible design goals • Evolved ADEOS to I-pipe layer (also used by RTAI) • Ported to 6 architectures Xenomai 3.0 • Released in 2015 after 5 years of development • Rework of in-kernel core (now POSIX-centric) • Support for native Linux NES SS2018 10
- 11. Xenomai NES SS2018 11 microkernel preempts Linux
- 13. Migration between schedulers • creating realtime task in application, over a normal Linux task • 2 threads for one task which share a lot of states • only one of them can run at the same time • Migration to RT: suspend Linux task, resume Cobalt thread • Migration to Linux (on syscall, fault/trap, signal): suspend Cobalt thread, resume Linux task • Every things that we do not handle in Xenomai will be directed to Linux NES SS2018 13
- 14. Hardware abstraction layer I-pipe Core NES SS2018 14
- 15. Issues of dual-kernel approach • Special API • Special tools and libraries • Microkernel needs to be ported for new HW and new Linux versions • Bad scaling on big platforms • Various number of not palatable patches • Changes to critical subsystems regularly cause regressions • Porting efforts consume core developer resources -Most work done by Philippe and Gilles so far -Time would be better spent on feature improvements... NES SS2018 15
- 16. Xenomai application • Machine control systems, PLCs • Printing machines (manroland) • Printers / copying machines • Network switches (e.g. Ruggedcom) • Magnetic resonance tomographs (Siemens Healthcare) • OROCOS (OSS robotics framework) • Robotic research projects • … (many, many incognito applications) NES SS2018 16
- 17. Preempt_RT • Founded 14 years ago by: Thomas Gleixner & Ingo Molnar • Huge community • In-kernel approach • Main idea: only code that, absolutely must be non-preemptible, is allowed to be non-preemptible • Most of the features already made it into the “mainline” • Interrupt handlers run in a kernel thread which has two advantages: - The kernel thread is interruptible - It shows up in a process list with a PID • First mainline integration 2006 NES SS2018 17
- 18. Preempt_RT Goals • 100% Preemptible kernel - not possible yet, but let’s try regardless - Removal of interrupts and preemption disabling - Fast “worst case” time. • Quick reaction time - Bring latencies down to a minimum • Minimize non-Preemptible kernel code - at the meantime Minimize code changing amount NES SS2018 18
- 19. Evolution of RT-Linux NES SS2018 19
- 21. Fully Preemptible kernel (the RT patch) • Preemption almost everywhere • Spinlocks turn into mutexes • No hard interrupt context • Preemptible critical section • Preemptible interrupt handlers • Preemptible “interrupt disable” code sequence • Priority inheritance • Deferred operation • Latency reduction NES SS2018 21
- 22. Mutex • Critical sections are still protected without disabling preemption • A Mutex guarantees data integrity while extending the operation of the spinlock - we have priority inheritance - All locks are assumed to be preemptible - This design eliminates the possibility that an inefficient locking implementation in a driver can introduce un-detected latency into the system. - Only a small subset of locks stay unpreemptible which are manageable. NES SS2018 22
- 23. Priority inversion • When a high priority task wants to run but it can not because a lower priority task is holding that lock • It is bad but we can not get rid of it, what we can get rid of is unbounded priority inversion NES SS2018 23
- 24. Unbounded priority inversion NES SS2018 24 A B C
- 25. Priority inheritance NES SS2018 25 A B C
- 26. local_irq_disable asmlinkage void do_entInt(unsigned long type, unsigned long vector, unsigned long la_ptr, struct pt_regs *regs) { struct pt_regs *old_regs; local_irq_disable(); switch (type) { case 0: #ifdef CONFIG_SMP NES SS2018 26 handle_ipi(regs); return; #else irq_err_count++; printk(KERN_CRIT "Interprocessor interrupt? "You must be kidding!n"); #endif break; 1 2
- 27. Preempt_disable NES SS2018 27 • Local_irq_disable younger sibling • Also does not give a hint to what does it do! • Has the exact same problems of local_irq_disable • Preempt_enable_no_resched - only should be used within preempt_disable locations
- 28. Why do we still need Co-kernel? Functional limitations of In-kernel • Emulation of RTOS scheduling behavior limited by Linux scheduler • Not all kernel+libc code paths used by In-kernel approaches are necessarily hard real-time under PREEMPT-RT • No detection of non-RT behavior of your application Performance limitations of In-kernel • Co-kernel is usually more light-weight on low-end platforms (limited caches vs. code path lengths) • PREEMPT-RT can have unwanted impact co-located non-RT workloads Xenomai adds value to Linux • it keeps the realtime stuff away from the complexities of Linux kernel • Co-kernel approach can be beneficial for low latencies and real-time application architecture NES SS2018 28
- 29. Latency measurement on xenomai & preempt_RT • Made by Jan Altenberg from Linutronix GmbH • On two ARM Cortex A9 SOC platforms • IRQ test with 10KHz frequency with the latency box • 100% CPU load with “hackbench” • They did the test in usespace and kernelspace • 12-hour duration NES SS2018 29
- 30. Hackbench • Starts n groups of 20 clients and 20 servers • Each client sends 100 messages to each server via a socket connection NES SS2018 30
- 31. Latency box NES SS2018 31
- 32. Xenomai latency userspace task NES SS2018 32
- 33. PREEMPT_RT latency userspace task NES SS2018 33
- 34. PREEMPT_RT latency userspace task (isolated CPU) NES SS2018 34
- 35. Latency: userspac- Comparison NES SS2018 35
- 36. Xenomai latency in kernelspace NES SS2018 36
- 37. PREEMPT_RT latency in kernelspace NES SS2018 37
- 38. PREEMPT_RT latency in kernelspace (isolated) NES SS2018 38
- 39. PREEMPT_RT latency in kernelspace (using FIQ) NES SS2018 39
- 40. Latency: Kernel - Comparison NES SS2018 40
- 41. Test results • Which approach to choose? - If timing is vital, choose Xenomai, it separates the realtime interrupt handling from complexities of Linux kernel • You can go with PREEMPT_RT using FIQ • Xenomai kernelspace is the best for 100% hard performance • Xenomai userspace is slower than it’s kernelspace • Tests only can help you in the field of latency which is dependent on the hardware • No guarantee for results repeatation NES SS2018 41
- 42. Conclusion • PREEMPT_RT is getting integrated in Linux mainline - Download here: https://www.kernel.org/ • PREEMPT_RT Increases Linux stability and quality • PREEMPT_RT is simple to use • Xenomai has it’s own advantages over PREEMPT_RT • Microkernels are hard to handle • For the most use cases the latency of the both approaches is almost the same • FIQ offers very fast latency but brings limitations NES SS2018 42
- 43. Thank you NES SS2018 43