Analysis of
                                                       interrupt latencies
                                                          in a real-time
             Università degli Studi di Trento                 kernel

   Facoltà di Scienze Matematiche Fisiche e Naturali
                                                       Real-time
                                                       operating systems

                                                       Real-time Linux

                                                       CFS
     Analysis of interrupt latencies in a              Recap

              real-time kernel



                   March 26, 2008


Laureando: Gabriele Modena (Mat. 108742)
Relatore: dott. Luigi Palopoli
Correlatore: dott. Luca Abeni
Analysis of
Real-Time Operating Systems (RTOS)                        interrupt latencies
                                                             in a real-time
                                                                 kernel



                                                          Real-time
                                                          operating systems

                                                          Real-time Linux

                                                          CFS

                                                          Recap




Ensure that an operation completes within a fixed amount
of time (deadline) =⇒ predictability.
When dealing with device drivers this implies
    managing race conditions;
    fulfill temporal constraints.
This thesis focuses on shared mixed resources.
Analysis of
Latency                                                    interrupt latencies
                                                              in a real-time
                                                                  kernel



                                                           Real-time
Latency is the measure of the difference between the       operating systems

theoretical (algorithm) and actual (implementation)        Real-time Linux

schedule.                                                  CFS

                                                           Recap
In a RTOS latency:
    should be bounded;
    the bound should not be too high.
The goal of a real-time kernel is to minimize latency by
using algorithms and technology that properly
implements them:

                  preemptible kernel.
Analysis of
Linux as an RTOS                                    interrupt latencies
                                                       in a real-time
                                                           kernel



                                                    Real-time
                                                    operating systems
Linux vanilla:
                                                    Real-time Linux
    since 2.6 preemptible kernel;                   CFS

The rt-preempt patch:                               Recap


    developed by Ingo Molnar;
    makes Linux hard real-time;
    uses the new (2.6.21) high resolution timers;
    preemptible critical sections;
    real-time mutexes;
    threaded interrput handlers.
Analysis of
Benchmarking rt-preempt                                   interrupt latencies
                                                             in a real-time
                                                                 kernel



                                                          Real-time
                                                          operating systems

                                                          Real-time Linux

                                                          CFS

                                                          Recap



Two goals:
    test the actual real-time performance;
    test the performance of I/O scheduling.
We look for a trade-off between latency and throughput.
Analysis of
Cyclictest                                                 interrupt latencies
                                                              in a real-time
                                                                  kernel



                                                           Real-time
                                                           operating systems

                                                           Real-time Linux

                                                           CFS
    high resolution test program written by Thomas         Recap
    Gleixner;
    measure latency;
    schedules periodic threads;
When it comes to real-time applications rt-preempt gives
the expected performance.
Analysis of
Vanilla vs. rt-preempt (1)                                                                                                  interrupt latencies
                                                                                                                               in a real-time
                                                                                                                                   kernel
 Probability of the latency (num. of samples / total samples)




                                                                                                                            Real-time
                                                                                       Preemptible Linux
                                                                                                                            operating systems
                                                                    1
                                                                                                                            Real-time Linux

                                                                                                                            CFS
                                                                   0.1
                                                                                                                            Recap

                                                                  0.01


                                                                 0.001


                                                                0.0001


                                                                1e-05


                                                                1e-06
                                                                         0   1000   2000    3000       4000   5000   6000
                                                                                       Latency (usecs)
Analysis of
Vanilla vs. rt-preempt (2)                                                                                         interrupt latencies
                                                                                                                      in a real-time
                                                                                                                          kernel
 Probability of the latency (num. of samples / total samples)




                                                                                                                   Real-time
                                                                                          RT Linux
                                                                                                                   operating systems
                                                                    1
                                                                                                                   Real-time Linux

                                                                                                                   CFS
                                                                   0.1
                                                                                                                   Recap

                                                                  0.01


                                                                 0.001


                                                                0.0001


                                                                1e-05


                                                                1e-06
                                                                         0   10   20       30       40   50   60
                                                                                       Latency (usecs)
Analysis of
I/O scheduling                                           interrupt latencies
                                                            in a real-time
                                                                kernel



                                                         Real-time
                                                         operating systems

                                                         Real-time Linux

                                                         CFS
    In traditional operating systems interrupt handler   Recap

    have priority above everything.
    rt-preempt =⇒ soft and hard irq handlers are
    threads:
             scheduling I/O ≡ scheduling tasks.
Analysis of
Benchmark network throughput                 interrupt latencies
                                                in a real-time
                                                    kernel



                                             Real-time
                                             operating systems

                                             Real-time Linux

                                             CFS
   netperf;                                  Recap

   UDP traffic;
   750B datagrams;
   schedule the real-time network threads;
   30 minutes execution.
Analysis of
Fixed priority scheduling                                    interrupt latencies
                                                                in a real-time
                                                                    kernel
It is not possibile to obtain low latency and high
throughput;                                                  Real-time
                                                             operating systems

                                                             Real-time Linux

 Net priority   min. lat (µs)    max. lat (µs)   Tp (Mbps)   CFS

      1              20              94            38.9      Recap

     50              22              96            39.9
     80              20             148            77.4
     99              22             164            73.4
      Table: Linux 2.6.24-rc2-rt1, 750B UDP datagrams


    high throughput and high latency;
    low throughput and low latency.
Fixed priority scheduling is not suitable =⇒ we look for
a method to assign CPU quota to tasks.
Analysis of
Completely Fair Scheduler              interrupt latencies
                                          in a real-time
                                              kernel



                                       Real-time
                                       operating systems

                                       Real-time Linux

                                       CFS

                                       Recap
    implemented by Ingo Molnar;
    merged in mainline since 2.6.23;
    weights tasks by ratio;
    group scheduling;
Analysis of
cfs & rt-preempt                                      interrupt latencies
                                                         in a real-time
                                                             kernel

My work
                                                      Real-time
    use CFS and modify the nice system call to set    operating systems

    weights (> 200 µs);                               Real-time Linux

                                                      CFS
    use CFS and tune Control Groups (> 1000 µs).
                                                      Recap
Results

          net   cyc   load   lat (µs)   Tp (Mbps)
           1     1      1     2931        49.9
           2     1      1     3911        67.3
           2     2      1     3721        71.1
           4     2      1     6117        80.9
           1     1      4     2871        35.2
       Table: Scheduling real-time threads with CFS
Analysis of
cfs & rt-preempt                                       interrupt latencies
                                                          in a real-time
                                                              kernel



                                                       Real-time
                                                       operating systems

                                                       Real-time Linux

                                                       CFS
    impact on the I/O throughput =⇒ the ratio system
                                                       Recap
    works;
    high latencies.
Short time possible solutions
    CFS clean up and tuning (in progress);
    EDF scheduling + time limit (Linux 2.6.25).
Analysis of
Overview of the work                                           interrupt latencies
                                                                  in a real-time
Step 1                                                                kernel

    litterature study on real-time computing;
                                                               Real-time
    study of the rt-preempt patch and real-time under Linux;   operating systems

                                                               Real-time Linux
Step 2                                                         CFS

                                                               Recap
    benchmark real-time performance;
    study the relationship between I/O and real-time
    applications;

Step 3
    test the CFS scheduler;
    try custom and standard solutions;
Future work
    results presented at the RTAS 2008 conference
    (http://www.rtas.org/ );
    tuning CFS and testing new scheduling policies.
Analysis of
Questions                   interrupt latencies
                               in a real-time
                                   kernel



                            Real-time
                            operating systems

                            Real-time Linux

                            CFS

                            Recap



            Any question?

More Related Content

PDF
Testing real-time Linux. What to test and how
PPTX
The Linux Scheduler: a Decade of Wasted Cores
PDF
Making Linux do Hard Real-time
PDF
HKG15-305: Real Time processing comparing the RT patch vs Core isolation
PPTX
Preempt_rt realtime patch
PDF
Mastering Real-time Linux
PDF
Linux Preempt-RT Internals
PDF
Solving Real-Time Scheduling Problems With RT_PREEMPT and Deadline-Based Sche...
Testing real-time Linux. What to test and how
The Linux Scheduler: a Decade of Wasted Cores
Making Linux do Hard Real-time
HKG15-305: Real Time processing comparing the RT patch vs Core isolation
Preempt_rt realtime patch
Mastering Real-time Linux
Linux Preempt-RT Internals
Solving Real-Time Scheduling Problems With RT_PREEMPT and Deadline-Based Sche...

What's hot (20)

ODP
Linux26 New Features
PPTX
Windows Internals for Linux Kernel Developers
PPT
PDF
NetBSD and Linux for Embedded Systems
PPTX
First steps on CentOs7
PDF
Running Applications on the NetBSD Rump Kernel by Justin Cormack
PDF
Systemd for developers
PDF
Introduction to NetBSD kernel
PPTX
Daemons
PDF
Linux cgroups and namespaces
PDF
Containers with systemd-nspawn
PPTX
Realizing Linux Containers (LXC)
PDF
Linuxcon Barcelon 2012: LXC Best Practices
PDF
Processes in unix
PDF
Linux Instrumentation
PDF
Namespaces and cgroups - the basis of Linux containers
PDF
Systemd: the modern Linux init system you will learn to love
PPT
Process and Threads in Linux - PPT
PDF
IJCER (www.ijceronline.com) International Journal of computational Engineeri...
PDF
Containers and Namespaces in the Linux Kernel
Linux26 New Features
Windows Internals for Linux Kernel Developers
NetBSD and Linux for Embedded Systems
First steps on CentOs7
Running Applications on the NetBSD Rump Kernel by Justin Cormack
Systemd for developers
Introduction to NetBSD kernel
Daemons
Linux cgroups and namespaces
Containers with systemd-nspawn
Realizing Linux Containers (LXC)
Linuxcon Barcelon 2012: LXC Best Practices
Processes in unix
Linux Instrumentation
Namespaces and cgroups - the basis of Linux containers
Systemd: the modern Linux init system you will learn to love
Process and Threads in Linux - PPT
IJCER (www.ijceronline.com) International Journal of computational Engineeri...
Containers and Namespaces in the Linux Kernel
Ad

Similar to Analysis of interrupt latencies in a real-time kernel (20)

PDF
Optimizing Linux Kernel for Real-time Performance On Multi-Core Architecture
PPT
PDF
Kernel Recipes 2016 - Wo needs a real-time operating system (not you!)
PPT
PDF
Enea Enabling Real-Time in Linux Whitepaper
PDF
The IRMOS Real-Time Scheduler
PPTX
How to Measure RTOS Performance
PPT
RTOS Basic Concepts
PDF
Making Linux do Hard Real-time
PPTX
Developing Real-Time Systems on Application Processors
PPT
ghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhjjjjjjjjjjjjjjj
PPT
PPT
Rtos
PDF
Analysis of Embedded Linux Literature Review Report
PDF
System on Chip Based RTC in Power Electronics
PDF
An Essential Relationship between Real-time and Resource Partitioning
PDF
Implementing Lightweight Networking
PDF
Implementing Lightweight Networking
PPTX
Real time Linux
PDF
Exploiting Linux Control Groups for Effective Run-time Resource Management
Optimizing Linux Kernel for Real-time Performance On Multi-Core Architecture
Kernel Recipes 2016 - Wo needs a real-time operating system (not you!)
Enea Enabling Real-Time in Linux Whitepaper
The IRMOS Real-Time Scheduler
How to Measure RTOS Performance
RTOS Basic Concepts
Making Linux do Hard Real-time
Developing Real-Time Systems on Application Processors
ghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhjjjjjjjjjjjjjjj
Rtos
Analysis of Embedded Linux Literature Review Report
System on Chip Based RTC in Power Electronics
An Essential Relationship between Real-time and Resource Partitioning
Implementing Lightweight Networking
Implementing Lightweight Networking
Real time Linux
Exploiting Linux Control Groups for Effective Run-time Resource Management
Ad

More from Gabriele Modena (6)

PDF
Resilient Distributed Datasets
PDF
Estimating mental states of a depressed person with Bayesian Networks
PDF
Analysis of grid log data with Affinity Propagation
PPT
Moving Towards a Streaming Architecture
PDF
Approximation algorithms for stream and batch processing
PDF
Full stack analytics with Hadoop 2
Resilient Distributed Datasets
Estimating mental states of a depressed person with Bayesian Networks
Analysis of grid log data with Affinity Propagation
Moving Towards a Streaming Architecture
Approximation algorithms for stream and batch processing
Full stack analytics with Hadoop 2

Recently uploaded (20)

PDF
OBE - B.A.(HON'S) IN INTERIOR ARCHITECTURE -Ar.MOHIUDDIN.pdf
PDF
Complications of Minimal Access-Surgery.pdf
PPTX
Introduction to pro and eukaryotes and differences.pptx
PDF
احياء السادس العلمي - الفصل الثالث (التكاثر) منهج متميزين/كلية بغداد/موهوبين
PDF
1.3 FINAL REVISED K-10 PE and Health CG 2023 Grades 4-10 (1).pdf
PDF
FORM 1 BIOLOGY MIND MAPS and their schemes
PDF
Practical Manual AGRO-233 Principles and Practices of Natural Farming
PDF
Uderstanding digital marketing and marketing stratergie for engaging the digi...
PDF
Chinmaya Tiranga quiz Grand Finale.pdf
PDF
David L Page_DCI Research Study Journey_how Methodology can inform one's prac...
PDF
MBA _Common_ 2nd year Syllabus _2021-22_.pdf
PPTX
CHAPTER IV. MAN AND BIOSPHERE AND ITS TOTALITY.pptx
PDF
LDMMIA Reiki Yoga Finals Review Spring Summer
PPTX
20th Century Theater, Methods, History.pptx
PDF
Paper A Mock Exam 9_ Attempt review.pdf.
PPTX
TNA_Presentation-1-Final(SAVE)) (1).pptx
DOCX
Cambridge-Practice-Tests-for-IELTS-12.docx
PPTX
History, Philosophy and sociology of education (1).pptx
PDF
Weekly quiz Compilation Jan -July 25.pdf
PDF
HVAC Specification 2024 according to central public works department
OBE - B.A.(HON'S) IN INTERIOR ARCHITECTURE -Ar.MOHIUDDIN.pdf
Complications of Minimal Access-Surgery.pdf
Introduction to pro and eukaryotes and differences.pptx
احياء السادس العلمي - الفصل الثالث (التكاثر) منهج متميزين/كلية بغداد/موهوبين
1.3 FINAL REVISED K-10 PE and Health CG 2023 Grades 4-10 (1).pdf
FORM 1 BIOLOGY MIND MAPS and their schemes
Practical Manual AGRO-233 Principles and Practices of Natural Farming
Uderstanding digital marketing and marketing stratergie for engaging the digi...
Chinmaya Tiranga quiz Grand Finale.pdf
David L Page_DCI Research Study Journey_how Methodology can inform one's prac...
MBA _Common_ 2nd year Syllabus _2021-22_.pdf
CHAPTER IV. MAN AND BIOSPHERE AND ITS TOTALITY.pptx
LDMMIA Reiki Yoga Finals Review Spring Summer
20th Century Theater, Methods, History.pptx
Paper A Mock Exam 9_ Attempt review.pdf.
TNA_Presentation-1-Final(SAVE)) (1).pptx
Cambridge-Practice-Tests-for-IELTS-12.docx
History, Philosophy and sociology of education (1).pptx
Weekly quiz Compilation Jan -July 25.pdf
HVAC Specification 2024 according to central public works department

Analysis of interrupt latencies in a real-time kernel

  • 1. Analysis of interrupt latencies in a real-time Università degli Studi di Trento kernel Facoltà di Scienze Matematiche Fisiche e Naturali Real-time operating systems Real-time Linux CFS Analysis of interrupt latencies in a Recap real-time kernel March 26, 2008 Laureando: Gabriele Modena (Mat. 108742) Relatore: dott. Luigi Palopoli Correlatore: dott. Luca Abeni
  • 2. Analysis of Real-Time Operating Systems (RTOS) interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap Ensure that an operation completes within a fixed amount of time (deadline) =⇒ predictability. When dealing with device drivers this implies managing race conditions; fulfill temporal constraints. This thesis focuses on shared mixed resources.
  • 3. Analysis of Latency interrupt latencies in a real-time kernel Real-time Latency is the measure of the difference between the operating systems theoretical (algorithm) and actual (implementation) Real-time Linux schedule. CFS Recap In a RTOS latency: should be bounded; the bound should not be too high. The goal of a real-time kernel is to minimize latency by using algorithms and technology that properly implements them: preemptible kernel.
  • 4. Analysis of Linux as an RTOS interrupt latencies in a real-time kernel Real-time operating systems Linux vanilla: Real-time Linux since 2.6 preemptible kernel; CFS The rt-preempt patch: Recap developed by Ingo Molnar; makes Linux hard real-time; uses the new (2.6.21) high resolution timers; preemptible critical sections; real-time mutexes; threaded interrput handlers.
  • 5. Analysis of Benchmarking rt-preempt interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap Two goals: test the actual real-time performance; test the performance of I/O scheduling. We look for a trade-off between latency and throughput.
  • 6. Analysis of Cyclictest interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS high resolution test program written by Thomas Recap Gleixner; measure latency; schedules periodic threads; When it comes to real-time applications rt-preempt gives the expected performance.
  • 7. Analysis of Vanilla vs. rt-preempt (1) interrupt latencies in a real-time kernel Probability of the latency (num. of samples / total samples) Real-time Preemptible Linux operating systems 1 Real-time Linux CFS 0.1 Recap 0.01 0.001 0.0001 1e-05 1e-06 0 1000 2000 3000 4000 5000 6000 Latency (usecs)
  • 8. Analysis of Vanilla vs. rt-preempt (2) interrupt latencies in a real-time kernel Probability of the latency (num. of samples / total samples) Real-time RT Linux operating systems 1 Real-time Linux CFS 0.1 Recap 0.01 0.001 0.0001 1e-05 1e-06 0 10 20 30 40 50 60 Latency (usecs)
  • 9. Analysis of I/O scheduling interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS In traditional operating systems interrupt handler Recap have priority above everything. rt-preempt =⇒ soft and hard irq handlers are threads: scheduling I/O ≡ scheduling tasks.
  • 10. Analysis of Benchmark network throughput interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS netperf; Recap UDP traffic; 750B datagrams; schedule the real-time network threads; 30 minutes execution.
  • 11. Analysis of Fixed priority scheduling interrupt latencies in a real-time kernel It is not possibile to obtain low latency and high throughput; Real-time operating systems Real-time Linux Net priority min. lat (µs) max. lat (µs) Tp (Mbps) CFS 1 20 94 38.9 Recap 50 22 96 39.9 80 20 148 77.4 99 22 164 73.4 Table: Linux 2.6.24-rc2-rt1, 750B UDP datagrams high throughput and high latency; low throughput and low latency. Fixed priority scheduling is not suitable =⇒ we look for a method to assign CPU quota to tasks.
  • 12. Analysis of Completely Fair Scheduler interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap implemented by Ingo Molnar; merged in mainline since 2.6.23; weights tasks by ratio; group scheduling;
  • 13. Analysis of cfs & rt-preempt interrupt latencies in a real-time kernel My work Real-time use CFS and modify the nice system call to set operating systems weights (> 200 µs); Real-time Linux CFS use CFS and tune Control Groups (> 1000 µs). Recap Results net cyc load lat (µs) Tp (Mbps) 1 1 1 2931 49.9 2 1 1 3911 67.3 2 2 1 3721 71.1 4 2 1 6117 80.9 1 1 4 2871 35.2 Table: Scheduling real-time threads with CFS
  • 14. Analysis of cfs & rt-preempt interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS impact on the I/O throughput =⇒ the ratio system Recap works; high latencies. Short time possible solutions CFS clean up and tuning (in progress); EDF scheduling + time limit (Linux 2.6.25).
  • 15. Analysis of Overview of the work interrupt latencies in a real-time Step 1 kernel litterature study on real-time computing; Real-time study of the rt-preempt patch and real-time under Linux; operating systems Real-time Linux Step 2 CFS Recap benchmark real-time performance; study the relationship between I/O and real-time applications; Step 3 test the CFS scheduler; try custom and standard solutions; Future work results presented at the RTAS 2008 conference (http://www.rtas.org/ ); tuning CFS and testing new scheduling policies.
  • 16. Analysis of Questions interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap Any question?