PREEMPT-RT Interrupt Threading and Interrupt Thread Priority Setting
-
PREEMPT-RT Interrupt Threading and Interrupt Thread Priority Setting
-
I. What is interrupt threading
- 1. General Linux interrupt handling
- 2. Shortcomings in real time
- 3. Interrupt threading
-
II. Interrupt Thread Priority Configuration
- Modifying interrupt thread priority with chrt
-
Modifying interrupt thread priority with rtirq
- Installing rtirq
- Instructions for using rtirq
- rtirq configuration
- III. Summary
-
I. What is interrupt threading
This paper describes one of the mechanisms by which the real-time linux program PREEMPT-RT improves the real-time performance of the system - theinterrupt threading, as well as interrupt thread priority configuration, hopefully providing some reference for interested readers.
I. What is interrupt threading
1. General Linux interrupt handling
An interrupt is an asynchronous event-handling mechanism used to respond to a hardware request, which interrupts the normal scheduling and execution of a process, and then calls the kernel interrupt handler to respond to the device's request.
Since interrupt handlers interrupt the operation of other processes, they need to run as fast as possible in order to minimize the impact on the scheduling of normal process operations. If the interrupt itself has little to do, it will not be too much of a problem to handle; but if the interrupt has a lot to deal with, the interrupt handler is likely to run for a long time. In particular, the interrupt handler will also temporarily shut down the interrupt when it responds to the interrupt. If the interrupts are heavily tasked, this can result in other interrupts not responding until the last interrupt processing is complete, or even lost interrupts.
So Linux divides the interrupt handling process into two phases, the first half and the second half:
- upper part: fast handling of hardware interrupts, inux does not support interrupt nesting.It runs in interrupt-disable modeIt mainly handles hardware-related or time-sensitive work, i.e., hard interrupts (Interrupt Request (IRQ)).
- lower part: Delayed processing of the first half of legacy tasks, usually run as a kernel thread, i.e., SoftIRQ.
The hard interrupt handler (ISR) responds quickly to hardware events and performs the least amount of work, deferring most tasks to the soft interrupt SoftIRQ or the task queue workqueue.
Soft interrupts are a mechanism for performing more time-consuming interrupt handling tasks at a later time. They are generally executed as kernel threads and each CPU corresponds to a soft interrupt kernel thread with the name "ksoftirqd/CPU number", e.g., the soft interrupt kernel thread corresponding to CPU 0 is ksoftirqd/0. Soft interrupts are run in the soft interrupt context, and the execution timing has 2 two.One is executed immediately after the return of the hardware interrupt, which, like the hardware interrupt, completely preempts the process context and does not allow it to be scheduled and preempted, and its execution time is no more than 2ms, and if the execution of the soft interrupt exceeds 2ms it will delay the soft interrupt to be executed in the soft interrupt thread "ksoftirqd/CPU No."。
Hardware and software interrupts in the system are viewed through the proc file system:
-
/proc/softirqs provides the operation of soft interrupts;
-
/proc/interrupts provides the operation of hard interrupts.
2. Shortcomings in real time
Although the traditional interrupt handling method improves the throughput of the system, it has the following shortcomings in terms of real-time task response real-time:
- disable interruptions: inux does not support interrupt nesting, and during hard interrupt processing, other interrupts are masked, increasing latency.
- Interrupt Serial Processing: inux does not support interrupt nesting, and peripheral interrupt processing on the same cpu is executed serially, which leads to uncertainty in emergency peripheral interrupt processing when there are many non-emergency peripheral interrupts.
- Unpredictable delays: Different drivers require different and unpredictable times for the execution of hard interrupt handlers, which may lead to delayed execution of high-priority tasks by interrupt preemption, affecting the real-time performance of the tasks.
- Uncertainty of soft interrupts: Soft interrupts pre-process after the exit of hard interrupts are fully preempted from the process context, and its processing time and scheduling order may also affect the execution of real-time tasks.
These shortcomings make it difficult for common linux interrupt handling methods to meet application scenarios with high real-time requirements.
3. Interrupt threading
To solve the above problems, the PREEMPT-RT patch introduced an interrupt threading mechanism, which was merged into linux mainline version 2.6.30 in 2009. as the name implies, theThe interrupt handling part of the original hardware interrupt context is also handled through a kernel threadThe interrupt thread is a real-time thread by default, with scheduling class SCHED_FIFO and real-time priority 50, except for interrupts that are explicitly specified in the code as non-interrupt threaded interrupts and system hardware timer interrupts, which are all handled through the interrupt thread.ksoftirqd/CPU
thread for execution.ksoftirqd/CPU
unchanged for ordinary non-real-time tasks. This is accomplished through theps
command to see the interrupt thread.
ubuntu@work-host:~$ ps -ef | grep irq
root 14 2 0 12moon10 ? 00:02:44 [ksoftirqd/0]
root 19 2 0 12moon10 ? 00:00:08 [irq_work/0]
root 25 2 0 12moon10 ? 00:00:00 [irq_work/1]
root 28 2 0 12moon10 ? 00:00:00 [ksoftirqd/1]
root 33 2 0 12moon10 ? 00:00:04 [irq_work/2]
root 36 2 0 12moon10 ? 00:02:47 [ksoftirqd/2]
root 41 2 0 12moon10 ? 00:00:01 [irq_work/3]
root 44 2 0 12moon10 ? 00:03:09 [ksoftirqd/3]
root 49 2 0 12moon10 ? 00:00:04 [irq_work/4]
root 52 2 0 12moon10 ? 00:02:42 [ksoftirqd/4]
root 57 2 0 12moon10 ? 00:00:02 [irq_work/5]
root 60 2 0 12moon10 ? 00:02:28 [ksoftirqd/5]
root 121 2 0 12moon10 ? 00:00:00 [irq/9-acpi]
root 136 2 0 12moon10 ? 00:00:00 [irq/120-PCIe PM]
root 137 2 0 12moon10 ? 00:00:00 [irq/121-PCIe PM]
root 138 2 0 12moon10 ? 00:00:00 [irq/121-aerdrv]
root 139 2 0 12moon10 ? 00:00:00 [irq/121-s-aerdr]
root 140 2 0 12moon10 ? 00:00:00 [irq/122-PCIe PM]
root 141 2 0 12moon10 ? 00:00:00 [irq/122-aerdrv]
root 142 2 0 12moon10 ? 00:00:00 [irq/122-s-aerdr]
root 143 2 0 12moon10 ? 00:00:00 [irq/123-PCIe PM]
root 144 2 0 12moon10 ? 00:00:00 [irq/123-aerdrv]
root 145 2 0 12moon10 ? 00:00:00 [irq/123-s-aerdr]
root 149 2 0 12moon10 ? 00:00:00 [vfio-irqfd-clea]
root 150 2 0 12moon10 ? 00:00:00 [irq/8-rtc0]
root 229 2 0 12moon10 ? 00:13:52 [irq/124-xhci_hc]
root 230 2 0 12moon10 ? 00:00:00 [irq/16-i801_smb]
root 281 2 0 12moon10 ? 00:00:00 [irq/126-ahci[00]
root 746 2 0 12moon10 ? 00:00:00 [irq/142-mei_me]
root 1089 2 0 12moon10 ? 00:00:05 [irq/143-i915]
root 1105 2 0 12moon10 ? 00:00:00 [irq/144-snd_hda]
root 1465 1 0 12moon10 ? 00:01:24 /usr/sbin/irqbalance --foreground
root 1731 2 0 12moon10 ? 00:00:20 [irq/125-eno1]
root 1734 2 0 12moon10 ? 00:00:00 [irq/127-enp2s0]
root 1735 2 0 12moon10 ? 00:00:04 [irq/128-enp2s0-]
....
Interrupt threading enhances the real-time performance of the Linux system in the following ways:
- Reduced interrupt disable time: From the original entire hardware interrupt processing need to mask global interrupts, to only respond to interrupts, wake up interrupt thread part will mask global interrupts, this part only performs the least amount of processing, reducing the time of interrupt disable, reducing the overall latency.
- Priority management: The interrupt handler, as a kernel thread, basically does not block global interrupts, and peripheral interrupt threads with different priorities can be given different priorities, thus avoiding low-priority peripheral interrupts from blocking the execution of high-priority peripherals and tasks.
- preemptibility: Interrupt threads act as schedulable kernel threads that can be preempted by high-priority tasks, making system response more flexible and real-time.
-
Increased predictability: Interrupt processing time becomes more controlled and predictable, improving the real-time performance of the system.
-Reduced system throughput: After interrupt threading, CPU context switching increases, and then the CPU processing efficiency will be discounted to some extent.
With these improvements, the interrupt threading mechanism of PREEMPT-RT effectively solves the shortcomings of traditional interrupt handling in real-time, so that the Linux system can better meet the needs of high real-time applications.
II. Interrupt Thread Priority Configuration
All peripheral interrupt thread default a real-time thread, scheduling class SCHED_FIFO, real-time priority of 50, for real-time requirements of high peripheral should be set higher priority, how can I modify the priority of the interrupt thread? There are two ways, the first is the most direct way, the driver code to add code to modify, but this is not flexible. The other is to modify it through the chrt tool.
Modifying interrupt thread priority with chrt
For example, to set the real-time priority of the thread with PID 1234 to 50, you can execute the following command:
chrt -f -p 50 1234
Here.-f
Indicates setting the real-time priority.-p
Indicates that the object of the operation is a process. Similarly, only the root user can modify the real-time priority of a thread.
Note: When using these commands, make sure that you have specified the PID correctly to avoid unnecessary effects on other processes.
But directly using chrt is still too primitive, here I recommend a tool - rtirq.
Modifying interrupt thread priority with rtirq
Installing rtirq
If you're using a debian-derived distribution, you can also use thertirq
This tool to modify the interrupt thread priority pulls directly from github
git clone /rncbc/rtirq
Or install it via the apt package manager
sudo apt install rtirq-init
take note ofThe rtirq tool relies on full-featured ps and crt, so in some embedded systems that use busybox, rtirq won't run due to the simplicity of the ps functionality.
Instructions for using rtirq
Let's see.rtirq-init
How it works.rtirq
is a script located in the/etc//rtirq
Located in/etc/
directory will be automatically executed after system startup, first read and parse the configuration file, and then modify the priority and scheduling policy of the interrupt thread according to the configuration file, and ultimately execute the samechrt
command, except that it offers more flexibility in configuration by way of a configuration file, and you can analyze the script source code yourself.
scripts/etc//rtirq
The parameters received arestart
、stop
、reset
、restart
(or)force-reload
、status
。
The configuration files used by the rtirq script to execute are, in order/etc/sysconfig/rtirq
、/etc/default/rtirq
、/etc/
, the first one exists and the second one does not take effect. where/etc/default/rtirq
It's a template file that is automatically generated after installation, and you can either modify it directly or create a new one based on it.
rtirq configuration
/etc/default/rtirq
The content is as follows:
# The name of the IRQ thread service we want to modify, configured in order of priority and step size as follows
# (high to low priority, space-separated list).
# RTIRQ_NAME_LIST="rtc snd usb i8042" # old version
RTIRQ_NAME_LIST="snd usb i8042"
# Highest priority.
RTIRQ_PRIO_HIGH=90
# Priority decrement step.
RTIRQ_PRIO_DECR=5
# Lowest priority level.
RTIRQ_PRIO_LOW=51
# Whether to reset all IRQ threads to SCHED_OTHER.
RTIRQ_RESET_ALL=0
# When supported by the kernel.
# Which services should not be threaded (space-separated list).
RTIRQ_NON_THREADED="rtc snd"
# Process names (99-91) that will be forced to the highest real-time priority range (high to low, space-separated list)
# RTIRQ_HIGH_LIST="timer"
- RTIRQ_NAME_LIST indicates the set of interrupt threads we want to modify, in order of priority from RTIRQ_PRIO_HIGH to RTIRQ_PRIO_LOW high, the examples are snd(90)>usb(85)>i8042(80);
- RTIRQ_PRIO_HIGH indicates the priority of the first interrupt thread in the first RTIRQ_NAME_LIST;
- RTIRQ_PRIO_DECR indicates the step from high to low interrupt thread priority in RTIRQ_NAME_LIST;
- RTIRQ_PRIO_LOW indicates the lowest priority;
- RTIRQ_RESET_ALL Does RTIRQ_RESET_ALL reset all IRQ threads other than RTIRQ_NAME_LIST to SCHED_OTHER?
If we don't know the name of the IRQ thread, we can get it by executing /etc//rtirq status and also see the default priority of the interrupt thread in the system:
ubuntu@work-host:~$ /etc//rtirq status
PID CLS RTPRIO NI PRI %CPU STAT COMMAND
1105 FF 90 - 130 0.0 S irq/144-snd_hda
229 FF 85 - 125 0.1 S irq/124-xhci_hc
121 FF 50 - 90 0.0 S irq/9-acpi
136 FF 50 - 90 0.0 S irq/120-PCIe PM
137 FF 50 - 90 0.0 S irq/121-PCIe PM
138 FF 50 - 90 0.0 S irq/121-aerdrv
150 FF 50 - 90 0.0 S irq/8-rtc0
230 FF 50 - 90 0.0 S irq/16-i801_smb
281 FF 50 - 90 0.0 S irq/126-ahci[00
746 FF 50 - 90 0.0 S irq/142-mei_me
1089 FF 50 - 90 0.0 S irq/143-i915
1731 FF 50 - 90 0.0 S irq/125-eno1
....
1757 FF 50 - 90 0.0 S irq/141-enp4s0-
14 RR 1 - 41 0.0 S ksoftirqd/0
28 TS - 0 19 0.0 S ksoftirqd/1
36 TS - 0 19 0.0 S ksoftirqd/2
44 TS - 0 19 0.0 S ksoftirqd/3
52 TS - 0 19 0.0 S ksoftirqd/4
60 TS - 0 19 0.0 S ksoftirqd/5
....
typical example
- First, configure all thread priorities to 0. The configuration file is as follows:
....
# Whether to reset all IRQ threads to SCHED_OTHER.
RTIRQ_RESET_ALL=1
...
Implementation:
ubuntu@work-host:~$ sudo /etc//rtirq reset
ubuntu@work-host:~$ sudo /etc//rtirq status
PID CLS RTPRIO NI PRI %CPU STAT COMMAND
14 TS - 0 19 0.0 S ksoftirqd/0
28 TS - 0 19 0.0 S ksoftirqd/1
36 TS - 0 19 0.0 S ksoftirqd/2
44 TS - 0 19 0.0 S ksoftirqd/3
52 TS - 0 19 0.0 S ksoftirqd/4
60 TS - 0 19 0.0 S ksoftirqd/5
121 TS - 0 19 0.0 S irq/9-acpi
137 TS - 0 19 0.0 S irq/120-PCIe PM
138 TS - 0 19 0.0 S irq/121-PCIe PM
139 TS - 0 19 0.0 S irq/121-aerdrv
140 TS - 0 19 0.0 S irq/121-s-aerdr
141 TS - 0 19 0.0 S irq/122-PCIe PM
142 TS - 0 19 0.0 S irq/122-aerdrv
143 TS - 0 19 0.0 S irq/122-s-aerdr
144 TS - 0 19 0.0 S irq/123-PCIe PM
145 TS - 0 19 0.0 S irq/123-aerdrv
146 TS - 0 19 0.0 S irq/123-s-aerdr
151 TS - 0 19 0.0 S irq/8-rtc0
251 TS - 0 19 0.0 S irq/125-xhci_hc
252 TS - 0 19 0.0 S irq/16-i801_smb
299 TS - 0 19 0.0 S irq/126-ahci[00
776 TS - 0 19 0.0 S irq/142-mei_me
1121 TS - 0 19 0.0 S irq/143-i915
1136 TS - 0 19 0.0 S irq/144-snd_hda
1798 TS - 0 19 0.0 S irq/124-eno1
1799 TS - 0 19 0.0 S irq/127-enp2s0
1800 TS - 0 19 0.0 S irq/128-enp2s0-
1801 TS - 0 19 0.0 S irq/129-enp2s0-
1802 TS - 0 19 0.0 S irq/130-enp2s0-
1803 TS - 0 19 0.0 S irq/131-enp2s0-
1808 TS - 0 19 0.0 S irq/132-enp3s0
1809 TS - 0 19 0.0 S irq/133-enp3s0-
1810 TS - 0 19 0.0 S irq/134-enp3s0-
1811 TS - 0 19 0.0 S irq/135-enp3s0-
1812 TS - 0 19 0.0 S irq/136-enp3s0-
1822 TS - 0 19 0.0 S irq/137-enp4s0
1823 TS - 0 19 0.0 S irq/138-enp4s0-
1824 TS - 0 19 0.0 S irq/139-enp4s0-
1825 TS - 0 19 0.0 S irq/140-enp4s0-
1826 TS - 0 19 0.0 S irq/141-enp4s0-
- Modify the interrupt thread priority of rtc0, enp4s0 with the following configuration file:
# IRQ thread service names
# (space separated list, from higher to lower priority).
# RTIRQ_NAME_LIST="rtc snd usb i8042" # old
RTIRQ_NAME_LIST="rtc enp4s0"
# Highest priority.
RTIRQ_PRIO_HIGH=90
# Priority decrease step.
RTIRQ_PRIO_DECR=5
# Lowest priority.
RTIRQ_PRIO_LOW=51
# Whether to reset all IRQ threads to SCHED_OTHER.
RTIRQ_RESET_ALL=0
# On kernel configurations that support it,
# which services should be NOT threaded
# (space separated list).
#RTIRQ_NON_THREADED="rtc snd"
# Process names which will be forced to the
# highest realtime priority range (99-91)
# (space separated list, from highest to lower priority).
# RTIRQ_HIGH_LIST="timer"
fulfillment
ubuntu@work-host:~$ sudo /etc//rtirq start
Setting IRQ priorities: stop [rtc] irq=8 pid=151: OK.
Setting IRQ priorities: stop [enp4s0] irq=137 pid=1822: OK.
Setting IRQ priorities: stop [enp4s0] irq=138 pid=1823: OK.
Setting IRQ priorities: stop [enp4s0] irq=139 pid=1824: OK.
Setting IRQ priorities: stop [enp4s0] irq=140 pid=1825: OK.
Setting IRQ priorities: stop [enp4s0] irq=141 pid=1826: OK.
ubuntu@work-host:~$ sudo /etc//rtirq status
PID CLS RTPRIO NI PRI %CPU STAT COMMAND
151 FF 90 - 130 0.0 S irq/8-rtc0
1822 FF 85 - 125 0.0 S irq/137-enp4s0
1823 FF 84 - 124 0.0 S irq/138-enp4s0-
1824 FF 83 - 123 0.0 S irq/139-enp4s0-
1825 FF 82 - 122 0.0 S irq/140-enp4s0-
1826 FF 81 - 121 0.0 S irq/141-enp4s0-
14 TS - 0 19 0.0 S ksoftirqd/0
28 TS - 0 19 0.0 S ksoftirqd/1
36 TS - 0 19 0.0 S ksoftirqd/2
44 TS - 0 19 0.0 S ksoftirqd/3
52 TS - 0 19 0.0 S ksoftirqd/4
60 TS - 0 19 0.0 S ksoftirqd/5
121 TS - 0 19 0.0 S irq/9-acpi
137 TS - 0 19 0.0 S irq/120-PCIe PM
138 TS - 0 19 0.0 S irq/121-PCIe PM
139 TS - 0 19 0.0 S irq/121-aerdrv
140 TS - 0 19 0.0 S irq/121-s-aerdr
141 TS - 0 19 0.0 S irq/122-PCIe PM
142 TS - 0 19 0.0 S irq/122-aerdrv
143 TS - 0 19 0.0 S irq/122-s-aerdr
144 TS - 0 19 0.0 S irq/123-PCIe PM
145 TS - 0 19 0.0 S irq/123-aerdrv
146 TS - 0 19 0.0 S irq/123-s-aerdr
251 TS - 0 19 0.0 S irq/125-xhci_hc
252 TS - 0 19 0.0 S irq/16-i801_smb
299 TS - 0 19 0.0 S irq/126-ahci[00
776 TS - 0 19 0.0 S irq/142-mei_me
1121 TS - 0 19 0.0 S irq/143-i915
1136 TS - 0 19 0.0 S irq/144-snd_hda
1798 TS - 0 19 0.0 S irq/124-eno1
1799 TS - 0 19 0.0 S irq/127-enp2s0
1800 TS - 0 19 0.0 S irq/128-enp2s0-
1801 TS - 0 19 0.0 S irq/129-enp2s0-
1802 TS - 0 19 0.0 S irq/130-enp2s0-
1803 TS - 0 19 0.0 S irq/131-enp2s0-
1808 TS - 0 19 0.0 S irq/132-enp3s0
1809 TS - 0 19 0.0 S irq/133-enp3s0-
1810 TS - 0 19 0.0 S irq/134-enp3s0-
1811 TS - 0 19 0.0 S irq/135-enp3s0-
1812 TS - 0 19 0.0 S irq/136-enp3s0-
As can be seen.irq/8-rtc0
Real-time priority is 90.irq/137-enp4s0
The real-time priority is 85 and the modification is successful.
III. Summary
This paper focuses on interrupt threading as a mechanism to improve the real-time performance of linux systems. By shifting peripheral interrupt processing from a hardware interrupt context, which masks global interrupts, to a thread context program execution, the priority and response time of interrupt processing can be better controlled, but at the expense of some system throughput. In addition, the article discusses in detail how to configure interrupt thread priorities for different peripherals to ensure mission-critical real-time performance. These configuration methods are important for system designers when planning and optimizing priorities for real-time applications.