Interrupts

Interrupts notify processors of asynchronous events and may occur between (almost) any two instructions. If an interrupt is detected, the processor temporarily pauses the execution of the currently scheduled task and executes a designatedinterrupt service routine(ISR) instead. This can cause the interrupted task to incur undesirable delays that must be accounted for when determining whether a task is schedulable, as discussed in detail in Chapter 3.

Most interrupts are maskable, i.e., the processor can be instructed by the OS to delay the invocation of ISRs until interrupts are unmasked again. However,non-maskable interrupts(NMIs), which can be used for “watch dog” functionality to detect system hangs, cannot be suppressed by the OS. In multiprocessor systems, some interrupts may further be local to a specific processor, whereas others may be serviced by multiple or all processors.

Interrupt categories. Interrupts can be broadly categorized into four classes: device interrupts

(DIs),timer interrupts(TIs),cycle-stealing interrupts(CSIs), andinter-processor interrupts(IPIs). We briefly discuss the purpose of each next.

DIs are triggered by hardware devices when a timely reaction by the OS is required or to avoid costly “polling” (see below). In real-time systems, DIs may cause jobs to be released,e.g., a sensor

may trigger a DI to indicate the availability of newly-acquired data, in response to which a job is released to process said data.

TIs are used by the OS to initiate some action in the future. For example, TIs are used to support high-resolution delays (“sleeping”) in Linux. In LITMUSRT, they are also used for periodic job releases and to enforce execution-time budgets. In networking protocols, TIs are commonly employed to trigger timeouts and packet re-transmissions.

CSIs are an artifact of the way that modern hardware architectures are commonly implemented and differ from the other categories in that they are neither controlled nor handled by the OS. CSIs are used to “steal” processing time for use by some component that is—from the point of view of the OS—hardware, but that is implemented as a combination of hardware and software (so called “firmware”) and that lacks its own processor. CSIs are intended to be transparent from a logical correctness point of view, but of course do affect temporal correctness. They are usually non-maskable and the OS is generally unaware if and when CSIs occur. A well-known example for the use of CSIs is the system management mode (SMM) in Intel’s x86 architecture (Intel Corporation, 2008a,c): when a system management interrupt (SMI) occurs, the system switches into the SMM to execute ISRs stored in firmware. For example, on some chip sets the SMM is entered to control the speed of fans for cooling purposes. CSIs can also occur in architectures in which raw hardware access is mediated by a hypervisor. For example, this is the case in Sony’s PlayStation 3 (Kurzak

et al., 2008) and Sun’s sun4v architecture (Saulsbury, 2008). In such systems, the hypervisor may become active at any time to handle interrupts or perform services for devices “invisible” to the OS. CSIs are especially problematic if the code that is being executed is unknown—for example, a CSI could flush instruction and data caches and thereby unexpectedly increase task execution costs.

In contrast to DIs, TIs, and CSIs, the final category considered, IPIs, are specific to multiprocessor systems. IPIs are used to synchronize state changes across processors and are generated by the OS. For example, the modification of an address space on one processor can require software-initiated TLB flushes on multiple processors (Intel Corporation, 2008c,d). IPIs are also commonly used to cause a remote processor to reschedule.

Xeon L7455. Figure 2.7 shows how interrupts are handled in our x86 platform used for the case studies presented in Chapters 4 and 7. Modern x86 systems use theadvanced programmable interrupt

device interrupt (DI) inter-processor interrupt (IPI) Core 1 Core24 APIC bus device device device device TPR LDID EOI local APIC TPR LDID EOI local APIC interrupt lines

timer temp timer temp

configuation table I/O APIC

Figure 2.7: Illustration of the I/O APIC and local APICs in the multiprocessor platform underlying the case studies discussed in Chapters 4 and 7. Devices are connected to the I/O APIC with physical interrupt lines. The I/O APIC polls the incoming interrupt lines and dispatches an interrupt to a local APIC when a signal is detected. The recipient is determined based on the interrupt-line-specific target LDID and the current priorities in the TPR registers. Local APICs are also used to send IPIs among processors and to handle per-processor devices (such as the depicted timer and temperature sensors).

controller(APIC) to manage interrupts (Intel Corporation, 1996, 2008c). The name derives from the legacyprogrammable interrupt controller(PIC), which was a single chip that was used in early x86-based designs. In contrast, the APIC is actually a collection of chips that collaborate to deliver interrupts.

Devices interface with theI/O APICthat handles interrupt distribution by sending DIs to one of the cores (more on DI distribution below). Each core has alocal APICthat performs the actual interrupt delivery when it receives a DI from the I/O APIC. Local APICs are also responsible for sending IPIs to each other, and also directly interface with processor-local interrupt sources (such as a timer, performance counters, and thermal sensors). Conceptually, the local APICs and the I/O APIC are connected by a dedicatedAPIC bus(Intel Corporation, 1996); however, current systems actually use the main system bus to transport APIC messages. A processor can temporarily disable the delivery of interrupts with thecliinstruction. This does not, however, stop the local APIC from accepting interrupts, which will be delivered once local interrupts are re-enabled by the OS with the

stiinstruction.

There are 15 distinct interrupt priorities. A local APIC accepts only higher-priority interrupts while an ISR is executing. If a higher-priority interrupt is received while executing an ISR for a lower-priority interrupt, then the higher-priority interrupt is delivered right away, interrupting the

lower-priority ISR (unless interrupt delivery has been disabled—see above). The OS must signal the end of an ISR to the local APIC by writing to a specialend-of-interrupt(EOI) register. Each local APIC has atask priority register(TPR) and will only accept interrupts that exceed the priority of the value in the TPR. This can be used by the OS to mark the local APIC as ineligible to process low-priority interrupts.

The I/O APIC polls 24 physical interrupt lines (each of which is connected to one or more devices) for interrupt signals. For each interrupt line, the OS can configure how a detected interrupt will be relayed to one or more local APICs. In particular, each interrupt can be sent to either a specific local APIC (and hence processor) or to alogical destination ID(LDID). Each local APIC belongs to exactly one LDID and multiple local APICs may share the same LDID; this allows interrupts to be distributed among several processors on a line-by-line basis. If there are multiple possible recipients for a logical destination ID, a TPR-aware arbitration protocol is used to dispatch the interrupt to the lowest-priority recipient processor,i.e., to the processor with the lowest priority value stored in its TPR register (the OS should avoid priority ties). The I/O APIC also allows each interrupt line to be individually masked, and to specify the interrupt priority for each line.

Taken together, these options provide the OS with considerable flexibility to determine which processor(s) will handle interrupts.