Pre-emptive Scheduling

This example explores using interrupts to implement a simple pre-emptive scheduler – a time-slicing round-robin scheduler. This can be achieved by

Figure 2.16: Output of running matrix mul8.c. Note how interrupts can occur at any time, independent of the presence or progress of consume calls on the resource.

implementing a custom scheduler, interrupt controller, and simulated timer peripheral that all work together to enforce the scheduling policy.

Figure 2.17 shows how these pieces fit together.

Each time the timer model wakes up it checks to see if it has reached the next scheduled timeout. If so, it raises an interrupt on the resource, notifies the scheduler of the interrupt, schedules the next timeout, and goes back to sleep. If the scheduler restarted the timer while it was sleeping it will wake before the newly scheduled timeout and go back to sleep without raising any interrupts. This behavior delivers the regular “timer tick” interrupts the scheduler needs to perform time slicing.

Meanwhile, a custom scheduler implements time slicing by only changing thread-resource mappings under two circumstances:

1. A thread blocks or terminates before its time slice expires

2. The timer “tick” interrupt marks the end of the current time slice.

When the first case occurs the scheduler must restart the timer so that the next thread gets a full time slice. In both cases the scheduler picks the least-recently scheduled (“LRS”) thread in the system to assign to the resource to enforce round-robin behavior. If neither of these cases holds the scheduler leaves the thread-resource assignment unchanged by reapplying the previous assignment. We describe the scheduler in more detail below.

State creation. First, we define a new struct to encapsulate state shared by the timer(s) and schedule:

20 t y p e d e f s t r u c t { 21 m e s h r e s o u r c e ∗ c r ; 22 g b o o l e a n i n t e r r u p t e d ; 23 d o u b l e i n t e r v a l ; 24 d o u b l e n e x t t i m e o u t ; 25 } t i m e r s t a t e ;

wait

for r in idle resources { if (last_thread eligible && !interrupted) reschedule(last_thread) else {

schedule(LRS_thread) reset timer

Figure 2.17: Components of a time-slicing MESH scheduler

The fields of the struct will be used to track resource-specific state within each timer and the scheduler that controls them all. The fields will be used as follows:

• cr. Pointer to the resource paired with this timer. Each resource has its own timer to allow flexible time slicing.

• interrupted. Flag set by the timer when it raises an interrupt, since there is no reliable way to infer interrupted status from thread or resource state.

• interval. Time slicing interval. This tells the timer how often to raise tick interrupts.

• next timeout. Next scheduled timeout/tick.

We also extend the helper function to accept the “tick” interval as an ar-gument, which it uses to initialize the state for the resource and set its time-slicing interval.

Resource-local storage. After creating the resource and state struct, the mesh resource

create entry() details on pg. 77

helper function places the state in resource-local storage so the scheduler can access and update the timer state of each resource as necessary.

278 // s t o r e the s t a t e i n the resource

279 m e s h r e s o u r c e c r e a t e e n t r y ( c r , TIMER KEY , s t a t e ) ;

MESH allows the user to associate local key-value pairs with resources, schedulers and threads as a way to flexibly extend these data types without changing their definitions. Each key-value pair must be initialized before

use by calling the appropriate function (mesh_resource_create_entry() in this case). After initialization values can be retrieved and changed using mesh_resource_get_entry() and mesh_resource_set_entry(), respec-tively (described later). The API reference contains complete descriptions for these functions and the corresponding ones for threads and schedulers.

Timer model. Next we replace the testbench thread from the last ex-ample with a timer model. It will raise the periodic timer interrupt the scheduler needs to properly time slice the threads it controls:

228 // A t e s t bench thread that a c t s as a hardware timer and r a i s e s 229 // p e r i o d i c i n t e r r u p s . The timeout i n t e r v a l i s supplied as the 230 // argument to the thread

231 v o i d ∗ t i m e r ( v o i d ∗ a r g ) {

The timer uses the state struct to determine how long to wait between in-terrupts. If the wait between now and the next scheduled timeout is greater than some EPSILON (necessary because of accuracy issues in floating point numbers) the thread will sleep (lines 241 – 246). This can occur multiple times if the scheduler updates the timer interval in the meantime. Once the wait is within ±EP SILON the timer will raise a tick interrupt on the resource to trigger scheduling. It then automatically begins a new timer interval, sleeping until the next potential time slice (lines 247 – 254).

Because each resource must have its own timer model, we move the test bench thread instantiation inside the create int res function:

285 // use a t e s t bench thread to represent a hardware timer

286 t i m e r n a m e = g s t r d u p p r i n t f ( ”%s t i m e r ” , name ) ;

287 m e s h c r e a t e t b t h r e a d ( t i m e r n a m e , & t i m e r , s t a t e , FALSE ) ; 288 g f r e e ( t i m e r n a m e ) ;

Time slicing scheduler. The time slicing scheduler makes thread-resource assignments in two phases.

93 // try to reschedule the same thread u n l e s s i t s time s l i c e expired 94 f o r ( i t e r = i d l e r e s o u r c e s ; i t e r && i d l e t h r e a d s ; i t e r = g s l i s t n e x t ( i t e r ) ) { 95 m e s h r e s o u r c e t h r e a d p a i r ∗ r t p a i r ;

96 m e s h r e s o u r c e ∗ c r = i t e r −>d a t a ;

First, it tries to reschedule threads whose time slice has not finished yet (lines 100 – 104). The scheduler will restart the timer for a resource every time it changes the thread assignment, so threads only get interrupted when their time slice has expired. The “interrupted” status is stored in the shared state struct, which the scheduler retrieves using mesh_resource_get_entry().

mesh resource get entry details on pg. 78

The scheduler then checks to see if the thread is still eligible to run by searching the eligible thread list (lines 106 – 108).

Once the scheduler has determined that the thread should be rescheduled, it makes a thread-resource pair and adds it to the list of assignments, as described in Section 2.3.5.

During the second phase of scheduling, the scheduler pairs up remaining idle threads and resources in least-recently scheduled order to enforce round robin scheduling across time slices.

It first checks to see if the resource was scheduled in the first phase (lines 128 – 130). Then, it updates the last scheduled status of the thread (line 141). Finally, if a time slice ended prematurely the scheduler updates the state struct so the timer can restart (lines 143 – 144).

The scheduler creates an entry in thread-local storage to track when each thread was last scheduled. Every time a thread begins a new time slice it stores the current simulation time in the thread-local entry. It later uses this information find the least-recently scheduled thread to assign:

31 v o i d s e t l a s t s c h e d u l e d ( m e s h t h r e a d ∗ c t , d o u b l e when ) {

This introduces a new function – mesh_thread_has_entry() – which gives mesh thread has entry()

details on pg. 82 a way to check whether a thread-local value already exists, since there is no reliable way to determine whether a thread is newly created or not. The entry union is necessary because double** and void** are incompatible types. After testing for the entry’s existence the code creates an entry if necessary.

Running the example Now that we’ve set up the infrastructure for time-slicing, there are a few small changes to make to complete the example. The main() function for this example is very similar to the last one. The only difference is that we now pass the time slicing interval (6 in this case) to the resource-creation helper function (see page 29), recalling that it uses the value to initialize the time state for each resource.

314 r e s o u r c e 1 = c r e a t e r e s o u r c e ( ” r e s o u r c e 1 ” , c f l ,

Because the timer testbench thread runs in an infinite loop, we must also mesh exit()

details on pg. 75 modify the boss thread to terminate the simulation when it completes, with a call to mesh_exit().

Now we can run the example, found in matrix_mul9.c. Figure 2.18 shows the resulting output in the MESH viewer. Note how consume calls no longer directly correspond to scheduling decisions. A single time slice might consist

Figure 2.18: Time-sliced program execution

of only part of a single consume call (ie slow resource at time 200), or many small consume calls together (ie resource MAC, also at time 200). Also note how the timer restarts when a different thread begins executing, ensuring that all threads get their full time slice. Finally, we can see how the timer tick can split consume calls at arbitrary points (again resource MAC at time 200), allowing interrupt handling to occur at exactly the right time.

Our new scheduler successfully decouples scheduling from consume calls, though threads can still force scheduling decisions by terminating or block-ing. This opens a new set of design variables to experiment with. Possibili-ties include varying time slice length (perhaps different lengths for different processors) or implementing purely pre-emptive schedules where the highest priority ready task always runs. Finally, interrupt modeling, combined with test bench threads, makes it possible to model external devices and their driver code (a timer in this case).