Implementation of Task Stealing in GPRM

As described above, inside GPRM every thread runs a tile with its own task manager. Each task manager consists of multiple queues for different purposes, such as exchanging packets (RX fifo and TX fifo) or storing results. As stated, one of these queues is the ready fifo. All tiles are initially in the non-stealing state, which means their task man- agers take tasks from their own ready fifo. If STEAL is enabled, then after running the last task in its ready fifo, the task manager searches for jobs in the ready fifos of other tiles (excluding the control tile). The tile remains in the stealing state and repeats this task-stealing process, unless there is no more job to steal after probing all ready fifos of all tiles once. Then it goes to the sleeping state, and waits for a reference packet to wake it up and start a new computation. This mechanism is shown in List. 5.3.

1T a s k T a s k M a n a g e r : : t a s k S t e a l ( ) { 2 f o r(i n t j = t i l e A d d r ; j < t i l e A d d r +NTILES ; ++ j ) { 3 i n t i = ( j % NTILES ) + 1 ; / / no s t e a l i n g f r o m t i l e 0 4 T i l e& v i c t i m = ∗ ( s y s t e m . t i l e s [ i ] ) ; 5 i f( v i c t i m . T a s k M a n a g e r . r e a d y f i f o . s i z e ( ) ! = 0 ) { 6 T a s k s t o l e n = v i c t i m . T a s k M a n a g e r . l o c k R e a d y Q ( 1 ) ; 7 / / 1 means a t h i e f i s l o c k i n g

8 i f( s t o l e n ! = EMPTY) { / / EMPTY i s an i n t number f o r n u l l t a s k s 9 . . . / / r e g i s t e r s i t t o t h e TCB o f t h e t h i e f 10 r e t u r n s t o l e n ; 11 } 12 } 13 } / / e l s e , l o o p c o n t i n u e s 14 r e t u r n EMPTY ; 15}

Listing 5.3: Task Stealing inside the task manager

The stolen task shown in List. 5.3 can be null (EMPTY), because we do not lock the queues only to check whether their sizes are greater than 0. This way, we do not interrupt the routine operations of other tiles. However, inside the member function called lockReadyQ(), we check if the size is still greater than 0 (or the owner has processed all of its tasks already). If the size of the ready queue is 0, the loop continues in order to find another victim.

The member function lockReadyQ() used in both Lists. 5.3 and 5.4 is responsible for locking the ready fifo and returning the top element, if any. As shown in the code, it

5.5. GPRM Task Stealing 78

gets an argument which specifies whether a thief has locked the FIFO or not. As a result, we can define different actions for the owners and thieves inside this member function. For instance, for research purposes, it is possible to define conditions for task stealing. We can set rules for thieves to steal only from the queues with larger sizes than X. However, for the experiments in this study we did not set any rules.

1v o i d T a s k M a n a g e r : : c o r e C o n t r o l (b o o l i s s t e a l s t a t e ) { 2 / / i s s t e a l s t a t e comes f r o m t h e T i l e c l a s s 3# i f d e f STEAL 4 i f( i s s t e a l s t a t e ) { 5 T a s k s t o l e n t a s k = t a s k S t e a l ( ) ; / / d e f i n e d e a r l i e r 6 i f( s t o l e n t a s k ! = EMPTY) { 7 c u r r e n t t a s k = s t o l e n t a s k ; 8 s t a t u s = r e a d y ; / / t h e s t a t u s o f t h e t a s k manager 9 } 10 } 11 e l s e i f ( s t a t u s == i d l e ) { 12 T a s k n e x t = l o c k R e a d y Q ( 0 ) ; / / 0 means t h e owner t i l e i s l o c k i n g 13 i f( n e x t ! = EMPTY) { 14 c u r r e n t t a s k = n e x t ; 15 s t a t u s = r e a d y ; 16 } 17 } 18# e l s e / / STEAL i s d i s a b l e d 19 i f( s t a t u s == i d l e and r e a d y f i f o . s i z e ( ) > 0 ) { 20 c u r r e n t t a s k = r e a d y f i f o . f r o n t ( ) ; 21 r e a d y f i f o . p o p F r o n t ( ) ; 22 s t a t u s = r e a d y ; 23 } 24# e n d i f 25 . . . 26}

Listing 5.4: Core Control inside the task manager

Another member function of the class TaskManager is called coreControl, which is the centre for main operations inside the task manager. As shown in List. 5.4, a macro called STEALis defined to ensure that if task stealing is disabled, no one locks the ready fifo. This way, we can measure the real overhead of stealing, compared to when tiles only take tasks from their own queues. Three cases are shown in this function:

1. If STEAL is defined, the tile is either in its stealing state or not. If it is, taskSteal() will be called, and if successful the status will be set to ready for processing.