Once we have HALLOCwe have to make some changes to get porting of LPEL on
the SCC complete. Since LPEL was originally designed for shared memory systems where all the cores are running under the same instance of OS, only one instance of S-Net RTS and LPEL is required, whereas in our case this does not hold anymore. In addition, the previously used synchronisation mechanism does not work either i.e. we can not usePTHREADmutex to achieve cross-core synchronisation on the SCC as each
core runs its own instance of OS. Furthermore, if the user does not specify the desired number of workers on program startup we can not infer available number of cores that will participate (in multi-core/many-core shared memory system with single system image this can be achieved by querying OS). And lastly, we have to make sure that all the calls to standardmalloc (for memory that is shared between cores) are replaced with scc_malloc.
3.4.1
Conductor/Worker Initialisation on the SCC
When deploying the LPEL on the SCC, it makes sense to create exactly as many workers as there are cores, as the cores of SCC are single-threaded. As there is no shared memory at the beginning, we can not just create conductor/workers on a single core and then distribute them amongst participating cores, which is the way it is done in a normal shared-memory machine. For this purpose, when the execution of a program starts, a configuration file is used to decide which core will be the conductor based on the physical core id.
As mentioned in Section3.2, to create a truly global shared memory, all the cores have to map part of the program’s address-space to the same virtual address range. Meta-data, including a flag necessary to establish communication between cores is located in a predefined location in MPB. This location is predefined and as such all the cores taking part in the computation will know this MPB location.
If a core is a conductor, it starts by initialising the shared memory, tasks, streams and the static parts of streaming network. If core is a worker, it will busy-wait on a flag located in MPB. Once the conductor has mapped the LUT entries and created the shared memory, it places the relevant LUT configuration in the MPB and sets the flag. Once the flag is set, the worker cores configure their LUTs to create mapping of the shared memory to the same virtual address range as the conductor.
workers. Each mailbox is protected by a lock to ensure that no messages are lost and operations are corruption free. Once we create a mailbox associated with conductor and each worker, it does not allow automatic establishment of communication channel between them. There are two ways this can be done: first the conductor and all workers will create their mailbox and then by using MPB communicate the location of their mailbox; the second way is to reserve some memory at the beginning of the shared memory and use it to statically allocate slots where each core will locate its own mailbox by means of core id. Since we have a fixed number of cores on the SCC, we opted to choose the second option. The conductor initialises all the mailboxes in the reserved section of the shared memory, all the workers has a private list of mailbox addresses that is populated with values by calculating offset from beginning of the shared memory.
Once the mailboxes are set-up, the workers request tasks to execute from the conductor and the conductor will fulfil these requests based on demand and task priority. When there are no more messages to be processed, the conductor sends a termination message to all the workers via the mailbox.
3.4.2
Synchronization Primitives
Having access to the atomic instruction to provide synchronisation and protect critical region is one of the requirement of the LPEL, which should be fulfilled by the under- lying hardware. LPEL requires synchronisation at different points, some examples are:
• During the initialisation phase, the conductor/workers use a shared flag to synchronise LUT remapping and the shared memory creation.
• The meta-data of the global allocator in HALLOC is accessed/modified by conductor/workers concurrently.
• The meta-data of the local allocator in HALLOC need to be protected against
concurrent access by multiple threads within the same core.
• The mailbox is an example of producer/consumer paradigm, where messages are added/removed from the queue concurrently. This queue needs to be protected against concurrent access to ensure messages are not lost and to avoid corruption of the queue.
• Streams are used to transfer messages between tasks. Streams are implemented as First In First Out (FIFO) buffers, and these buffers need protection to ensure
integrity of data (during reading/writing to the stream) and its associated counters (used to derive data demand for scheduling purposes).
We already use all the hardware registers provided by SCC for synchronisation. The MPB is used to store the shared flag. We use the T&S registers to implement locks that protect the meta-data of the global allocator. We use the atomic counter registers to implement locks to protect the garbage list and the mailboxes.
We still need more synchronisation primitives to protect the streams and for the core local memory allocator. For this purpose we use POSIX (PTHREAD) mutexes. The SCC runs an OS instance on each core, so we create mutexes with the process sharedattribute set. This allows different workers to access the same mutex, as this access will be seen as it was accessed by different processes. In addition, the function that is called to acquire lock on this mutex i.e. pthread_mutex_lock is replaced by calls to the non-blocking version of the same function pthread_mutex_trylock.
3.5
Chapter Summary
This chapter provided a detailed description of the design and implementation of LPEL for the SCC. We have covered memory management mechanisms including, LUT remapping, memory mapping and shared memory allocation/deallocation. We also covered the concept and the design criteria of HALLOC, a novel hierarchical memory
allocator. We provided an account of how conductor and workers are initialised on the SCC and how we use synchronisation primitives provided by the SCC hardware.
Taxonomy of Resource-aware
Execution
This chapter describes the resource model and different resource load scenarios. We study the influence of load balance on power consumption in § 4.2. We provide formal definitions of load ranges for Reactive Stream Programs (RSPs) with different arrival rates in § 4.5. Furthermore, in § 4.6these definitions are then extended with inclusion of Dynamic Voltage and Frequency Scaling (DVFS), and as such we take into account the impact DVFS has on available resource and system state.