Experiment: Heuristics vs ILP multi-cycle

With the model extended to multiple minor cycles some more experiments were performed to assess the performance of the Worst Fit when compared with the ILP model. The aim of this experiment is simply to assess the performance of all WF and ILP when allocating to multiple minor cycles. In addition, timing data is presented considering the time taken by each approach to find a solution.

Setup

A larger number of synthetically generated task sets were produced. 1000 task sets were generated per 5% utilisation increment. The standard features of the experimental setup are described below:

[25] which provides an unbiased distribution of utilisation values, following standard practice in synthetic task set generation.

• The minor cycle length was set at 25, with the major cycle length set at 100

(TF = 25, TM = 100)

• Task periods were selected at random from either 25, 50 or 100.

• Deadlines were set equal to periods. Di = Ti.

• The LO execution times of each task were produced as follows: Ci(LO) =

Ui/Ti

• For tasks with a criticality greater than the lowest, their HI execution times

were determined by Ci(Li) = Ci(LO) ∗ CF - CF is the criticality factor, a

random value between 1.2 and 2.

• Timing data was recorded to find the average time each heuristic took to find a solution. All timing data was recorded on a 4 core Intel i7 4790K.

• The barrier protocol was implemented for all allocation techniques.

Additionally, we extended Worst Fit to be able to account for multiple minor cy- cles. Allocation begins at the highest criticality level, tasks are allocated to the core with the greatest available capacity, the range of cores available for this allocation

is determined by the period of the task. If TM _{= 100 and T}F _{= 25, a task with}

a period of 50 will be allocated to the core with the most spare capacity in the first two minor cycles and the same in the second two. Once the highest criticality

level is allocated a value of Smaxis found for each minor cycle, allocation of the next

Results

Experiment One Parameters:

• 20 tasks were generated per task-set. • Allocation was made to a 2 core platform.

• Tasks were evenly distributed over two criticality levels. • 4 minor cycles were included per major cycle.

The first experiment is presented as a standard schedulability plot detailing the performance of both techniques. The results are shown in Figure 3.21:

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Utilisation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Schedulability ILP WF

Figure 3.21: A comparison of the performance of all allocation approaches.

We observe WF performing significantly worse than in the single-cycle case. This result was expected as WF allocates tasks in order, an allocation for the pre- vious criticality level may be undesirable for the next, the heuristic is not able to re-allocate. However the ILP approach does not rely on a sequential allocation of tasks, as such it is able to allocate tasks at all criticality levels in the most optimal fashion.

Experiment Two Parameters:

• Multiple experimental runs were performed, each increasing the number of tasks per set in increments of 20, from 20 to 100.

• Allocation was made to a 2 core platform.

• Tasks were evenly distributed over two criticality levels. • 4 minor cycles were included per major cycle.

This experiment investigates the effect of scaling the number of tasks from 20 per set to 100. We present the results in 3.22.

20 40 60 80 100 Number Of Tasks 0 0.2 0.4 0.6 0.8 1 Weighted Schedulability ILP WF

Figure 3.22: A plot illustrating the performance as the number of tasks per set is increased.

Figure 3.22 illustrates that scaling the number of tasks provides a significant boost in schedulability for both ILP and WF. While ILP always out performs WF, we see the results narrowing as the number of tasks is increased. The general increase is due to an increased granularity in the allocation, the utilisation of each task set at each set is the same regardless of the number of tasks, thus the exe- cution time of a task in a task-set of size 100 is likely to be relatively small. This results in task sets which are easier to allocate, this ease of allocation explains the gains made by WF and the overall gains made by both approaches.

Experiment Three Parameters:

• Multiple experimental runs were performed, each increasing the number of cores per set in increments of 2, from 2 to 8.

• All task sets contained 20 tasks.

• Tasks were evenly distributed over two criticality levels. • 4 minor cycles were included per major cycle.

We then investigated the performance when the number of cores are scaled. The results are shown in Figure 3.23.

2 4 6 8 Number Of Cores 0 0.2 0.4 0.6 0.8 1 Weighted Schedulability ILP WF

Figure 3.23: A plot illustrating the performance as the number of cores is increased.

As we have observed previously (see Section 3.1.2) if the number of cores is scaled up without an increase in the number of tasks a significant loss in schedu- lability is observed. We illustrate this with two plots in Figure 3.24. These results again illustrate how additional locations for allocation, created by increasing the core count, require a suitably large number of tasks in order to be efficiently allo- cated.

0 1 2 3 4 5 6 7 8 Utilisation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Schedulability ILP-100T-8C ILP-20T-8C

Figure 3.24: A plot illustrating the improved Schedulability on an 8 core platform when the number of tasks is also increased.

Experiment Four Parameters:

• Multiple experimental runs were performed, each increasing the number of criticality levels by 1, from 2 to 5.

• 20 tasks were generated per task-set. • Allocation was made to a 2 core platform. • 4 minor cycles were included per major cycle.

Experiment four scales the number of criticality levels. The results of this experi- ment are shown in Figure 3.25. In this figure we observe the greatest weakness of WF, it scales poorly as the number of criticality levels is increased. A steady drop in schedulability may also be observed from ILP. A general decrease in schedula-

bility is expected as for each criticality level an additional barrier (point of Smax) is

2 3 4 5

Number Of Criticality Levels

0 0.2 0.4 0.6 0.8 1 Weighted Schedulability ILP WF

Figure 3.25: A plot illustrating the performance as the number of criticality levels is increased.

Experiment Five Parameters:

• Multiple experimental runs were performed, each increasing the number of minor cycles from 4 to 16 in increments of 4.

• 20 tasks were generated per task-set. • Allocation was made to a 2 core platform.

• Tasks were evenly distributed over two criticality levels.

Finally, we investigate the performance as the number of minor cycles is increased. The periods of tasks remain fixed at 25, 50 or 100. Thus we increase the number of minor cycles from 4 to 8, 12 and 16. The results are shown in Figure 3.26. The results here show little change across all techniques. This is likely to be due to the same periods being used for each number of minor cycles. Thus if a set is schedulable on 4 minor cycles, it is also schedulable on 8 if the largest period of any task is equal to the length of 4 minor cycles.

4 8 12 16

Number Of Minor Cycles

0 0.2 0.4 0.6 0.8 1 Weighted Schedulability ILP WF

Figure 3.26: A plot illustrating the performance as the number of minor cycles is increased.

Timing

The schedulability results required further investigation. In short the question be- comes: How efficient is ILP in comparison to Worst Fit? Timing data was taken for both the ILP solver and the WF implementation. While Gurobi does record the time taken to solve the model, we used Matlab’s basic timing tools to take a reading for both the heuristic and the ILP solver to remain consistent. We took execution time data taken from the results shown in Experiment One is presented in Table 3.3:

WF ILP

Average 0.0058 0.0064

Median 0.0058 0.0017

Max 0.0185 58.618

Table 3.3: The average, median and max execution times of WF and ILP (in sec- onds).

The timing data in Table 3.3 illustrate that WF maintains a very low average, median and maximum execution times. All heuristic approaches allocate task sets within a fraction of a second. The ILP approach has an average execution time only slightly higher than the heuristic and a median execution time well below. A small number of outliers push the average execution time of ILP well above the median. However given than the median is very low it is clear that the vast majority of task sets are allocated very quickly within a fraction of a second. We observed the timing outliers at utilisations where task sets are borderline schedulable. While we included no timing cut-off for the ILP solver in Experiment One, we included a cut-off of 60 seconds for the scalability investigation. We observed only 0.3% of models generated reaching this limit over all of the timing data presented. This is a practical consideration as a large number of different experimental runs are required to scale all parameters. We accept that a number of outliers will occur but again observe a consistently low median execution time indicating that the vast majority of task sets are allocated very quickly.

We scaled the size of the task set, the number of cores, the number of criticality levels and the number of minor cycles and considered the execution time increase. We utilised box plots to present the resulting execution time data, the red centre line in each box is the median execution time, with the bottom and top whiskers being the 25th and 75 percentiles respectively. While a number of outliers exsist, this experiment illustrates that the majority of task sets are allocated very quickly. The data is presented in Figures 3.27, 3.28, 3.29 and 3.30 respectively.

It is clear from the average execution times and the box plot data that all ap- proaches, no matter the scaling, take little time to execute for the vast majority of cases. However, we observe that the ILP based approach proves to be quicker to execute in all cases of comparable scale. As all timing values are taken using Matlab’s in-built timing tools, the run-time recorded for ILP accounts for creation of the ILP model and its time running in the solver. The creation of the model is ex- tensive but not exceedingly time consuming, it largely revolves around constructing a sparse matrix to represent the constraints.

WF20 WF100 ILP20 ILP100

Number Of Tasks & Allocation Technique

0 0.01 0.02 0.03 0.04 0.05 0.06

Execution Time (Seconds)

Figure 3.27: A box plot illustrating the range of execution times taken for each approach as the number of tasks is increased.

WF2 WF8 ILP2 ILP8

Number Of Cores & Allocation Technique 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02

Execution Time (Seconds)

Figure 3.28: A box plot illustrating the range of execution times taken for each approach as the number of cores is increased.

WF2 WF5 ILP2 ILP5 Number Of Crit Levels & Allocation Technique 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02

Execution Time (Seconds)

Figure 3.29: A box plot illustrating the range of execution times taken for each approach as the number of criticality levels is increased.

WF4 WF16 ILP4 ILP16

Number Of Minor Cycles & Allocation Technique 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04

Execution Time (Seconds)

Figure 3.30: A box plot illustrating the range of execution times taken for each approach as the number of minor cycles is increased.

As our ILP models have no optimisation function, the solver is able to quickly find a solution and thus the overall runtime is very low (See the below ’Feasibility vs Optimisation’ section for more insight). On the other hand, worst fit relies solely on a large number of iterations to find suitable allocations, as such its execution time is slightly larger than the ILP solution in this case.

We have shown that scaling the number of tasks, cores, criticality levels and minor cycles does have an impact on the execution time. However, the time taken to find feasible solutions remains extremely fast. In addition, given that this multiple minor cycle case is more complex than the single cycle, the results illustrating very fast performance also apply to the prior experiments in this chapter. Timing data was recorded for the experiment in Section 3.1.4, it shows extremely fast performance across all techniques, we included the details in Appendix A.

Summary

To conclude we summarise the results of the experiment, both the schedulability result and the timing result.

• Firstly: Our ILP implementation outperforms Worst Fit by a significant margin, particularly when scalability via the number of criticality levels is taken into account. This makes ILP an more effective candidate for allocation.

• Secondly: The timing data illustrates that, in this case our ILP approach ac- tually out-performs WF in terms of execution time. This is likely down to implementation, ILP models are generated and run as feasibility tests using Gurobi[45], WF must undergo a large number of iterations in Matlab in order to find successful allocations. It may be possible to produce a faster imple- mentation of worst fit, however both approaches are equally fast with a very small number of outliers.

While we do not claim that no better heuristic might exist, with the increase in schedulability offered by ILP and the very low execution times the logical choice is to make use of the optimal solver.

Feasibility vs Optimisation

They key reason for the efficient run-time of our ILP models, is their lack of an optimisation function. In effect these become feasibility tests rather than optimisa- tions, the question becomes ’is there a search space?’ rather than ’which point maximises a particular function?’. In these cases we are interested in constraint satisfaction rather than optimisation. The ability to generate models using tools such as Matlab makes ILP a compelling option when investigating the feasibility of a given task set on a mixed criticality cyclic executive. Model generation tools allow for the rapid development of system models which provide a system designer with a means of investigating possible task allocations during all stages of the design and development process.

Task Ordering With Criticality Level

During task allocation, tasks are allocated to the appropriate number of minor cy- cles, and within these to a core. Within this, each minor cycle on each core, is split via the barrier separating the execution of each criticality level. The ordering of execution within tasks of the same criticality level is not explicit, schedulability simply requires the tasks to fit in some order. As the barrier is a dynamic structure, even if a criticality change occurs, the barrier will still be invoked and some work of the following criticality level will execute. In a dual criticality context, logically, work scheduled later in the LO criticality execution is more likely to be effected by a criticality change (HI work overrunning the pre-computed point S). If a critical- ity change occurs, HI work overruns the pre-computed S point by some amount, once that work completes LO work begins execution. Work scheduled first in the LO mode is less likely to suffer during a criticality change. The notion of Impor- tance could be used as a means of ordering tasks within a criticality level. Such an ordering could be applied to any criticality levels below the highest, with tasks assigned a higher importance executing earlier in the schedule. Ordering these tasks is simple, as the schedule is viable no matter which order the tasks execute. In addition to simply ordering tasks once a schedule has been established, importance could be used as a factor for optimisation. The solver could be tasked with ensuring that where possible, higher importance tasks execute before lower

importance. This could either be modelled as firm constraints, where such an order is mandatory or as a desired order which might suffer if the only schedulable solution breaks it. It is worth noting that if modelled as firm constraints optimality is lost (in that it may not find a feasibility schedule, even if one exsists).We return to the issue of optimisation in Chapter 5.