Experiment Results and Discussion

We performed four sets of experiment. In each set, one independent variable varied while the rest were set to the default values presented in Table 8.1. The experiment was performed five times for each value of the selected independent variable. The dependent variables, i.e. solving time and total cost, were recorded after each run.

Varied Numbers of Jobs

In this set of experiment, we selected 10 different values corresponding to the different numbers of job which varied from 5 to 14. The results are illustrated by Figure 8.1.

Figure 8.1a presents the solving times of three approaches when the number of jobs varies. The x-axis shows the number of jobs which goes from 5 to 14. The y-axis shows the solving time in milliseconds. Notably, log-10 scale is used for the y-axis.

It is evident that the solving times of the Exact approach are higher than the Hybrid approach’s which are higher than the solving times of the Heuristic approach. However, the difference between the Exact and Hybrid approaches is substantial compared to the difference between the Hybrid and Heuristic approaches. For instance, when there are 5 jobs, the solving time of the Exact approach (804.2 milliseconds) is nearly 14 times higher the solving time of the Hybrid approach (58.4 milliseconds), which is only 1.4 times higher than the solving time of the Heuristic approach (41 milliseconds). Furthermore, when there are 14 jobs, the solving time of the Exact approach (545362.8 milliseconds) is about 7400 times higher the solving time of the Hybrid approach (73 milliseconds), which is only 1.2 times higher than the solving time of the Heuristic approach (62.4 milliseconds).

The Exact approach’s solving time not only are higher than other approaches but also suffers substantial growth. As the number of jobs increases from 5 to 14, the solving time of the Exact approach grows 67714%, from 804.2 milliseconds to more than 540000 milliseconds, i.e. more than 9 minutes. On the other hand, the solving times of the Hybrid and Heuristic approaches only increase 25% and 52% respectively.

Figure 8.1b presents the total cost of the scheduling plans produced by the three ap- proaches when the number of jobs varies. The x-axis shows the number of jobs while the y-axis shows the total cost. It can be seen that as the number of jobs increases, the cost increases as well.

The Exact approach always manages to find the scheduling plan with the cheapest cost. Interestingly, the costs of Hybrid and Heuristic approaches are nearly identical. In average, the cost of using the Exact approach is 3.5% and 4.9% cheaper compared to the Hybrid and

Heuristic approach respectively. Using the Hybrid approach results in the cost which is 1.1% cheaper compared to the Heuristic approach.

Varied Numbers of Tasks

In this set of experiment, we selected 10 different values corresponding to the different numbers of tasks which varied from 100 to 1000. Since there are 5 jobs by default, the total number of tasks varies from 500 to 5000 tasks. The results are illustrated by Figure 8.2.

Figure 8.2a presents the solving times of three approaches when the number of jobs varies. The x-axis shows the number of tasks which goes from 500 to 5000. The y-axis shows the solving time in milliseconds. Notably, log-10 scale is used for the y-axis.

It is evident that the solving times of the Exact approach are higher than the Hybrid approach’s which are higher than the solving times of the Heuristic approach. However, the difference between the Exact and Hybrid approaches is substantial compared to the difference between the Hybrid and Heuristic approaches. For instance, when there are 500 tasks, the solving time of the Exact approach (927 milliseconds) is more than 15 times higher the solving time of the Hybrid approach (58.8 milliseconds), which is only 1.6 times higher than the solving time of the Heuristic approach (37.4 milliseconds). Furthermore, when there are 5000 tasks, the solving time of the Exact approach (259813.2 milliseconds) is nearly 3000 times higher the solving time of the Hybrid approach (86.4 milliseconds), which is only 1.05 times higher than the solving time of the Heuristic approach (82.6 milliseconds).

The Exact approach’s solving time not only are higher than other approaches but also suffers substantial growth. As the number of tasks increases from 5 to 14, the solving time of the Exact approach grows 27900%, from 927 milliseconds to more than 250000 milliseconds, i.e. more than 4 minutes. On the other hand, the solving times of the Hybrid and Heuristic approaches only increase 47% and 120% respectively.

Figure 8.2b presents the total cost of the scheduling plans produced by the three ap- proaches when the number of tasks varies. The x-axis shows the number of tasks while the y-axis shows the total cost. It can be seen that as the number of tasks increases, the cost increases as well.

Figure 8.2b shows that the total costs of three approaches are almost identical. In average, the cost of using the Exact approach is 1.8% and 2.5% cheaper compared to the Hybrid and Heuristic approach respectively. Using the Hybrid approach results in the cost which is 0.7% cheaper compared to the Heuristic approach.

(a) Average solving times of three scheduling approaches when the number of jobs varies. The x-axis presents the number of jobs, the y-axis presents the solving time in log scale.

(b) Average cost of three scheduling approaches when the number of jobs varies. The x-axis presents the number of jobs, the y-axis presents the total cost of a cloud cluster constructed to execute all the jobs. The error bars illustrate the standard errors.

Fig. 8.1 Experiment Result of Three Scheduling Approaches When the Number of Jobs Varies

(a) Average solving times of three scheduling approaches when the number of tasks varies. The x-axis presents the number of tasks, the y-axis presents the solving time in log scale.

(b) Average cost of three scheduling approaches when the number of tasks varies. The x-axis presents the number of tasks, the y-axis presents the total cost of a cloud cluster constructed to execute all the jobs. The error bars illustrate the standard errors.

Fig. 8.2 Experiment Result of Three Scheduling Approaches When the Number of Tasks Varies

Varied Numbers of Instance Type

In this set of experiment, we selected 10 different values corresponding to the different numbers of instance types which varied from 5 to 14. The results are illustrated by Figure 8.3.

Figure 8.3a presents the solving times of three approaches when the number of jobs varies. The x-axis shows the number of types which goes from 5 to 14. The y-axis shows the solving time in milliseconds. Notably, log-10 scale is used for the y-axis.

It is evident that the solving times of the Exact approach are higher than the Hybrid approach’s which are higher than the solving times of the Heuristic approach. However, the difference between the Exact and Hybrid approaches is substantial compared to the difference between the Hybrid and Heuristic approaches. For instance, when there are 5 instance type, the solving time of the Exact approach (1611.6 milliseconds) is more than 26 times higher the solving time of the Hybrid approach (60.8 milliseconds), which is only 1.3 times higher than the solving time of the Heuristic approach (44.6 milliseconds). Furthermore, when there are 14 instance type, the solving time of the Exact approach (1494 milliseconds) is more than 37 times higher the solving time of the Hybrid approach (39.8 milliseconds), which is only 1.7 times higher than the solving time of the Heuristic approach (23.2 milliseconds).

Figure 8.3a also shows that increasing the number of available instance types does not significantly increase the solving time. In other words, the number of instance type does not affect the solving time as much as the number of jobs and the number of tasks do.

Figure 8.3b presents the total cost of the scheduling plans produced by the three ap- proaches when the number of tasks varies. The x-axis shows the number of instance types while the y-axis shows the total cost. Similar to the solving time, the total cost seems to be not affected by the increase in the number of instance types. Notably, the plot seems to vary due to the random process that creates applications, jobs, and task execution times.

The figure shows that the Exact approach always has the lowest cost in comparison to the other approaches, whose costs are almost identical. In average, the cost of using the Exact approach is 4.8% and 5.3% cheaper compared to the Hybrid and Heuristic approach respectively. Using the Hybrid approach results in the cost which is 0.5% cheaper compared to the Heuristic approach.

Varied Deadlines

In this set of experiment, we selected 10 different values corresponding to the different deadlines which varied from from 600 to 1500 seconds. More specifically, each value

(a) Average solving times of three scheduling approaches when the number of instance types varies. The x-axis presents the number of instance types, the y-axis presents the solving time in log scale.

(b) Average cost of three scheduling approaches when the number of instance types varies. The x-axis presents the number of instance types, the y-axis presents the total cost of a cloud cluster constructed to execute all the jobs. The error bars illustrate the standard errors.

Fig. 8.3 Experiment Result of Three Scheduling Approaches When the Number of Instance Types Varies

denoted the amount of time within which all tasks of a job must be fully executed. The results are illustrated by Figure 8.4.

Figure 8.4a presents the solving times of three approaches when the number of jobs varies. The x-axis shows the deadline which goes from 600 to 1500. The y-axis shows the solving time in milliseconds. Notably, log-10 scale is used for the y-axis.

It is evident that the solving times of the Exact approach are higher than the Hybrid approach’s which are higher than the solving times of the Heuristic approach. However, the difference between the Exact and Hybrid approaches is substantial compared to the difference between the Hybrid and Heuristic approaches. For instance, when the deadline is 600 seconds, the solving time of the Exact approach (2122.6 milliseconds) is more than 41 times higher the solving time of the Hybrid approach (51.6 milliseconds), which is only 1.1 times higher than the solving time of the Heuristic approach (46.6 milliseconds). Furthermore, when the deadline is 1500 seconds, the solving time of the Exact approach (197.6 milliseconds) is more than 19 times higher the solving time of the Hybrid approach (10.2 milliseconds), which is only 2 times higher than the solving time of the Heuristic approach (2.04 milliseconds).

Figure 8.4a also shows that the solving time decreases as the deadline increases. In average, the solving times of the Exact, Hybrid, and Heuristic approaches decreases 90.7%, 80.2%, and 89% respectively. This can be explained as follow: since the deadline increases, it is possible to assign more tasks to be executed on each instance. This results in the lower number of instances required to execute all tasks. As a result, the scheduling approaches need to consider the lower number of instances, which reduces the solving time.

Figure 8.4b presents the total cost of the scheduling plans produced by the three ap- proaches when the deadline varies. The x-axis shows the deadline while the y-axis shows the total cost. Similar to the solving time, the total cost decreases as the deadline increases with the same reason, i.e. lower number of VMs is required to execute all tasks, which results in the lower total cost.

The figure shows that the Exact approach always has the lowest cost in comparison to the other approaches, whose costs are almost identical. In average, the cost of using the Exact approach is 6.6% and 7.6% cheaper compared to the Hybrid and Heuristic approach respectively. Using the Hybrid approach results in the cost which is 1% cheaper compared to the Heuristic approach.

8.2.3 Discussion

In the previous section, we have presented the experiment results comparing three proposed scheduling approaches. Based on the results, this section presents the remarks and conclusion, some of which are previously hypothesised.

(a) Average solving times of three scheduling approaches when the deadline varies. The x-axis presents the length of deadlines, the y-axis presents the solving time in log scale.

(b) Average cost of three scheduling approaches when the length of deadlines varies. The x-axis presents the length of deadlines, the y-axis presents the total cost of a cloud cluster constructed to execute all the jobs. The error bars illustrate the standard errors.

The total cost is affected by the intensity of the submission. As shown by Figures 8.1b and 8.2b, intensive workloads, i.e. higher number of tasks/jobs or shorter deadlines, also results in higher costs, since it requires more instances. On the contrary, high deadline, i.e. low urgency, can reduce the number of instances and result in lower cast. as shown by Figure 8.4b

The solving time is affected by the intensity of the submission.As shown by Figures 8.1a and 8.2a, the solving times of all three approaches increase when the number of jobs or tasks increases. Moreover, as shown by Figure 8.4a, the solving time can decrease if the deadline decrease, i.e. the urgency decreases.

The number of instance types does not affect the solving time and cost.As shown in Figure 8.3, there is no correlation between the number of instance types and the solving time. This can be explained as all approaches are able to quickly focus on the certain instance types which discarding the rest. Similarly, the total cost is not affected by the number of instance types.

The Exact approach is the most cost efficient. In all of our experiment, the Exact approach is able to achieve the lower cost than the other two. It can reduce cost as much as 6.6%.

The Exact approach is not scalable. As shown by Figures 8.1a and 8.2a, the solving time of the Exact approach grow significantly. It can take few minutes to find a solution. This is not ideal for a real time system which should schedule job execution as soon as possible.

Both the Hybrid and Heuristic approaches are scalable. In our experiments, both approaches are able to produce the scheduling plan within 100 milliseconds at any type of workload intensity. In other words, they both scale well and are suitable for a real time system which requires quick scheduling decision making.

The Hybrid approach is the best of both world.Even though the Hybrid approach is able to find a cheaper scheduling plan compared with the Heuristic approach, its cost saving is not significant since it always around 1%. However, taking into account that the solving time of the hybrid approach is still very fast (less than a second), there is no disadvantage of using this approach.

As a conclusion, we believe the both the Hybrid and Heuristic approach are suitable for a real time system which not only needs to handle intensive workload but also is required to make quick scheduling decision. On the other hand, even though the Exact approach can achieve cost saving up to 6.6%, its solving time is too high to be suitable highly intensive system.