Checkpointing Overheads

Pre- and post-execution overheads may minimal, however there are other overheads that may increase the total completion time of a cloud job during its execution on the ad hoc guest. We perceived that such overheads existed by comparing the job execution time (see Figure 6.7) and the completion time recorded by BOINC system as part of

6.5. Platform Performance 169

the Job Service project; we define the latter as the BOINC execution time. Note that the former, measures only the time the job spent executing and not the wallclock time between the start and end time of the cloud job.

By default, BOINC records the time when it sends a job to an ad hoc guest and the time when it receives the job’s the results. The difference between the start and end times is therefore the BOINC execution time and this can obtained either by querying the Job Service database or through the project’s administration web interface.

In order to determine the performance overheads of our ad hoc client while an ad hoc guest executes a cloud job, we submitted each benchmark to our EDIM1 ad hoc cloud five times and collected their BOINC execution times once completed. During the execution of the benchmark, the ad hoc client was set to periodically checkpoint once per minute and distribute the compressed checkpoint to three other ad hoc hosts.

The results of all experiments were averaged and these are shown in Table 6.3.

Benchmark

Table 6.3: Overheads Unaccounted for when Executing Cloud Jobs

We see that by subtracting the cloud job execution time from the BOINC execution time, we obtain the time introduced by the overheads associated with the ad hoc client.

The BOINC execution time does however include the time for the ad hoc server to send the cloud job to the ad hoc guest and for the guest’s BOINC client to return the results. We know from Figure 6.9 that the time to perform these operations on EDIM1 is 34 seconds and 1 second respectively. Therefore we can update the additional time introduced by the ad hoc client to exclude these overheads. This leaves an unaccounted period of time introduced by other ad hoc client overheads.

Table 6.3 also shows that these overheads differ dependent on the type of cloud job running. Although an ad hoc client is composed of many threads concurrently executing various tasks, we can rule out this as a primary factor of variable overheads

between each benchmark. The only feature of an ad hoc client that could introduce such overheads is periodic checkpointing. We investigate what affect periodic check-pointing has on the total completion time of a cloud job.

6.5.3.1 Checkpoint Downtime

Periodic checkpointing is one of the many important processes used improve the re-liability of the inherently unreliable ad hoc cloud model. While we show later in the chapter that this is an effective method of providing reliability, periodic ing does however have one potentially significant limitation; during each checkpoint-ing operation, the virtual machine must be paused therefore suspendcheckpoint-ing the executcheckpoint-ing cloud job; we define this as the checkpoint penalty.

Previously in Section 3.4.2 of Chapter 3, we investigated what affects of taking checkpoints each minute had on the storage space of a volunteer host that uses V-BOINC. We found that for CPU, Memory and I/O stress benchmarks, the average time to take a checkpoint was approximately 1.5 seconds while the same operation took the Disk benchmark on average 25 seconds to complete; the exact figures can be found in Table 3.2.

For the respective benchmarks, the downtime incurred while taking checkpoints is low, however taking per minute checkpoints may not be optimal for both the cloud job or the ad hoc host. For example, as a cloud job consumes a larger portion of storage space or memory, or the checkpointing frequency decreases, the size of the checkpoint may increase, in turn increasing the time to take the checkpoint.

In order to determine the potential checkpoint times for a range of checkpointing frequencies, we varied the frequency over the period of one hour for each benchmark.

This was performed by instantiating our V-BOINC virtual machine with 1 GB of mem-ory on our MacBook Pro 2007 general purpose host and running each of our stress benchmarks individually while taking either 1, 2, 3, 4, 6, 12, 30 or 60 equally spaced checkpoints per hour.

For example, we started by taking 60 checkpoints per hour while the CPU bench-mark was executing. In the second hour, the same benchbench-mark was executed while 30 checkpoints were taken, and so on; this process was repeated for each benchmark.

After each hour, the virtual machine was terminated and a new one was instantiated for the following hour to ensure each experiment was independent and that no virtual machine or host caching affected the results of the subsequent experiment.

Checkpoint times were obtained by the ad hoc client via the UNIX-based time

com-6.5. Platform Performance 171

mand when executing the VirtualBox snapshot function. Note that as this experiment was performed on a different host which had a different virtual machine configuration from the V-BOINC checkpoint experiment (see Section 3.4.2) , the per minute check-point times for each benchmark are slightly different to those outlined in Chapter 3.

Table 6.4 shows the minimum and maximum checkpoint times experienced when exe-cuting the benchmarks while a varying number of checkpoints were taken each hour.

Benchmark Min. Checkpoint Time (s) Max. Checkpoint Time (s)

CPU 2.3 (CF 60) 13.21 (CF 1)

Memory 5.83 (CF 4) 16.46 (CF 2)

I/O 2.37 (CF 60) 13.05 (CF 4)

Disk 43.4 (CF 30) 54.71 (CF 4)

Table 6.4: Checkpoint Frequency and Time Relationship

Table 6.4 only displays the minimum and maximum checkpoint times as there is no strict correlation between the checkpointing frequency and checkpoint time; we indi-cate which checkpoint frequency (CF) produces the minimum and maximum check-point times in Table 6.4. We sometimes see that a pattern can emerge where lower checkpoint frequencies produce higher checkpoint times and vice versa. For exam-ple, we see that the maximum checkpoint times are produced when the checkpoint frequency is 4 or lower, however the checkpoint frequencies that produce the minimal checkpoint times are not consistent for each benchmark.

We see that a checkpoint frequency of 60 produces the minimal checkpoint times for the CPU and I/O benchmarks while a checkpoint frequency of 4 produces the min-imal checkpoint time for the Memory benchmark. Therefore the differences between the minimum and maximum checkpoint times are partially dependent on the check-point frequency, but not for all classes of application.

Despite the checkpointing frequency employed, we can be reassured that the check-point downtime penalties of periodic checkcheck-pointing are minimal for the CPU, Memory and I/O benchmarks. However, we see that the Disk benchmark has a high checkpoint downtime penalty of at most 54.71 seconds per checkpoint. If per minute checkpoints were taken while the ad hoc guest executes a disk-intensive benchmark, we would see that for every minute, the cloud job could potentially be suspended for the following 54.71 seconds.

6.5.3.2 Checkpoint Size

During the same experiment, we also recorded the checkpoint sizes dependent on the checkpoint frequency used. The results are shown in Table 6.5. Note that as this experiment was performed on a different host which had a different virtual machine configuration from the V-BOINC checkpoint experiment (see Section 3.4.2), the per minute checkpoint sizes for each benchmark are slightly different to those outlined in Chapter 3.

Benchmark Checkpoint Size (MB) Snapshots per Hour CPU Memory I/O Disk

60 37.2 56.6 36.9 89.5

30 37.3 43.5 37.7 89.7

12 37.8 44.0 39.0 90.1

6 38.2 45.0 38.5 92.2

4 38.6 43.3 41.2 91.0

3 38.2 43.3 37.0 2370.9

2 36.5 57.8 38.4 1574.8

1 37.0 44.3 36.9 1393.3

Table 6.5: Checkpoint Interval and Size Relationship

We see that in most cases, a decrease of the checkpoint frequency does not necessarily increase the size of the checkpoint; this is particularly true for benchmarks that do not write to disk or consume large amounts of memory. Surprisingly, the Memory benchmark also follows this pattern and produces checkpoint sizes of approximately 44 MB despite utilizing over 90% of the virtual machine’s memory. Unsurprisingly, as the Disk benchmark continues to write data to the virtual disk, the checkpoint size increases. A dramatic increase is observed when the checkpoint frequency is three per hour or fewer; we attribute this to the behaviour of the benchmark. We can therefore be reassured that if a checkpoint frequency of 4 per hour or higher is used by each ad hoc client, the checkpoint sizes and downtime penalties will be relatively small.

6.5.3.3 The Near-Optimal Checkpoint Frequency

This therefore introduces the problem of how to decide which checkpoint frequency is best with the aim of reducing both the checkpoint time and size; we define this

6.5. Platform Performance 173

as the near-optimal checkpoint frequency. However, the near-optimal checkpointing frequency is not only a factor of checkpoint sizes and capture times, but also the desired reliability of the ad hoc cloud and the load placed on the ad hoc host’s CPU and network by checkpoint compression and distribution respectively.

A high checkpoint frequency will produce a greater number of checkpoints that must be distributed to other ad hoc hosts. This in turn will increase the likelihood of a cloud job completing if the job is interrupted by ad hoc host or guest termina-tions or failures. This will however introduce greater CPU and network loads. A low checkpoint frequency will produce fewer checkpoints, which in some cases may be larger in size, but the reliability of the ad hoc platform would suffer. Calculating the near-optimal checkpoint frequency for each application submitted to the ad hoc cloud would be extremely difficult, therefore we only outline a possible solution to provide a rough estimate of the near-optimal frequency for each class of stress benchmark. We base our assumptions on the data presented in Tables 6.4 and 6.5.

Our analysis first begins by discarding checkpoint frequencies that are 3 or less for the Disk benchmark due to sudden increase of checkpoint size; it would be unwise to distribute this amount of data over the network periodically. Furthermore, we also discount checkpoint frequencies 4 or less for all benchmarks as it is unreasonable to assume that if the cloud job fails and its ad hoc guest is restored on another ad hoc host, the ad hoc cloud user would have to wait an additional 15 minutes while the cloud job returns to the previous state before the ad hoc guest failed; we define this as the re-computation overhead.

As checkpoint frequencies greater than 4 per hour result in similar checkpoint sizes and capture times, reducing network load and achieving a desired reliability level are the only entities that must now be factored into determining the near-optimal check-point frequency; we omit CPU load in this decision making process as we assume that compressing checkpoints no greater than 100 MB consumes little CPU resources. We also assume that by compressing checkpoints as a .tar.gz file with a low compression ratio of 5:4, produces an 80 MB compressed file, we can estimate the amount of likely checkpointing traffic sent over the network to other ad hoc hosts per hour. Figure 6.10 shows our estimations.

We see that if 60 checkpoints are taken per hour and each is sent to one other ad hoc host, 4.6 GB will be distributed every hour until the benchmark completes. Similarly, in the case when 30, 12 and 6 checkpoints are taken every hour, the total amount of data distributed to a single ad hoc host would be 2.3 GB, 0.9 GB and 0.4 GB respectively.

Figure 6.10: Estimated Checkpoint Data Transfer Per Hour

However, our implementation specifies that the P2P Scheduler selects at least three other ad hoc hosts to receive a single checkpoint to provide a reliable environment for cloud job execution. Therefore an ad hoc host must transfer a total of 14.06 GB, 7.03 GB, 2.81 GB or 1.4 GB if 60, 30, 12 or 6 checkpoints are taken every hour, respectively.

We believe that in most cases, an ad hoc host will not have to send checkpoints to more than six other ad hoc hosts at a time due our ad hoc scheduler’s scheduling rule that the most reliable hosts are selected first to receive checkpoints. For example, in the event that the six most reliable ad hoc hosts each have a probability of 60% of failing or terminating at any time, each ad hoc host in the ad hoc cloud must send every checkpoint to these six ad hoc hosts in order to satisfy the 95% successful completion rate for cloud jobs.

In this possible worst case scenario, we see that if the checkpoint frequency is 60 or 30 per hour, an ad hoc host must transfer a total of 28.12 GB and 14.06 GB respectively per hour to other ad hoc hosts. This is in contrast to a total 2.81 GB of data transferred per hour when the checkpoint frequency is 6 per hour. With the aim of striking a balance between desired reliability and reducing network load, we believe that taking and distributing 60 or 30 checkpoints per hour results in a significant amount of data being transferred over the network. Furthermore, we assume that a checkpointing frequency of 6 per hour (i.e. 1 checkpoint every 10 minutes) does not provide enough reliability and that the re-computation overhead is still quite high.

6.5. Platform Performance 175

Therefore we believe that a reasonable estimation of the near-optimal checkpoint-ing frequency based on checkpoint size, capture time, desired reliability for a job and the reduction of load for CPU and network resources is 12 checkpoints per hour for our particular benchmarks. It is important to note however, that this estimation is based on the results from our benchmarks, which may be atypical in terms of checkpoints sizes compared with normal workloads and have significantly different resource demands or usage patterns. Furthermore, we realize that our chosen static near-optimal checkpoint frequency may not be optimal for small cloud jobs that complete in under five minutes, for example. We also realize that the network of EDIM1 may be significantly different from those of commodity networks, however as mentioned in Section 1.5 of Chapter 1, we do assume that in most cases the ad hoc cloud will be deployed on a LAN that offers reasonable performance, e.g organization and research institution networks.

Currently, the chosen near-optimal checkpoint frequency is a static value set within our ad hoc cloud implementation however we aim to dynamically adjust this frequency based on the reliability of the ad hoc host executing the cloud job. For example, as the reliability of the ad hoc host increases, fewer checkpoints can be taken and subse-quently distributed, and vice versa; we leave the addition of this functionality as future work.