Benchmarking the ad hoc Cloud

We now investigate the performance of running cloud jobs in variety of different con-figurations on EDIM1 in order to determine the extent of overheads introduced by the ad hoc client. Similar to the V-BOINC experiments outlined in Section 3.4 of Chapter 3, we ran the CPU, Memory, I/O and Disk benchmarks in the following configurations:

• native execution on the EDIM1 node,

• execution on a virtual machine (VM) on an EDIM1 backend node. The virtual machine has 2 CPUs and 2 GB of memory,

• execution via V-BOINC and the V-BOINC client. The V-BOINC virtual ma-chine also has 2 CPUs and 2 GB of memory,

• execution via the ad hoc cloud. A single ad hoc client’s components and threads execute while periodic 1 minute checkpoints are distributed to three other ad hoc hosts,

• execution on Amazon EC2. A single m1.medium instance has 1 vCPU with 2 ECUs and 3.75 GB of memory.

Each benchmark was executed five times in each of the configurations above and on its completion, each benchmark would output its execution time, i.e the time it spent exe-cuting; we define this as the cloud job execution time. Note that each stress benchmark was executed as a single process and that the Memory benchmark allocated 2 GB of memory, e.g. –vm-bytes 2048M. In the case of running each benchmark via the ad hoc

6.5. Platform Performance 163

client, the cloud job execution time does not include other factors that may affect the total time a cloud job is present in the ad hoc cloud. For example, the time a job awaits to be scheduled to an ad hoc host, the time to configure and instantiate a virtual ma-chine, the time taken by periodic checkpointing or the time caused by virtual machine migration; we discuss such overheads later in the chapter. The average cloud job exe-cution times of each benchmark are shown in Figure 6.7. We display 95% confidence intervals to show that in most cases, the true mean will lie within the specified range.

Figure 6.7: EDIM1 Benchmark Results and ad hoc Client Overhead

Figure 6.7 shows the cloud job execution times of the benchmarks running in their respective configurations. As expected, we see that running each benchmark natively typically offers the lowest execution time with the exception of the I/O benchmark.

Similar to the results obtained by evaluating V-BOINC, this phenomenon may be caused by the virtualization technology having better caching mechanisms than the native host. We also see that running a virtual machine on EDIM1 introduces sig-nificant virtualization overheads. This is shown by the difference of execution times between the Native and VM configurations; the greatest virtualization overhead occurs when the Memory benchmark takes almost half an order of magnitude longer to com-plete when using virtualization. The virtualization overheads experienced on EDIM1 are significant and are due to the use of Intel Atom CPUs in the backend nodes where hardware virtualization is not supported.

As expected, we also see that the cloud job execution time is typically lower on Amazon EC2 than any configuration that uses virtualization on EDIM1, with the ex-ception of the I/O benchmark. As outlined in Chapter 4, the overhead between the VM and V-BOINC configurations is minimal, but we see that executing the same

bench-marks via the ad hoc client, the overhead is substantially greater for the Memory and Disk benchmarks.

At first glance, it would reasonable to assume the additional functionality of an ad hoc client in comparison to a V-BOINC client would explain the additional over-heads. For example, periodic checkpointing, compressing and distribution may con-sume memory and disk space that otherwise would be used by the executing cloud job. Furthermore, the additional operations that occur in the background, such as the Resource Monitor, Running Detector, Accessible Detector and many of the Listener components waiting on a specific event to occur, may also consume resources required by the virtual machine and cloud job.

Although these additional functionalities do introduce overheads, they are negligi-ble in comparison to the additional and significant overheads introduced by executing the Memory and Disk benchmarks via the ad hoc client on EDIM1. We hypothesise that during these benchmarks, the combination of a large number of instruction calls to the hypervisor (which must be trapped by the hypervisor, translated by the software, sent to the hardware and the request returned via the same route), as well as the ex-ecution of the virtual machine, the internal cloud job and the ad hoc clients are large enough to introduce the significant slowdown. However, in the event that a workload demands more resources from the ad hoc host, the overhead of the ad hoc client will not increase as the virtual machine will only be offered resource capacity that is not used by host processes or by BOINC and the ad hoc client. It is worth noting that if insufficient resources are available for the ad hoc client to consume, the execution times for workloads may increase, however by migrating the virtual machine to a more viable host, workload performance reductions can be minimized.

However, in order to prove the ad hoc cloud does not introduce significant over-heads when executing both of these benchmarks, we executed the same benchmarks on four other hosts each with different hardware specifications. We define these hosts as general purpose hosts as they have hardware specifications comparable to standard models of laptops and Desktops; the class of hosts we believe host owners are likely to deploy an ad hoc cloud on. The specifications of these hosts are shown in Table 6.2.

Table 6.2 shows the variety of hardware specifications of our general purpose hosts, however most importantly, hardware-assisted virtualization is available on all of these hosts to help increase the performance of the hypervisor. In order to confirm that our ad hoc client does not introduce significant overheads, we executed the stress benchmarks on an ad hoc cloud deployed on our general purpose hosts and recorded the cloud job

6.5. Platform Performance 165

Name Processor Type Memory VT-x/AMD-V

MacBook Pro 2007 2.2 GHz Intel Core 2 Duo 2 GB Yes MacBook Pro 2010 2.4 GHz Intel Core 2 Duo 8 GB Yes Dell Optiplex 755 3 GHz Intel Core 2 Duo 4 GB Yes Dell Optiplex 790 3.1 GHz Intel Core i3-2100 4 GB Yes

Table 6.2: General Purpose Host Benchmark Results

execution times. As the results obtained were similar for each host, we only outline the results obtained from an ad hoc cloud consisting of a server and client operating on the Dell Optiplex 790 and 755 hosts respectively. We do not display results for the CPU benchmark as it failed to complete on a virtual machine running on our general purpose hosts. Our results are shown in Figure 6.8.

Figure 6.8: Dell Optiplex Benchmark Results and Ad hoc Client Overheads

Firstly we see that by executing these benchmarks on a general purpose host, the cloud job execution time is substantially reduced. For example, executing the Memory, I/O and Disk benchmarks via the VM configuration on the Dell Optiplex 755 takes approx-imately 11, 13.4 and 12.8 minutes less respectively.

Secondly, we see that in comparison to EDIM1, the performance overheads intro-duced when executing the Memory benchmark via the ad hoc client on a general pur-pose host is lower; this overhead is reduced from a cloud job taking 4.5 times longer to complete on EDIM1 to 2.5 on the general purpose host. Similarly, we also see that the

performance overheads introduced when executing the Disk benchmark via the ad hoc client on a general purpose host is lower in comparison with running the same bench-marks on EDIM1; this overhead is reduced from a cloud job taking 3.3 times longer to complete on EDIM1 to 2.5 on the general purpose host.

Conversely, the overhead increases when executing the I/O benchmark via the ad hoc client on the general purpose host; this overhead previously took 1.3 times longer to complete on EDIM1 but now takes 4.2 times longer. Note that the execution time of this benchmark on an ad hoc client is still lower than the native execution on the general purpose host.

Despite the more realistic overheads observed from a general purpose host, the cloud job execution times for each benchmark are lower when executing on a general purpose host in comparison to executing the same benchmarks on an Amazon EC2 m1.medium instance that has similar resources to the Dell Optiplex 755. The ad hoc client running on this host executes the Memory, I/O and Disk benchmarks, approxi-mately 68%, 98% and 21% faster than on the m1.medium instance.

By executing the series of stress benchmarks on EDIM1 and a number of hosts that are likely to be used by a large majority who employ ad hoc cloud computing, we have shown that the overheads introduced by the ad hoc client are minimal for a cloud platform of this nature. Furthermore, we have shown that an ad hoc client running on a general purpose host is able to perform better than executing on an Amazon EC2 instance with comparable resources.

We investigate whether the ad hoc cloud is still able to outperform Amazon EC2 later in the chapter, after we outline the affects of the many unique overheads associated with the ad hoc cloud have on a cloud job’s execution time.