Cloud Computing Performance. Benchmark Testing Report. Comparing ProfitBricks vs. Amazon EC2

(1)

Cloud Computing Performance

Benchmark Testing Report

Comparing ProfitBricks vs. Amazon EC2

April 2014

(2)

Topics ……….Page

Introduction ... 1

Executive Summary ... 2

Benchmark Testing Methodology ... 2

UnixBench Results ... 4

DBENCH Results ... 7

Iperf Results ... 10

Next Steps ... 13

We invite you to contact one of our Cloud Computing consultants by emailing inbound-‐

[email protected] or calling 866-‐852-‐5229. We also have a 14-‐day, no obligation trial account option that does not require a credit card. Please visit www.ProfitBricks.com to get started.

(3)

Introduction

At ProfitBricks, Cloud Computing performance is a primary focus for our teams of engineers. We take a comprehensive view on IaaS performance benchmarking. We contract with 3rd_parties, explore, research and develop new testing methodologies in-house, and discover the bugs and limitations of traditional and new benchmarking tools. As a Cloud Computing service provider we realize that we are an edge case – our responsibility for high performance extends from hardware and network architecture to the virtualization layer, to the software we develop to manage the environment. Cloud Computing’s multi-tenant architecture requires that we keep and maintain performance levels that exceed our customers’ expectations, and we strive to remain the highest performance cloud available in the market. The ProfitBricks performance engineering team continually tests the performance of other cloud platforms and services. Performance is our passion – and the team is dedicated to publishing accurate and repeatable results.

Benchmark testing Cloud Computing environments that are not similar requires a thorough understanding of the respective environments, the effects that each component may have on performance, in addition to ensuring that the configurations are similar. It’s also essential that each benchmark test is run on top of a similar stack and software configuration.

We welcome questions about our methodologies. ProfitBricks strives to share our expertise with the community and to engage in an open dialogue about cloud performance.

(4)

Executive Summary

In this report, the ProfitBricks performance engineering team presents our latest series of standardized benchmark testing results (UnixBench, DBENCH, Iperf), comparing ProfitBricks virtual data centers and instances with Amazon’s EC2 instances. We strive to create an apples to apples comparison of virtual servers/instances at both ProfitBricks and Amazon EC2, and we believe our methodologies are fully transparent (see description in Section 3). In January 2014 Amazon introduced a new line of instance sizes, based on new CPUs and new architecture. ProfitBricks engineers spent eight weeks configuring and testing these new EC2 “m3” instance types and compared them to similar ProfitBricks instances alongside Amazon’s legacy “m1” instances. The results show that ProfitBricks continues to be the performance leader in the cloud on all tested configurations, using all three standard benchmarks, showing results that are at least twice the performance of Amazon. In some cases the performance difference between ProfitBricks and Amazon EC2 is an astounding 17x (Iperf).

Benchmark Testing Methodology

All benchmarking tools were compiled on Ubuntu 12.04 (64-bit) servers running a 3.2.0.58 Linux kernel, and each test was performed three times on three separate days with the results averaged. By taking advantage of ProfitBricks’ granular scaling features, we were able to compare Amazon’s latest EC2 offerings to ProfitBricks servers with similar resource specifications. For example, we contrasted the performance of an EC2 m3.large with 2 vCPUs and 7.5GB of RAM with a ProfitBricks server containing 2 dedicated cores and 7.5GB of dedicated RAM (and equivalent InfiniBand-powered double redundant storage devices).

In all cases, these are standard configurations at both ProfitBricks and Amazon. No special or optional services were added to the configurations. For each report, three test runs were completed on each instance for three days. Note: we continually see performance variations (sometimes large) on Amazon EC2 services that vary by instance size, by hour, and by data center. This is the main reason for multiple runs – we want to ensure that we’ve determined a representative composite result.

(5)

Server Configuration:

Amazon virtual instances are sold and packaged with a pre-packaged quantity of vCPUs, RAM and temporary storage.

ProfitBricks virtual instances are not pre-packaged, thus every instance can be uniquely configured. We’ve done our best to match equivalent ProfitBricks instances to Amazon instances on a hardware basis – not a cost basis.

OS Configuration:

Operating System: Ubuntu 12.04 (64-bit) with 3.2.0.58 Linux kernel

Each instance had the following packages installed if the provider did not include them with their images: make, GCC, automake, libpopt-dev, zlib1g-dev, g++, git

All benchmark software used in these tests are open source – and we have provided the information you would need (including configuration details) to run these tests. You can download the tools from these URLs:

UnixBench: https://byte-unixbench.googlecode.com/files/UnixBench5.1.3.tgz

DBENCH: Use git to create a local branch of the official git repository and download the tree: `git clone git://git.samba.org/sahlberg/dbench.git dbench`

Iperf: http://downloads.sourceforge.net/project/iperf/iperf-2.0.5.tar.gz

For DBENCH and UnixBench, both of which test storage performance, a separate block device (EBS on EC2 vs. ProfitBricks block storage) was formatted with ext4 and mounted:

• mkfs.ext4 /dev/second_device • mount /dev/second_device /bench

(6)

UnixBench Results

UnixBench is one of the most popular open source benchmark performance testing tools, which combines multiple tests to assess various aspects of a system’s performance in a Unix-‐like system. ProfitBricks performance engineers run these tests multiple times over numerous days. The entire set of index values is then combined to calculate an overall performance index for the system.

In this UnixBench benchmark test, ProfitBricks exceeds the performance of Amazon’s new m3 instances by 2 to 4.4x, and Amazon’s m1 instances by 2.2 to 3.1x depending on the instance size. These tests were run over three separate days and the results and test configuration can be found in the UnixBench test results on the next two pages.

UnixBench test configuration:

Get HZ environment variable to what is configured as "CONFIG_HZ_?" in the kernel configuration: "grep 'CONFIG_HZ_' /boot/config-3.2.0-58-virtual"

• In case of Ubuntu 12.04 LTS: 250 HZ

Edit Makefile and change the following line:

-‐ from "OPTON = -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall"

-‐ to "OPTON = -march=native -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall"

Compile UnixBench:

-‐ -HZ="250" make

Run UnixBench:

-‐ HZ="250" ./Run -c NR_OF_CORES

-‐ NR_OF_CORES equals the number of virtual cores available to each VM

(7)

UnixBench Results:

ProfitBricks Cloud Server Instances vs. Amazon AWS EC2 Instances – type m3

Instance Size ProfitBricks UnixBench score AWS EC2 UnixBench score ProfitBricks Performance Advantage EC2: m3.medium, ProfitBricks equivalent (1 CPU Core/3.75GB RAM)

1213.16 274.31 4.4x

EC2: m3.large,

ProfitBricks equivalent (2 CPU cores/7.5GB RAM)

1766.58 759.50 2.3x

EC2: m3.xlarge,

ProfitBricks equivalent (4 CPU cores/15GB RAM)

2475.73 1218.80 2.0x 0 500 1000 1500 2000 2500 3000

m3.medium m3.large m3.xlarge

UnixBench A

verage Score ProfitBricks UnixBench score

(8)

UnixBench Results:

Instance Size ProfitBricks UnixBench score AWS EC2 UnixBench score ProfitBricks Performance Advantage EC2: m1.medium, ProfitBricks equivalent (1 CPU Core/3.75GB RAM) 1213.16 389.10 3.1x EC2: m1.large, ProfitBricks equivalent (2 CPU cores/7.5GB RAM) 1766.58 762.82 2.3x EC2: m1.xlarge, ProfitBricks equivalent (4 CPU cores/15GB RAM) 2475.73 1103.38 2.2x 0 500 1000 1500 2000 2500 3000

UnixBench A

verage Score ProfitBricks UnixBench score

(9)

DBENCH Results

DBENCH is a popular open source performance testing tool that generates I/O workloads for a file system or networked CIFS of NFS server. It is used to stress a file system to see which workload becomes saturated.

In this DBENCH benchmark test, ProfitBricks exceeds the performance of standard EBS volumes on Amazon’s new m3 instances by 10.5 to 16x and Amazon’s m1 instances by 9.5 to 12.7x depending on instance size. These tests were run over three separate days and the results and test configuration can be found on the DBENCH test results on the next two pages.

DBENCH test configuration:

Compile DBENCH:

-‐ CFLAGS=”-‐march=native” ./autogen.sh

-‐ CFLAGS=”-‐march=native” ./configure

-‐ make

Run DBENCH:

-‐ ./dbench –backend=fileio –t 60 –D /benchmark/bench –loadfile=loadfiles/client.txt 48

(10)

DBENCH Results:

Instance Size _{DBENCH (MB/s)}ProfitBricks _{DBENCH (MB/s)}AWS EC2 _{Performance Advantage}ProfitBricks

EC2: m3.medium,

ProfitBricks equivalent (1 CPU Core/3.75GB RAM)

344.85 32.87 10.5x

EC2: m3.large,

505.05 31.13 16.2x

EC2: m3.xlarge,

641.31 48.63 13.2x 0 100 200 300 400 500 600 700

DBENCH A

verage Score (MB/s)

ProfitBricks DBENCH (MB/s) AWS EC2 DBENCH (MB/s)

(11)

DBENCH Results:

Instance Size _{DBENCH (MB/s)}ProfitBricks _{DBENCH (MB/s)}AWS EC2 ProfitBricks Performance _Advantage

EC2: m1.medium,

344.85 27.13 12.7x

EC2: m1.large, ProfitBricks equivalent (2 CPU

cores/7.5GB RAM)

505.05 41.74 12.1x

EC2: m1.xlarge,

641.31 67.80 9.5x 0 100 200 300 400 500 600 700

DBENCH A

verage Score (MB/s)

ProfitBricks DBENCH (MB/s) AWS EC2 DBENCH (MB/s)

(12)

Iperf Results

Iperf is a popular open-source network testing tool that creates TCP and UDP data streams and measures the throughput of the network that it is running on.

In this Iperf benchmark test, ProfitBricks exceeds the performance of Amazon’s new m3 instances by 7.3 to 17x and Amazon’s m1 instances by 4.9 to 7.2x depending on the instance size. These tests were run over three separate days and the results and test configuration can be found on the Iperf test results on the next two pages.

Iperf test configuration:

VMs were configured on both providers for both the client and the server – instances appeared to run on separate hardware to ensure that a local bridge was not used in these tests.

Compile iperf: • ./configure • make Run iperf: • Server: ./src/iperf –s

• Client: ./src/iperf –c IP_OF-‐SERVER –f m –t 60 –P 4

(13)

Iperf Results:

Instance Size ProfitBricks Iperf (Mbit/s)

AWS EC2 Iperf (Mbit/s)

ProfitBricks Performance Advantage

EC2: m3.medium,

5112.67 300.11 17.0x

EC2: m3.large,

5189.67 678.78 7.6x

EC2: m3.xlarge,

5660.89 778.89 7.3x 0 1000 2000 3000 4000 5000 6000

Iperf (Mbit/s)

ProfitBricks Iperf (Mbit/s) AWS EC2 Iperf (Mbit/s)

(14)

Iperf Results:

Instance Size ProfitBricks Iperf (Mbit/s)

AWS EC2 Iperf (Mbit/s) ProfitBricks Performance Advantage EC2: m1.medium, ProfitBricks equivalent (1 CPU Core/3.75GB RAM) 5112.67 1033.56 4.9x EC2: m1.large, ProfitBricks equivalent (2 CPU cores/7.5GB RAM) 5189.67 717.67 7.2x EC2: m1.xlarge, ProfitBricks equivalent (4 CPU cores/15GB RAM) 5660.89 1120.89 5.1x 0 1000 2000 3000 4000 5000 6000

Iperf (Mbit/s)

ProfitBricks Iperf (Mbit/s) AWS EC2 Iperf (Mbit/s)

(15)

Next Steps

ProfitBricks is here to help you evaluate us as your cloud provider, from benchmark testing to pricing questions our team of knowledgeable Cloud Computing Consultants and Sales Engineers are available by emailing [email protected] or calling 866-852-5229.

ProfitBricks offers a 14-day, no obligation trial account that does not require a credit card for activation. Visit www.profitbricks.com/trial today!

Finally, for up-to-the-minute information about our products and services, be sure to check out our Resource Center at http://www.profitbricks.com/press-and-info-center

ProfitBricks Inc. 15 Tudor Street Cambridge, MA 02139 Phone: +1 866 852 5229 Fax: +1 888 620 3376 Email: [email protected] www.profitbricks.com twitter.com/profitbricksusa blog.profitbricks.com

©2014 ProfitBricks, Inc. All rights reserved. ProfitBricks, the ProfitBricks logo and Data Center Designer are trademarks of ProfitBricks Inc. All other trademarks are the property of their respective owners. ProfitBricks reserves the right to make changes without further notice.

Cloud Computing Performance. Benchmark Testing Report. Comparing ProfitBricks vs. Amazon EC2