Cloud Computing:
Amazon Web Services
overview & hands-on experience
What is AWS / Cloud Computing?
Cloud Computing:
No clear definition, but can be seen as a cloud of
resources available to you
Available in this sense means: Unlimited resources!
Pay only for what you actually use
AWS:
Started out ”internally” within Amazon, and with first
SOAP interfaces to amazon.com for public use
Is now a mixture of actual WS operations and
What does it offer?
Storage (S3: Simple Storage Service)
Processing capabilities (EC2: Elastic Cloud
Computing)
... and a bit of glue/auxiliary services:
SimpleDB
SQS: Simple Queue Service
A sample use case
End users
submit videos to be transcoded
to the website.
The videos are
stored in
Amazon
S3
, and a message (“the request message”) is placed in an
Amazon SQS queue (“the
incoming queue
”) with a pointer to the video and to the target video
format in the message.
The transcoding engine, running on a set of Amazon
EC2 instances
,
reads
the request
message from the
incoming queue
,
retrieves
the video from Amazon
S3
using the pointer,
and transcodes the video into the target format.
The converted video is
put back
into Amazon
S3
and another message (“the response
message”) is placed in another Amazon SQS queue (“the
outgoing queue
”) with a pointer to
the converted video.
At the same time,
metadata
about the video (e.g., format, date created and length) can be
indexed into Amazon
SimpleDB
for easy querying.
During this whole workflow, a dedicated Amazon EC2 instance can constantly
monitor
the
incoming queue and, based on the number of messages in the incoming queue, is able to
dynamically adjust the number of transcoding Amazon EC2 instances
to meet customers’
response time requirements.
EC2: Pricing
Instances
* $0.10 per Small Instance (m1.small) instance-hour (or partial hour)
* $0.40 per Large Instance (m1.large) instance-hour (or partial hour)
* $0.80 per Extra Large Instance (m1.xlarge) instance-hour (or partial hour)
* $0.20 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour)
* $0.80 per High-CPU Extra Large Instance (c1.xlarge) instance-hour (or partial hour)
Data Transfer
* $0.10 per GB Internet Data Transfer - all data transfer into Amazon EC2
* $0.17 per GB Internet Data Transfer - first 10 TB / month data transfer out of Amazon EC2
* $0.13 per GB Internet Data Transfer - next 40 TB / month data transfer out of Amazon EC2
* ...
Storage — New Pricing (effective November 1st)
* $0.15 per GB – first 50 TB / month of storage used
* $0.14 per GB – next 50 TB / month of storage used
* ... (Storage in EU is slightly more expensive - $0.18/GB)
SQS Requests
* $0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request)
Data transferred between Amazon EC2
instances in the same Availability Zone
using private IP addresses is free of
charge. Data transferred between
Amazon EC2 and Amazon S3-Europe will
be charged as Internet data transfer. Data
transferred between Amazon EC2 and
Amazon S3-US, Amazon SDB, and
Amazon SQS is free of charge.
Small: 1.7GB RAM, 160GB HDD, 1 CU*,32bit
XL: 15GB RAM, 1.6TB HDD, 4x2 CUs*,64bit
High XL: 7GB RAM, 1.6TB HDD, 8x2.5 Cus*
Interfaces
WSDL/SOAP
REST
Command-line tools (Java programs using SOAP)
Security:
X509 certificates (EC2)
Or simpler ”Public ID/shared secret for signing” (S3)
'SHA1 digest of the concatenation of "AmazonS3" +
OPERATION + Timestamp'
S3
Objects are grouped into ”
buckets
” and identified
by
keys
:
http://
christoph
.s3.amazonaws.com/
obj/ec2.png
”URL-friendly” scheme is no coincidence: objects
can be retrieved by plain HTTP (if access
permissions allow)
In addition, append
?torrent
to get a
BitTorrent download
May turn out cheaper if many requests
SOAP: PutObject uses DIME attachments,
PutObjectInline limited to 1MB :-)
S3 provides redundant storage (>3 copies),
automatic and dynamic placement
No guarantees wrt ”freshness”/concurrency
control etc.
Currently > 29 billion objects stored in S3
Petabytes of data
EC2
All instances are virtual (probably XEN on some
really fat machines)
Instances are started using an imageId (image
stored in S3)
After an instance is terminated, all data on it is lost
(but persisted across reboots)
For persistent storage, use S3 or EBS (Elastic block
storage)
EBS is like a real block device
Storage slightly cheaper than S3, but also
transfers are billed ($0.10 per 1M I/O requests)
Can be mounted only on one EC2 at a time
Tools provided for creating custom images
Start from a predefined one or from scratch
Can pass up to 16K of user-defined data as
EC2: hands-on!
christoph@uri:~/aws$
ls
ec2-api-tools-1.3-24159 ec2-api-tools.zip private.pem public.pem
christoph@uri:~/aws$
export EC2_CERT=~/aws/public.pem
christoph@uri:~/aws$
export EC2_PRIVATE_KEY=~/aws/private.pem
christoph@uri:~/aws$
export EC2_HOME=~/aws/ec2-api-tools-1.3-24159/
christoph@uri:~/aws$
export PATH=$PATH:$EC2_HOME/bin
christoph@uri:~/aws$
ec2-describe-images -o self -o amazon|grep machine
IMAGE ami-2c5fba45 ec2-public-images/demo-paid-AMI-v1.07.manifest.xml amazon available public A79EC0DB i386 machine IMAGE ami-bd9d78d4 ec2-public-images/demo-paid-AMI.manifest.xml amazon available public A79EC0DB i386 machine
IMAGE ami-2f5fba46 ec2-public-images/developer-image-i386-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-26b6534f ec2-public-images/developer-image.manifest.xml amazon available public i386 machine
IMAGE ami-f51aff9c ec2-public-images/fedora-8-i386-base-v1.06.manifest.xml amazon available public i386 machine aki-a71cf9ce ari-a51cf9cc IMAGE ami-2b5fba42 ec2-public-images/fedora-8-i386-base-v1.07.manifest.xml amazon available public i386 machine aki-a71cf9ce ari-a51cf9cc
IMAGE ami-f21aff9b ec2-public-images/fedora-8-x86_64-base-v1.06.manifest.xml amazon available public x86_64 machine aki-b51cf9dc ari-b31cf9da IMAGE ami-2a5fba43 ec2-public-images/fedora-8-x86_64-base-v1.07.manifest.xml amazon available public x86_64 machine aki-b51cf9dc ari-b31cf9da IMAGE ami-a21affcb ec2-public-images/fedora-core-6-x86_64-base-v1.06.manifest.xml amazon available public x86_64 machine aki-a53adfcc ari-a23adfcb IMAGE ami-2d5fba44 ec2-public-images/fedora-core-6-x86_64-base-v1.07.manifest.xml amazon available public x86_64 machine aki-a53adfcc ari-a23adfcb IMAGE ami-225fba4b ec2-public-images/fedora-core4-apache-mysql-v1.07.manifest.xml amazon available public i386 machine
IMAGE ami-25b6534c ec2-public-images/fedora-core4-apache-mysql.manifest.xml amazon available public i386 machine IMAGE ami-2e5fba47 ec2-public-images/fedora-core4-apache-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-23b6534a ec2-public-images/fedora-core4-apache.manifest.xml amazon available public i386 machine
IMAGE ami-215fba48 ec2-public-images/fedora-core4-base-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-20b65349 ec2-public-images/fedora-core4-base.manifest.xml amazon available public i386 machine
IMAGE ami-205fba49 ec2-public-images/fedora-core4-i386-base-v1.07.manifest.xml amazon available public i386 machine aki-9b00e5f2 IMAGE ami-255fba4c ec2-public-images/fedora-core4-mysql-v1.07.manifest.xml amazon available public i386 machine
IMAGE ami-22b6534b ec2-public-images/fedora-core4-mysql.manifest.xml amazon available public i386 machine IMAGE ami-36ff1a5f ec2-public-images/fedora-core6-base-x86_64.manifest.xml amazon available public x86_64 machine
