Running Amazon EC2 instances. Juan Lago, July (Document )

Running Amazon EC2 instances

Juan Lago, July 2011

(Document 2011.05)

This guide shows how to run an Amazon EC2 instance both from the command line and the Web interface, i.e, Amazon Web Service (AWS) Management Console.

Prerequisites

•

•

•

•

•

Creating security credentials

You need security credentials before using AWS tools. Amazon provides an EC2 Getting Starting Guide for setting up an AWS account and accessing EC2 Service in the following link:

http://docs.amazonwebservices.com/AmazonEC2/gsg/2007-01-03/

First of all, I’m going to create new X.509 certificate and a private key. Both, the certificate and the private key, are needed for setting up the command line tools and must be downloaded into the computer that will access AWS API.

Please, sign in into the AWS Management Console (http://aws.amazon.com/console) using your login and password. Then, click the “Account” link located in the upper part of the Management Console Web page as shown in figure 1.

Figure 1. AWS Management Console. “Account” link highlighted.

The account page will be displayed. From this page, you can manage different aspects of your account like account activity, identity and access management, payment methods, security credentials, etc. Click the link called “Security Credentials” and sign in again.

In the “Access Credentials” section, choose “X.509 Certificates” tab and press “Create a new Certificate”. A new X.509 Certificate will be created as well as a new private key (see figure 2). Please download both files using the corresponding download links. In this case, I put these files in a folder called .ec2 inside my hadoop user home directory.

$ cd $ pwd /home/hadoop $ mkdir .ec2 $ mv ./Downloads/*.pem .ec2 $ cd .ec2 $ ls -la cert-HTVO5VSZSLLJP2YW2C6QCHEDXIMOMWJ2.pem pk-HTVO5VSZSLLJP2YW2C6QCHEDXIMOMWJ2.pem

Figure 2. X.509 Certificate created. Private key file generated and prepared for downloading.

Also, in the same “Security Credentials” web page you can find your AWS Account ID under the “Account Identifiers” section. You should use this value whenever you need to provide and EC2 user ID. The account ID is of the form XXXX-YYYY-ZZZZ. Your AWS Account ID, with the hyphens removed is your EC2 user ID. The example above would be XXXXYYYYZZZZ.

Installing Amazon Web Services Tools

AWS Tools must be installed in order to use the AWS API from the command line. In this guide, a Biolinux distribution (Dabian based) machine will be used for interacting with AWS API.

Download the command line tools from the Amazon EC2 Resource Center:

http://aws.amazon.com/developertools/351?_encoding=UTF8&jiveRedirect=1

A file called ec2-api-tools.zip will be downloaded. These tools are written in Java and include shell scripts for both Windows and Linux/Mac OSX.

This guide assumes you have SUN Java 6 and ssh client installed and JAVA_HOME environment variable is correctly set. For example, in Biolinux and with the hadoop user, check the Java installation:

$ echo $JAVA_HOME

/usr/lib/jvm/java-6-sun $ java -version

java version "1.6.0_26"

Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)

Unzip the downloaded file and move the lib and bin directories (just created after unzipping) inside .ec2 directory.

$ cd

$ cd Downloads

$ unzip ec2-api-tools.zip $ cd ec2-api-tools-1.4.4.1 $ ls

drwxr-xr-x 2 hadoop hadoop drwxr-xr-x 2 hadoop hadoop -rw-r--r-- 1 hadoop hadoop -rw-r--r-- 1 hadoop hadoop -rw-r--r-- 1 hadoop hadoop $ mv bin ../.ec2

$ mv lib ../.ec2 20480 Aug 24 01:49 bin 4096 Aug 24 01:49 lib 4852 Aug 24 01:47 license.txt 539 Aug 24 01:47 notice.txt 46468 Aug 24 01:47 THIRDPARTYLICENSE.TXT

Create $EC2_HOME environment variable and add the bin directory to $PATH environment variable. For this, edit the file .bashrc for user hadoop:

$ cd

$ vim .bashrc

export EC2_HOME=~/.ec2

export PATH=$PATH:$EC2_HOME/bin

The command line tools need access to the the private key and X.509 certificate you generated after signing up for the Amazon EC2 service. It's possible to provide this information on the command line every time you invoke the tools. But it's far simpler to set up some environment variables and be done with it.

Again, edit your user’s .bashrc file and add the following environment variables: export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem`

export EC2_CERT=`ls $EC2_HOME/cert-*.pem`

Running an instance

Now we are ready for running an instance from the command line. The first thing is to find a suitable AMI (Amazon Machine Image) to run. For this purpose, you can use the following command:

$ ec2-describe-images -a --filter image-type=machine | grep BioLinux IMAGE ami-90d32af9 678711657553/CloudBioLinux Ubuntu 10.04 LTS 64bit

20110602 678711657553 available public x86_64 machine aki-427d952b ebs paravirtual xen

IMAGE ami-ad8e4ec4 678711657553/CloudBioLinux Ubuntu 11.04 64bit 20110819 678711657553 available public x86_64 machine

aki-427d952b ebs paravirtual xen

This example will find all AMIs of type “machine” containing the string “BioLinux”. We need the AMI ID to run the instance. This can be found in the second column of the output (in bold) from the ec2-describe-images command. In the previous example, ami-90d32af9 and ami-ad8e4ec4.

See the command line reference for more details:

http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/

Generating a keypair

In order to login securely to a public AMI instance, you will need a public/private keypair. Amazon EC2 public images use this feature to provide secure access without passwords. There is a command in the API tools for this. You only have to choose a name for the keypair as parameter:

$ ec2-add-keypair keypair_for_biolinux

KEYPAIR keypair_for_biolinux32:96:68:6d:1d:7f:0e:93:df:94:9d: 59:3e:ac:f3:a2:d3:59:5b:75

---BEGIN RSA PRIVATE

KEY---MIIEpAIBAAKCAQEA2MUcek2371xrZRVipZx+p4f4lU+8/02mLZLEQFfqVCKjajSy3ASrAFB02Ul+ SBGQ4OaX495+buyf+DNHFrXAyF8zXY/oF2o9A9AmFeuSdtXZB3lZeBBvI3//cT4L1bRQKoKZ2+tm KcqW5rcaDkWKJbKRBsCRd3EWXhUfvM8y5umwzPLgbj7st7Gfcn2g4b70OJyg604D5/8lCnYMNfxO hSEaeIRe/4EcFmnk9dHZ6xeyRdpsU82guFO45FUq3sbqCeGzXqt5dwybILtcUUfzRxxpfuYrx9G2 GzWJr9Siw86tm9quQzATdZBaFvTefH5YdiRm0k7tFOsyWhRxN6wrJwIDAQABAoIBADrKCpUo6s+h 8wZxLFXaAYeSGtrzy0A6YO1PNqZ0sXqsAkHVlQSSx4XITEh55CPLj81O/LayKvwOdLwtvAPuuyAc oaOoWbuv9mhIVWvwfcJ48OlUAe3MFzg7iJ2/xBnrUyqUTQIV75k59n8P/aMNPRBIQ4jfRRRtuTE+ 7WKZlyaEjqh/jr/GO0pITlzCWtJ8qp1anyApCBB9Oq+fPdURE3EX0lBaXN/GKeVJ44bBHj+T82O0 biNfQO05JmnqJtopW662uDHyHGNZrEHXmgFjGxPazad+kGw1V90Q9WdDH0RQ/jxIkAwlQa1IA/nU ow+MJz5UjlBMuYVKp0nrdDN/XMECgYEA9bcXpqR9dtx1lHiD0f0HpRPXw9h9dmkkoUdokeDL0Rov hk43I2E2cZSDzXkD19AziEAV5thyW+q7UsSf8r0ygn6ciNXUbeO0QRC8yQT/i/cqEwRMxQ9y2c3I hnbXAb7vWPzd2o3bwL6xzjjP3au/rKEMm01dQnEV42HSLmGzHPECgYEA4dfcysdohUKgIoZHtLmv 496M+QhxM4MZJq1VeONVpHn2Vx9f2k8s0+GCaSwjZ2f5PH4By1bs7+Az9QtkNJVglaH+Dz7citlJ FxhhCXJKeI2mq9PAQwwOQo7rSrdbCiQCEBKpV5KSB0NrN6QQBP1ZFuXVCryGXcj2Obs8nQObqZcC gYAd/nukm++00rSHqy5sM7b7L7Dm90ZZbfxKs6zQIeC/ZbLnciTI7NT4iGiObHdLoN6pgykpRMKw 9bZoJBhiSTYhN9Nj4LZNs74NMz3YTAPl0gYQzDWS2I/shnIz983HKMeO1U8/npF0tdJ+mfDF3nFH YWFok8kOkz9Ywlgl0UdMAQKBgQCQBhs17gkHwr9ghQXBiRFUXh7Eyc93LEmj0rJJT6DULTsdwFJe ZFOQI6YI8C77yOC25Ry70glzyCUaRlN2/TQE+LOOgWYf3gWwhsmLPqmjS6JYvcdEerG1XHNMyLV0 OBZgraX0xPH/U81SRHw60l44LgW4vRVrRbdQhclX0G86rQKBgQCL0YFBClOz9M9oanUFGAOSrXPl TsQjqWbsiq/RxW9QZnwB4TK2g8WkrNM2Wem3n3uiU5/ixYayIXSBzjSGjtpoexF8pmQdhOdurwmI MhzTKp75fOrYth4Q+7ve1VhJQmYrsuv9kbkcmV/Jux3lefmhdGkAoKrVafLN+BGzLhxgBA== ---END RSA PRIVATE

KEY---Copy the content between “---BEGIN...” and “---END...” in your local machine. In my case, I’ve copied it in a file called id_rsa-keypair_for_biolinux in .ec2 directory. For security reasons, change this file permissions to 600.

$ chmod 600 id_rsa-keypair_for_biolinux $ ls -la \bin cert-HTVO5VSZSLLJP2YW2C6QCHEDXIMOMWJ2.pem id_rsa-keypair_for_biolinux \lib pk-HTVO5VSZSLLJP2YW2C6QCHEDXIMOMWJ2.pem

Running an instance

Now we are going to launch a CloudBioLinux instance using an AMI ID found in our previous search:

$ ec2-run-instances ami-90d32af9 -k keypair_for_biolinux -t t1.micro RESERVATION r-a1d098ce 413426948323 default

INSTANCE i-27494b46 ami-90d32af9 pending

keypair_for_biolinux 0 t1.micro2011-08-26T08:02:29+0000 us- east-1a aki-427d952b monitoring-disabled ebs paravirtual xen sg-20d6b149 default The instance ID returned by ec2-run-instances command is an identifier and will be used later to manipulate this instance (in this example, i-27494b46)

Authorizing network access to the instance

Before you can reach your instance over the internet, you have to authorize traffic to it. SSH port is 22, Web access is 80, etc:

$ ec2-authorize default -p 22 GROUP!! default

PERMISSION! default! ALLOWS! tcp 22 22 FROM CIDR 0.0.0.0/0 ingress

From the AWS management console (Web interface), you can define a security group in Amazon to tell which ports are enable from specific IPs. My default security group is shown in figure 3. You can also do it from the command line with ec2-create-group command. Also, you can use ec2-describe-group to find out what groups you already have and how they are configured.

Figure 3. Default security group in AWS Management Console.

You can think of security groups as firewall policies, sets of rules on what ports are to be allowed and not on a given group. By default every port is blocked.

It’s a good idea to add a security group before lunching any instances as you will have to specify a security group for the instance that you are lunching. And without specifying one you won’t be able to access the instance through ssh.

Connecting to the instance

Now you can login as root (or any other user) and reach access to the instance as you would do with any other host, using your private key. We also need to know the instance IP or hostname. This value is returned in ec2-describe-instances:

$ ec2-describe-instances i-27494b46

RESERVATION r-a1d098ce 413426948323 default INSTANCE i-27494b46 ami-90d32af9 ec2-50-16-74-104.compute-1.amazonaws.com ip-10-117-90-41.ec2.internal running keypair_for_biolinux 0 t1.micro

2011-08-26T08:02:29+0000 us-east-1a aki-427d952b monitoring-disabled 50.16.74.103 10.117.90.41 ebs paravirtual xen sg-20d6b149 default

BLOCKDEVICE /dev/sda1 vol-969830fc 2011-08-26T08:02:51.000Z

In this case, the hostname is ec2-50-16-74-103.compute-1.amazonaws.com. Because this instance is an Ubuntu platform, I will connect with the user ubuntu:

$ ssh -i id_rsa-keypair_for_biolinux ubuntu@ec2-50-16-74-103.compute-1.amazonaws.com Linux ip-10-117-90-41 2.6.32-312-ec2 #24-Ubuntu SMP Fri Jan 7 18:30:50 UTC 2011 x86_64 GNU/Linux

Ubuntu 10.04.2 LTS

Welcome to Ubuntu

* Documentation: https://help.ubuntu.com/

System information as of Fri Aug 26 08:33:34 UTC 2011

System load: 0.0

Usage of /: 45.4% of 19.69GB Memory usage: 22% Swap usage: 0%

Processes: 95 Users logged in: 0

IP address for eth0: 10.117.90.41

Graph this data and manage this system at https://landscape.canonical.com/

---At the moment, only the core of the system is installed. To tune the system to your needs, you can choose to install one or more

predefined collections of software by running the following command: sudo tasksel --section server

---6 packages can be updated. 6 updates are security updates.

A newer build of the Ubuntu lucid server image is available. It is named 'release' and has build serial '20110719'. Ubuntu 10.04.2 LTS

Welcome to Ubuntu!

* Documentation: https://help.ubuntu.com/

System information as of Thu Jun 2 11:16:23 UTC 2011

System load: 0.06!! ! Processes: 80 Usage of /: 3.6% of 19.69GB! Users logged in: 1

Memory usage: 1%! ! ! IP address for eth0: 10.113.17.228 Swap usage: 0%

Graph this data and manage this system at https://landscape.canonical.com/

---At the moment, only the core of the system is installed. To tune the system to your needs, you can choose to install one or more

predefined collections of software by running the following command: sudo tasksel --section server

---0 packages can be updated. 0 updates are security updates.

Last login: Thu Jun 2 15:21:54 2011 from sph184-59.harvard.edu ubuntu@ip-10-117-90-41:~$

Terminating your instances

Amazon will charged you for the time your instance is running per hour. Charging starts when you launch the instance, not when you log into. If, for instance, your instance is running during 20 minutes, you will be charge a hole hour. But you also have to consider the starting and finishing time, not only the duration. If the instance started at 9:55 am and finished at 11:03 am, you will be charge 3 hours (from 9 to 10, 10 to 11, 11 to 12).

Terminating an instance can be done with ec2-terminate-instances. You need to indicate which instance to terminate with its ID:

$ ec2-terminate-instances i-27494b46

INSTANCE i-27494b46 running shutting-down

A useful bash script

Here you are a useful bash script for performing most of the commands needed from launching to connecting an EC2 CloudBioLinux instance:

#!/usr/bin/env bash

# Script for launching Amazon EC2 CloudBioLinux instances # Run: $ source launch_biolinux.sh

# Sets some usefull system variables (env | grep MY_EC2)

# Amazon CloudBioLinux instances

# ami-90d32af9 - Ubuntu 10.04 LTS 64bit # ami-ad8e4ec4 - Ubuntu 11.04 64bit

INSTANCE='ami-90d32af9' #INSTANCE='ami-ad8e4ec4'

echo "Launching instance..."

echo ">> ec2-run-instances $INSTANCE -k keypair_for_biolinux -t t1.micro" RUN=`ec2-run-instances $INSTANCE -k keypair_for_biolinux -t t1.micro`

set $RUN

# Now $6 has the instance ID export MY_EC2_INSTANCE=$6 echo "Instance ID = $6"

echo "Retrieving $6 description..." echo ">> ec2-describe-instances $6" DESCRIBE=`ec2-describe-instances $6` set $DESCRIBE

# Now $8 has the instance hostname or the string "pending" STATUS=$8

while [ $STATUS = "pending" ] do

echo -n "." sleep 5

DESCRIBE=`ec2-describe-instances $6` set $DESCRIBE

# Now $8 has the instance hostname STATUS=$8

done

export MY_EC2_HOSTNAME=$8

export MY_EC2_CONNECT="ssh -i id_rsa-keypair_for_biolinux ubuntu@$8" export MY_EC2_TERMINATE="ec2-terminate-instances $MY_EC2_INSTANCE"

Using the Web interface: AWS Management Console

All the topics that we have seen for running an instance from the command line, can be done easily using the AWS Web interface, i.e., the AWS Management Console.

In the following screenshots, you can see how to launch and terminate an instance. If you see a screen not showed in the screenshots, accept the default values and keep it unmodified (press “continue”).

Figure 4. Launch instance: click 1 to access EC2 service and 2 to launch an instance.

Figure 5. Choose AMI: click 1 for community AMIs and type search string in 2. Click “select” in your favorite AMI (3).

Figure 6. Select number of instances (1) and type (2).

Figure 8. Security groups: choose an existing group or create a new one.

Figure 10. Instance in pending status while launching.

Figure 11. Connect help. Inside instance actions (1) press “connect”. ssh command line showed in (2).