Continuous Integration and
Deployment
Version 1.0
Copyright © 2013, 2014 Amazon Web Services, Inc. and its affiliates.
All rights reserved.
This work may not be reproduced or redistributed, in whole or in part,
without prior written permission from Amazon Web Services, Inc.
Commercial copying, lending, or selling is prohibited.
For corrections or feedback on the course, please email us at
[email protected].
For all other questions, please email us at
[email protected].
DO
NOT
Table of Contents
Introduction ... 4
Overview ... 4
Technical Knowledge Prerequisites ... 4
Topics Covered ... 4
Sign in to the AWS Management Console ... 4
Using qwikLABS
tmto sign in to the AWS Management Console ... 4
Module 1 Source Code Management & Automated Testing ... 6
Setting Up Github And Creating Our Own Sample Application and Data Store
Repositories ... 6
Creating a GitHub account ... 6
Forking the sample web application repository ... 6
Forking the data sources repository ... 7
Connecting to the development environment ... 7
Setting up the development environment ... 8
Setting up GitHub SSH keys ... 8
Telling Git about ourselves ... 10
Getting the sample Python web application ... 10
Getting the latest version of the data sources configuration ... 11
Testing the sample Python web application ... 12
Additional Resources ... 14
DO
NOT
Introduction
Overview
In this lab you will use a sample Python web application to learn about source code
management (SCM) techniques and some common approaches to automated testing. This will then be used as a foundation to build a continuous integration pipeline to automate this using Atlassian Bamboo.
Technical Knowledge Prerequisites
To successfully complete this lab, you should be familiar with the following:
Basic source code management concepts
Git as a source code management system
Automated application testing concepts, including automated unit testing
Python web application programming
Topics Covered
This lab will take you through source code management techniques and automated testing, including:
Forking a version or the sample Python web application on GitHub
Using Git as your SCM tool and common operations with Git
Unit testing a sample Python web application
Test driven development (TDD) concepts
Sign in to the AWS Management Console
Using qwikLABS
tmto sign in to the AWS Management Console
Welcome to this self-paced lab! The first step is for you to sign in to Amazon Web Services. 1. To the right of the lab title, click Start Lab. If you are prompted for a token, use the one
you received or purchased.
Note: A status bar shows the progress of the lab environment creation process. The AWS Management Console is accessible during lab resource creation, but your AWS resources may not be fully available until the process is complete.
2. On the lab details page, notice the lab properties.
a. Duration - The time the lab will run before automatically shutting down. b. Setup Time - The estimated time to set up the lab environment.
DO
NOT
c. AWS Region - The AWS Region in which the lab resources are created.
Note: The AWS Region for your lab will differ depending on your location and the lab setup.
3. In the AWS Management Console section of the qwikLAB™ page, copy the Password to the clipboard.
4. Click the Open Console button.
5. Log into the AWS Management Console using the following steps. a. In the User Name field type awsstudent.
b. In the Password field, paste the password copied from the lab details page. c. Click Sign in using our secure server.
Note: The AWS account is automatically generated by qwikLAB™. Also, the login credentials for the awsstudent account are provisioned by qwikLAB™ using AWS Identity Access Management.
DO
NOT
Module 1 Source Code Management & Automated
Testing
In this module we’ll ensure that our development environment is configured correctly and ready to use. This will include ensuring that we have Git tools configured, and the correct unit test libraries installed and configured.
Because we’re using the Flask web application framework for Python, we’re going to use some standard approaches to testing Flash applications. This means we’ll also be using the ‘unittest’ test library available for Python.
A development environment has been provided containing all the tools and libraries you’ll need to work with the example web application and use Git for this lab.
We’ll be using GitHub to fork a version of the sample Python web application and clone that forked repository to work on it locally. There are lots of options for hosting your source code, both public and private. Many businesses choose to host their source code “behind the firewall”, and this would work just fine. In fact, Atlassian Bamboo has great support for “behind the firewall” options like Atlassian Stash or your own Git server. In this lab we use Github for ease of use and to simplify setup and configuration.
Setting Up Github And Creating Our Own Sample Application and Data Store
Repositories
We’ll be using GitHub to fork the sample Python web application and also to push our
changes to when we’ve completed our test and development work. We’re also going to fork a GitHub repository that will be responsible for managing the configuration and lifecycle of our relational MySQL database.
If you don’t have a GitHub account already, you’ll need to create one. Even if you do have a GitHub account, you might want to create a new one for the purposes of completing these labs.
Creating a GitHub account
1. Browse to https://github.com/join.
2. Fill out the form to create your personal account 3. Choose a personal plan, e.g. Free ($0/month) 4. Click Finish sign up
This will have created a GitHub personal account and logged you in. Forking the sample web application repository
To work with the sample Python web application, we’ll fork our own version of it. This will let us push any changes we make in subsequent labs to our own GitHub repository. The version of the web application we’ll be working with is https://github.com/aws-tools/py-flask-signup
1. Open a new browser tab or window go to https://github.com/aws-tools/py-flask-signup
2. In the upper right corner click on Fork
3. You could be prompted to select where the repository will be forked. Make sure you select the GitHub account you just created if you are.
4. You will now have a completely new version of the py-flask-signup repository under your GitHub account.
DO
NOT
Whenever you want to clone this forked repository you can do so, using the HTTPS clone URL on the repository home page, e.g. https://github.com/<your_username>/py-flask-signup.git or the SSH clone URL, e.g. [email protected]:<your_username>/py-flask-signup.git. We’ll be doing this later in the lab.
Forking the data sources repository
To create and update our relational database environment for our web application, we’ll fork another repository containing a baseline definition of our relational database environment. This repository also includes the database schema used to support our web application. In fact, we’ll be continuously delivering schema changes for our database in subsequent labs. The repository we want to fork is https://github.com/aws-tools/py-flask-signup-datasources
1. Browse to https://github.com/aws-tools/py-flask-signup-datasources
2. In the upper right corner click on Fork.
3. You could be prompted to select where the repository will be forked. Make sure you select the GitHub account you just created if you are.
4. You will now have a completely new version of the py-flask-signup-datasources repository under your GitHub account.
Whenever you want to clone this forked repository you can do so, using the HTTPS clone URL on the repository home page, e.g. https://github.com/<your_username>/py-flask-signup-datasources.git or the SSH clone URL, e.g. [email protected]:<your_username>/py-flask-signup.git-datasources.git. We’ll do this in subsequent labs.
Connecting to the development environment
This lab will make use of the development environment that you’ve used in previous labs. You will connect to the instance using your local SSH client, typically Putty for Windows or the native SSH client for OSX/Linux. The following steps remind you how to connect to this Development & Build instance. We have created a new Development & Build instance for you, so you’ll need to complete the same steps again. If you are comfortable connecting to the Development & Build instance on your own, feel free to skip this section.
The development instance will be reachable using its public DNS address. The public DNS of the instance can be found by going to the EC2 Management Console of the AWS
Management Console. In your list of running EC2 instances, select the instance with name
‘Linux DevOps Instance’ to display the instance details. Identify the public DNS value in
instance details pane at the bottom of the screen, and copy it to the clipboard. Alternatively you can find the public DNS name of the management instance in the qwikLAB™
environment information screen under the Additional Lab Information section as the value for the DevOpsServerLinuxDNS key. The public DNS name will look something like:
ec2-54-69-64-114.us-west-2.compute.amazonaws.com
Windows SSH clients: Connect to your Development & Build EC2 instance
1. Return to your open browser that has the qwikLAB™ lab information. Download the qwikLABS™ provided EC2 Key Pair private key file in the PuTTY compatible PPK format.
DO
NOT
2. Click on the down arrow next to the Download PEM/PPK drop-down. 3. Click on the Download PPK option
4. Open PuTTY.exe that you downloaded in a previous lab.
5. Enter ec2-user@<your public DNS> into the Host Name field (Ctrl+v). 6. Expand the SSH category by clicking on it.
7. Select the Auth category by clicking on it (not the + symbol next to it).
8. Click Browse and locate the PPK file (ending in .ppk) in your Downloads directory or whatever other location you chose.
9. Click Open.
10. Click Yes when prompted to allow a first connection to this remote SSH server. Since you are using a key pair for authentication, you will not be prompted for a password.
Common Issues
If PuTTY fails to connect to your EC2 instance, check that:
You have entered ‘ec2-user@<hostname>’ in Putty.
You have downloaded the .PPK file for this lab from qwikLAB™
You are using the downloaded .PPK for ‘Private key file for authentication’
The network you are on allows for outbound TCP connections to destination port 22 OSX & Linux SSH clients: Connect to your Development & Build EC2 instance 1. Return to your open browser that has the qwikLAB™ lab information. Download the
qwikLABS™ provided EC2 Key Pair private key file in the PEM format. 2. Click on the down arrow next to the Download PEM/PPK drop-down. 3. Click on the Download PEM option.
4. Save the file to your Downloads directory, or another directory of your choice. 5. Open the OSX/Linux Terminal application.
6. Enter the below commands substituting the path/filename for the .pem file you downloaded from qwikLABS™ and pasting ec2-user@<your EC2 hostname> to substitute the example below.
chmod 600 ~/Downloads/qwiklab-l33-5018.pem
ssh –i ~/Downloads/qwiklab-l33-5018.pem
ec2-54-69-64-114.us-west-2.compute.amazonaws.com
Setting up the development environment
Once connected to your development environment we’re going to create and setup SSH keys for use with our new GitHub account.
Setting up GitHub SSH keys
We also need to set up our development environment with our GitHub SSH keys and make sure GitHub knows about them. This will mean we can authenticate and push our changes to our newly forked GitHub repository.
DO
NOT
GitHub has documentation on how to do this, https://help.github.com/articles/generating-ssh-keys/. It’s important that we generate SSH keys to be able to authenticate without passwords when we interact with our GitHub account.
You can either follow the instructions outlined at https://help.github.com/articles/generating-ssh-keys/ or follow the basic steps outlined below in your Development environment:
1. Check for existing SSH keys ls –la ~/.ssh
You shouldn’t see any files with names like ‘id_rsa.pub’ and ‘id_rsa’. This is because we’ve created a new development Linux environment for you. If you were doing this on your laptop or another machine you might see keys already created. In that case you could choose to reuse those keys or create new ones for your Github account.
2. Generate a new SSH key, and use the defaults
ssh-keygen -t rsa -C [email protected]
This will generate a private and public key in ~/.ssh for you. You should see something like:
Your identification has been saved in /home/ec2-user/.ssh/id_rsa.
Your public key has been saved in /home/ec2-user/.ssh/id_rsa.pub.
3. Now we’ll add our new SSH key to our Github account. To do this you’ll need to copy the content of the public key to your clipboard. To do this:
cat ~/.ssh/id_rsa.pub
which will display the contents of your public key. Copy the contents from the beginning of the key starting with ‘ssh-rsa’ to the end of the email address you entered, e.g. ‘[email protected]’. Make sure not to include any other characters or lines.
4. Now we add the key to our Github account. To do this:
a. Log in to Github.com and click on the top right gear icon, ‘Settings’ b. On the left hand navigation click on ‘SSH keys’
c. Click the ‘Add SSH key’ button
d. In Title field enter something like ‘AWS lab development environment’ e. In the Key field paste the contents of your public key copied in the previous
setp
f. Click ‘Add key’
Once you’ve configured GitHub with your SSH keys, you should be able to run: $ ssh -T [email protected]
DO
NOT
Hi <username>! You've successfully authenticated, but GitHub does not provide shell access.
Telling Git about ourselves
We’re going to tell our local Git install about ourselves as well. This will help identify
ourselves when we push changes to our remote repository on Github too. To do this we want to run the following commands:
git config --global user.name "Your Name"
git config --global user.email [email protected]
Git will store this information in our ~/.gitconfig file in our development environment.
Getting the sample Python web application
1. Now that we have configured our Git environment and can successfully authenticate against our new GitHub account, we’ll get the latest version of our sample Python application.
2. We have our own version of the forked sample application located in a new public Github repository. On your development environment,
a. Make a directory in your home directory called ‘git’ and change to that directory
mkdir $HOME/git
cd $HOME/git
b. Get the latest version of the sample web application. To do this you’ll need to find the SSH clone URL for your GitHub repository. To do this, browse to
Github.com and your py-flask-signup repository. On the right hand side you’ll see a navigation bar like below:
Click on the SSH link and copy the SSH clone URL into your clipboard.
Once you’ve done that go back to your development environment and clone your
DO
NOT
fork of the same Python web application git repository. E.g.,
git clone [email protected]:<your_username>/py-flask-signup.git
c. You should see a new directory, ‘py-flask-signup’. Change to that directory. cd py-flask-signup
3. We want to change to a branch that has the latest application changes in it, called ‘age-collection'. To do this on the CLI,
git checkout age-collection
4. If you list all git branches in your repository now you should see the micro-blog branch, and it should have an asterisk next to it as the current branch. E.g.
git branch
* age-collection
master
Getting the latest version of the data sources configuration
1. From your $HOME/git directory, get the latest version of your data sources repository. As you did for the sample web application, retrieve the SSH clone URL for your data sources forked repository and clone it, e.g.
git clone [email protected]:<your_username>/py-flask-signup-datasources.git
2. You should see a new directory, ‘py-flask-signup-datasources’. Change to that directory.
cd py-flask-signup-datasources
3. Change to the age-collection branch, e.g. git checkout age-collection
git branch
* age-collection
master
4. Configure the remote repository, e.g.
git config --global push.default simple
DO
NOT
git push --set-upstream origin age-collection
5. You should now have a cloned version of the py-flask-signup-datasources repository inside your git working area. We’ll use this for a later lab.
Testing the sample Python web application
1. If you inspect the contents of the py-flask-signup directory you’ll find our web application along with lots of other supporting configuration. The part we’re most interested in is the test suite. This is located in the tests/ directory.
a. We’ll run our tests from this directory, and use the Python unittest library. To run your test case, run python tests/application-tests.py.
b. Try running the test harness, python tests/application-tests.py. You should now see a Python import error, pointing to the fact that our application ‘module’ can’t be loaded. Something like:
ImportError: No module named application
To fix this error we need to export an environment variable, $PYTHONPATH c. To do this, type the following in your bash shell:
export PYTHONPATH=/home/ec2-user/git/py-flask-signup
If you now echo that environment variable you should see your working directory for the cloned py-flask-signup web application repository
echo $PYTHONPATH
/home/ec2-user/git/py-flask-signup
We’ll need to set this environment variable later on when we automate running these tests from our CI environment.
d. Try running the test harness again, python tests/application-tests.py. You should now see a Python module import error similar to,
ImportError: No module named flask
This message tells us that our development environment doesn’t have the necessary Python modules installed. In fact, we’re missing at least two modules for our Flask application,
e. To fix this, let’s install the Python package manager, pip, and then use pip to install the missing Flask modules. Run the following:
sudo yum install python26-pip
sudo pip install Flask Flask-SQLAlchemy
When we come to doing the work to automate the creation of our deployable application artifact in a later lab, we’ll have to bear this in mind as well.
f. Finally, try running the test harness again, python tests/application-tests.py. You should see something like the following:
DO
NOT
python tests/application-tests.py ... --- Ran 5 tests in 0.016s OK
Inspect the output. You should see tests run successfully, i.e. ‘OK’.
In a test driven development model, you would often see failing tests until bugs are fixes, or features are built against tests. We’re not trying to simulate test driven development here, just to point out the importance of having a test harness for your applications, how you might go about running it. The main goal will be to automate this test effort in subsequent labs, so we always test our code on every build and deploy.
2. Open tests/application-tests.py. Have a look at all the methods starting with “test_”. These are our tests in the test case we’re running.
3. Have a think about other tests you’d like to run. If you have time and are ambitious try implementing some of your own custom tests.
4. If you made any changes to the application or test harness, and making sure that you have an application that passes all the tests you can commit your changes. This will also version them for you and give you an audit trail. To check which files you’ve modified since working on your test harness run:
git status
This will show you the status of your local Git repository. You should for example see something like:
modified: application.py
modified: tests/application-tests.py
We’ll stage these files to be committed:
git add application.py tests/application-tests.py
And then commit them:
git commit –m “Bug fixes to ensure all tests pass successfully.”
5. The final task we want to do perform is push our changes back to the remote Github repository. Git makes this easy provided that we have the right permissions and have set up our SSH keys properly at the beginning of this lab.
DO
NOT
To do this, run these commands:
git config --global push.default simple
git push --set-upstream origin age-collection
You should see something like the following output if successful: Total 0 (delta 0), reused 0 (delta 0)
To [email protected]:aws-tools/py-flask-signup.git * [new branch] age-collection -> age-collection
Branch micro-blog set up to track remote branch age-collection from origin.
Congratulations! You’ve now implemented some basic source code management and built an automated test harness for your Flask application using the unittest library.
Having good development processes will help improve the quality of the software you’re building and delivering to your customers. In the next lab we’ll automate this process and build it into a continuous integration pipeline.
Additional Resources
AWS Training and Certification.
For feedback, suggestions, or corrections, please email: [email protected]
