Brian Connolly Systems Engineer, LabKey Software LabKey Server in the Cloud

(1)

LabKey Server in the Cloud

Brian Connolly

Systems Engineer, LabKey Software [email protected]

(2)

Agenda

 _{What is the Cloud?}

 _{Why would I want to use the cloud?}  _{What will it cost?}

 _{Using LabKey in the cloud}

 _{How does LabKey use the cloud?}  _{Other scientific tools in the Cloud}

(3)

Introduction

 _{Who am I}

(4)

(5)

What is the cloud Wikipedia says

Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet).

- http://en.wikipedia.org/wiki/Cloud_computing

What does this mean to me?

 _{rent vs buy}

 only pay for what you use

(6)

Types of Clouds

 _{Datacenter as a Service*}  _{Platform as a Service*}  _{Software as a Service*}

 _{How non-IT folks see the cloud vendors.}

(7)

Datacenter as a Service

From an API or GUI, you are able to provision and manage all pieces of a datacenter (servers, network hardware, storage, databases, etc).

What do you get?  _{Full control over:}

 _{servers (install and configure any way you want)}  _{network hardware (firewall, load balancing)}

 _{Pay as you use, self-service, and support custom configs}  _{Access to all the vendors services}

 _{Storage, Database, Message Queues, CDN, load balancers, firewall,etc}

(8)

Platform as a Service

“delivers a computing platform and/or solution stack as a

service, often consuming cloud infrastructure and sustaining cloud applications.”

- Wikipedia (http://en.wikipedia.org/wiki/IaaS#Platform)

What do you get?

 _{servers (or their equivalent)}  _{storage service}

 _{database service, etc}

 _{no firewall, load balancing, etc}

 Failover, clustering, custom configs…. maybe

(9)

Platform as a Service (cont)

There are two types of Platform as a Service  _{You get a server(s)}

 _{essentially the old hosted server model}

 _{existing hosting and VPS companies are in this space.}  _{major vendors: Rackspace, GoGrid, IBM, etc}

 You write some code and hit the deploy button.

 _{you never interact with the servers directly}

 _{your application code is bundled with “deployment descriptors” and}

sent to “cloud” via API

(10)

Software as a Service

“deliver software over the Internet, eliminating the need to install and run the application on the customer's own computers and simplifying maintenance and support.”

- Wikipedia (http://en.wikipedia.org/wiki/IaaS#Application)

 _{What do you get?}

 _{you are the end-user for this software}  _{personalization available}

 _{pay as you use}

(11)

(12)

Why would I want to use the cloud?

To meet a deadline….

 _{A reviewer asked for the samples processed using a new method.}  _{I need to process large number of samples for a grant application}

Prototyping: Try a new processing method

 _{Proteomics: Use an updated FASTA file or additional parameter}  _{Genomics: Reference sequence has changed}

 _{I have new hypothesis and want to quickly re-process my data}

(13)

Why would I want to use the cloud? (cont)

I want to try out new software to see if it meets my needs

 _{LabKey Online}

 _{Galaxy’s free public server}  _{UCSC Genome Browser}

I want to automate my pipeline

 _{Cyclecomputing.com (Push button HPC in the cloud)}  _Starcluster

 _{CloudBioLinux}

(14)

Why would I NOT want to use the cloud? (cont)

 _{Processing huge amounts of data}  _{Data transfer time is too long}

 _{small network pipe to the internet}

 _{transfer time + processing time in the cloud >= processing time in on}

your laptop

 _{I have a long running study (year or more) and I need to the}

computing around 24x7

(15)

(16)

What will it cost?

 _{What will I be changed for?}  _{How will I be billed?}

(17)

What will I get charged for?

When using the “cloud” you are renting the computers you need.

 _{Most clouds bill by hour}

 _{vendors: AWS, Rackspace,Windows Azure, Google App Engine, etc}  _{some do not (Heroku)}

 SAAS usually bills by the month

 _{vendors: Salesforce, etc}

(18)

How will I be billed?

 _{Billed monthly}

 _{Billed to credit card}

 _{Large institutions or large companies use purchase orders}  _{Monitoring usage and cost during the month}

(19)

How to estimate your costs

In general things you will get charged for are:

 _{Servers (instances)}  _{Network usage}

 _{ie what you send into and out of cloud}

 _Storage

(20)

Estimating Costs: Servers  _{What do I mean by Servers?}

 Called instances at AWS, Google App Engine and Windows Azure

 Called Dynos at Heroku  _{Usage is charged per hour}

 _{Price goes up with the size of the server}  _{How to estimate:}

 how many servers will you need?

 _{what type of servers do you need?}

 _{windows or linux}

 _{what will they be doing?}

 how big a server to do you need?

 where should they be located

 AWS: Spot instances

(21)

Estimating Costs: Network

 _{What do I mean by Network}

 _{Bandwidth into and out of the “cloud”}

 _{You are changed only for Bandwidth out of “cloud”}  _{Bandwidth into “cloud” is generally free}

 _{Bandwidth between servers is generally free}  _{Bandwidth between datacenters (not free)}  _{For most scientific applications}

 _{This is usually small compared to Servers}  _{100GB of traffic in a month = $15}

(22)

Estimating Costs: Storage

 _{What do I mean by Storage?}

 _{Amount of data you have stored in the cloud}  _{Windows Azure}

 _{$0.15/GB per month based on daily average}  _{You are charged for # of transactions}

 AWS

 _{$0.10/GB per month}

 _{You are charged for # of I/O requests}  _{For most scientific applications:}

(23)

(24)

Using LabKey in the cloud

 _{Who is doing it}

 _{Which clouds can run a LabKey Server}  _{Installing LabKey from scratch}

(25)

Who is running LabKey in the cloud?

 _LabKey

 _{LabKey Online}  _{Test servers}

 _{Non-Profit Research Institute}  _{Seattle based BioTech company}

(26)

Which clouds can run LabKey Server?

 _{Datacenter as a Service clouds}  _{Amazon Web Services}

 _{Some Platform as a Service clouds}  _Rackspace

 _GoGrid

 _{IBM Smartcloud}

LabKey currently cannot be used on

 _{Window Azure}

 _{Google App Engine}  _Heroku

(27)

Installing the LabKey in a cloud

1. Start a new instance at your cloud provider

1. Download the LabKey Server installer

 _{Windows Installer}

 _{Linux Installer (coming in 11.3)}

2. Install LabKey Server

 _{Instructions at}_{http://www.labkey.org}

(28)

(29)

How does LabKey do it?

 _{Use Amazon Web Services and Rackspace Cloud offerings}  _{Operating Systems}

 _{Linux: Ubuntu 10.04 LTS}

(30)

How does LabKey do it? (cont)

 _{Installation/Configuration}

 _{Choose latest Ubuntu AMI (} http://uec-images.ubuntu.com/releases/10.04/release/)

 Use EBS backed instances

 AWS: Use Cloudformation to provision

 Instances

 Networks (firewalls)

 Disks

(31)

How does LabKey do it? (cont)

 _{Data upload/download speeds}  _{what do we see here at FHCRC}  _{the “ship us a hard drive” option}

 _{Processor /memory combinations}  _{test and measure}

 _{Pipelines in the Cloud}  _{our experience}

(32)

What does it cost us?

Lets use LabKey Online as an example:

 _{Server stats}

 _{instance type: m1.large}

 _{(2) EBS volumes: 85GB total}  _{Operating System: Linux}  _{Datacenter: us-east-1c}

 _{Cost break-down (average monthly price: July->Oct 2011)}

Cost Price Percentage of Total

Instance $250.92 95.8%

(33)

(34)

Other scientific tools in the cloud

 _Galaxy

 _{Both SAAS and install on your own instances in the cloud}  _GenomeSpace

 _{Cytoscape, Galaxy, GenePattern, Genomica, Integrative Genomics}

Viewer (IGV), and the UCSC Browser in the cloud

 _{The Gaggle}

 _{The Gaggle is a framework for exchanging data between}

independently developed software tools and databases….

 _{CloudBioLinux}  _Starcluster

(35)

Key Messages

 _{LabKey has been run successfully in the cloud by both LabKey}

and a number of other customers

 _{We would love to help you get started using LabKey in the}

(36)

Any questions?

Brian Connolly

[email protected] 206-667-7521

(37)

If you use LabKey Server for your research, please reference one of these publications about the platform:

General Use: Nelson EK, Piehler B, Eckels J, Rauch A, Bellew M, Hussey P, Ramsay S, Nathe C, Lum K, Krouse K, Stearns D, Connolly B, Skillman T, Igra M. LabKey Server: An

open source platform for scientific data integration, analysis and collaboration. BMC

Bioinformatics 2011 Mar 9; 12(1): 71.

Proteomics: Rauch A, Bellew M, Eng J, Fitzgibbon M, Holzman T, Hussey P, Igra M, Maclean B, Lin CW, Detter A, Fang R, Faca V, Gafken P, Zhang H, Whitaker J, States D, Hanash S, Paulovich A, McIntosh MW: Computational Proteomics Analysis System (CPAS): An Extensible, Open-Source Analytic System for Evaluating and Publishing

Proteomic Data and High Throughput Biological Experiments. Journal of Proteome

Research 2006, 5:112-121.

Flow Cytometry: Shulman N, Bellew M, Snelling G, Carter D, Huang Y, Li H, Self SG, McElrath MJ, De Rosa SC: Development of an automated analysis system for data from

flow cytometric intracellular cytokine staining assays from clinical vaccine trials.