Cloudmesh: Software Defined Distributed Systems as a Service SDDSaaS

(1)

Cloudmesh: Software Defined

Distributed Systems as a Service

SDDSaaS

Workshop on the Development of a Next-Generation, Interoperable,

Federated Network Cyberinfrastructure

Washington DC

October 1 2014

Geoffrey Fox, Gregor von Laszewski

[email protected]

http://www.infomall.org

School of Informatics and Computing

Digital Science Center

(2)

Origins and Future of Cloudmesh

• _Past:

_{Needed to move back and forth between}

_{Bare Metal}

_{and different}

_VM

managers

in FutureGrid using emerging DevOps ideas like

Chef

and templated

(software defined)

image libraries

– _{Address many different changing tools with abstractions}

• Integrate

new metrics

in form consistent with XSEDE at execution (user) and

job summary levels

• _{Current Focus/Futures:}

_{Preserves and builds on user/project}

/experiment/provisioning/metrics structure of FutureGrid

• _{Now linking of}

_{system definition}

_and

_{system execution}

_{steps in a common}

Python environment while future additions could include

Software Defined

Networking

(as described in previous talks)

– _{System execution classically called orchestration or workflow i.e. our view of SDDS}

includes infrastructure and software including multiple workflow steps

• _{Now used to support}

_laboratories

_for

_{online classes}

_{in data science and for}

several large scale

data analytics research

,

education

and

standards

projects

including

RDA

(Research Data Alliance) &

NIST

Public Working Group in Big

Data

(3)

FutureGrid

(4)

4 M a n a g e m e n t S e c u r it y & P r iv a c y

Big Data Application Provider

Visualization

Visualization _Access_Access Analytics Analytics Curation Curation Collection Collection System Orchestrator DAT A SW DAT A SW I N F O R M AT I O N V A L U E C H A I N

IT V A L U E C H A IN D at a C o n su m e r D at a P ro vi d e r

Horizontally Scalable (VM clusters)

Vertically Scalable Horizontally Scalable

Vertically Scalable Big Data Framework Provider

Processing Frameworks (analytic tools, etc.)

Platforms (databases, etc.)

Infrastructures

Physical and Virtual Resources (networking, computing, etc.)

D A TA D A TA SW

K E Y :

SW SW Service Use Data Flow Analytics Tools Transfer DATA

Instantiate/Test NIST Big Data Reference Architecture

http://bigdatawg.nist.gov/V1_output_docs.php

Strong Industry Participation

(5)

Kaleidoscope of (Apache) Big Data Stack (ABDS) and HPC Technologies

Cross-Cutting Functionalities

Message and Data Protocols: Avro, Thrift, Protobuf Distributed Coordination: Zookeeper, Giraffe, JGroups Security & Privacy: InCommon, OpenStack Keystone, LDAP, Sentry Monitoring: Ambari, Ganglia, Nagios, Inca

Workflow-Orchestration: Oozie, ODE, Airavata, OODT (Tools), Pegasus, Kepler, Swift, Taverna, Trident, ActiveBPEL, BioKepler, Galaxy, IPython, Dryad, Naiad, Tez, Google FlumeJava, Crunch, Cascading, Scalding, e-Science Central,

Application and Analytics: Mahout , MLlib , MLbase, CompLearn, R, Bioconductor, ImageJ, Scalapack, PetSc, Azure Machine Learning, Google Prediction API, Google Translation API

High level Programming: Kite, Hive, HCatalog, Tajo, Pig, Phoenix, Shark, MRQL, Impala, Presto, Sawzall, Drill, Google BigQuery (Dremel), Microsoft Reef, Google Cloud DataFlow, Summingbird

Basic Programming model and runtime, SPMD, Streaming, MapReduce: Hadoop, Spark, Twister, Stratosphere, Llama, Hama, Giraph, Pregel, Pegasus

Streaming: Storm, S4, Samza, Google MillWheel, Amazon Kinesis

Inter process communication Collectives, point-to-point, publish-subscribe: Harp, MPI, Netty, ZeroMQ, ActiveMQ, RabbitMQ, QPid, Kafka, Kestrel

Public Cloud: Amazon SNS, Google Pub Sub, Azure Queues

In-memory databases/caches: GORA (general object from NoSQL), Memcached, Redis (key value), Hazelcast, Ehcache

Object-relational mapping: Hibernate, OpenJPA and JDBC Standard

Extraction Tools: UIMA, Tika

SQL: Oracle, MySQL, Phoenix, SciDB, Apache Derby, Google Cloud SQL, Azure SQL, Amazon RDS

NoSQL: HBase, Accumulo, Cassandra, Solandra, MongoDB, CouchDB, Lucene, Solr, Berkeley DB, Riak, Voldemort. Neo4J, Yarcdata, Jena, Sesame, AllegroGraph, RYA, Parquet, RCFile, ORC

Public Cloud: Azure Table, Amazon Dynamo, Google DataStore

File management: iRODS

Data Transport: BitTorrent, HTTP, FTP, SSH, Globus Online (GridFTP), Flume, Sqoop

Cluster Resource Management: Mesos, Yarn, Helix, Llama, Condor, SGE, OpenPBS, Moab, Slurm, Torque

File systems: HDFS, Swift, Cinder, Ceph, FUSE, Gluster, Lustre, GPFS, GFFS

Public Cloud: Amazon S3, Azure Blob, Google Cloud Storage

Interoperability: Whirr, JClouds, OCCI, CDMI

DevOps: Docker, Puppet, Chef, Ansible, Boto, Libcloud, Cobbler, CloudMesh

IaaS Management from HPC to hypervisors: Xen, KVM, OpenStack, OpenNebula, Eucalyptus, CloudStack, VMware vCloud, Amazon, Azure, Google Clouds

Networking: Google Cloud DNS, Amazon Route 53

(6)

Cloudmesh: from IaaS(NaaS) to Workflow

(Orchestration)

(SaaS Orchestration)

Workflow

(IaaS Orchestration)

Virtual Cluster

Components

Infrastructure

• _IPython

• _{Pegasus etc.}

• _Heat

• _Python

• _chef

• _apt-get/yum

• _{VMs, Networks,}

Baremetal

Im

ag

es

Im

ag

es

D

at

a

(7)

Cloudmesh and SDDSaaS Stack for HPC-ABDS

SaaS

PaaS

IaaS

NaaS

BMaaS

Orchestration

Mahout, MLlib, R Mahout, MLlib, R

Hadoop, Giraph, Storm Hadoop, Giraph, Storm

OpenStack, Bare metal OpenStack, Bare metal

OpenFlow OpenFlow

Just examples from 150 components

Cobbler Cobbler

Abstract

Interfaces removes tool dependency

IPython, Pegasus, Kepler, FlumeJava, Tez, Cascading IPython, Pegasus, Kepler, FlumeJava, Tez, Cascading

One Chef recipe per IU CS Masters Student …. Data Distributed and Streaming …

(8)

(9)

CloudMesh Architecture

• _{Cloudmesh is a}

_SDDSaaS

_{toolkit to support}

– A software-defined distributed system encompassing virtualized and bare-metal

infrastructure, networks, application, systems and platform software with a unifying goal of providing Computing as a Service.

– The creation of a tightly integrated mesh of services targeting multiple IaaS

frameworks

– The ability to federate a number of resources from academia and industry. This includes existing FutureSystems infrastructure, Amazon Web Services, Azure, HP Cloud, Karlsruhe using several IaaS frameworks

– The creation of an environment in which it becomes easier to experiment with

platforms and software services while assisting with their deployment and execution.

– The exposure of information to guide the efficient utilization of resources. (Monitoring)

– Support reproducible computing environments

– IPython-based workflow as an interoperable onramp

• _{Cloudmesh exposes both hypervisor-based and bare-metal provisioning to}

users and administrators

(10)

Cloudmesh Functionality

User On-Ramp

Amazon, Azure, FutureSystems, Comet, XSEDE, ExoGeni, Other Science Clouds

Cloudmesh

Information

Services

• _CloudMetrics

Information

Services

• _CloudMetrics

Provisioning

Management

• _Rain

• _{Cloud Shifting} • _{Cloud Bursting}

Provisioning

Management

• _Rain

• _{Cloud Shifting} • _{Cloud Bursting}

Virtual Machine

Management

• _{IaaS Abstraction}

Virtual Machine

Management

• _{IaaS Abstraction}

(11)

Building Blocks of Cloudmesh

• _{Uses internally}

_{Libcloud and Cobbler}

• _{Celery Task/Query manager (AMQP - RabbitMQ)}

• _MongoDB

• _{Accesses via abstractions}

_{external systems/standards}

• _{OpenPBS, Chef}

• _{OpenStack (including tools like Heat), AWS EC2, Eucalyptus,}

Azure

• _{Xsede user management (Amie) via Futuregrid}

• _Implementing

_{Docker, Slurm, OCCI, Ansible, Puppet}

(12)

SDDS Software Defined Distributed Systems

• _Cloudmesh

_{builds infrastructure as SDDS consisting of one or more virtual clusters}

or slices with extensive built-in monitoring

• _{These slices are instantiated on infrastructures with various owners}

• _{Controlled by roles/rules of Project, User, infrastructure}

Python or REST API User in Project User in Project CMPlan CMPlan CMProv CMProv CMMon CMMon Infrastructure (Cluster, Storage, Network, CPS) Infrastructure (Cluster, Storage, Network, CPS)

Instance Type

Current State

Management Structure

Provisioning Rules

Usage Rules (depends on user roles) Results Results CMExec CMExec User RolesUser Roles

User role and infrastructure rule dependent security

checks

User role and infrastructure rule dependent security

checks

Request

Execution in Project

Request SDDS

Select

Plan Requested SDDS as _{federated Virtual} Infrastructures Requested SDDS as

federated Virtual Infrastructures #1Virtual infra. Linux #2 Virtual infra. Windows #3Virtual infra. Linux #4 Virtual infra. Mac OS X Repository Repository Image and Template Library SDDSL SDDSL 

_{One needs general}

hypervisor and

bare-metal slices to

support research



_Gives an

experiment

management

system

that

enables

(13)

What is SDDSL?

• _{There is an active OASIS standard activity}

_TOSCA

_(Topology

and Orchestration Specification for Cloud Applications)

• _{But this is similar to mash-ups or workflow (Taverna,}

Kepler, Pegasus, Swift ..) and we know that workflow itself

is very successful but workflow standards are not

–

_{OASIS WS-BPEL}

_{(Business Process Execution Language) didn’t}

catch on

• _{As basic tools (Cloudmesh) use Python and Python is a}

popular scripting language for workflow, we suggest that

Python

could be

SDDSL

–

_{IPython Notebooks are natural log of execution provenance}

–

_{Explosion of new Commercial (Google Cloud Dataflow) and}

(14)

Cloudmesh as an On-Ramp

• _{As an On-Ramp, CloudMesh deploys recipes on}

multiple platforms so you can test in one place and do

production on others

• _{Its multi-host support implies it is effective at}

distributed systems

• _{It will support traditional workflow functions such as}

–

_{Specification of an execution dataflow}

–

_{Customization of Recipe}

–

_{Specification of program parameters}

• _{Workflow quite well explored in Python}

_https://

wiki.openstack.org/wiki/NovaOrchestration/Workflo

wEngines

(15)

Cloudmesh: Integrated Access Interfaces

(Horizontal Integration)

GUI

(16)

(17)

… Register clouds

Multiple clouds are registered

(18)

… Work with VMs

VMs

Panel with VM Table (HP)

Search

(19)

(20)

Provisioning OpenStack

View the parallel provisioning tasks execution from AMPQ

(21)

Monitoring and Metrics Interface

• _{Service Monitoring}

• _{Energy/Temperature}

Monitoring

• _{Monitoring of}

Provisioning

• _{Integration with other}

Tools

–

_{Nagios, Ganglia, Inca, FG}

Metrics

–

_{Accounting metrics}

(22)

Cloudmesh MOOC

(23)

(24)

Infra

structure

IaaS

 Software Defined

Computing (virtual Clusters)

 _{Hypervisor, Bare Metal}  Operating System

Platform

PaaS

 Cloud e.g. MapReduce

 _{HPC e.g. PETSc, SAGA}  _{Computer Science e.g.}

Compiler tools, Sensor nets, Monitors

Software-Defined Distributed

System (SDDS) as a Service includes

Network

NaaS

 Software Defined Networks

 _{OpenFlow GENI}

Software

(Application Or Usage)

SaaS

 Use HPC-ABDS

 _{Class Usages e.g. run}

GPU & multicore

 Applications

 _{Control Robot}

FutureGrid used SDDS-aaS Tools

 Provisioning

 Image Management

 IaaS Interoperability

 NaaS, IaaS tools

 Expt management

 Dynamic IaaS NaaS

 DevOps

FutureGrid used SDDS-aaS Tools

 Provisioning

 Image Management

 IaaS Interoperability

 NaaS, IaaS tools

 Expt management

 Dynamic IaaS NaaS

 DevOps

CloudMesh is a

SDDSaaS tool that uses Dynamic Provisioning and Image Management to provide custom

environments for general target systems

Involves (1) creating, (2) deploying, and (3) provisioning

of one or more images in a set of machines on demand

http://mycloudmesh.org/

24

(25)

Cloudmesh Architecture

• Cloudmesh

Management

Framework

for

monitoring and

operations, user and

project management,

experiment planning

and deployment of

services needed by an

experiment

• _{Provisioning and}

execution

environments to be

deployed on resources

to (or interfaced with)

enable experiment

management.

• _Resources

_.

(26)

CloudMesh Administrative View of SDDS aaS

• _CM-BMPaaS

_{(Bare Metal Provisioning aaS) is a systems view and allows}

Cloudmesh to dynamically generate anything and assign it as permitted by user

role and resource policy

– _{FutureGrid machines India, Bravo, Delta, Sierra, Foxtrot are like this}

– _Note this_{only implies user level bare metal access}_{if given user is authorized and this is}

done on a per machine basis

– _It_{does imply dynamic retargeting of nodes}_{to typically safe modes of operation}

(approved machine images) such as switching back and forth between OpenStack, OpenNebula, HPC on Bare metal, Hadoop etc.

• _CM-HPaaS

_{(Hypervisor based Provisioning aaS) allows Cloudmesh to generate}

"anything" on the hypervisor allowed for a particular user

– _{Platform determined by images available to user} – _{Amazon, Azure, HPCloud, Google Compute Engine}

• _CM-PaaS

_{(Platform as a Service) makes available an essentially fixed Platform}

with configuration differences

– _{XSEDE with MPI HPC nodes could be like this as is Google App Engine and Amazon HPC}

Cluster. Echo at IU (ScaleMP) is like this

– _{In such a case a system administrator can statically change base system but the}

(27)

CloudMesh User View of SDDS aaS

• _{Note we always consider virtual clusters or slices with nodes}

that may or may not have hypervisors

• _{Well defined user and project management assigning roles}

• _BM-IaaS

_{: Bare Metal (root access) Infrastructure as a service}

with variants e.g. can change firmware or not

• _H-IaaS:

_{Hypervisor based Infrastructure (Machine) as a Service.}

User provided a collection of hypervisors to build system on.

–

_{Classic Commercial cloud view}

• _PSaaS

_{Physical or Platformed System as a Service where user}

provided a configured image on either Bare Metal or a

Hypervisor

–

_{User could request a deployment of Apache Storm and Kafka to}

(28)

Cloudmesh Components I

• _Cobbler:

Python based provisioning of bare-metal or hypervisor-based systems

• _{Apache Libcloud:}

_{Python library for interacting with many of the}

popular cloud service providers using a unified API. (One Interface To

Rule Them All)

• _Celery

_{is an asynchronous task queue/job queue environment}

based on RabbitMQ or equivalent and written in Python

• _{OpenStack Heat}

_{is a Python orchestration engine for common}

cloud environments managing the entire lifecycle of infrastructure

and applications.

• _Docker

_{(written in Go) is a tool to package an application and its}

dependencies in a virtual Linux container

• _OCCI

_{is an Open Grid Forum cloud instance standard}

• _Slurm

_{is an open source C based job scheduler from HPC community}

(29)

Cloudmesh Components II

• _Chef

_{Ansible Puppet Salt}

_are system

configuration managers. Scripts are used to define system

• _Razor

_{cloud bare metal provisioning from EMC/puppet}

• _Juju

_{from Ubuntu orchestrates services and their}

provisioning defined by charms across multiple clouds

• _Xcat

_{(Originally we used this) is a rather specialized (IBM)}

dynamic provisioning system

• _Foreman

_{written in Ruby/Javascript is an open source}

project that helps system administrators manage servers

throughout their lifecycle, from provisioning and

(30)

(31)

Background - FutureGrid

• _{Some requirements originate from FutureGrid.}

– _{A high performance and grid testbed that allowed scientists to collaboratively develop}

and test innovative approaches to parallel, grid, and cloud computing.

– _{Users can deploy their own hardware and software configurations on a public/private}

cloud, and run their experiments.

– Provides an advanced framework to manage user and project affiliation and propagates this information to a variety of subsystems constituting the FutureGrid service

infrastructure. This includes operational services to deal with authentication, authorization and accounting.

• _{Important features of FutureGrid:}

– Metric framework that allows us to create usage reports from all of our IaaS frameworks. Developed from systems aimed at XSEDE

– _{Repeatable experiments}_{can be created with a number of tools including Cloudmesh.}

Provisioning of services and images can be conducted by Rain.

– _{Multiple IaaS frameworks}_{including OpenStack, Eucalyptus, and Nimbus.}

– _{Mixed operation model}_._{a standard production cloud}_{that operates on-demand, but also}

a set of cloud instances that can be reserved for a particular project.

(32)

Functionality Requirements

• _{Provide virtual machine and bare-metal management in a}

multi-cloud

environment with very different policies and including

–

_{Expandable resources,}

–

_{External clouds from research partners,}

–

_{Public clouds,}

–

_{My own cloud}

• _{Provide multi-cloud services and deployments controlled by users}

_&

provider

• _Enable

_raining

_of

–

_{Operating systems (bare-metal provisioning),}

–

_Services

–

_Platforms

–

_IaaS

• _{Deploy and give access to}

_Monitoring

infrastructure across a multi-cloud environment

(33)

Cloudmesh Provisioning and Execution

• _{Bare-metal Provisioning}

– _{Originally developed a provisioning framework in FutureGrid based on xCAT and Moab.}

(Rain)

– _{Due to limitations and significant changes between versions we replaced it with a}

framework that allows the utilization of different bare-metal provisioners.

– _{At this time we have provided an interface for cobbler and are also targeting an interface}

to OpenStack Ironic.

• _{Virtual Machine Provisioning}

– An abstraction layer to allow the integration of virtual machine management APIs based on the native IaaS service protocols. This helps in exposing features that are otherwise not accessible when quasi protocol standards such as EC2 are used on non-AWS IaaS frameworks. It also prevents limitaions that exist in current implementations, such as libcloud to use OpenStack.

• _{Network Provisioning}

_(Future)

– _{Utilize networks offering various levels of control, from standard IP connectivity to}

completely configurable SDNs as novel cloud architectures will almost certainly leverage NaaS and SDN alongside system software and middleware. FutureGrid resources will make use of SDN using OpenFlow whenever possible though the same level of