Open Source based C-DAC Cloud Infrastructure & Services

Open Source based

C-DAC Cloud Infrastructure &

Services

19 Mar 2014

NISG DeitY Industrial Cloud Workshop

New Delhi

Centre for Development of Advanced Computing

(C-DAC) Bangalore

19 Mar 2014 1

Dr. Prahlada Rao B.B

Joint Director

C-DAC

•

CDAC was launched as India’s initiative to Design & Develop

High

Performance Computers Indigenously in

1988

•

Premier R&D organization under the

Department of Electronics and

Information Technology

(DeitY)

,

Govt. of India

•

C-DAC is engaged in

R&D and Development of PARAM

Series of

High

Performance Machines

and also Specialized in

:

–

Cloud/Grid Computing Technologies

–

E-Governance Applications

– Multilingual , Ubiquitous Computing

–

Health Informatics

–

Embedded, Mobile computing

–

GIS…

–

Educational Research: ACTS

• Centre for Development of Advanced Computing :

http://

www.cdac.in

Presentation Plan

•

About C-DAC

•

C-DAC Cloud Computing Activities

o

C-DAC Cloud Infrastructure

o

Scientific Cloud Computing

o

SuMegha-Cloud Lab Kit

•

Service Delivery Model

•

Cloud Disseminations

•

Conclusions

Supercomputing

Evol

ution @ C-DAC

•

Param 8000

•

Param 9000

•

Param 10000

•

ParamPadma

•

GARUDA Grid

Comp

•

Param Yuva II

•

Cloud Comp

•

PetaFlop Comp

70 + Partners

10000 CPUs –

550TF

EGI,

CHAIN-REDS

220TB Storage

1500 +

Certificates

NKN

Focus Areas

•

PAN C-DAC Cloud Infrastructure

•

C-DAC Cloud Stack

•

Enablement of multi-centric applications on C-DAC Cloud

•

Cloud Interoperability

Highlights

•

Infrastructure as a service

•

Cloud Stacks :

–

Meghdooth

–

SuMegha (Scientific Cloud)

•

Storage as a Service

•

Enablement of Multi centric Applications

HPC as a Service:

MPI/Hadoop Virtual Cluster(VC) Problem Solving Environments

Bangalore:

HPC Applications

PAN C-DAC Cloud Initiatives: Overview

Chennai:

Meghdooth Cloud Stack

Bangalore:

SuMegha:

Scientific Cloud Stack

Noida

:

Mohali:

Courses

Mumbai:

Disaster

Recovery

Hyderabad:

E-learning

e-shikshak

Pune:

Bangalore: PAN C-DAC Cloud Infrastructure

PSE NGS PSE SFM SaaS

IaaS

Pan C-DAC Cloud Computing



C-DAC local afforts to setup cloud with limited infrastructure.

Cloud Stacks implemented



Based on open source.



Eucalyptus, Nimbus, OpenStack as the core engine.



Many extensions to meet production requirement.

Monitoring, metering and billing supported.



Elasticity based on runtime parameters.



Scientific Cloud Implemented

(

IaaS, SaaS, PaaS)

.

Scientific Cloud Stack-

SuMegha

Released

.



Cloud Storage as a Service (

CloudVault

) Developed



SuMegha_Cloud LabKit (

to Build Privite Clouds)

Developed



MeghaSikshak Cloud based eLearning Tool Developed

Currently Applications or Enabled on local infrastructure.

Pan C-DAC Cloud Infrastructure

•

Physical Cloud infrastructure Set up

•

Host Cloud Middleware Stacks:

Meghdoot, Sumegha-Scientific Cloud

•

Availability of the Cloud Infrastructure for applications.

•

Based on

Meghdoot/SuMegha

Cloud Stacks.

•

Physical Security, Power, Air Conditioning Available.

•

C-DAC Centres able to access resources for building and experimenting

with cloud applications.

•

Repository:

for Application Snapshots, Tools, and common tools.

•

Provide tool

s for repositories, cloud management.

•

Disaster Recovery

, backup for projects.

10 19 Mar 2014 C-DAC Cloud Services 2 EGov

PAN C-DAC Cloud: Applications

Pan C-DAC Cloud Computing Supports:



Pan C-DAC Cloud infrastructure Operating with: Meghdoot /

Sumegha Stacks.



Cloud Enable Applications of Pan C-DAC Cloud

Project

o

Hospital Information System(HIS) C-DAC, Noida

o

HIS- Health Care Applications C-DAC, Thiruvananthapuram

Learning Management System (e-Sikshak) C-DAC, Hyderabad

MANTRA (MAchiNe assisted TRAnslation tool) C-DAC Pune

Disaster Recovery as a Service (DRaaS) C-DAC Mumbai

o

eSanjeevani (A web based health care solution) C-DAC Mohali

11 19 Mar 2014 C-DAC Cloud Services 2 EGov

Private C-DAC network

Public network (Internet) over VPN

12 19 Mar 2014 C-DAC Cloud Services 2 EGov

Highlights

•

Infrastructure as a service

•

Cloud Stacks :

–

Meghdooth

–

SuMegha (Scientific Cloud)

•

Storage as a Service

•

Enablement of Multi centric Applications

C-DAC Scientific Cloud

C-DAC Scientific Cloud

Scientific Cloud

is a model for

on-demand

access to a

shared pool of

HPC resources

(e.g.,

networks, servers, storage, applications)

that

can be easily provisioned

as and when needed by the researchers

and scientists.

Benefits of Scientific Cloud

•

Easy access to large infrastructures

On demand access to HPC resources

Virtual ownership of cloud resources

•

Ease of deployment

•

Scientific Cloud provide an alternative for HPC outreach

SuMegha: Scientific Cloud

www.sumegha.in

Provides on demand access to Compute, Storage resources

and enables parallel environments.

Services Supported

:



Infrastructure as a service(IaaS)



Storage as a service(StaaS)



Software as a service(SaaS)

Stakeholders



Faculty and Students from engineering colleges



Researchers

SuMegha: Scientific Cloud

www.sumegha.in

•

Cluster as an Infrastructure Service (IaaS

)

•

Provisioning of on demand HPC clusters

•

Provisioning of Map reduce clusters to process large data sets

•

Support for MPI as a tightly-coupled applications category.

•

Design methodology for providing/supporting installation of MPI

libraries in multiple machines simultaneously

•

Storage as a Service (StaaS): CloudVault

•

SaaS:

–

Job Submission Portal (C_JSP) ,

Cloud Access Portals

–

Monitoring tools

•

PaaS:

Program development and analysis Tools

•

Applications as Service (AaaS)

PSEs

–

PSE for Climate Modelling

–

PSE’s for Bio Informatics

•

Types of applications supported

–

Sequential & Concurrent Applications

–

Data Parallel Applications -

Map Reduce

–

Parallel Programs:

OpenMP ,Pthreads, MPI

19 Mar 2014 C-DAC Cloud Services 2 EGov

SuMegha Stack

Managemen t

SCIENTIIFC APPLICATIONS (Bioinformatics, Climate modeling)

Cloud Middleware (NIMBUS, Openstack Swift)

SLA & Policy Mana-gement Monitori ng (NAGIOS + Ganglia) Accounti ng, Metering & Billing

Operating System (RHEL) Hypervisors (XEN) A U T H & S E C U R I T Y ) SaaS (Job submissi on portal) PaaS (HADOOP, MPI with High Perf IO)

IaaS (VM, Virtual Clusters , storage)

Interconnect ( Ethernet & infiniband) Cluster Virtualization (Contextualization)

Scientific Cloud portal and Problem Solving Environments

Computational and Storage resources

18 StaaS

(Storage as a Service)

SciCloud Service Offerings

IaaS

Cloud Portal

Virtual

Machine

(OS, Time, Type)

Virtual

Cluster

(No. of nodes, type, Time)

Hadoop

Cluster

MPI Cluster

Cloud Vault

(StaaS)

Web

interface

Infrastructure as a Service(IaaS)

•

Accessible using Self service portal

http://scientificcloud.cdacb.in/

•

Infrastructure as a Service

:

Provisioning of virtual machines(VM) &

virtual clusters (VC)

– VM Specs

• Small, medium, large images

• Access to cloud storage(cloud vault)

• Access to additional elastic storage, through CIFS gateway

– VC Specs

• Upto 8 nodes in a cluster

• Each node: Small, medium, Large

• Type of cluster : MPI, HADOOP

• Access to cloud storage(cloud vault)

• Access to additional elastic storage, through CIFS gateway

•

Secure Access :

Secure ssh based access

to the VMs & VCs through internet using

secure key

VM Size Details Size Detail

Size No. Of Virtual CPU RAM (GB)

Small 1 1

Medium 2 2

Large 4 4

Job Submission Portal for Virtual Clusters

•

Web based interface to

provide an easy & user

friendly mechanism for job

submission to MPI based

virtual clusters.

•

Enables users to run

Sequential & Parallel jobs

Features

•

Job submission through Torque job scheduler.

•

Job monitoring to know the Status of submitted jobs

•

Output/Error files can be Viewed/Downloaded from the Portal

•

User Management on Virtual Cluster

Cloud Vault

End-User

Swift

Physical

Storage

OpenStack Object

Storage

Cloud Vault

is your personal hard drive in the

cloud. Store any kind of file like Documents,

Music, Videos, Photos, and Virtual Machine

Images on

CDAC Cloud Vault

secure servers.

Cloud Vault provides access to storage space for storing and accessing data from anywhere, anytime

http://cloudvault.cdacb.in:8084

Features

– Easy access to Storage : Supports easy access

mechanisms such as Web interface, Linux clients

– File and directory operations : list, upload, download,

delete

– Data Sharing: Enables users to share their data with

the other registered users of cloud vault in a controlled way

– Interfacing with the Virtual Machines & Virtual Clusters

– Reliability & Availability: Provides 99.5% up-time guarantee and 3-way redundancy for data.

– Huge file support : Allows users to upload files from

range of MBs, GBs (upto 40GB)

– Data isolation and security: Data of one user is

isolated & secured from other

Storage as a Service : Cloud Vault

PSE for SFM

•

Web

based

problem

solving environment to

run

seasonal

forecast

model on MPI based

virtual cluster provisioned

by SuMegha

•

Enables

on

line

visualization of the results

•

Facility is provided to run

GrADS commands from

the

browser,

which

visualize the necessary

forecast parameters.

PSE-SFM: JOB SUBMISSION:

PSE-SFM: JOB MONITORING

PSE-SFM: VISUALIZATION

PSE for NGS

•

Problem Solving Environment for Next Generation

Sequencing (PSE for NGS)

•

Salient Features:

–

Executes the complete workflow Preprocessing(Quality of

input data),Indexing, Alignment , post processing,

Visualization

–

Run-Time Logs for debugging and Directory Trees to navigate

easily among the projects or different output files/directories.

–

Huge Datasets uploading via Internet (More than 10GB

supported).

–

Common View window for visualisations and other textual

outputs.

–

Notification Centre for the user to know which step is going

on at any given time and other information like which project

is active and which is pending.

Next Generation Sequencing

SuMegha Cloud Kit

•

Packaged software kit to enable the automated deployment of

private scientific clouds on the hardware.

•

Suitable for academic institutes & research organizations who

want to setup cloud labs.

Benefits



Automated deployment of cloud setup for Infrastructure as a service



Provides web based tools that supports easy and quick access to the

Virtual machines and Virtual HPC clusters



Simple and quick installation of all the cloud components.



Provides Pre-built OS images

with complex parallel environments like MPI

and Map Reduce to cater the HPC needs.



Provides Golden images

with preloaded HPC applications like Seasonal

Forecast Model ,Cloud Next generation pipeline with the required

visualization tools and job submission portal for easy parallel job

submission



Enables the users to setup cloud lab without having expertise of system

administration.

SuMegha Software Stack & Supported Images

Components :



Hypervisor



Xen



Cloud Middleware



Nimbus, Openstack Swift



Storage



Glusterfs



Portal



SuMegha portal



Job Submission Portal



Cloud Vault Portal



PSE



PSE for SFM



PSE for NGS Pipeline

Supported Images:



Virtual Machines



CentOS (small, large,

medium)



Job Submission Portal



Virtual Clusters



MPICH enabled with job

submission portal



HADOOP enabled



PSE enabled images



SFM



NGS Pipeline

Enabling Applications on Meghraj

using SuMegha Cloud Lab Kit

SuMegha Cloud Lab kit

Sophisticated Packaged software Kit to enable the automated

deployment of private clouds on the hardware.

Highlights

– Open source components: Xen Hypervisor , Nimbus(cloud middleware) and GlusterFS

– Cloud tools & Dashboard: Cloud Portal, Job Submission Portal and Problem Solving Environment (PSE), Cloud monitoring.

– Automated Installer: Installation and configuration of private cloud on the recommended hardware to setup Cloud Lab without needing expert knowledge of System Administration.

– Secured Accessto VMs through SSH using private keys

– Golden Images: VM’s of different sizes - small, medium, large with CentOS operating system

– Parallel Programming environmentswith MPI libraries and HADOOP framework.

Advantages

• Provides a hand holding mechanism for moving existing applications & building new applications on cloud

• Lower cost to application developers :Application developer need not pay for the cloud resources at the application development & testing phase

• Enhanced cloud adoption in several domains

Phases of Cloud Application Enablement using

SuMegha Lab kit

•

Enabling Existing Application

–

Setup cloud testing environment using SuMegha lab kit auto

installer

–

Test application on local cloud setup using VMs

–

Deploy application on Meghraj Cloud

•

Developing New Application

–

Developing Application on SuMegha Development Environment

(SDE) using Cloud APIs(openstack, eucalyptus,etc)

–

Deploy & Test Application on local setup

–

Deploy Application on Meghraj cloud

Elastic Framework for Cloud Applications

Elasticity Framework

allows cloud resources

(VMs, Storage) to dynamically scale-up or down as

required by the application at run-time

Building Blocks

•

User defined application configurations.

•

Cloud monitoring.

•

Cloud Scalar.

TENANT APP STOR E

SuMegha

Elasticity

Framework

Monitoring

Configuration

Deployment

Configuration

and Scripts

Production

Cloud A

Production

Cloud B

Developer

(Development Environment) SuMegha Lab kit

1

2

3

1

2

3

4

4

•

Developer Of cloud Application

:-1.

Create his application

2.

Provide Deployment configuration and scripts to the Elasticity

frame work

3.

Test on private cloud Created Using

SuMegha Cloud Labkit

4.

List that in AppStore

•

User of Cloud Application

1.

Choose the cloud application and provide the monitoring

threshold listed in app store

2.

The deployment configuration will be taken from Elasticity

Framework.

3.

Setting the threshold in the motoring component of Elasticity

Framework.

4.

Deployed on the cloud chosen by user.

19 Mar 2014 C-DAC Cloud Services 2 EGov 37

Steps to Cloud Application Enablement using

SuMegha Lab kit

Points to Ponder

•

Licensing for commercial cloud platforms

supported by SuMegha

•

Billing & Pricing based on cloud platform

chosen by the users

Cloud Publications/Patents

• Provisional patent on “A CLOUD STORAGE SERVICE FOR CLOUD COMPUTING”, Indian Patent Office, Provisional Patent Application No. 1392/CHE/2013, Patent Filed on: 28 March 2013.

• Shivay Veer Sharma, Anil Bhargava, Payal Saluja, Prahlada Rao B. B., Accessing E-Infrastructures using CDAC Scientific Cloud (CSC) Services , 2nd_{IEEE CCEM 2013,} Bangalore (Poster)

• Deepanshu S, Aman A, Ramesh Naidu Laveti, Arunachalam B, Vineeth Simon Arackal, Prahlada Rao B B; Problem Solving Environment for Seasonal Forecast Model on CDAC Scientific Cloud; ACC 2013, Bangalore

• AmanArora, Pavan Kumar, Vineeth Simon Arackal, Janaki Chintalapati, Arunachalam B,Prahlada Rao B B, Cloud NGS Tool: An Online Pipeline for Next Generation

Sequencing Data; ACC 2013, Bangalore

• Vineeth Simon Arackal, AmanArora, Deepanshu Saxena, Arunachalam B, Prahlada Rao B B; SciInterface: A Web-Based Job Submission Mechanism for Scientific Cloud Computing; IEEE CCEM 2013, Bangalore

• Payal Saluja, Prahlada Rao BB, Ankit Mittal, Rameez Ahmad, C-DAC Scientific Cloud: On demand provisioning of HPC resources for Scientific Applications, paper

published & presented at 18th International conference of parallel and distributed processing techniques and applications(PDPTA 2012) 6th-19th July 2012 at Las Vegas, Nevada, USA

• Prahlada Rao B. B, Payal Saluja, Neetu Sharma, Ankit Mittal, Shivay Veer Sharma, Cloud Computing for Internet of Things & Sensing Based Applications poster paper published and presented at 6th International conference on Sensing Technology (ICST 2012), 18-21st Dec 2012, Kolkata, India

Dissemination: Workshops/Trainings

Workshops Conducted: (3)

o Indo_Taiwan International Workshop on Cloud Computing & Its Applications, 7-8th Nov 2013, Bangalore

o Prahlada Rao B. B. , National Workshop on Cloud Computing, conducted at CDAC Knowledge Park, Bangalore, 21st_{May 2011 .(80+ participants from Industry, Academia)}

o National Work Shop on Virtualization, Feb 2011, C-DAC Bangalore ((70+ participants from Industry, Academia)

Tutorials

o Prahlada Rao B.B., Payal Saluja, Vineeth Simon Arackle, Half-Day Tutorial “Open Source Platforms for Scientific Clouds” CSI’s Annual Conference on Advances in Cloud Computing (ACC2012), 26th July 2012, Bangalore.India (60+ participants from Industry, Academia)

o Prahlada Rao, Vineeth Simon Arackal, Payal Saluja Tutorial on “Cloud Computing for HPC” C-DAC’s National Conference on Parallel Computing Technologies (PARCOMPTECH 2013) 21-23 Feb ,

Bangalore, India

o Prahlada Rao B.B , Tutorial on “Scientific Clouds” International Conference on Advances in

Computing, Communications and Informatics (ICACCI-2013) , Mysore

o Prahlada Rao B.B , Invited Talk on “Trends in Cloud Computing” International conference on Advances

in Cloud Computing (ACC 2013), Bangalore, 19-20 Oct 2013, India

o Cloud Computing Talks: 15+ Invited /KeyNote Talks in Intl Conferences/ Workshops, and TEQiP programs

Dissemination: Technical Talks Given

• Prahlada Rao B.B., “Resent Trends in Cloud Computing” Technology day Talk C-DAC Bangalore 10th _{May 2013.}

• Prahlada Rao B.B., “CDAC Cloud Initiatives”Workshop on National Cloud Innovation Council, C-DAC Bangalore 29th _{April 2013.}

• Prahlada Rao B.B., “HPC for BigData”International Workshop on Data Analytics and Applications (IWDAA) 2013, BITS, Goa Campus, 1st _Mar2013.

• Prahlada Rao B.B, Payal Saluja, Neetu Sharma, Ankit Mittal, Shivay Veer Sharma, “Cloud Computing for IOT & Sensing Based Applicationss”, in IEEE International Conference on Sensing

Technologies, ICST 2012, Kollkata, India, Dec 2012.

• Prahlada Rao B.B., Tutorial On “Cloud Computing”, International Conference on Trends

in Knowledge and Information Dynamics, ICTK 2012, organized by ISI, NIMHANS Conv Centre, Bangalore, India, 10 July 2012.(150 participants from Industry, Academia)

• Payal Saluja, Prahlada Rao, Ankit Mittal, Rameez Ahmad, “CDAC Scientific Cloud: On Demand Provisioning of Resources for Scientific Applications”, in Proceedings of 18th _{International}

Conference PDPTA 2012, Las vegas, Nevada, USA, 16-19 July 2012.

• Dr. Prahlada Rao BB, Cloud Computing for Telemedicine, National Workshop on Telemedicine Today & Tomorrow, C-DAC Mohali, An Invited Talk, 6th_{July 2012.}

• Vineeth Simon Arackal, Arunachalam B, Payal Saluja, Prahlada Rao B B, Co-existence of Cloud and Grid: A Case Study in Service Oriented Grid GARUDA. The International Conference on Information Technology- New Generations (ITNG 2011), Las Vegas, USA. 11-13April 2011.

• Vineeth Simon Arackel, “ Coexistence of Grid and Cloud” at "Grid To Cloud (G2C)- Challenges and Applications" Workshop at VIT, Vellore, 2nd April 2011.

41 19 Mar 2014 C-DAC Cloud Services 2 EGov

Conclusions

19 Mar 2014 C-DAC Cloud Services 2 EGov 42

•

Middleware Development for Cloud, Grid

&

HPC:C-DAC Strength

•

C-DAC Cloud Activities

o

Open Source based C-DAC Cloud Testbed

o

Cloud Applications of National Importance

o

Scientific Cloud Computing Research

o

SuMegha- Cloud LabKit for Privite Cloud Building

o

Cloud Interoperability Research

o

Elastic Cloud Applications Development

•

C-DAC Founding member of IntercloudTestbed-IEEE

Thank You