An Experiment in Data Centre Automation

(1)

An Experiment in Data Centre

Automation

SmartFrog Dependency-Based Orchestration and Model Distribution

Patrick Goldsack, Andrew Farrell, Paul Murray, Peter Toft

HP Laboratories, Bristol

(2)

SmartFrog is…

A framework for describing, deploying and managing

distributed live-model management components.

•A description

language

for describing live models

•A

runtime

for realising the model descriptions

•A

component model

for defining live model components

•Predefined

components

to model specific things

sfConfig extends WebService { WebServer extends Apache {

port 8080; }

AppServer extends Jboss; }

(3)

Live Models

live model

reality

may pre-exist (physical) or be created when model is created (virtual, software)

management relationship: •create, configure, modify, monitor, terminate, recover •co-located & shared fate or remote & resilient

models are:

•fine-grained compositional representations of desired and actual reality

•components collaborate to achieve state, including coordination of actions •components may provide APIs for interactions

(4)

The Context

DB backend

(5)

Abstract Problem

peer-to-peer federated management systems security domains no central point for management

(6)

Abstract Problem

Orchestration of

virtual machine creation

virtual network endpoint management virtual storage element management

(7)

Abstract Problem

Orchestration of software and service configuration

(8)

Distributed Automation Framework

• A set of technologies aimed at providing

model-based orchestration in a distributed context

• Break into two aspects:

−

Federation of management systems:

model exchange

−

Orchestration within a management system:

compositional fine-grained

live state models

overlaid

with

dependencies

• Based on, and extends, SmartFrog runtime and

(9)

Federated Models

resource manager model injector model projector …

federation protocol endpoint

storage manager model yyy model xxx model VM0 model _model model

communications security and model-data provenance required.

VM0 model VM0

model

distribution through protocols with differing semantics for consistency

(10)

Security Issues

• How do we authenticate model data?

• signing

• network layer

• How do we delegate distribution?

• signing that survives transformation

• How do we guarantee model data freshness?

(11)

Local Orchestration

model

actuators or scripts parameterised live dependency model templates fine-grained live state components dependencies transform models create templates deploy/update/terminate interact with low-level actuators read/write model model data

injectors & projectors

protocol end-points

(12)

Live Models with Dependencies

• Define live models with their live components

managing “reality”

• Consider the states through which these live

components can transition

• Consider pre-conditions for these transitions

−

e.g. dependencies on other components having already

transitioned to new states

• Guard the live components with the

pre-conditions

−

Relevant for certain transitions

(13)

Example

size 500; create (targetVolumeState = “volCreated”); status “volNotCreated”;

targetVolumeState “volCreated”;

targetVMState “vmCreated”;

on by relevant  (targetVMState = “vmCreated”) enabled  (on:status = “volCreated”) memory 512; OS “win XP sp2”; install (targetVMState = “vmCreated”); status “vmNotCreated”; on by relevant  (targetVolumeState = “volNotCreated”) enabled  (on:status = “vmNotCreated”)

VM

volume

(14)

Example – Component View

!created Vol created _!created VM created

(15)

Properties

• Composition of models

• Abstraction and structural hiding

(16)

Model Checking SF Orchestrations

• Models checked for safety and liveness

conditions

−

essential for confirming intuition of model authors

−

check for absence of deadlock, and

−

arbitrary domain-dependent conditions as CTL/LTL

propositions on component/model state, for example:

• G(Xos:installed 

vm:created)

For the os to be installed, the vm must have already been created

• Done at model parsing stage, prior to deployment

• Uses NuSMV model checker

(17)

(18)

CMDB

actuators or scripts parameterised live dependency model templates transform models create templates deploy/update/terminate interact with low-level actuators read/write model fine-grained live state components dependencies

(19)

Example 2

create (state in {“installed”, ”prepared”}) memory service:vm:mem; status “pending”; MAC “”; install (state in {“installed”, ”prepared”} ) os service:OS; MAC VM:MAC; installed false;

install (state in {“installed”}); package service:name; installed false; software OS VM Radia HP VMM

four possible desired states:

ready: components launched, no action

prepared: VM created, OS installed, no Application installed: VM created, OS and Application both installed removed: uninstall and the model removed

service extends {memory 512; name “apache”; OS “winXP”;} desiredState “installed”;

(20)

Example 2

state desiredState;

enabled 

on:status == “created” relevant 

(21)

Example 2

state desiredState;

enabled 

on:status == “created”

relevant 

state in {“installed”, “prepared”}

relevant 

state in {“installed”}

enabled  on:installed

(22)

Example 2

state desiredState;

enabled 

relevant 

state in {“installed”}

enabled  on:installed

relevant 

state in {“removed”, “ready”} enabled !on:installed

enabled !on:installed relevant 

(23)

Example 2

state desiredState;

enabled 

relevant 

relevant  state in {“installed”} enabled  on:installed enabled !on:installed relevant 

state in {“removed”, “ready”}

relevant 

enabled !on:installed

relevant 

state in {“removed”} enabled  !on:created

(24)

Example 2 with failure handling

state (IF VM:status == “failed” THEN “ready” ELSE desiredState FI;

enabled 

relevant 

relevant  state in {“installed”} enabled  on:installed enabled !on:installed relevant 

relevant 

enabled !on:installed

relevant 

state in {“removed”}

(25)

As SmartFrog Orchestration Model

ManageAppDeployment extends Model {

service extends StateComponent { dstate ...; ...}; vm extends StateComponent {

tCreate extends Transition {

dependency "!created && (service:dstate == installed || ...prepared)"; statefunction extends {status "created";}

} ... status “pending”; memory service:memory; MAC "xxx"; } os extends StateComponent {...}; app extends StateComponent {...};

sfTerminateCondition “service:dstate == removed && !vm:status == created“; } StateComponents Dependency condition Orchestration State Non-orchestration state



Model Termination Condition StateComponents



Transition

State function determines next values for attributes

(26)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

Define: • configuration data • system composition • templates • parameterisation Imagine:

…a syntactically simpler and semantically richer equivalent to

(27)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

WebServer extends { portNumber 80; maxThreads 50; vDirectories extends { pages “/usr/local/pages”; cgi “/usr/local/cgi”; photos “/usr/photos”; } }

(28)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

WebServer extends { … } System extends {

server1 extends WebServer { portNumber 8080;

}

server2 extends WebServer { maxThreads 10;

sfProcessHost “s2.hp.com” }

(29)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

WebServer extends { … } LoadBAlancer extends { … } System extends {

server extends WebServer { portNumber 80; } lb extends LoadBalancer { sPort server:portNumber; } }

(30)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

System extends { port 80;

server1 extends WebServer { portNumber port;

}

server2 extends WebServer { portNumber (port + 1); }

}

anotherSystem extends System {

port 8080; }

(31)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

System extends {

port LAZY PROPERTY webport;

server1 extends WebServer { portNumber port;

}

server2 extends WebServer { portNumber (port + 1); }

(32)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

WebService extends { noWebServers TBD; noAppServers TBD;

cardinalities extends Assertions { web (noWebServers > 0); app ((noAppServers > 0) &

(noAppServers < 5)); all ((noWebServers +

noAppServers) < 10); }

}

aWS extends WebService { noWebServers 2;

noAppServers 6; }

 

(33)

Modelling

• Data, not functional

• Aggregation

• Prototyping

• Linking

• Functions

• Delayed binding

• Assertions and

Schemas

• Constraints

A prototype capability… aSystem extends { x VAR; y VAR; z VAR; } #suchThat# x :: [1..3], y :: [2..5], z :: [1..2], allDifferent([x,y,z]) #

mySystem extends aSystem { x 2;

(34)

access tier web tier application tier database tier edge routers routing switches authentication, DNS, intrusion detect, VPN 1st level firewall 2nd level firewall load balancing switches web servers web page storage

(NAS) database SQL servers storage area network (SAN) application servers files (NAS) switches switches min/max no. of web servers

min no. of app servers specific EJB’s size of data, no. of tables Template parameters transaction rate response times

constructed from templates for web server

application server …

prototyping and aggregation example of multiple domains (sub-)system templates require

strong notion of validation collections of sub-templates are

a common feature

(35)

Deployment

appServer extends Jetty { db LAZY database; log LAZY logging; }

appServer

logging extends Log { level = 3;

dir = “/logs”; logging

database extends MySql { log LAZY logging;

users [ [“web, “”] [“admin”, “secReT”] ]; } database log _log db

System extends Compound { logging extends …;

database extends …; appServer extends …; }

(36)

Anubis: A Model Federation Protocol

Anubis

System Components

Anubis servers use UDP multicast to:

discover other nodes

time pair-wise communication

determine partitions and stability properties

Comp

(37)

Anubis Model Transfer

Anubis

System Components

X = { information }

Listener

Wants to

know X

Provider

knows X

Anubis servers establish TCP connections on demand to communicate time-sensitive messages

Leader election determines placement of location service: servers register provider and listener names locator binds providers and listeners