Why we Picked CF as the Basis for our Public Cloud Multi-Tenant Platform

Why we Picked CF as the Basis for our

Public Cloud Multi-Tenant Platform

Mike Root @mikersj

THE

ANNOYING

WE ONLY WANT TO BUILD APPLICATIONS

Our Stuff is Complex

tenants

app

lica

THE

ANNOYING

WE ONLY WANT TO BUILD APPLICATIONS

Our Stuff is Complex

s

TENANT

PARTITIONING

Shared

Services

V7

Shared

Services

V7

Shared

Services

V6

Shared

Services

V6

Supplier Exchange

v15

Tenant Admin

v1

Supplier Exchange Client

acme.emcond.com

xDB cluster (metadata)

xDB

Server

Cassandra cluster

Cassandra

Server

Cassandra

Server

Cloud Blob Store

SWIFT

SWIFT

SWIFT

Acme

Supplier Exchange

v15

Supplier Exchange

v16

Router

Supplier Exchange Client

FBPortal.emcond.com

FooBar

acme.emcond.com

: se v16

FBPortal.emcond.com : se v15

xDB

Server

PRODUCTION

SETUP

CloudFoundry

BOSH managed VM’s

Secure Full Text

Engine

CF Service

Cassandra

CF Service

xDB

Swift Blob Store

ClamAV

(virus check)

Firestone

Authentication

CF Service

RabbitMQ

Analytics Engine

Transformation

Services

Metadata Service

Platform Mgt

Console

Tenant Mgt

Console

Supplier Exchange

Other VM’s

HAWQ

Pivotal HD

Windows VM

AD/ADFS/vCenter etc

Warden Containers

Concurrent

Authoring Service

BPM Service

Case Management

Service

Retention Mgt

NETWORK

SEGREGATION

BOSH

CF Other

CF Router

DEA

CF

Services

CLOUDFOUNDRY

BENEFITS

BOSH provisions the entire datacenter



Repeatable at any time



No downtime

Cloud Foundry



Handles application scalability



Tenant (Customer) modifiable URL

Upgrade tool



Uses the CF API



Blue Green upgrade (0 downtime)



Dynamically move/configure tenants

Promotion process is fully automated

Shellshock, OS Security vulnerability

Exa

m

ple

1

2

3

We updated the stem-cell (OS)

Ran “bosh deploy”



16 CF environments updated (dev/test/pre-prod/prod)



16x30 VM’s updated



No down time

Drank beer (optional)

CLOUDFOUNDRY

BENEFITS

WE DIDN’T BUILD

ANY OF THIS

Monitoring



Monitoring VMs



Monitoring applications

Resource scaling

High Availability



VM HA



Application HA

Log collection

Health metrics

Shellshock, OS Security vulnerability

SPIFF

GENERATION

OF BOSH

MANIFESTS



16 Deployments managed with auto generated

manifests



Core templates



16 Instance specific deployments

$> spiff merge cf-jobs.yml cf-network.yml vcenter.yml ci.yml

dev.yml

ci_merged.yml

dev_merged.yml

preprod_merged.yml

prod_merged.yml

preprod.yml

prod.yml

Support for multiple networks

CLOUDFOUNDRY

UPGRADE TOOL

BLUE

GREEN

DEPLOYMENT



CloudFoundry API to deploy applications



SaaS REST to move/configure tenants



e.g. new security rules, enable features etc



Old and new versions running in parallel

CloudFoundry

NGIS Blue

NGIS Green

App Blue

App Blue

App Green

T0-A1

T0-A4

T1-A1

T2-A1

T3-A1

T5-A1

T4-A1

T3-A2

T0-A2

GAPS

WE

CONQUERED



OS hardening; Ubuntu patch version, ssh permissions, file access etc etc



Deploying in multiple networks with firewalls



CF/BOSH/DEA/Router/Services



Spiff not setup for multiple networks



Keeping up to date with latest CF release



Converting v1 services to v2 services



DR setup



CF cli is constantly changing, but CF api is more stable



We use the API for the upgrade tool



Debugging applications in CloudFoundry (step through code)



Filesystem options/type for persistent storage (swift)



Filesystem size of /tmp is not configurable



BOSH builds are more difficult because BOSH doesn’t support artifactory as a BLOB

store



Managing multiple BOSH deployments



Log files other than stderr/stdout



Collecting them



Limited disk space



SPIFF manifests are harder to read by a human



Spiff diff helps



Application needs to report ready before ready for large application



Router networking resources exhausted



keep-alive disabled



CF staging blobs not cleaned up properly



CF routes not cleaned up properly

CONTRIBUTIONS

TO

CLOUNDFOUNDRY

WE ONLY WANT TO BUILD APPLICATIONS

Bosh releases



ClamAV



First EMC Contribution to CF.org



SWIFT



HA Proxy



Deployment VM



BOSH CLI



CF CLI



User management ability



Cassandra as a CF Service



Zabbix Agent (open source monitoring tool)



Network yml

Service Brokers



Cassandra

PROMOTION

PROCESS

Upgrade Tool / BOSH: Configuration Management, Orchestration

SaaS Continuous Deployment Process

QE

Validation

Performance

Validation

Integration

Validation

Upgrade Validation

Pre-Prod

Validation

Production

CI

DEV

Validation

#804 Functional Test

Build #806

Build #805

Build #804

Build #803

Build #802

#804 L10N/I18N Test #804 Performance Test #804 Longevity Test #804 Integration Test #804 Upgrade Test #804 Environment Test #804 Datacenter 1 #804 Datacenter 2 #804 Datacenter 3 #804 Datacenter 4

1

Cloud Foundry

Cloud Foundry

2

Cloud Foundry

3

Cloud Foundry

4

Cloud Foundry

5

Cloud Foundry

6

Cloud Foundry

7-8

12 hours

6 hours

28 mins

30 minutes

DEPLOYMENT

HISTORY

GA

29 releases in 37 weeks since GA



8 BOSH upgrades



2 CloudFoundry upgrades



22 upgrades of 3 applications

Q & A