Comparing Cloud Computing Resources for Model Calibration with PEST

(1)

Comparing Cloud Computing Resources

for Model Calibration with PEST

Charles Brush

Modeling Support Branch, Bay-Delta Office

California Department of Water Resources, Sacramento, CA

CWEMF Annual Meeting

March 10, 2015

(2)

Outline

Problem Statement

Computing Options

Performance Issues & Metrics

Systems Tested

Test Results

Cost Comparison

Conclusions

(3)

3

(4)

IWFM Parameters

Saturated Zone

–

Conductivity (K

_h

, K

_v

)

–

Storage (S

_y

, S

_s

)

–

Subsidence (S

_e

, S

_i

)

Unsaturated Zone

–

Porosity

–

Conductivity

Water Management

–

Diversion losses

–

Irrigation efficiency

–

Re-use factors

Land Surface

–

Field capacity

–

Porosity

–

Recharge factor

–

Curve Numbers

River nodes

–

Conductance

Small Watersheds

–

Field capacity

–

Porosity

–

Conductivity

–

Recession coefficients

(5)

PEST Optimization Iteration

Calculate Jacobian matrix

(run model at least once for each

adjustable parameter)

Calculate several sets of

new parameter values

Select the parameter set with the

lowest objective function value

(6)

Pilot Points

Saturated Zone

–

Conductivity (K

_h

, K

_v

)

–

Storage (S

_y

, S

_s

)

–

Subsidence (S

_e

, S

_i

)

Unsaturated Zone

–

Porosity

–

Conductivity

Water Management

–

Diversion losses

–

Irrigation efficiency

–

Re-use factors

(7)

(8)

(9)

(10)

(11)

One PEST Iteration

Pilot Points

_Parameters

PEST

Compter

Run Time

Total

137 3,772

15 PCs

3 min

1 week

394 8,912

15 PCs

3 min

3 weeks

1,393

28,892

15 PCs

3 min

16 weeks

(12)

One PEST Iteration

Pilot Points

_Parameters

PEST

Compter

Run Time

Total

137 3,772

15 PCs

3 min

1 week

394 8,912

15 PCs

3 min

3 weeks

1,393

28,892

15 PCs

3 min

16 weeks

1,393

28,892

Carver

1.5 min

2 weeks

(13)

Calibration Options

• Calibration Strategy

–

Pilot Points

–

Regularization

–

SVD Assist

–

Super-Parameters

–

?

• More Robust Computers

–

In-House Linux Cluster

(14)

14

Cloud Computing Options

• Infrastructure as a Service (IaaS)

• Platform as a Service (PaaS)

• Software as a Service (SaaS)

• Desktop as Service (DaaS)

• Backend as a Service (BaaS)

• Information Technology Management

as a Service (ITMaaS)

(15)

15

Hardware

Software Kernel

HyperVisor

(16)

16

Hardware

Software Kernel

HyperVisor

Computational Resources (IaaS)

(17)

17

Hardware

Software Kernel

HyperVisor

Computational Resources (IaaS)

Software Environment (PaaS)

(18)

18

Hardware

Software Kernel

HyperVisor

Computational Resources (IaaS)

Cloud Applications (SaaS)

Software Environment (PaaS)

(19)

19

Hardware

Software Kernel

HyperVisor

Computational Resources (IaaS)

Cloud Applications (SaaS)

Software Environment (PaaS)

(20)

Computing Options

• Multiple Office PCs

• Private Cluster

• USDOE NERSC Cluster

• Cloud Computer Systems

–

Commodity Cloud System

(21)

21

(22)

(23)

(24)

(25)

(26)

(27)

27

Performance Issues

• Contention

– Two or more processes

attempting to use one resource at the

same time

• Latency

– Time interval between

(28)

28

(29)

29

(30)

30

(31)

31

(32)

32

Performance Issues

• Contention

– No CPU sharing:

(33)

33

Data Center Servers

Servers

Network Interface

(34)

34

(35)

35

Data Center Servers

(36)

36

(37)

37

(38)

38

(39)

39

(40)

40

Performance Issues

• Contention

– No CPU sharing,

guarantee only one App per core.

• Latency

– All allocated servers must

(41)

41

Performance Metrics

• Wall Time

t

• CPU Time

c = t n*

• Speedup

s = t

₁

/ t

_n

• Scaling Efficiency η = c

₁

/ c

_n

(42)

Systems Tested

• BeoPEST on Office PCs

• MPEST on USDOE super computers

• PPEST on Amazon EC2

(43)

Systems Tested

System Processor Cores RAM (GB) OS Storage Network

Office PC BDOHY-034 Intel® Xeon® CPU E5-2690

@ 2.9 GHz 8 16 Windows 7 64-bit Hard disk DWR

BDOHY-024/025 Intel® Xeon® CPU 5140 @

2.33 GHz 4 4 Windows 7 64-bit Hard disk DWR

BDOHY-027 Intel® Xeon® CPU X5472 @

3.0 GHz 4 4 Windows 7 64-bit Hard disk DWR

Windstream VM

Windstream VM unknown 8/VM 32 Windows 2012

Server 40GB HD Cisco Nexus

Amazon AWS VM

c3.8xlarge unknown 32/VM 60 Windows 2012

Server 2x320GB SSD 10 Gigabit

NERSC Carver Intel Xeon X5550 @ 2.67

(44)

44

Test Problem

• C2VSim-CG calibration exercise

• 100 hydraulic conductivity parameters

• Two parameter estimation iterations

• Run PEST with master plus 1 to 64 clients

(45)

Wall Time

0.1 1.0 10.0 0 10 20 30 40 50 60 W all tim e ( hr ) Number of CPUs

(46)

0 5 10 15 20 25 30 35 40 0 10 20 30 40 50 60 Co mp ut at io n t ime (h r) Number of CPUs

NERSC Carver Windstream Office PCs Amazon EC2

(47)

0 5 10 15 20 25 0 5 10 15 20 25 30 35 40 45 50 55 60 65 Sp ee du p Number of CPUs

(48)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 Sc alin g E ffic ie nc y Number of CPUs

(49)

49

Costs - AWS

Item Description Unit Quantity Total

ELASTIC COMPUTE CLOUD

US East (Northern Virginia) Region

EBS $0.095 per GB-Month of snapshot data stored - US East $0.095 0.5 $0.05 US West (Northern California) Region

ECC Windows $0.752 per On Demand Windows c3.2xlarge Instance Hour $0.752 6080 $4,572.16 EBS $0.08 per GB-month of Magnetic provisioned storage - US West $0.080 100 $8.00 $0.105 per GB-Month of snapshot data stored - US West $0.105 100 $10.50 $0.12 per GB-month of General Purpose SSD storage - US West $0.120 100 $12.00

DATA TRANSFER

US West (Northern California) Region

AWS Data Transfer USW1-CloudFront-Out-Bytes $0.02 per GB data transfer out of US West to CloudFront $0.020 0.1 $0.00 AWS Data Transfer USW1-EU-AWS-Out-Bytes $0.02 per GB - US West data transfer to EU (Ireland) $0.020 0 $0.00 AWS Data Transfer USW1-USE1-AWS-Out-Bytes $0.02 per GB - US West data transfer to US East $0.020 1 $0.02 AWS Data Transfer USW1-USW2-AWS-Out-Bytes $0.02 per GB - US West (CA) data transfer to US West (OR) $0.020 10 $0.20 Bandwidth $0.000 per GB - data transfer in per month $0.000 2 $0.00 $0.000 per GB - first 1 GB of data transferred out per month $0.000 1 $0.00 $0.010 per GB - regional data transfer - in/out/between EC2 AZs $0.010 1 $0.01

TOTAL $4,602.94

(50)

50

Costs – Windstream HS

Item Description Unit Quantity Total

SETUP

WCS Enterprise Virtual Data Center Setup Fee $300.00 1 $300.00 WCS Anti-virus/Anti-malware Setup fee $10.00 16 $160.00

WCA SAN Storage Setup fee $0.00 16 $0.00

RECURRING

WCS Enterprise Virtual Data Center Virtual firewall, private VLAN, secure VPN capability,

secure SSL, 600GB data transfer $340.00 1 $340.00 WCS Enterprise Virtual Data Center Data transfer >600GM/mo, per GB $2.00 0 $0.00 WCS Compute vCPU w/2.5GB RAM, base 32 vCPUs $64.00 32 $2,048.00 WCS Compute vCPU w/2.5GB RAM, additional resources per vCPU-hour $0.088 0 $0.00 WCS Anti-virus/Anti-malware Base $10.00 16 $160.00 WCS Anti-virus/Anti-malware Additional VMs >32 $10.00 0 $0.00 WCS SAN Storage Primary storage for VMs, including daily backups

w/30-day retention $0.50 500 $250.00

SETUP $460.00

RECURRING $2,798.00

(51)

51

Costs - AWS

32 vCPUs/1 mo 64 CPUs/1 mo 96 vCPUs/1mo

BASE 0 vCPUs base 0 vCPUs base 0 vCPUs base

ADDITIONAL 32 vCPUs 64 vCPUs 96 vCPUs

30 days 30 days 30 days

US East Storage 0.5 $0.05 0.5 $0.05 0.5 $0.05 US West Instance-hrs 5,760 $4,331.52 11,520 $8,663.04 17,280 $12,994.56 Storage 100 $8.00 100 $8.00 100 $8.00 100 $10.50 100 $10.50 100 $10.50 100 $12.00 100 $12.00 100 $12.00 US West Data 0.1 $0.00 0.1 $0.00 0.1 $0.00 0 $0.00 0 $0.00 0 $0.00 1 $0.02 1 $0.02 1 $0.02 10 $0.20 10 $0.20 10 $0.20 Bandwidth 2 $0.00 2 $0.00 2 $0.00 1 $0.00 1 $0.00 1 $0.00 1 $0.01 1 $0.01 1 $0.01 Cost/Mo $4,362.30 $8,693.82 $13,025.34 Cost per CPU-Hour: $0.19 $0.19 $0.19

(52)

52

Costs - Windstream HS

32 vCPUs/1 mo 64 CPUs/1 mo 96 vCPUs/1mo

BASE 32 vCPUs base 32 vCPUs base 32 vCPUs base

Additional 0 vCPUs 32 vCPUs 64 vCPUs

0 days/mo 30 days/mo 30 days/mo

CPU-Hours 23,040 CPU-hours 46,080 CPU-hours 69,120 CPU-hours

VDC/data 1 $340.00 1 $340.00 1 $340.00 Compute 32 $2,048.00 32 $2,048.00 32 $2,048.00 Compute 0 $0.00 32 $2,027.52 64 $4,055.04 Anti-virus 16 $160.00 16 $160.00 16 $160.00 Anti-virus 0 $0.00 32 $320.00 64 $640.00 Storage 500 $250.00 500 $250.00 500 $250.00 Cost/Mo $2,798.00 $5,145.52 $7,493.04 Cost per CPU-Hour: $0.12 $0.11 $0.11

(53)

53

Scale-Up Costs

$0 $20,000 $40,000 $60,000 $80,000 $100,000 $120,000 $140,000 $160,000 0 2 4 6 8 10 12 Es tim at ed C os t