J.Farres – EOP-GS ESRIN
6/6/2012
Cloud Computing and Content Delivery Network
use within Earth Observation Ground
Agenda
1.
Introduction
2.
ESA Experiences
1.
EO Dissemination on Level 3
2.
EO Re-processing on Amazon
3.
Dissemination and Processing on Hetzner
4.
Exploitation platform with Helix Nebula
Introduction
Processing bursting Dissemination peaks
ICT Costs savings
Collaboration platform Cloud Computing IaaS SaaS Hosting (VPS, Rental) CDN PaaS
A model for enabling convenient, on-demand network
access to a shared pool of configurable computing
resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and
released with minimal management effort or service provider interaction [NIST]
Agenda
1.
Introduction
2.
ESA Experiences
1.
EO Dissemination on Level 3
2.
EO Re-processing on Amazon
3.
Dissemination and Processing on Hetzner
4.
Exploitation platform with Helix Nebula
Case 1: EO Dissemination on Level 3 (1)
Purpose
• Allow for very performing dissemination of EO products at the event
of natural disasters.
Project / Service
• Timeframe: 2008-2012
• Provider: Level 3, off-the-shelf services
• Data: 6-7TB, SAR, GOCE
• System: Portal / EOLI + EO products secured by ESA login
Pros
Cons
Case 1: EO Dissemination on Level 3 (2)
Pros
1. Excellent dissemination
performance
2. from US and Europe
3. Good integration with ESA user
authentification + EOLI
4. Very good monitoring and
reporting
5. allowing to detect and stop
abuse
Cons
1. High Price
2. Price based on bytes stored and
bytes downloaded, hence penalizes large EO products
Case 2: EO Re-processing on Amazon(1)
Purpose
• Fast re-processing of large EO products collections for CalVal
purposes.
Project / Service
• Timeframe: 2009 and 2011
• Provider: Amazon, EC2, S3
• Data: ERS SAR Wave, MIPAS (30,000 products)
• System: 200 Virtual Servers
configured as Working Nodes to an ESA grid.
Pros
Cons
Case 2: EO Re-processing on Amazon(2)
Pros
1. Excellent processing scalability
2. Efficient bulk-in/out data service
(via HD)
3. “The faster the cheaper” as it
reduces storing costs
Cons
1. Complex and changing pricing,
e.g. periods with cheaper hosts and free data upload.
Case 3: Dissemination and Processing on
Hetzner (1)
Purpose
• Couple large processing capability with fast dissemination for low
cost.
Project / Service
• Timeframe: 2011
• Provider: Hetzner (Hosting services)
• Data: 60TB
• System: 1 Head: Catalogue, Processor Register, Cluster Head
n Nodes: Data Dissemination point, Cluster Node Designed as back-end for multiple front-ends
• Usage: GeoHazards SuperSites (38,000 SAR images and 3,000 users)
Pros
Cons
Case 2: Dissemination and Processing on
Hetzner (2)
Pros
1. Good archive scalability (chunks
of 8TB)
2. Much cheaper than Level 3 or
Amazon services
3. Synergy of processing
-dissemination: processing peaks followed by dissemination peaks
4. Storage costs service two
functions: dissemination and
Cons
1. System scales storage,
dissemination and processing capabilities simultaneously.
2. Lower service levels than Level
Page 11
Case 4: Exploitation platform with Helix
Nebula – SSEP (1)
Purpose
• Pilot a collaborative platform for EO exploitation using multisourced
cloud provisioning.
Project / Service
• Timeframe: 2012-2013
• Providers: ATOS, CloudSigma, Interoute, T-Systems
• Data: GeoHazards data (ESA, CNES, DLR, …)
• Processors: ESA, CNR, Gamma, …
• System (TBD): Large capabilities: storage, dissemination &
processing
Platform for deployment of partner data, processors Based on existing GS sw components
Case 4: Exploitation platform with Helix
Nebula – SSEP (2)
Expectations
• ESA can distribute data dissemination and processing functions over
several cloud providers (multi-sourcing) at best value.
• Other agencies can do the same on the same platform/cloud.
• Users can provision their ICT from the cloud “near” the data/services
provided by ESA and other agencies.
• Users can deploy EO value added services directly on the cloud, i.e.
Agenda
1.
Introduction
2.
ESA Experiences
1.
EO Dissemination on Level 3
2.
EO Re-processing on Amazon
3.
Dissemination and Processing on Hetzner
4.
Exploitation platform with Helix Nebula
Lessons Learnt: ICT provisioning
1. As soon as ICT needs can be predicted and planned, IaaS is
more expensive than other hosting solutions like (VPS, Renting).
2. Flexibility of Public IaaS is less appealing when internal
resources are pooled, virtualized and managed as an internal cloud.
3. On the other hand, IaaS services allow to size down internal ICT
resources to the “fixed” need and ensure their maximum
utilization; e.g. using external provisioning for the “variable” need.
Lessons Learnt: Service Levels
• Terms & Conditions in Public Clouds express surprising low
commitment.
• Cloud opportunities can become risks when applied to critical
systems.
⇒ Develop multi-sourcing
⇒ Plan contingency scenarios for services hosted in Public
Lessons Learnt: Application Areas in EO GS
• Dissemination and on-demand processing
– because is very variable (depending on user demand)
• Secondary archive and re-processing
– because is limited in time
• Temporary resources for integration, testing and demonstration
– because is limited in time
• System sizing
– because needs are unknown
Lessons learnt: Users
• User expectations on Cloud Computing are very high
– All data discoverable and accessible online with same performances as consumer services.
– Availability of long time series of coherent data from different providers.
– Availability of collaboration platforms and tools where to exchange experiences and information and where data from different providers and even different disciplines can be accessed, combined and exploited.
– Users will be able to perform processing directly on the cloud using virtual servers.
• The Cloud provides a well accepted paradigm for pay-per-use
• User communities adopt social networks patterns: sharing objects,
providing feed-back