Cloud-based Infrastructures. Serving INSPIRE needs

(1)

Cloud-based Infrastructures

Serving INSPIRE needs

INSPIRE Conference 2014

Workshop Sessions

Benoit BAURENS, AKKA Technologies (F)

Claudio LUCCHESE, CNR (I)

(2)

The Digitized Earth

Avalanche of data generated by various

institutions all over Europe in numerous

domains:

• Geography,

• Earth observation,

(3)

Data Quantity and Quality

Visibility, Accessibility and Sharing

We cannot act here

We can « push up » here for improving

necessary services

Why considering cloud

technologies?

(4)

Geospatial sciences & applications

Enabling IT Resources

More and more needs for

computations, simulations,

peak responses to

emergencies….

Why considering cloud

technologies?

(5)

• Data intensity scenarios: storage but not only (e.g.

performance of queries, synchronisation and integrity of

data)

• Computing‐intensity scenarios: geo‐processing and

computation on demand, procurement and release of

resources

• Access‐intensity scenarios: World‐wide and/or concurrent

access to information in case of particular events where

maps / data/ services are required well beyond normal usage

Why considering cloud technologies?

Some common characteristics about

Geo‐Applications Usage

(6)

Data must be provided / shared

• Efficiently

• Ubiquitously

• In various forms

• At no (or cheap) price

• Following standards and

recommendations

• « packed » into relevant

services

(7)

• Cloud computing comes from the convergence of:

–

service oriented architectures

• ... loose coupling of services with operating systems and technologies ...

–

parallel computing

• large scale data analysis, up to thousands of machines

–

virtualization

• independence from physical hardware

What is Cloud Computing?

Cloud computing is a model for enabling ubiquitous,

convenient, on-demand network access to a shared pool of

configurable computing resources (e.g., networks, servers,

storage, applications, and services) that can be rapidly

provisioned and released with minimal management effort

or service provider interaction. (NIST)

(8)

Why considering the Cloud?

Promises of the Cloud: Costs Reduction and

simplification

Reduced Total Cost of Ownership

Technical staff, power supply, physical space

hardware, cables,

Scale economy among partners

Share of Databases, servers, CPUs, …

Pay‐as‐you‐go: Operations Costs versus

Infrastructure Costs

(9)

Why considering the Cloud?

Promises of the Cloud: Not only about the

Money!

A cloud platform shall also provide

Large computing power with ad‐hoc machines and network

Various up‐to‐date Operating Systems and technologies

Ubiquitous access

and Quality of Service

(10)

About Cloud Computing

(11)

Hardware

Operating System

App

Traditional Stack

About Cloud Computing

(12)

Hardware

Operating System

App

Traditional Stack

Hardware Operating System App App App

Traditional Stack Hardware Operating System App App App

Traditional Stack

About Cloud Computing

(13)

About Cloud Computing

Towards perfect capacity management?

• Starting costs

remain reasonable

• Adaptive capacity:

scaling‐down is

as important as scaling‐up

(14)

• Diverse software requirements

• Diverse resource requirements

• Resource requirements vary over time

• Reduce costs

Challenges of GeoData Services

Infrastructures

(15)

• Diverse software requirements

<‐>

Virtualization

• To support a larger number of software requirements

• Diverse resource requirements

<‐>

Scalability

• To support

large data volumes

and

high throughput

• To support

increasing dataset sizes

• Resource requirements vary over time

<‐>

Elasticity

• To support a

varying number of users

• To support

on demand computations

• Reduce costs

<‐>

Pay‐as‐you‐go

• To reduce

infrastructural cost

during low platform usage

Challenges of GeoData Services

infrastructures and Cloud

(16)

(17)

Hardware

Operating System

App

Traditional Stack

Moving to the Cloud

(18)

• Identify a common

software stack

–

Homogeneous infrastructure

–

Scale economies

–

Cloud compliance

• Define

data volumes

requirements

–

Capacity planning

• Scalability

challenges and requirements

–

High‐throughput services

–

Reliability

–

Guarantee

QoS

for INSPIRE/OGC services

Moving to the Cloud

Think Layers !.

Geo‐Spatial Stack

Operating System

Spatial Data Storage

Map Server

Web Server

(19)

InGeoCLOUDS

Architecture:

Auto‐Scaling

Layers

(20)

The Architecture

SCALABILITY

+

ELASTICITY

+

ON-DEMAND

+

MEASURED SERVICE

InGeoCloudS Achievements

(21)

InGeoCloudS scalable services:

Elastic File Server

Elastic Database Server

Elastic Web Server

Elastic Map Server

Elastic Linked Data Store

All of the above are

hot topics

from a

technological and

scientific

point of view.

(22)

(23)

Elastic File Server

We evaluated several technologies:

S3FS, S3Backer, pNFS, LUSTRE, …

Our choice was

GlusterFS

No single point of failure

No file metadata server

Scalable

C

an add as many servers as needed at any time.

Support

standard protocols

(

e.g. NFS)

Includes some optimizations, e.g., read ahead, write behind, async I/O,

scheduling, caching

It is currently

Other Cloud‐based storage solutions are based on the

key‐value

access pattern, which is incompatible with every other

technology on the Geo‐Spatial Software stack

(24)

(25)

Elastic File Server Scalability

55

77

210

344

78

125

342

730

0

100

200

300

400

500

600

700

800 Thr

o

ughput

(MB/

s)

GlusterFS ‐ write

GlusteFS ‐ read

(26)

• PostgreSQL (+PostGIS)

• PgPool

Load balancer

–

Master/Slave architecture

–

Streaming replication

• Scalability

–

Parallel read operations

–

Can add as many servers

as needed at any time.

• Reliability

–

Automatic fail‐over

(27)

INSPIRE services : different part of the system potentially not in relation

with data publication

The classical approach of the environmental data

dissemination…

InGeoCloudS Context and

Challenges

(28)

The INSPIRE Architecture

Applications and Geoportals

r

Servic

e

Laye

r

Application

s

Laye

r

Spatial Data Set

“Invoke” service

Spatial Data Service

Registry

Registers

View

Service

Download

Service

Transformation

Service

Metadata:

Discovery

Service

Service Bus

Service

metadata

(29)

A cloud‐based infrastructure for

scientific publication

Metadata

OGC services

INSPIRE

Data

model

(30)

InGeoCloudS project Focus

Applications and Geoportals

r

Servic

e

Laye

r

Application

s

Laye

r

Spatial Data Set

“Invoke” service

Spatial Data Service

Registry

Registers

View

Service

Download

Service

Transformation

Service

Metadata:

Discovery

Service

Service Bus

Service

metadata

(31)

Simplify the process of “transforming”

geo‐data as geo‐

services

Guarantee the geo‐service compliance with

OGC

standards

and

INSPIRE

requirements

3 components in the Data Publication :

Read Only services with OGC:WMS (image) and OGC:WFS

(data)

CRUD API to manage the configuration of each service by

data‐provider

Metadata management (ISO 1911 + OGC:CSW)

Geo Publication – A Key Service in

InGeoCloudS

Objectives

(32)

« Software As A Service » for INSPIRE and GIS team

Insights for GeoPublication :

A SaaS approach

for GIS+INSPIRE objectives

The service in

the cloud

The software

GIS

desktop

Share my

_maps

with Internet

Provide my

INSPIRE

services

Describe and

share your

datasets

(33)

The process of geo‐datasets publication is greatly simplified

Mask all the technical aspects for the Web publication of

geospatial datasets

Do not worry about the performance requirements (WMS in less

<5s), the capability (>20 requests/s) or the availability of your

web services (+99%)

Benefit of the improvement of the service over time without

“install/update” process: WFS 2.0, new GIS functionalities,

stats of use,…

You are in your workspace and you master your publication

The strengths of the SaaS approach for

GIS+INSPIRE Publication for data providers

and users

(34)

(35)

GeoPublication Component

Architecture

ELASTIC GEOSPATIAL

SERVER CLUSTER

Mapserver

Server

WMS

_WFS

Mounting FS for

all data provider

ReadOnly

Access DB

3306 port

Mapserver

Server

Mapserver

Server

…

HTTP load balancer

HTTP/API

Mounting FS for

all data provider

Write

Data

publication

(36)

Example with the number of requests

with a WMS GetMap

Small Amazon

instance

6 WMS

Performance

GetMap 800x600 <5

s

Capacity

simultaneaus

requests > 20/s

Availability

_99%

Large Amazon

instance

50

(37)

(38)

30

40

50

60

70

80

90

100 4000

6000

8000

10000

12000

e

ra

ge

CPU

Utiliz

ation

R

e

ques

ts

/

min

Issued Requests

System Load

No. Servers

Load Threshold

Elasticity Experiment:

Elastic Web Server

(39)

10

20

30

40

50

60

70

80

90

100 2000

4000

6000

8000

10000

12000

Av

e

ra

ge

CPU

Utiliz

ation

R

e

ques

ts

/

min

1 server

2 servers

3 servers

4 servers

System load

increases quickly

_{increases slowly:}

System load

the system can sustain

peak loads more easily

(40)

ID‐Card of the Project

Who we are?

5 Geological Surveys bringing in 6 initial Use Cases (datasets and applications)