Cloud-based Geospatial
Data services and
analysis
Xuezhi Wang
Scientific Data Center
Computer Network Information Center Chinese Academy of Sciences
Outlines
1
Introduction of Geospatial Data Cloud
2
Cloud-based Geospatial Data Analysis platform
3
Cloud-based Geospatial Data visualization
platform
Geospatial Data Cloud
I.
Platform overviews
II.
Geospatial data services
III.
Geospatial model services
IV.
Services achievements
Why do we do?
In 2008, USGS opened all geospatial datasets to the public for free.
• Landsat, MODIS is the most popular remote sensing dataset for scientists and researchers. Especially, the
Landsat series.
• Due to the low international network speed, many Chinese scientists cannot take advantage of these resources
easily.
• Based on the CNIC’s mass storage and high speed network( CSTNET: Chinese Science and Technology Network), we mirrored the geospatial datasets
covered China and surrounding area, and provided online data services for
Design a new cloud-based geospatial analysis and
visualization platform USGS opened its Landsat datasets.
we contacted the USGS staffs through email.
We got an oral permit for Landsat redistribution from USGS. we download the Landsat datasets, and provide public
services for Chinese scientists
Cooperated with National Center for Atmospheric Research (NCAR)
Mirrored the NCAR datasets provide public services for Chinese
Scientists
We got a permit for SRTM dataset redistribution from International Centre for Tropical Agriculture(CIAT). We cache
SRTM global elevation data, and provide online data services.
Accroding to the user requirements, We make some remote sensing products from Landsat dataset and Modis dataset.
such as vegetation index, water cover dataset.
Dowload MODIS dataset covered CN Host the first user conference Design a new cloud-based data platform
Geospatial data cloud opened for public services Host the second user conference with ESRI China
Platform objectives
Based on our cloud infrastructure, we hope to construct an 'one-stop'
geospatial data service platform for Chinese scientists and researchers.
An open cloud-based platform meeting the requirements of data
search, data acquisition, online storage, scientific models, online
calculation and map visualization.
Model Online Calculation Search Acquisition Storage Visualization
http://www.gscloud.cn
General framework
Database
cluster Online storage
Near-line storage Data integration Automatic Metadata crawling Automatic data entities cache Mass Storage Cloud environments Distributed file system Virtual Machines Task scheduling Data query Data category Integrated search Complex search One-key indexes Data services Smart data Order Data application
Favorite data Batch
download Model services Model package Online execution Web map
services Map publish
Map visualization Map style edit Model management User data spaces Data management Task management Map management Model flow Map share Service management System managem ent Users Data Models Indexes Tasks Services Resources monitor
Geospatial Data Cloud
I.
Platform overviews
II.
Geospatial data services
III.
Geospatial model services
IV.
Services achievements
Online data resources
Open geospatial datasets: LANDSAT、MODIS、EO-1、DEM、NCAR、
NOAA and LUCC
More than 350TB data entities, and over 7 million data records.
Covered China and surrounding area
Online services for Chinese scientists and researchers with free of charge
Landsat1-3 MSS Landsat 4-5 MSS Landsat 4-5 TM Landsat 7 SLC-on Landsat7 SLC-off Landsat 8 OLI TERRA AQUA TERRA AQUA GDEM 30m SRTM 90m Ocean elevation HYP_L1R HYP_L1G AL1
Generated geospatial products
Water-cover of China from Landsat TM/ETM in Summer
Slope, Aspect, Topographic position indexes et. of China
Land Surface Temperature, Vegetation indexes (5 days, 15
days, and 30 days) of China, from 2000 – now
ASPECT
Geospatial data query
Massive geospatial metadata
I need Landsat data in summer, 2000
I need DEM data of Beijing
Input: geo-location or address
PostgreSQL+PostGIS Cluster Based on geospatial
storage cluster and search engine, we provide three ways for users to access their preferred data:
1: by address name 2: by geo-location
Platform Screenshots
Platform Screenshots
Platform Screenshots
Platform Screenshots
Platform Screenshots
The result page that matched the query criteria
Extra attribute
Geospatial Data Cloud
I.
Platform overviews
II.
Geospatial data services
III.
Geospatial model services
IV.
Services achievements
Geospatial model services
Why do we need this services?
For example:
If we need Beijing DEM data, there have two ways.
Traditional way: search DEM data, download DEM data,
mosaic DEM data with ArcGIS, and then clip DEM data
with Beijing shape mask .
Convenience way: select Beijing administrative unit in
the web, submit a clip task to system, and then download
Beijing DEM data.
Which is the best way?
I prefer the second way.
That’s why we provide online geospatial model
We developed an elastic online task processing system with cloud techniques.
Users fill model parameters on the web page and submit tasks to the system;
The tasks were run under cloud virtual machines;
The results were stored in cloud storage.
Task scheduling Task Task Task Computing
resource pool Storage pool
VM VM
VM
virtual machines in Cloud
Overall frameworks
Online models and tools
DEM tools:
DEM clip and mosaic; Reprojection;
Aspect, Slope, Topographical index calculation of
specified region
LANDSAT tools:
Online gap-fill model for Landsat ETM7 SLC-off
dataset
MODIS tools:
Online Land Surface Temperature calculation of user
specified time range
Online vegetation indexes calculation of user specified
time range
Private cloud storage
We provide user with private cloud storage to store
their own model inputs and outputs data.
User manage their own data here, including upload, download, rename, delete, create folder and move
Platform Screenshots
Platform Screenshots
Model parameters page of DEM clip and mosaic tool
Platform Screenshots
Task tracking page
list user’s submitted tasks
Platform Screenshots
Task result page
When the task was
finished, the results can be download from here, or from user private cloud storage
Geospatial Data Cloud
I.
Platform overviews
II.
Geospatial data services
III.
Geospatial model services
IV.
Services achievements
Our platform was wildly used in many research fields in China: earth
science, environmental science, ecology, hydrology, meteorology,
oceanography, natural disasters, city planning ...
Based on statistics in November 2013
User characteristics
Currently, over 70,000 users
Doctor Master Bachelor Others University CAS Government Company Others
Service statistics
Avg. downloaded over: 200 GB/day;
online downloaded over: 250 TB;
Outlines
1
Introduction of Geospatial Data Cloud
2
Cloud-based Geospatial Data Analysis platform
3
Cloud-based Geospatial Data visualization
platform
Cloud-based Geospatial Data
Analysis Platform
Why do we need this platform?
For example: If we want to calculate Chinese monthly
temperature variation from 2000 to 2014, how can I
achieve this?
Traditional way: download all data, and calculate with workstations. It
may spent several months to download data. This is a very bad thing.
Shortcuts: using cloud computing technology. Apply virtual machines
in the cloud; Write code in the virtual machines; read all data from
cloud storage; Run program in parallel
Which way is the best? I prefer the second way.
That’s why we want to construct this platform.
Overall frameworks
...
Storage resource pool
Cloud manager with
OpenStack
Distributed file system
Data pool
Computing resource pool
Public geospatial
data
User private
geospatial data
virtual machine pool
VM
VM
VM
Currently progress of the platform
We have finished the prototype system. The system is
under inner tests for access pressure and stability. In the
end of this year, it will be provided for the public access.
Outlines
1
Introduction of Geospatial Data Cloud
2
Cloud-based Geospatial Data Analysis platform
3
Cloud-based Geospatial Data visualization
platform
Cloud-based Geospatial Data
Visualization Platform
Why do we need this platform?
In many cases, users want to publish their geospatial data,
model results, vector map to the web site;
They also want to customize the map layer styles through
the web pages;
They also want to customize the components and styles of
the webgis pages;
Optimized Web Map Service
Computing
resource pool Storage pool
VM VM
VM
Dynamical render map in Cloud
HTML5 map client ... Tiles Storage Dynamical cache Map styles editor Map request
Overall frameworks
Currently progress of the platform
Most of system functions were already finished,
under inner tests and bug fixed.
Published services provided:
OGC WMS（web map service）, Google TMS , OGC
WFS （web feature service）
System screenshots
I.
Render multi-bands raster layer with RGB false-color,
System screenshots
System screenshots
System screenshots
System screenshots
System screenshots
II.
Render vector layer with colorful symbol by unique
System screenshots
III.
Render vector layer with colorful symbol by classified
System screenshots
IV.
