Oklahoma s Open Source Spatial Data Clearinghouse: OKMaps

Oklahoma’s Open Source

Spatial Data Clearinghouse:

OKMaps

MAGIC 2014 Symposium

State Geographic Information Coordinator Oklahoma Office of Geographic Information

Why OKMaps?



Sub-section D-3 of OS 82:1501.205.3 : Establish a

central statewide geographic information clearinghouse



A definite need existed for a central, authoritative source

of geospatial data but no funding was available



Growing awareness at OK Office of Homeland Security

that GIS can help fulfill missions



In 2008 funding became available through OK OHS from

US Department of Homeland Security to support

geospatial data development and distribution

System Goals



Low or no ongoing maintenance costs



Not technology or software-specific  No license fees



Open Geospatial Consortium (OGC) Web Service Standards (WMS, WFS,

WCS, KML,CSW, SQL)



Core software code is extensible and open- Java,JavaScript, AJAX,

ASP.NET, HTML, CSS



Integrated Catalog, Metadata Manager and Search Engine

Output for Google Earth visualization



Point, Line, Polygon, and Raster (including floating point for DEM/LIDAR)

support for storage, view, download, and web services



Download by area (state, county, USGS quad, PLSS, visible extent) with

many data format choices in any commonly recognized projection



Support for frequently updated layers (vehicle tracking, weather radar)

Security (Data and system functionality limited by Role/User)

High-Level Architecture

Data Catalog Data Layer Selection

Metadata Clients

Google Earth Users Web Service Consumers

Data Downloaders

OGC Standard Web Services (KML, WMS, WFS, WCS)

Data Storage

Architecture Detail

Metadata Raster Storage Vector Storage KML, WFS Feature Server WMS Server File Download Generator Feature Input/ Update Inter-layer analysis (eg buffer etc.)

Raster Tiling or Caching

(optional)

Raw Data Data Conversion

Open-Source Architecture



Web Server



GeoServer



GeoWebCache



Catalog



GeoNetworks



Database



PostgreSQL/PostGIS



Data Translator



GeoTools-GDAL/OGR



GUI-Viewer



OpenLayers

Clearinghouse

User will select different layers (both raster and vector) from the catalog. These will then appear in GE. It is possible to have GE request updates every few seconds or minutes to display realtime data. GE handles all the network connections and layers once they are set up.

Web Service Client

A web service client, such as OpenLayers, ArcMap, AutoCAD Map, Global Mapper, QGIS, uDig, and many others make standard WMS, WFS, WCS requests to the warehouse and gets back data. The user will first select layers from the catalog, and add these to their client. Then, the client will talk directly to the WMS, WFS and WCS server(s).

Clearinghouse

Users that need downloaded files in a variety of formats,

projections, datums, etc, will first select the data layers, formats, options and AOI using the catalog. The request is passed to the file download generator which returns the file for download.

Feature I/O Client

Adds and updates features in the warehouse. This can be done using a feature client like OpenLayers, QGIS, TinyOWS, or another client that supports WFS-T, or input-output of KML. The user would select an updateable layer from the catalog, then the software

Clearinghouse

Metadata

Data Warehouse Catalog

The catalog is the central location where users choose their data from the warehouse. Metadata about the geographic data within the

warehouse is presented in an easily searchable, easy to read fashion. The data can then be previewed on a map within the catalog. Once the user is happy with their choice, they can choose the method of receiving the data, in Google Earth, files, or via web services.

The catalog has a secure way of allowing a fine degree of access rights depending on userid/password or other access control method. Metadata is published to FGDC standards. GeoRSS subscription service is available to notify user when new data layers are added or existing ones updated.

Open Source: – an open source Java server that does most of these tasks.

Clearinghouse

WMS Server

A WMS (Web Map Service) server creates images

of the

underlying vector or raster data according to a standard

request. The images can then be displayed in most clients,

including Google Earth, OpenLayers, ArcGIS, AutoCAD Map,

uDIG, and QGIS. In addition, WMS images are used in the

Catalog to show previews of the warehouse data.

Open Source: – an open source Java server

provides this service. - a Java web application

used to

cache tiles

from the WMS that is bundled with

Clearinghouse

WFS Server

A WFS (Web Feature Service) server returns the actual

source of the underlying vector data

according to a standard

request. This raw data can then be displayed and analyzed

in most clients, including OpenLayers, ArcGIS, AutoCAD

Map, uDIG and QGIS.

Open Source: – an open source Java server

provides this service.

Clearinghouse

Server

WCS Server

The OGC Web Coverage Service (WCS) defines a standard interface and operations that enables interoperable access to geospatial “grid coverage's". The term "grid coverage" typically refers to content such as satellite images, digital aerial photos,

digital elevation data, and other phenomena represented by values (color, intensity, elevation) at each measurement point (pixel).

Clients include OpenLayers and Global Mapper.

Open Source: -an open source Java server provides this service.

Clearinghouse

File Download Generator

This component constructs downloadable vector and raster files in a wide variety of formats, projections, and options. Given the user's requirements, it creates the files, then makes them available for download.

This component can read from both the underlying raster and vector storage, as well as from WFS, KML, WMS and WCS sources.

Open Source: -an open source Java library coupled with GDAL/OGR libraries provides this functionality.

Clearinghouse

Vector (Feature) Storage

Vector features can be stored either in flat files (e.g. SHP files), or in a

geodatabase. A geodatabase has a number of advantages over flat files, such as built-in analysis and query tools that make them much preferable in a data

warehouse application. Additionally, a geodatabase is not fragmented into many different files.

Interoperability: If the geodatabase supports standards like the OGC SQL standards, one geodatabase can be substituted for another if required.

Open Source: – the spatial-enabled database used worldwide in many applications.

Clearinghouse

Storage

Raster Storage

Large raster images can be stored as flat files in a variety of formats, such as GeoTiff, MrSID, JPEG2000, ERDAS, JPEG . There are also ways to store them in databases, or in proprietary servers.

Besides the image data itself, each image file needs to have metadata information that describes the image, such as spatial location,

projection, date, source.

Aerial photo/DRG/DEM images in OKMaps are stored as jpeg compressed GeoTiff quarter-quad files bound together in mosaic coverage's generated by GeoServer.

LiDAR LAS data is stored in a database utilizing PDAL (Point Data Abstraction Library) and the pgpointcloud extension for

PostgreSQL developed by Paul Ramsey in early 2013. Data is stored in the database as patches where a patch = 1 row and a patch consists of 400 LiDAR points. Currently there are 171.9 Million rows in the

Clearinghouse

Raw Data Data Conversion

Data Conversion of Raw Data

Data for the warehouse will need to be converted into common formats for storage. The same tools that are used for the file download generator can be used for this.

Additionally, standardized metadata needs to be tracked, and added to the catalog for the data as it is input into the warehouse. Open Source: - -GDAL/OGR with some custom control software.

Clearinghouse

or Caching

Raster Tiling/Caching

This component provides faster access to raster data. Many client software, including Google Earth, and OpenLayers can consume

standard sized image tiles much more quickly and seamlessly than arbitrarily sized images.

A tiling/caching system can increase the perceived speed of both WMS servers, and raw imagery access.

In particular, Google Earth provides a way to consume tiles that appears much more seamless than access to WMS servers. Open Source:

Clearinghouse

Inter Layer Analysis

It may be useful to include a component that allows for inter-layer analysis. For example, you may have a road feature that comes from one WFS server, and you need to find all schools that are

within 1km of this road which is on a KML server (buffer analysis). It is possible to provide an intermediate analysis component that has access to both servers and can perform the operation.

Possible Software:

Custom software using tools such as GDAL/OGR and a GeoDatabase like PostGIS or new OGC-WPS extension to GeoServer.

Clearinghouse