Analysing Big Data in ArcGIS

(1)

7th November 2014

Karl John Pedersen

OIL & GAS

Analysing Big Data in ArcGIS

1

AIS based risk modelling

(2)

Introduction

 Large amounts of data are produced from monitoring systems  Processing and analysing this data provides a challenge

 How does DNV GL use ‘Big data’ – AIS vessel tracking

2

Background and

(3)

Background

3

400

offices

100

countries

16,000

employees

150

years

 Environmental Modelling and GIS unit within Oil & Gas, DNV GL, Oslo – DNV GL - International company headquartered in Norway

– Global classification, certification, technical assurance and advisory company  GIS used in risk modelling and visualisation for customers

 Many projects use AIS data

 How have we adapted to increasing amounts of AIS data  Examples from Oil & Gas and Maritime sectors

(4)

What is AIS

 AIS - Automatic Identification System - vessel tracking data  Create lines – append lines spanning multiple days

 Join to ship database to get attributes e.g Vessel type & size, engine & fuel type

(5)

Analysis examples

 Examples where AIS data are used in risk modelling

(6)

View vessel traffic around an offshore installation

(7)

Calculate vessel density around installation – collision risk

(8)

Risk to pipelines from fishing activity

8

(9)

Number of fishing vessels per grid cell – quantifies risk

(10)

Fishing activity vs. depth – risk from vessel anchors

(11)

Developing risk models in ArcGIS

 AIS data used in further analysis

– AIS data from ArcGIS often used as input to DNV GL risk models – Results visualised/mapped in ArcGIS

 Risk models increasingly developed in ArcGIS:

11 AIS Database Process AIS data in ArcGIS Run risk model Map/ Visualisation Data export for users Excel/Access ArcGIS

(12)

ArcGIS modelling example

 Input accident frequencies from Excel  Run risk model - Python

 Arctic Shipping Risk

(13)

How we previously handled AIS data

– Data in SQL Server accessed via IBM Cognos – Imported text files from Cognos into ArcGIS  Challenges:

– Large text files, lacking data structure

– Many steps - possibility of errors

– Limited geographic selection – Time consuming

– Increasing data quantities – Historical data

– Global data



Need to improve access to AIS ‘Big data’

(14)

What is Big data?

 Big data – …a collection of data sets so large and complex …difficult to process

using … traditional data processing applications (Wikipedia)

Example from DNV GL:

Database of > 2,5 billion AIS points

(15)

Solution for handling AIS data

 IBM Netezza data warehouse implemented to improve data access  Implemented in existing Cognos environment

What is Netezza

:

 Data warehouse appliance from IBM

 Provides rapid access and analysis of ‘Big data’

 Includes a Spatial Esri package

 AIS data readable directly in ArcGIS

(16)

Making data available to ArcMap from Netezza

16

Netezza

Spatial Esri

package

AIS data

imported

from

supplier

Spatial

data

created in

Netezza

Polygon

data

loaded via

WKT

format

(17)

Accessing data in Netezza from ArcGIS

 Install IBM Netezza ODBC driver

 Add data as normal – geographic data has own icon

Data access via:

 ArcToolbox Query Layer

 Python

(18)

ArcToolbox Query Layer

 Processing carried out on server  Select geographical area

 Points mapped directly in ArcGIS  Data downloadable to ArcGIS

 Work reduced from days to minutes

(19)

Allows analysis of global AIS data

(20)

Serving maps to users - ArcGIS Server

 ArcGIS Server service for AIS data created  Issues creating service for whole dataset

(21)

Serving maps to users - Cognos

 Cognos use spatial data from Netezza  Esri Maps for Cognos

 Implemented simple mapping tool

(22)

Limitations

– ArcGIS cannot write to Netezza – Issues creating ArcGIS service – No support for raster

– Limited user base

(23)

Summary

 Advantages of implementing solution:

– Enables significantly improved access to AIS data – Allows analysis which was not previously possible – Part of existing data warehouse

– Ability for existing non-GIS users to analyse data using familiar tools

(24)

SAFER, SMARTER, GREENER

www.dnvgl.com

That’s all ! Questions?

24

Contact details:

Karl John Pedersen

Principal Specialist, Environmental Modelling and GIS, Environmental Risk Management, Høvik, Norway E-mail [email protected]