Data Data Everywhere, We are now in the Big Data era

(1)

www.fugro.com

Data Data Everywhere,

Mike Liddell 13th_{March 2014}

UUVs @ OI 2014

(2)

www.fugro.com Date

Contents Menu

Background

Big Data

What is Generating our Big Data Physical Management of Big Data Optimisation in Data Processing Techniques for Handling Data Conclusions

(3)

www.fugro.com Date

Contents Menu

Background

Big Data

(4)

www.fugro.com

Current Status - Maturing Subsea Vehicle Market

 AUV’s moving towards commodity

– In most categories there is growing choice of supplier – More manufactures moving into the market

 ROV’s at commodity status

(5)

www.fugro.com

Current Status - Sensor’s and Payloads

 Commodity Market ( Just look around this show…)

– Growing richness in range and suppliers of acoustic sensors. – Growing range of Optical and Laser sensors.

– Rapidly growing range Positioning and Navigation Sensors.

 Some of the bigger players becoming acquisitive

 Some of the manufactures starting to connect the dots in their portfolios from sensors to solutions.

 Some vehicle providers tightly integrating sensors to provide a package solution

(6)

www.fugro.com

Current Status - Summary

 Subsea Platforms are commodities ( just not cheap ones)  Rich sensor / payload market

The toolbox is available

 Operational Challenges Remain

– Experienced People. – L & R ( AUVs)

– Autonomy



The big challenge for the Survey and Inspection contractors is the

efficient handling of all this data. Big Data.

(7)

www.fugro.com Date

Contents Menu

Background

Big Data

(8)

www.fugro.com

What is Big Data

 http://en.wikipedia.org/wiki/Big_Data

 Big data is the term for a collection of data sets

so large and

complex

that it becomes

difficult to process using

on-hand

database management tools or

traditional data processing

applications

. The

challenges include capture,

curation, storage, search, sharing, transfer, analysis,

and visualization

. The trend to larger data sets is due to the

additional information derivable

from analysis of a single

large set of related data

, as compared to separate smaller sets

with the same total amount of data,

allowing correlations to be

found

to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time

(9)

www.fugro.com

What Big Data Challenges do we face

 Data Management.

– Protect – Transport – Deliver

 Efficient Data Handling (Processing of the Data) – Rapid Delivery of Results

• More Automation. Less Intervention – Integration of Data Sets

– Temporal Analysis

(10)

www.fugro.com Date

Contents Menu

Background

Big Data

What is Generating our Big Data

Physical Management of Big Data Optimisation in Data Processing Techniques for Handling Data Conclusions

(11)

www.fugro.com

What is our Big Data

 Imagery

– Digital Video -> HD Digital Video -> 4K – Photo Stills

 New Survey and Inspection Sensors – Laser Profilers

– Synthetic Aperture Sonars – MBE Water Column Logging – Optical Sensors

 Peripheral Sensors

– Inertial Navigation Sensors – Hydrocarbon Sniffers

– Magnetometers

– Forward Looking sonars – SVP/ CTD

(12)

www.fugro.com

Big Data Generators

 VIDEO

– Typical ROV Inspection 3-4 Camera

– Data Volumes SD per Month DVM 6TB – Data Volume HD per Month DVM 11TB – Growth in move from SD to HD to 4K??? – Move away from Video to Stills? Fast ROV

 Photo Stills

– 172GB per day. ( 1 MB Image) DVM 3.4TB – Extensive Processing ( duplications) DVM 12TB

• Each process results in a new image

(13)

www.fugro.com

Unique Challenges of Big Data Offshore

 Cruise lengths of 30-45 days not uncommon. Typically with limited bandwidth on vessels. Not commercially practical to stream all data to beach.

 Port calls are expensive “down time” so need to turn vessel around fast.

 Port calls are often in in remote locations. Arctic, West/East Africa. Shipping logistics can be difficult or time consuming.

 Long duration uptime. 24/7 acquisition.

 Complex data that needs to be rapidly accessible often with random access. – (Jump to the middle of a video file)

 How and what data products we deliver to clients

(14)

www.fugro.com Date

Contents Menu

Background

Big Data

What is Generating our Big Data

Physical Management of Big Data

Optimisation in Data Processing Techniques for Handling Data Conclusions

(15)

www.fugro.com

Data Storage Optimisation

RAID array of disks for

redundancy and capacity

Academic Definition:

Redundant Array of

Inexpensive Disks

Manufacturers Definition:

Redundant Array of

Independent Disks

(16)

www.fugro.com

Data Storage Optimisation - Speed

Tiered Storage

approach for speed

optimisation.

Layers of disks

optimised for speed or

storage.

Optimised solutions fast

become expensive

(17)

www.fugro.com

Data Storage Optimisation - Backup

Solution for DR and

Replication to another

RAID.

LTO Tape Drives

6.25TB on a tape

(18)

www.fugro.com

Data Storage Optimisation - Office

 Data Live on Vessel for Weeks or Months

 Data Live in Office for Longer. Final Report issued. Client Comments  Office Data Storage may need to be significantly larger. 100-1000s TB

 Disaster Recovery Solutions.

– What if the building burnt down…. – What if the building got flooded…. – What if the power went off….

 How will this data be archived for the long term • Medium

• Duration

• (and who pays?)

(19)

www.fugro.com Date

Contents Menu

Background

Big Data

What is Generating our Big Data Physical Management of Big Data

Optimisation in Data Processing

Techniques for Handling Data Conclusions

(20)

www.fugro.com

Data Processing Optimisation

 Data is Secure and Backed Up but we need fast access to this data

 Often software vendors recommend storing data locally on SSD disks for fast access.

– Non starter in a large Survey Data Centre • Danger of data loss.

• Danger of sync issues. Where is the most recent data

• Single user access. What if it’s a large project and needs multiple user access

– Not a responsible method of managing data – But we still need fast access?

(21)

www.fugro.com

Data Access Optimisation - Network

Workstation PCs with

Fast Ethernet

connections to server

Minimised bottle necks

in network to maintain

available Ethernet

connection from PC to

Storage.

(22)

www.fugro.com

Data Access Optimisation - Software

Optimised Software

enables cluster

processing

Processing tasks

distributed across

multiple cores on multiple

PC.

Significant time savings

on computational

intensive tasks.

(23)

www.fugro.com Date

Contents Menu

Background

Big Data

What is Generating our Big Data Physical Management of Big Data Optimisation in Data Processing

Techniques for Handling Data

(24)

www.fugro.com

Delivery of Data to the Client

 Generally there is a requirement to deliver significant number of data products to the client.

 With Inspection activities that has meant dozens of hard disks or RAID arrays full of video files.

(25)

www.fugro.com

(26)

www.fugro.com

(27)

www.fugro.com

Video Deliverables

 Both VHS Tapes and Digital Video Files are essentially RAW products  Digital Video provides significant improvements for data review and QC

(28)

www.fugro.com

Video Deliverables

 Digital Video integration with Survey Data tends to have a limited life span and user base.

 Event Listing becomes “the product” that is circulated.

– Compact

 There is a better way

– Convert the Digital Video into a Geo-referenced Image – Image can be loaded into GIS software

• Desktop or web based

– Product that can be shared within a company, access available to all that require it.

(29)

www.fugro.com

Video Processing

• INPUT: Digital Video and

USBL Positioning

• OUTPUT: Referenced Image

for GIS

(30)

www.fugro.com

Video Processing

 As-Built photo record of a pipeline.  Enables Temporal Comparisons

– Layers in GIS

– Automated Classification/ Comparison

 Comparisons with other platform image data sets (AUV Stills Camera)

 Benefits

– Compact deliverable (relatively) – Optimised for GIS

• Optimised for Web delivery

– Resource that can be widely distributed – Longer life.

• As-Built record

• Annual Comparisons – Spatial

(31)

www.fugro.com

(32)

www.fugro.com

AUV Image Handling

 FliMap. Image and Laser Profiling Solution from Helicopter  Very close analogy to AUV

– 50 knots v 4 knots

– 50m altitude v 4m altitude – Still images

– High resolution topographic profile.  Very large image datasets

– 2Hz - 24 hours =172,000 Images  Optimised Image Handling Tools

– Radiometric and Topographic Corrections  GIS friendly outputs. Tiled Images

(33)

www.fugro.com

(34)

www.fugro.com

Image Handling

 Convergence in Products - GIS Tiled Image – ROV – Video Input

– AUV - Still Image Input

 Processing Flows that evolve towards minimal human interaction

 Challenges Remain

– Computationally intensive.

(35)

www.fugro.com Date

Contents Menu

Background

Big Data

What is Generating our Big Data Physical Management of Big Data Optimisation in Data Processing Techniques for Handling Data

(36)

www.fugro.com

Conclusions

 Big Data is here and its only going to get

BIGGER.

 Rich sensor market of acoustic and now laser/optical sensors.

 Tools are available for acquiring the data but the processing is the big challenge.

 Investment in infrastructure for secure management of data is not insignificant. ( Should not be overlooked).

 New approaches required for the handling of these large data volumes.  Emerging techniques for improved handling of Video and creating new

geospatial products

(37)

www.fugro.com

Data Data Everywhere, We are now in the Big Data era