SRS BIO OPTICAL WORKFLOW

(1)

SRS – BIO‐OPTICAL WORKFLOW

Version 2.0

22

nd

_{March 2013}

Data Workflows

eMII, the data management facility for IMOS, has developed workflows for each IMOS sub‐facility to describe the flow of IMOS data from planning through data collection to data delivery and public data access. The primary goals of this workflow are to:  Improve data flow and data handoff, making tracking of data status easy and preventing data loss  Identify and delimit precisely the responsibilities of each person involved  Improve communication at the interface between IMOS facilities (i.e. between eMII and other IMOS facilities)  Improve transparency for end users by providing more details to populate metadata records (i.e. limitations and processing methods applied to datasets)  Assist in reporting planned deployments against actual deployments and data delivery The workflow is available on the next page of this document. Additional information (i.e. timeline, input, output, step description) for each operation step is available in the ‘Supporting Information’ section. The role and contact details of people involved in the workflow are summarised in a table and suggested potential improvements are listed at the end of the document. An Appendix describes access permissions for IMOS data directories and lists useful links.

(2)

eMII

Data delivery on the IMOS portal

SRS ‐ Bio‐optical database Prepare Excel template SRS Data officer eMII Project officer

Data collection and data processing

Create/update a WMS layer in Geoserver Project officer Software engineer Create a file per cruise using the Excel template Data officer Configure IMOS portal menu Project officer Data and metadata publicly available through the IMOS portal Publish data on the Data Fabric Project officer List of acronyms:  BODBAW: Bio‐Optical Database of Australian waters  CSIRO: Commonwealth Scientific and Industrial research Organisation  eMII: eMarine Information Infrastructure  MEST: IMOS Metadata Catalogue  QA/QC: Quality Assurance/Quality Control  SRS: Satellite Remote Sensing  WMS: Web Map Service Send files to the eMII Project officer by email Data officer Create/update database tables Project officer Software engineer Convert data into NetCDF format Project officer Gather data collected on various cruises Data officer Perform QA/QC on the data Data officer Create a metadata record for each cruise Data officer Harvest metadata records into IMOS MEST Software engineer

(3)

Supporting information

Phase Operation step Timeline Input Output Step description Step operator

Da ta collection an d processing Prepare Excel template Excel template created Creation of an Excel template to store the data. This template contains both metadata and data. SRS Data officer eMII Project officer Gather data collected on various cruises Data collected during previous cruises Data from previous cruises gathered SRS Data officer Create a file per cruise using the Excel template  Created Excel template  Data from previous cruises gathered One Excel data file

populated per cruise Copy the data in the Excel template. SRS Data officer

Create a metadata record for each cruise Data from previous cruises gathered One metadata record created per cruise Metadata records once created are available on the MarLIN CSIRO website (MarLIN format) and then duplicated in Geonetwork in order to be harvested by the IMOS instance of Geonetwork. SRS Data officer Perform QA/QC on the data One Excel data file

populated per cruise QA/QCed Excel data files SRS Data officer

Send files to the eMII Project officer by email QA/QCed Excel data files QA/QCed Excel data files at eMII SRS Data officer Convert data into NetCDF format Two to three weeks Excel files at eMII  NetCDF files  Excel files  .csv files Run a Matlab script to convert automatically the Excel data files into IMOS compliant NetCDF files. eMII Project officer Harvest metadata records into IMOS MEST One metadata record created per cruise Metadata records populated Harvest the CSIRO MEST catalogue and publish the records on the IMOS MEST. Software engineer Publish data on the Data Fabric  NetCDF files  Excel files  .csv files NetCDF, Excel and .csv files publicly available in the “Opendap” and “Public” folder of the Data Fabric  Move manually NetCDF files into the “Opendap” folder of the Data Fabric.  Move manually Excel and .csv files into the “Public” folder of the Data Fabric. eMII Project officer Create/update database tables  Excel files at eMII  Metadata records populated Database tables populated Run a Matlab script to update the IMOS portal database using information extracted from the Excel files (e.g. filename, platform code, start/end time and location, bounding box, data and metadata links). eMII Project officer   

(4)

Da ta delivery on the IM O S portal Create/update a WMS layer in Geoserver Creation: One week Update: One day Populated database tables WMS layer created in Geoserver  Use of a database table as a data source for the creation of a WMS layer in Geoserver.  Configure the pop‐up window displaying information from the database table (e.g. start/end time and location, link to data and metadata).  Create filters.  Create a link to the metadata records. eMII Project officer and Software engineer Configure IMOS portal menu Creation: One week Update: One day WMS layer created in Geoserver Data and metadata accessible on the IMOS portal  Access to the admin interface of the IMOS portal.  Select and include newly created layers to the portal menu.  Save configuration.  Update internet browser and check that the portal menu has been updated. eMII Project officer

(5)

Contact details

Role Name Institution Email address Phone Comments

SRS Bio‐optical

database

Facility leader Dr. Edward King CSIRO [email protected] (02) 6246 5894

Sub‐facility leader Dr. Vittorio Brando CSIRO [email protected] (02) 6246 5716

Data officer Dr. Lesley Clementson CSIRO [email protected] (03) 6232 5337

eMII

Project officer Mr. Laurent Besnard UTAS [email protected] (03) 6226 8570

Data services team leader Mr. Sebastien Mancini UTAS [email protected] (03) 6226 8571

(6)

Suggested improvements

 Make available monthly reports created by eMII to the Facility (eMII suggestion).

(7)

Appendix

Access permissions for each directory of the Data Fabric

IMOS datasets are stored on the ARCS Data Fabric. The Data Fabric is a “virtual file system” that allows data to be distributed across sites, but appear under a uniform structure. The base directory for IMOS is located at http://df.arcs.org.au/ARCS/projects/IMOS/public/. Under this path the directories are as follows:

 Staging ‐ the place for facilities to upload their data (processed and unprocessed); accessed by facility and eMII; no public access  Archive ‐ for raw unprocessed files and other materials; accessed by facility and eMII; no public access  Public ‐ for processed QA/QC data for general access which is not suitable for OPeNDAP (like AUV images, excel spreadsheets, PDF document …); accessed by facility, eMII and the public.  Opendap ‐ location for files to be accessible via THREDDS/OPeNDAP; accessed by facility, eMII and the public

Supporting links

IMOS portal: http://imos.aodn.org.au/webportal/ IMOS THREDDS server (access to NetCDF files): http://opendap‐vpac.arcs.org.au/thredds/catalog/IMOS/SRS/catalog.html Data Fabric: http://df.arcs.org.au/ARCS/projects/IMOS/public/SRS/ IMOS MEST (Metadata catalogue): http://imosmest.aodn.org.au/geonetwork/srv/en/main.home IMOS website: http://www.imos.org.au/ SRS facility: http://imos.org.au/srs.html SRS‐Bio‐optical database: http://imos.org.au/bwg.html

Directory

Facilities eMII End users

Staging read/write read/write not visible

Public read read/write read

Opendap read read/write read