Management of Water Quality Data Using EnviroSys and SharePoint

(1)

Management of Water Quality Data Using EnviroSys and SharePoint

Wet Tropics Major Integrated Project Standard Operating Procedure

WTMIP SOP 016

Version 1.0

Wet Tropics Major Integrated Project 22 June 2021

(2)

WTMIP SOP 016 - Management of Water Quality Data Using EnviroSys and

SharePoint

1. Purpose and scope

EnviroSys is an environmental data management software solution that has been implemented to collect, store, manage, analyse and share/present water quality data for Local Scale Monitoring (LSM), Catchment Repair (CR) and Farm Services (FS) activity areas of the Wet Tropics Major Integrated Project (WTMIP).

The procedures outlined in this document apply to the operation and maintenance of water quality data management software EnviroSys and the file storage and sharing software SharePoint. While other systems, for example eagle.io, are referred to in the procedure, the management of telemetered data from field-deployed water quality instruments using eagle.io is covered in ‘WTMIP SOP 14 - Management of Telemetered Water Quality Data on eagle.io’.

This document includes:

▪

Background information for systems and methods in place.

▪

Operational details for interacting with EnviroSys, SharePoint and other software tools used by the WTMIP.

▪

Backup, support and quality checking procedures.

2. Training, competency and responsibilities

Program staff participating in water quality measurements, data quality checking and water quality analysis are provided with training in operating EnviroSys. Records of participant competency are maintained within the Terrain Natural Resource Management (NRM) file management system (SharePoint).

Table 1: Roles and responsibilities of WTMIP program participants

Position Responsibilities

WTMIP Project Leader Provide resources for the implementation and continued development of this method.

WTMIP Water Quality Project Officers (Leader/Technicians) and Basin Co-ordinators

Ensure methods described in this method document are followed. Train new staff members in the methods.

Continually review and develop the methods where appropriate.

WTMIP program staff, contract staff, landholders and external stakeholders

Follow the methods described in this document. Provide feedback to WTMIP Water Quality Project Officers for continued development of this method document.

3. Workplace health and safety

Desk based work activities adhere to Terrain NRM work health and safety (WHS) requirements as prescribed in the Joint Corporate Nature, Terrain NRM, Cape York NRM and Northern Gulf Resource Management Group Health and Safety Policy and Procedures Manual.

(5)

4. Solution overview

4.1. Description of system components

The data management solution created for the WTMIP utilises a combination of software products to fulfil all current and future needs for water quality data management and analysis. Figure 1 shows the various components of the data management solution and is followed by a description of each component.

(6)

EnviroSys

EnviroSys is the WTMIP’s central database which holds all water quality data. It provides secure data storage, automated processes, real-time notifications on system health, access to water quality data and quality control (QC) reports. The WTMIP team members have access at various levels.

Bureau of Meteorology (BOM)

Data from a range of Bureau of Meteorology stations is imported automatically or manually into EnviroSys, depending on the BOM monitoring station.

HOBO

HOBO depth sensors are deployed in CR and FS monitoring locations. Data is downloaded via USB and emailed to the EnviroSys monitored email address for extraction into EnviroSys.

Water Quality and Investigations (WQI) eagle.io

Data from the Department of Environment and Science (DES), Water Quality and Investigations (WQI) eagle.io workspace is retrieved via an API (Application Programming Interface) script running on the EnviroSys server.

EnviroSys mobile app

The mobile app for iPhone and Android is used to collect water quality data from mobile water quality devices such as TriOS OPUS and TriOS NICO nitrate sensors, NECi photometers and ProDSS multi-parameter meters. Data is manually entered in the app by field staff and directly submitted to EnviroSys. Site photos taken during grab sampling may also be saved in the database via the EnviroSys mobile app.

Cairns Regional Council Laboratory Services – email upload

Grab sampling data is sent to the EnviroSys monitored email address by the CRC Laboratory. EnviroSys extracts results from the lab reports and stores additional attachments from the email (i.e. laboratory certificates and electronic chain of custody (eCoC) forms).

eagle.io

eagle.io provides the initial storage location for high frequency data received from in-situ LSM and CR sampling sites.

eagle.io data is made available in EnviroSys and is backed up on SharePoint (CSV format).

eagle.io is used in its own right (e.g. for changing the configuration of loggers and quality coding data) while EnviroSys is used to integrate the in-situ logger data with WTMIP data from other sources.

1622.farm

http://1622.farm is a web page for water quality data from Queensland sugarcane growing regions and hosts data from multiple projects. Nitrate and stream level data from WTMIP’s eagle.io sampling locations is retrieved via the eagle.io API and easily accessed by farmers via mobile device (iPhone, iPad, Android) or a computer.

SharePoint

SharePoint is used to store time-lapse photographs, lab project tracking sheets, analysed data, raw lab result files and to back up eagle.io data.

(7)

4.2. EnviroSys user interfaces

Portal

The EnviroSys portal can be reached via https://data.terrain.org.au and all WTMIP team members can access water quality data in read-only (basic user) mode. The Water Quality Leader, Data Manager and Water Quality Technicians have access to retrieve data, invalidate data, set quality codes and leave comments. The different elements of the portal are described in Screenshot 1.

Screenshot 1

(1) The ‘Insight’ panel provides information about compliance and completeness of sample schedules. It is part of the Sample Management module of EnviroSys.

(2) The ‘Realtime’ panel displays alerts for unexpected disruptions to logging for the in-situ monitoring stations

(3) ‘Sample Management’ allows the scheduling of monitoring runs. ‘Data Browser’ allows the user to explore all monitoring sites via Esri maps

(4) A ‘Data View’ is the most common form of data query/search. The main search window is entered through

‘Adhoc Data View’. Customised Data Views can be created, saved and shared by the users.

(5) Excel or PDF reports can be generated to report on system usage, to extract data for presentation and to report on data quality.

(8)

Mobile app

The mobile app allows for the collection of field data, such as nitrate results from hand-held devices and field observations associated with grab samples, using Android and iPhone/iPad devices. Site photos taken during grab sampling may also be saved within the app and transmitted to EnviroSys via the mobile app. The annotations for Screenshot 2 provides an overview of the mobile app.

Screenshot 2

(1) The ‘Enter’ tab provides the means to enter data into custom built forms. Results are directly transmitted to EnviroSys.

(2) The ‘Schedule’ displays when results are expected to be retrieved from loggers, BOM weather stations and grab sampling events.

(3) The ‘Realtime’ tab allows the user to check the status and current results of loggers.

(4) The ‘More’ tab provides application settings, account settings and access to log files.

(9)

Admin interface

The admin interface can be reached via https://envirosys.terrain.org.au. A simple version of the admin interface for the most common tasks (Screenshot 3) is complemented by an advanced admin interface which allows the user to gain more control over the setup (Screenshot 4). As part of their product development roadmap, acQuire (the EnviroSys vendor), is moving more functions into the simple admin interface over time. The advanced admin interface can be accessed by clicking on the icon marked in Screenshot 3.

Screenshot 3 Screenshot 4

(10)

4.3. EnviroSys terminology

Data can be navigated by ‘Site’ or ‘Aspect’ (Screenshot 5).

A ‘Sample Site’ (1) is a geographical area in which monitoring occurs.

An ‘Aspect’ (2) is the combination of ‘Program’ (3) and a ‘Data Type’ (4). A ‘Program’ (3) means ‘Monitoring Program’ (e.g.

Catchment Repair, Farm Services, Local Scale Monitoring). The ‘Data Type’ (4) relates to the way the data is collected (e.g.

via Loggers or Grab Sampling) or the intended purpose of collection (e.g. QC).

Screenshot 5

A ‘Sample Point’ (5) is the specific point within a ‘Sample Site’ where monitoring occurs. In Screenshot 6, MIP-CR-BR01-B1 is the complete location code with BR01 representing the ‘Sample Site’ and B1 referring to the ‘Sample Point’.

A ‘Test’ / ‘Test Method’ (6) is the name of the result delivered by the CRC Laboratory or from a field-based sensor or gauge.

It may also be a calculated value based on other results.

Screenshot 6

(11)

5. Viewing and exporting data

5.1. Searching for data

Searching by navigation

This search is useful to see the ‘Insight’ and ‘Realtime’ notifications for (a) specific sample site(s) or program(s).

Select a ‘Site’ (1) or ‘Aspect’ (2) (Screenshot 7) and the page in the background will refresh with up-to-date metrics of the relevant sample points.

Screenshot 7

To browse locations and results via map click on the ‘Data Browser’ icon which can be found in the ‘Analysis’ panel (Screenshot 8).

Screenshot 8

(12)

Searching via an ‘Adhoc Data View’

The ‘Adhoc Data View’ is the most common tool for searching as it offers many different possibilities to search for and filter data. To view data via this method, click on the ‘Adhoc Data View’ button under ‘Select a Data View’ in the ‘Analysis’

panel (Screenshot 9).

Screenshot 9

As per Screenshot 10,

(1) – Select either a predefined past timeframe or a custom date range for which results are required.

(2) – At least one ‘Aspect’ must be chosen for data to be displayed.

(3) – If a specific location should be queried, select it from the ‘Locations’ link. Otherwise, leave empty to search for all.

(4) – This can be used to display specific ‘Tests’ (e.g. Nitrate, DIN). If no selection is made, all ‘Tests’ available for that Aspect-Location combination will be displayed.

(5) – After selecting the above criteria, click the ‘Apply’ button to display results.

(6) – To save a search that is to be performed on a regular basis (e.g. show all data of a specific sample site to for the past two weeks) click the ‘Save As’ button and type the name you want to save it as.

(7) – Clicking the ‘More’ button or ‘Save As’ (6) button, brings up the ‘Advanced Search’. See ‘Searching by project number’

for more details.

Screenshot 10

(13)

Searching by ‘Project Number’

On occasion there may be a need to search for data using additional metadata associated with samples, such as the CRC Laboratory ‘Project Number’. This is done using an ‘Advanced Search’. To reveal the ‘Advanced Search’ option (Screenshot 11), go to Searching via an ‘Adhoc Data View’and click the ‘More’ (7) button (Screenshot 10).

To retrieve or check if a particular project has already been loaded into the system click on ‘Sample Metadata Criteria’ (1).

Screenshot 11

This will open the Criteria Builder (Screenshot 12). Next, select and fill in the ‘Project Number’ (2), click ‘OK’ (3). Once the

‘Sample Metadata Criteria’ has been entered, select the ‘Apply’ (4) button on the ‘Advanced Search’ screen (Screenshot 11).

Screenshot 12

(14)

A list of all selected ‘Test Methods’ (5) are displayed in columns in Screenshot 13. Results are ordered by ‘Sample Date’

and ‘Location’ (6)

Screenshot 13

5.2. Displaying data as a graph

Click on the ‘Graph’ (1) (Screenshot 14) symbol to bring up the graphing section. Assign the parameters of interest to the left and/or right axes of the graph by clicking on the ‘L’ or ‘R’ under the ‘Test Method’ name (2). This will display the data on a graph (3). To modify the way data is displayed, click on the cog (4).

Screenshot 14

6 5

(15)

In the ‘Chart Options’ (Screenshot 15) it is possible to change the style (5) of lines and colours (6) for each ‘Location’.

Colours can be standardised across graphs by choosing the same colour codes (7).

Screenshot 15

5.3. Exporting and sharing

To export the displayed data into an Excel file (Screenshot 16) click on the download button (1) in the ‘Table’ section. To export the graph as a .png file, click on the download button (2) in the ‘Graph’ section.

Screenshot 16

To export data including a formattable graph within an Excel file, click on the Share (3) button on the ‘Advanced Search’

(Screenshot 17).

(16)

Screenshot 17

In the resulting pop-up window (Screenshot 18), select the ‘Cog’ (4) and make sure the ‘Graph’ (5) box is ticked.

Screenshot 18

Navigate to the ‘Share’ (6) section and click on the ‘Download’ (7) button twice (wait until prepared after first click) to download the data and graph to your computer.

Alternatively, to send the exported data/graph via email, fill in the name of ‘internal users’ (8) and/or email addresses of

‘external users’ (9), followed by clicking the ‘Send Email’ (10) button (Screenshot 19).

(17)

Screenshot 19

In cases when larger amounts of data are to be exported, it is recommended to use the ‘Data Extract’ (1) report in Screenshot 20. This is because it takes some time to generate the export and the portal cannot be used when following the method described above. In contrast, the data export method can happen ‘behind the scenes’ while the user continues to navigate EnviroSys.

Select the ‘Data Extract’ (1) report to bring up the report configuration page.

Screenshot 20

In the report configuration page (Screenshot 21),

(2) – Select the desired ‘Program’ and, if relevant, a specific ‘Data Type’.

(3) – Insert a valid timeframe using the calendar icons.

(4) – Select the ‘Quality Code(s)’ that you would like to filter by – leave empty if all should be exported.

(5) – Select desired ‘Sample Points’ to export.

(18)

(7) – To let a background process take care of the generation of the export file, click the ‘Email Report’ button. This provides the ability to continue using the EnviroSys portal while the report is generated.

Screenshot 21

5.4. Generating routine reports

EnviroSys allows creation of system usage related (1) and custom/data related (2) reports. To generate a report, click the desired report in Screenshot 22 (e.g. ‘Monthly usage statistics’ (3) and fill in the required input fields (4), then, click on

‘Run Report’ (5). This produces a PDF or Excel report (6) as shown in Screenshot 23. Please note, required input fields vary from report to report.

(19)

(20)

6. Data management– Grab sampling

The analysis of all grab samples for the WTMIP water quality monitoring program is contracted to the CRC Laboratory. A range of conventions and tools have been developed and are used to facilitate the tracking of, and the communication around, samples and results. Section 6.1 to Section 6.4 describe each data management convention/tool and how they relate to each other. The data management process undertaken during grab sampling, including communications with CRC Lab and other preparatory work, is illustrated in a workflow diagram in Section 6.5. Data management procedures followed during grab sampling are detailed in Section 6.6 to Section 6.8.

6.1. Water quality sampling master spreadsheet

The ‘_MIP WQ Sampling Info Master.xlsx’ file, referred to as the ‘WQ Sampling Master’ in this document, is in SharePoint at ‘Project Centre – MIP – Knowledge & Info – Data Storage’.

The purpose of the WQ Sampling Master is to:

• Document parameters/aspects relevant to the WTMIP such as sample points, project descriptions and other relevant references (e.g. CRC Laboratory rapid log programs).

• Track the status of grab samples being submitted to and received from CRC Laboratory.

• Cross-check the sample information (i.e. Sample Points and Project Descriptions) used by EnviroSys and CRC Laboratory on a biannual basis.

Available sheets are:

Screenshot 24

The sheets shown in Screenshot 24 have the following content:

Introduction To provide purpose, context and instructions

References Contains lists that are used in communication with the CRC Laboratory Lab Project Tracking Is used to track all grab sampling to make sure nothing falls through the cracks ALL Sample Points Central record of sites and sampling locations

Tracks the history of Sample Points

Used to cross check against the sample points and project descriptions used by EnviroSys to ensure EnviroSys is configured to successfully load all result files delivered by CRC Laboratory.

(21)

6.2. Project description

The ‘Project Description’ is of paramount importance in managing the grab sampling data as it is used throughout the entire grab sampling process, including:

•

To request bottles for a laboratory project (i.e. each batch of samples collected and submitted to CRC Laboratory is linked to a unique project number, which enables traceability of the data from the point of collection to reporting).

•

To be recorded on the eCoC

•

To name the eresults file

•

To import eresults into the correct monitoring program in EnviroSys

•

To identify projects when filing and retrieving laboratory-issued documents into/from SharePoint

The ‘Project Description/s’ for each ‘Sample Point’ can be found in the ‘ALL Sample Points’ sheet (Screenshot 25) of the WQ Sampling Master and a list of all ‘Project Descriptions’ can be found in the ‘Reference’ sheet (Screenshot 26).

To make sure no mistakes have been made in the recording of Project Descriptions and Sample Points in the WQ Sampling Master, a biannual cross-check is run against the names used by EnviroSys (Section 9.5).

(22)

6.3. CRC Laboratory ‘Rapid Log Programs’

The CRC Laboratory utilises a Laboratory Information Management System (LIMS) to manage all samples and associated data. The LIMS provides the ability to create ‘Rapid Log Programs’ to enable rapid generation of new ‘Projects’ by spelling out the details (e.g. the site names, analyses required, distributions lists etc) of a particular project type (e.g. LSM routine stream sampling, or CR bioreactor monitoring). A ‘Project’ is a batch of samples taken from one or multiple locations, analysed for a prescribed suite of parameters, using a particular eCoC format with results being distributed to a particular group of people.

Screenshot 27 shows an example of a ‘Rapid Log Program’.

(1) – ‘Rapid Log Program’ number – The internal LIMS code for a ‘Rapid Log Program’.

(2) – ‘Project Description’ – Corresponds to the ‘Project Description’ recorded in the ‘WQ Sampling Master’. It is used when requesting bottles from the CRC Laboratory and for naming the eresults files. It is then also picked up by EnviroSys to allocate the data to the correct monitoring program at import stage.

(3) – ‘Distribution List’ – A predefined list of email accounts that should receive communication regarding a particular project.

(4) – ‘e-CoC template’ – A chain of custody template containing specific fields to be recorded in the field.

(5) – ‘Test List’ – A defined list of parameters to be tested for a particular project.

(6) – ‘Sampling Point’ – The equivalent to the ‘Sample Point’ in the ‘WQ Sampling Master’.

Screenshot 27

(23)

6.4. Naming conventions

Naming conventions have been introduced to facilitate:

• Identification of a location/project from the file name.

• Presentation of data.

• Compatibility and processing of file names by computer programs or scripts.

Sample sites

Sample sites names are unique identification codes across all activity areas (CR, FS, LSM) and contain one or multiple sample points.

The nature of CR activities is to pilot different treatment systems to test their effectiveness at improving water quality in the wet tropics. CR has therefore chosen to start the name of each sample site according to the type of treatment system (e.g. BR = Bioreactor, CW = Constructed Wetland, LW = Landscaped Wetland).

The naming convention for CR’s sample sites is:

[TREATENT SYSTEM TYPE][SEQUENTIAL NUMBER]

Examples:

BR01 BR02 CW01 HES01

For LSM and FS, sampling takes place at locations in the Johnstone and Tully basins at either naturally occurring waterways or on farms and paddocks. FS and LSM sample sites are distinguished by activity area and basin.

The naming convention for FS and LSM is:

[ACTIVITY AREA]-[BASIN CODE][SEQUENTIAL NUMBER]

Examples:

FS-T1 FS-J4 LSM-T1 LSM-J3

Sample points (Locations)

Sample points are the specific locations within a sample site where samples are taken. Results in EnviroSys relate to sample points. The following general rules apply to the naming of sample points:

• Different types of information about the sample point is separated by an underscore (_)

• Spaces can be used within one piece of information (e.g. In PP)

• Hyphens to be used for joining words or numbers only (e.g. 900-1) The general naming convention for sample points is:

[PROJECT]_[ACTIVITY AREA]_[SAMPLE SITE]_[SAMPLE POINT CODE]

Where there is only one sample point at a site, the sample point name is shortened to:

[PROJECT]_[ACTIVITY AREA]_[SAMPLE SITE]

Examples:

MIP_CR_BR01_B1 MIP_LSM_T1 MIP_FS_J6

(24)

Sample points (quality control (QC))

Sample point codes are also used for quality control samples, even though the samples are rarely associated with a geographical site.

There are two main type of naming conventions for QC samples: those where the location should not be revealed to the CRC Laboratory to avoid any bias by the analyser; and those, where the location or instrument is included in the name to easily verify or find related QC results for a location or instrument.

The naming convention for non-disclosed locations is:

[PROJECT]_[ACTIVITY AREA]_[optional: INSTRUMENT]_[TYPE OF QC]

Examples:

MIP_LSM_FB MIP_LSM_RSS_BL1 MIP_FS_P_FB

The naming convention for disclosed locations is:

[PROJECT]_[ACTIVITY AREA]_[SAMPLE SITE]_[SAMPLEPOINT CODE or INSTRUMENT]_[TYPE OF QC]

Examples:

MIP_LSM_T8_NSV MIP_LSM_J6_NSV

Note that some of the older QC sample point names do not comply with the naming conventions outlined above (e.g.

(MIP_FS_NSV_NICO3 or MIP_CR_QC_04_FB) as they were created before the data management system had been formalised. Metadata stored in the ‘ALL Sample Points’ worksheet of the WQ Sampling Master provides details of what each QC code represents. All future QC sample points codes, however, should comply with the naming convention documented here.

Project Description

As some operating systems, programs and scripts have issues with processing special characters in file names, the ‘Project Description’ should not contain any special characters such as “?, !, %, /, &”. The ‘Project Description’ is case sensitive.

The naming convention for ‘Project Description’ is:

[ACTIVITY AREA] [DESCRIPTION] [(SITE REFERENCE)]

with ‘SITE REFERENCE’ being used only when sampling occurs for one sample site only.

Examples:

CR Bioreactor (BR01)

LSM Event Grab Sampling Tully (T5) FS Tully Piezometers Routine

(25)

6.5. Grab sampling data – process

Figure 2 illustrates the process for managing grab sampling data from the initiation of a sampling request to the analysis of data, as well as an accompanying process for collecting site photos during sampling events.

Figure 2: Data management process for water quality grab sampling

(26)

6.6. Communication with CRC Laboratory

Requesting a new ‘Project’

Prior to each sampling occasion, monitoring staff are responsible for contacting the laboratory and requesting a new Project. This is done by sending an email referencing the Rapid Log Project Description and/or the CRC Rapid Log Code found in the References sheet in the WQ Sampling Master. The laboratory will then issue a set of bottles for that particular project type (i.e. pre-labelled with the relevant project number, site names and bottle numbers). A Chain of Custody (CoC) form, specific to that project and bottle pack, is emailed to MIP team members on the Distribution List specified in the Rapid Log Program. The same procedure is followed regardless of whether routine or event monitoring is being planned¹. Typically allow two weeks for bottle orders to be fulfilled. If a large bottle order is being submitted, or a rapid turnaround time is required for bottle issue, it is recommended to call ahead to enquire about CRC Laboratory workloads and ability to meet the order. Should the laboratory staff be unable to fulfil the whole bottle order in the required timeframe, the order may need to be split, with sub-orders prioritised.

A CoC form is to be used for all sampling activities where samples are being transferred to another party (i.e. couriers, laboratory, other sampling staff) so that if any quality control issues arise, reference can be made back to how the samples were transferred from the point of collection to the point of analysis. The CoC also captures important metadata such as field measurements, site observations, information on sample preservation/storage conditions and notes regarding why a sample was not collected. An example of the CoC form is shown in Screenshot 28 below.

Screenshot 28

1 Typically for events, a maximum of 20 samples are included in the project with the first 10 bottle sets labelled. Blank bottles with loose labels provided for the remaining 10 bottle sets (so they can be used for other projects if the event is brief). Should the event last longer than expected, more bottles and labels can be ordered and added to the eCoC form.

(27)

In the field, each sample is identified by a unique sample number which can be found in the CoC. A single sample may be collected into a range of different containers depending on the parameters being analysed. Each sample container has a unique number, which is also identified on the CoC form. Samplers should always check they are sampling into the correct container number for each sample point (location code). The Description on the CoC form represents the WTMIP location code.

A unique ‘Project Number’ is assigned to each CoC form which is tracked and referenced in the ‘WQ Sampling Master’ (sheet ‘Lab Project Tracking’). For more information on the tracking procedure, please refer to Section 6.9.

An example of the ‘Lab Project Tracking’ sheet is shown in Screenshot 29.

Screenshot 29

(28)

Communication following bottle request

Following the bottle request, the previously issued ‘Project Number’ serves to link all communications relating to a particular project and enables the tracking of eresults. The different stages of communication following bottle request are illustrated in Screenshot 30.

(1) – CRC Laboratory issues an (‘incomplete’) electronic Chain of Custody (CoC) Excel sheet to the relevant distribution list.

(2) – Sampler submits a completed CoC back to CRC Laboratory via email ([email protected]).

(3) – CRC Laboratory issues a sample acknowledgement (‘SAMACK’) to the distribution list. The sampler is responsible for checking that the dates, project number and other recorded data from the CoC has been correctly recorded and reports errors to the lab within 48 hours of receiving the SAMACK.

(4) – CRC Laboratory issues a ‘Certificate of Analysis’ along with final results (eresults) and other relevant documentation.

An EnviroSys monitored email account ([email protected]) is included in all distribution lists. When eresults are found in an email attachment with a valid/known ‘Project Description’ in the file name, the data from the eresults and associated attachments (i.e. laboratory certificates and a copy of the CoC) are imported into EnviroSys.

If site photos have been taken, these are also now linked to the sample/’Project Number’ by EnviroSys, provided the data and time entered into the mobile app entry for that Sample Point is identical to the date and time recorded on the CoC.

(5) – When a mistake has been made during the sampling process or the recording of the results, the CRC Laboratory reissues a ‘Replacement’ of the final results.

Screenshot 30

(29)

Setting up a new project type

If unfamiliar with setting up a new CRC Laboratory project type, discuss your needs with the Water Quality Leader or EnviroSys Admin prior to commencing the process below.

Step 1

Review the outlined in Section 6.4 of this SOP to define your ‘Project Description’ and ‘Sample Point’ names.

Step 2

Prepare an email for the CRC Laboratory containing:

• Project Description

• Sample Point(s)

• Test List (from ‘WQ Sampling Master’) or define a new parameter suite that suits your project

• eCOC format (from ‘WQ Sampling Master’) or define your own

• Distribution list (from ‘WQ Sampling Master’) or define your own See Screenshot 31 for an example:

Screenshot 31

Step 3

Send email from Step 2 to Water Quality Leader and Data Manager for review.

Step 4

The Data Manager or Water Quality Leader approve or amend the new project(s) and associated metadata. The Data Manager updates the ‘WQ Sampling Master’ by adding the new ‘Sample Point(s)’ (if any) and new ‘Project Description(s)’.

To create a new Sample Point additional metadata, such as the site co-ordinates, the reason the site is being added to the program and the monitoring activities planned for the site should also be recorded in the WQ Sampling Master. The Data Manager also provides the ‘Project Description’ and ‘Sample Points’ to the EnviroSys support team so that Envirosys is configured to receive results for the new project.

Step 5

The Water Quality Leader sends the email requesting a new project type to [email protected].

Step 6

CRC Laboratory will create new ‘Rapid Log Program’ from the information supplied and will send new ‘Rapid Log Program’

for confirmation in their reply.

(30)

6.7. Use of the EnviroSys mobile app during grab sampling

For installation of the EnviroSys mobile app and general instructions on how to collect data with the mobile app, please refer to Appendix 1 and Appendix 2.

Collecting site photos via the mobile app

During each sampling occasion, site photos are taken to document the current state of the sample point at the time of sampling. In the event of data being inconclusive or showing abnormal values, sites photos, along with the site observations from the CoC, can be reviewed to identify potential reasons for the anomalies.

Follow the steps outlined in Screenshot 32 to collect and submit site photos:

(1) – Make sure to have the correct ‘Form’ selected.

(2) – Select the ‘Location’ that is being sampled.

(3) – Take a photo in the field by clicking the ‘Camera’ icon.

or

(4) – Select a photo from your photo library by clicking the ‘Image’ icon (e.g. in case a photo was taken with the phone camera app and form is completed in the car or office after the sampling event).

(5) – Adjust the date and time according to the relevant entry in the CoC by clicking the ‘Calendar’ and ‘Clock’ icons.

Correct the minutes manually in the field ‘Sample Date’ in case the 30 min interval offered through the icon does not match the time recorded in the CoC.

(6) – Click the ‘Save’ button to save the data entry on your phone.

(7) – When ready, click the ‘Send Now’ button (Screenshot 33) to submit results directly to EnviroSys. Results will be linked directly to the photos once the eresults are received from the CRC Laboratory. This is done based on a match of date/time and ‘Sample Point’. It is critical that the date and time are consistent between the mobile app entry and the CoC form for EnviroSys to link the laboratory results to the photo. This cannot be corrected retrospectively (once imported, EnviroSys will consider them separate records) so the sampler needs to take a high level of care.

Screenshot 32

Screenshot 33

(31)

6.8. Use of waterproof cameras for collection of site photos

In some instances, adverse weather conditions may require the use of a waterproof camera. In these cases, the photos from the camera will need to be downloaded from the SD card, manually named, and uploaded into SharePoint for storage. Steps to rename and upload photos into SharePoint are detailed below.

Collecting the SD card from the waterproof camera

1. Take the camera to the Toughbook/Laptop and unclip the lid at the bottom of the camera. This will open the camera and show the SD card inside.

2. Push in on the SD card to enable the card to be released. Once the card has popped up slightly, pull the SD card out of the camera.

3. Insert the SD card into the SD card port on the right-hand side of the Toughbook/Laptop.

4. The computer should recognise a new device has been connected and open the corresponding folder. If it has not, go into File Explorer and scroll down until you see the SD card on the far left-hand side and select SD card.

5. Copy images from the SD card to the desktop in a new folder with the location name and date (i.e.

MIP_LSM_TullyRoutine_20200408)

6. Once the copy is complete go back into SD Card in file Explorer and delete all old images off the “old image”

folder in the SD card.

7. Replace images in “old image” folder with recent images.

8. Once the copy is complete delete images off the recent image folder.

By doing this, you have created a copy of the most recent images to be stored on SharePoint on the Toughbook Desktop and left a back-up copy of those images in the “old image” folder on the SD card. This means that if there are any technical issues with the copy to the Toughbook desktop or into SharePoint, you have a copy on the SD card which you can transfer back across to the Toughbook.

DO NOT delete any images out of “old images” on the SD card if they have not first been transferred safely to SharePoint.

Naming images for storage on SharePoint

For the security of data, images need to be backed up and traceable. Each image must be named so that it is uniquely identified with site location and time before being moved onto SharePoint.

Use the Bulk Rename Utility from https://www.bulkrenameutility.co.uk to rename images for storage by performing the steps outlined below and in Screenshot 34:

Navigate to the location of the stored images (e.g. \Documents\SITENAME\).

1. Go to Actions in top bar and click “select all”

2. Name: Fixed, then type in site location (i.e MIP_LSM_T).

3. Auto Date (8): Fill in the fields "Mode" (Prefix), "Type" (Taken (Original))), "Fmt" (Custom) and "Custom" (%Y-

%m-%d - %H.%M.%S_).

4. Click the "Rename" button and confirm.

(32)

Screenshot 34

This will bulk rename all photos to the corresponding activity area (LSM), basin (Tully (T) or Johnstone (J)) and the date and time the photo was taken. The individual photos now need to be identified via site location and direction of the photo (upstream or downstream).

1. Open the file created on the desktop containing the bulk renamed photos.

2. Select the tab ‘View’ on the top left-hand side of file explorer.

3. Right click on the first photo, scroll down the pop-up bar and select ‘Rename’.

4. Enter in the site location and the photo direction separated by an underscore just before the recorded date and time (i.e. MIP_LSM_T11_US_20200104 – 15.32.44)

5. Select ‘Enter’

6. Repeat for all photos

Uploading images to SharePoint

The location of grab sample stream photos in SharePoint is ‘Data Storage - MIP Data Storage’ which can be accessed via the Terranium homepage.

To upload the recently named photos:

1. Go into ‘Data Storage - MIP Data Storage – Photos - Down and Up stream photos’ (Screenshot 35)

2. Select the grab sampling activity area (i.e. Event based grab sampling, North Johnstone River or Routine grab sampling).

3. Select ‘Upload’ on the tool bar across the top of the data storage page (Screenshot 36).

(33)

4. Select ‘Folder’ which will open a brower window where you can select the relevant folder from the computer desktop to upload. Select ‘Upload’.

5. The pop-up browser will close, and an icon will appear on the far right-hand side of the data storage toolbar which will give you information on the progress of the upload. Once the upload has completed you are free to close SharePoint.

Screenshot 35

(34)

Screenshot 36

(35)

6.9. Tracking Cairns Regional Council Laboratory Services projects

The following diagram (Figure 3) illustrates the process followed to track the timely and correct delivery of electronic results (eresults) and accompanying laboratory documentation. All tracking is done via the ‘WQ Sampling Master’ in the

‘Lab Project Tracking’ sheet. This process also includes the backing up of all lab data on SharePoint. The process is executed once a week.

Eresults, CoCs and laboratory certificates are received from the CRC Laboratory via email distribution lists (see Section 6.6). The Water Quality Leader, Data Manager and EnviroSys are on all email distribution lists. Other team members that were involved in the sampling for a particular activity area (FS, CR or LSM) are on distribution lists for specific projects that relate to their work.

Figure 3: Process diagram for tracking Cairns Regional Council Laboratory projects

The complete process described in the following sections can also be viewed as a video training session

First-time preparations

Step 1 – Sign-up to receive all emails related to laboratory results

In order to execute the procedure described, the person responsible for tracking and filing needs to be included on all CRC Laboratory distribution lists and all samplers should be advised to cc the responsible person into all CoC submissions to the CRC Laboratory.

Step 2 - Install macro and setup rules in Outlook

A macro needs to be implemented in Outlook to allow attachments from multiple emails to be saved at one time.

Instructions can be found here https://www.youtube.com/watch?v=nBHi4s9MN8M or via SharePoint. Written instructions can be found in Appendix 3.

Samplers are advised to put ‘CoC’ and the ‘Project Number’ into the subject line of the email when sending ‘Complete

(36)

For the macro to be useful, rules in Outlook need to be set up to have all emails from the lab and samplers moved into one folder. To do so:

(1) Select ‘Rules – Manage Rules & Alerts…’ from the ‘Home’ ribbon in Outlook (Screenshot 37)

Screenshot 37

(2) Select ‘New Rule’ in Screenshot 38

Screenshot 38

(37)

As outlined in Screenshot 39,

(3) Select ‘Move messages with specific words in the subject to a folder’

(4) Click on ‘specific words’

Screenshot 39

(5) Fill in the highlighted search terms, individually, and click the ‘Add’ button. When done, click ‘OK’ (Screenshot 40)

Screenshot 40

(38)

(6) Click on ‘specified’ (Screenshot 41)

Screenshot 41

(7) Create a ‘New…’ folder called ‘Lab Results’ (if not already done), select the ‘Lab Results’ folder and click ‘OK’

(Screenshot 42)

Screenshot 42

(8) If CRC Laboratory results are already in the inbox, click the ‘Run Rules Now…’ button (Screenshot 43) and follow the onscreen prompts. Otherwise click on ‘OK’.

Screenshot 43

(39)

Step 3 - Sync SharePoint folder

The sync feature in SharePoint allows files to be synchronised with a local desktop computer via OneDrive. This feature makes the upload to SharePoint easier by being able to move files locally.

In Screenshot 44,

(1) Navigate to ‘Data Storage’ – ‘MIP Data Storage’ – ‘CRC Lab Results’

(2) Click the ‘Sync’ button and follow the instructions on screen

Screenshot 44

(3) Click on ‘Open Microsoft OneDrive’ and follow the instructions on screen (Screenshot 45)

Screenshot 45

(40)

Other useful tools (optional)

QuickLook

QuickLook (Figure 4) is a tool that allows preview of the contents of a file (works for CSV and PDF files) without having to open it. To preview a file, simply press the spacebar when a file is selected. QuickLook is available free and can be downloaded here:

https://www.microsoft.com/en-au/p/quicklook/9nv4bs3l1h4s

Figure 4: QuickLook software tool

Directory Opus

Directory Opus (Figure 5) is an advanced file manager that allows customisation of the way file operations are performed. This makes file management more efficient and provides additional information. For example, when

replacing files, it shows more information about the existing file and the file to be replaced. That way, before replacing a file, the operator can be certain the files are identical.

The Pro edition can be downloaded for a fee at https://www.gpsoft.com.au/

Figure 5: Directory Opus software tool

(41)

Project tracking and file backup procedure

Step 1 - Saving attachments from multiple emails in Outlook

(1) Highlight all emails with eresults by highlighting the first email, holding the Shift key and then selecting the last new email (Screenshot 46)

(2) Select ‘Developer’ from the Ribbon, then ‘Macros’ and the script ‘SaveAllAttachments’

Screenshot 46

(3) You will be prompted to select a location to save the attachments on your local computer (e.g. under ‘My Documents – Outlook Attachments – Emails’ (Screenshot 47)

Screenshot 47

(4) Flag or categorise saved emails to mark the starting point for the next run (Screenshot 48)

Screenshot 48

(42)

Step 2 - Check project files As outlined in Screenshot 49,

(1) Cleanup - In the file system, remove all images that have been extracted from signatures

Please note: The CRC Laboratory delivers all correspondence relating to a project with the project number first in the file name in order to make it easier for sorting and finding. Should this have been missed, rename the files at this stage.

As technology can fail (e.g. information in ‘WQ Sampling Master’ not saved) or one may be called away to do another urgent job, process projects in order of completeness of laboratory project (A), (B), (C). It also helps by not having to jump between individual steps of the procedure.

(A) - Projects with eresults (B) - Projects only with SAMACKs and

(C) - Projects only with CoCs

Screenshot 49

(43)

Step 3 - Check WQ Sampling Info Master to see if project exists (for (A), (B), (C))

All tracking takes place in the ’Lab Project Tracking’ sheet of the ‘WQ Sampling Master’ (Screenshot 50).

Use the Find function (CTRL+F) in Excel to determine whether details for a project have already been recorded or if a new project needs to be added.

Screenshot 50

(44)

Step 4 - Update ‘WQ Sampling Master’

For (A) - eresults received

(1) Check Outlook (Screenshot 51) for date received and fill in column ‘Final results received’

Screenshot 51

For (B) - only SAMACK received

Open SAMACK (Screenshot 52 and Screenshot 53) and retrieve dates to enter into columns:

(2) - CoC + Samples submitted (3) - SAMACK issue date

(4) - Sample Date (page 2 or 3 of SAMACK) - If multiple dates are listed, the earliest one is to be chosen (this is not necessarily the top one in the list)

Screenshot 52

Screenshot 53

(45)

For (C) - only (In)complete CoC exists

Open the CoC (Screenshot 54) and retrieve:

(5) Sample Date (complete CoCs only) (6) Project Description

(7) Project#

Screenshot 54

(46)

Step 5 - Backup all lab files in ‘Data Storage - MIP Data Storage - CRC Lab Results’

(1) In the file browser, (Screenshot 55), determine which folder the unprocessed files should be moved to by using the name of eresults files for each set of CRC Laboratory project files. For example, the file named

‘100782_eresults_MIP_CR_Bioreactor (BR-02)_22-4-2020' should be filed in the ‘CR’ activity area subfolder, under ‘BR02’.

For an overview of the complete folder structure, please refer to Appendix 4.

(2) Use the Ctrl or Shift key to select multiple files that are to be moved to the identified location. Move the files by dragging and dropping the files into the desired location.

Screenshot 55

Once the files have been pasted in their final destination, check that each set contains a complete CoC, eresults, SAMACK and Certificate of Analysis (CoA) file. Supplementary CoA files will also exist for projects where a sub-contracting laboratory has been used (i.e. if pesticides or dissolved organic carbon were also analysed).

Please note, incomplete CoCs are not filed on SharePoint as they are a template for samplers to populate in the field and do not contain any sample-specific information. The arrival of a CoC, however, marks the commencement of the tracking process and requires the person responsible for tracking to enter the project number into the WQ Sampling Info Master sheet. This is important because some samplers use the tracking sheet to take stock of what sample bottle packs are in storage at the Terrain and Canegrowers offices.

Step 6 - Update WQ Sampling Master Update field ‘Results filed on SharePoint Y/N’.

Please note, when an eresult has to be re-issued, the file name of the eresult has an ‘-R’ appended to the end. When this occurs, record the re-issue in the column ‘Results re-issued? (Y/N)’ of the ‘WQ Sampling Master’. Re-issued eresults are filed in SharePoint in addition to the original issue as described in Step 5.

Inform the Water Quality Leader if any holding periods have been exceeded, if results are overdue or if any of the file types cannot be found for a project.

Step 7 – Invoice checking

The WTMIP Corporate Support Coordinator receives all invoices from CRC Laboratory. During invoice processing the Corporate Support Coordinator adds ‘Y’ in column ‘Invoice received’ and adds new project numbers to the spreadsheet if the project is not already recorded.

The Corporate Support Coordinator uses the tracking sheet to determine whether results have been received and therefore, whether it is alright to pay the invoice. As such, it is important for the tracking sheet to be updated on a weekly basis. Should an update be delayed, the Corporate Support Coordinator needs to be informed of the delay so that an alternate method can be employed to check that results have been received, and CRC Laboratory can still be paid.

(47)

7. Data Management - High frequency logger data

7.1. Transferring data from eagle.io to EnviroSys and SharePoint

eagle.io is a cloud-based platform to collect data from, and submit commands to, remotely deployed environmental sensors. Data collected from in-situ equipment is submitted via telemetry directly into the eagle.io portal and accessible via https://app.eagle.io/ui/camp51/Terrain_NRM_SO18000986/. Three separate eagle.io workspaces exist for the WTMIP, each set-up and administered by the vendors contracted to install the monitoring equipment (i.e. Xylem, Campbell Scientific Australia and Turbid). The Xylem workspace relates to the LSM in-stream high-frequency monitoring stations, the Campbell Scientific workspace is for the LSM paddock run-off station data and the Turbid workspace relates to the in-situ equipment installed at the high efficiency sediment basin. Some WTMIP staff also have access to the Department of Environment and Science Water Quality and Investigation team’s eagle.io workspace containing the end- of-catchment data for the Great Barrier Reef Catchment Loads Monitoring Program.

Screenshot 56 shows the eagle.io user interface.

Screenshot 56

The eagle.io and SharePoint/EnviroSys connection

eagle.io provides the ability to extract data via an Application Programming Interface (API) by utilising a unique API key for each equipment vendor (Xylem, Turbid, Campbell Scientific Australia) or other external sources (e.g. WQI eagle.io data). A script is used to extract the data via the API and then saves the data to a destination system as desired.

All scripts are stored on SharePoint in ‘Project Centre – MIP – Knowledge and Info - Technical Papers – Choosing an Environmental Data Management System - EDMS – 004 Implementation – Scripts’. The scripts are provided and maintained by acQuire.

eagle.io SharePoint backup script

While eagle.io backs up all data in their own data centres for six months, there is no means for a customer to perform an automatic backup without purchasing a premium account or purchasing extra online storage (Amazon S3). Prior to the EnviroSys implementation, a decision was made to secure the WTMIP data stored on eagle.io by contracting Future

(48)

EnviroSys now provides the primary back up of WTMIP eagle.io data (see below) and the SharePoint script is no longer maintained or checked. The information below is retained in this SOP for future reference, should there be a need to transition away from EnviroSys in the future.

The ‘eagle.io SharePoint backup script’ provides an automatic backup for all eagle.io data which can be easily accessed in SharePoint at https://terrainnrm.sharepoint.com/Eagleio/. The script is installed on the EnviroSys server and executes once per day at 1am. To check the script is working, conduct a visual check in SharePoint once a week. To do so, follow the above link, open the vendor’s folder and check to see if the ‘Date Modified’ is ’24 hours ago’ or less.

eagle.io EnviroSys import script

To have all data available in one database, the ‘eagle.io EnviroSys import script’ imports all data from eagle.io into EnviroSys. The script runs on the EnviroSys server every 30 minutes and data is directly imported into EnviroSys.

Ensure the script is working by regularly checking the ‘Realtime’ panel in EnviroSys (Screenshot 1), to check for alerts and confirm that all data is current.

eagle.io QC data download script

eagle.io data is checked by the Water Quality Technician for abnormal data and is categorised into various subcategories of ‘Good’, ‘Bad’ and ‘Uncertain’. For more specific information on categorisation, refer to ‘WTMIP SOP 14 - Management of Telemetered Water Quality Data on eagle.io’.

The ‘eagle.io QC data download script’ is required because the quality coding takes place once a month and quality codes are therefore not captured by the ‘eagle.io EnviroSys import script’ which imports data from the most recent 30 minutes only.

Also, a decision was made to perform the quality coding at the source, in eagle.io, as this platform accommodates rapid batch coding of multiple records at one time.

The ‘eagle.io QC data download script’ is executed by the user after the quality coding in eagle.io has been completed for the month.

To use the script:

(1) In Screenshot 57, download the script from SharePoint (from ‘Project Centre – MIP – Knowledge & Info – Technical Papers – Choosing an Environmental Data Management System - EDMS – 004 Implementation – Scripts – eagle io QC data download script’) and save it locally on your computer. Double click on the highlighted ‘Download eagleio QC data.bat’ file.

Screenshot 57

(2) Select a destination folder (e.g. ‘Documents/Quality Codes’) to save the exported data (Screenshot 58).

(49)

Screenshot 58

(3) Enter a ‘Start Date’ using format YYYY-MM-DD (Screenshot 59).

Screenshot 59

(4) Enter an ’End Date’ using format YYYY-MM-DD (Screenshot 60).

Screenshot 60

(5) Select which logger the quality codes should be exported from, click OK and wait until the export is finished (Screenshot 61). This may take several minutes depending on how long of a date range has been selected.

(50)

Screenshot 61

(6) Send the resulting file to '[email protected]'

(7) Visually confirm in EnviroSys portal that quality code has been applied to sample

(51)

7.2. Transferring data from HOBO data loggers to EnviroSys

The Catchment Repair program uses HOBO water level data loggers. Data collected by the HOBOs is stored internally on the device. Data is retrieved manually and saved to SharePoint before being imported to EnviroSys though emailing Excel files to [email protected].

The installation, operation, and maintenance of HOBO loggers as well as the retrieval of data is described in ‘WTMIP SOP 018: HOBO deployment, download and processing’.

7.3. Transferring data from Catchment Repair rain gauges to EnviroSys

Rain gauges are used by Catchment Repair and the operation, maintenance of the equipment as well as the retrieval of data are described in ‘WTMIP SOP 017: Operation of RainLog 2.0 data loggers and RL loader software for recording rainfall’.

Data logged by CR rain gauges are stored in EnviroSys. Import of data into EnviroSys is done by sending the exported file from the ‘RL-Loader’ software (see WTMIP SOP 017) to [email protected].

(52)

7.4. Storing time-lapse imagery on SharePoint

The WTMIP operates five paddock run-off sampling stations at two locations. Autosamplers are used to collected paddock run-off water from flume structures (Figure 6) positioned low in the landscape. Time-lapse photographs of each flume are taken every 15 minutes to enable water quality technicians to check what has been happening at the flumes when personnel were not able to be present on paddock (e.g. that there was sufficient water passing through the flume when the autosampler triggered or how foreign objects made their way into the flume). For more information on paddock run-off monitoring, please refer to ‘WTMIP SOP 001: Wet Tropics Major Integrated Project, Local Scale Monitoring Program: Monitoring design overview and rationale’ and ‘WTMIP SOP 007: Paddock surface water monitoring for nutrients, total suspended solids and pesticides.’

The procedure below provides instruction on retrieving, processing and storing time-lapse imagery.

Figure 6: Flume with autosampler intake hose and flume wing walls to direct run-off into the flume

Step 1 – Retrieval of SD card

During a site visit, detach the time-lapse camera (Figure 7), unscrew the cover and remove the SD card.

Figure 7: Time lapse camera with SD card slot

(53)

Step 2 – Copying SD card content to PC

Insert the SD card into your computer and copy the content of it to \Documents\SITENAME\ (e.g. \Documents\T2\

or \Documents\J1\, depending on the site location (MIP_FS_T2 or MIP_FS_J1)).

Step 3 – Confirm all files have been transferred

First, identify the oldest file on the SD card by sorting files by ‘Date Created’. Then, in your file browser or in SharePoint, identify the same file in the upload location. Look at the adjacent file and confirm that it is 15 minutes older. This confirms the continuity from the current upload to the most recent one.

To confirm completeness, keep files sorted by ‘Date Created’ and scroll from the oldest to the most recent file. Visually check that images appear in 15-minute increments. Compare the oldest and the newest file with the data on the SD card.

If operating in the file browser, another method of checking is to select all recently copied files, retrieve the file count from the status bar of your file browser and make sure that it is the same as the number of files on the SD card.

Step 4 – Deleting SD card content

Delete the images on the SD card, remove the SD card from computer, insert it back into the time-lapse camera, close the cover of the time-lapse camera and re-attach it to the star picket in front of the flume, making sure the camera is positioned so that the entire flume is visible within the field of view, the solar panel power source is connected, and the time and date is correct.

Step 5 – Bulk renaming files

Once back at the office, select the images with the date and time range of interest (i.e. an hour before through to an hour after an active sampling event). All other photos can be discarded if they do not relate to an event, false trigger or other activities on paddock.

Bulk rename all images using ‘Bulk Rename Utility’ from https://www.bulkrenameutility.co.uk following the steps detailed below:

As per Screenshot 62,

(1) Navigate to the location of the stored images (e.g. \Documents\SITENAME\) and select all available files (2) ‘Add (7)’: Fill in location in ‘Suffix’ leading with an underscore to add the site location to the file name (e.g. _T2) (3) ‘Auto Date (8)’: Fill in the fields ‘Mode’=Prefix, ‘Type’=Creation (Current), ‘Fmt’ = Custom and ‘Custom’ = %Y-%m-

%d - %H.%M.%S_

(4) Click the "Rename" button and confirm

The resulting file name will now have the creation date and time, original file name and the location of the site it was taken in the file name for easy sorting and identification – e.g. 2019-04-03 – 15.32.44_SYCR0081_T2.

(54)

Screenshot 62

Step 5A – Uploading files onto SharePoint

For the few users who perform this task on a regular basis, it is recommended to “sync” files with your laptop. For casual users, follow the instructions provided in ‘Step 5B’.

To synchronise a SharePoint folder with your laptop:

(1) Navigate to ‘Data Storage’ – ‘MIP Data Storage’ – ‘Photos’ – ‘Time Lapse Photographs’

(2) Click the ‘Sync’ button

(55)

(3)On the resulting pop-up window, click on ‘Open Microsoft OneDrive’ and follow the prompts on screen

(4)If setting up the sync functionality for the first time, wait for the syncing of existing files to be completed. Then, copy new photographs into the appropriate year folder of the location on your computer ‘Data Storage - MIP Data Storage’ –

‘Photos’ – ‘Time Lapse Photographs’ – ‘[SITENAME]’. OneDrive will automatically sync the copied files to SharePoint.

Step 5B -- Alternative upload method

If this task is executed by a user who does not perform this procedure on a regular basis, it is recommended to upload images directly into SharePoint as synced data takes a while to download, fills up space on the computer unnecessarily and, when too many users are syncing the same folder with SharePoint, file version conflicts may occur.

To upload photographs, navigate to the following location in SharePoint ‘Data Storage - MIP Data Storage’ – ‘Photos’ –

‘Time Lapse Photographs’ – ‘[SITENAME]’ – ‘YEAR’.

Verify that the upload is complete by clicking on the "Show Progress" icon (Screenshot 63) immediately after starting the upload (otherwise the notification will disappear when upload is finished).

Screenshot 63

Step 7 - Delete files

Delete photographs that were initially copied onto your laptop at \Documents\SITENAME\.

(56)

8. Data Management – High frequency data from external sources

8.1. Importing Queensland Government water quality data into EnviroSys

The Water Quality and Investigations (WQI) team from the Department of Environment and Science (DES) conducts the Great Barrier Reef Catchment Loads Monitoring Program and uses eagle.io for storing and accessing data. The data is shared with WTMIP team members via the ‘DISTI WQI’ eagle.io workspace (Screenshot 64).

Screenshot 64

To enable the WTMIP team to easily access the DES data relevant to the Tully and Johnstone basins, data from the following WQI sites is retrieved from eagle.io and imported into EnviroSys by the same method described in Section 7.1.

JRI Logger (Johnstone River at Innisfail)

JRC Logger (Johnstone River at Coquette Point) TRE Logger (Tully River at Euramo)

TRG Logger (Tully River at Tully Gorge National Park)

(57)

8.2. Importing BOM rainfall data into EnviroSys

Rainfall data is collected to evaluate sampled data and to better understand and communicate the relationship between fertiliser application timings/amounts and excess run-off into rivers. Data collected from loggers, in-situ monitoring stations and/or grab sampling is therefore often overlayed with rainfall data. The following section describes how rainfall data sourced from the Bureau of Meteorology (BOM) is transferred into EnviroSys. The management of rainfall data generated by the WTMIP is described in Section 7.1 and Section 7.3.

Table 2 Details of Bureau of Meteorology stations

Station Name BoM station code Data source/owner Retrieval Method

Innisfail 032025 BOM Automatic import

Innisfail Wharf Alert 032163 BOM Import via CSV file

Innisfail Aerodrome 032197 BOM Automatic import

South Johnstone 032037 BOM Automatic import

Tully Sugar Mill 032042 BOM Import via CSV file

Bolinda 531057 Cassowary Coast Council Request from council

Innisfail, Innisfail Aerodrome and South Johnstone

Data from Innisfail and South Johnstone weather stations are made accessible in EnviroSys via API import from the BOM.

Only some BOM stations support the ability for data to be directly imported into third party systems such as EnviroSys. No user interaction is required.

No data has been available from the Innisfail station (BOM station code 032025) since June 2020. The Innisfail Aerodrome and Innisfail Wharf Alert sites were subsequently added to the suite of BOM sites used by the WTMIP.

Tully Sugar Mill and Innisfail Wharf Alert

For some BOM weather stations only manual downloads of historic data is possible. The weather stations ‘Tully Sugar Mill’

and ‘Innisfail Wharf Alert’ fall into this category.

To retrieve data for ‘Tully Sugar Mill’, go to http://www.bom.gov.au/climate/data/ and find Tully Sugar Mill in section 2 as shown in ‘Screenshot 65’ (1) and click on ‘Get Data’ (2).

(58)

On the resulting web page (Screenshot 66), click on ‘1 year of data’ (3). This will download a zip file which contains a csv file for the current year.

Screenshot 66

Unzip the file and open the csv file in Excel. Copy all contents, or just the date range you require, and then paste into cell B2 in the Excel template ‘BoM Rainfall.xlsx’ (Screenshot 67). The ‘BoM Rainfall.xlsx’ is available by double clicking on this icon:

BoM Rainf all.xlsx

Screenshot 67

To combine the ‘Year’ (column D), ‘Month’ (column E) ‘Day’ (column F) into a single column, copy the formula in column A down the entire data set as shown in Screenshot 68. This step is required so that EnviroSys can read the date in the correct format.

Screenshot 68

Save the file as a csv, name it ‘Tully Rainfall.csv’ and email the csv file to [email protected].

Management of Water Quality Data Using EnviroSys and SharePoint