1. Data collection and dissemination in hydrology

(1)

WMO Data Operations and Management

Global initiatives in hydrological data sharing

The ultimate goal of data collection in hydrology, be it precipitation measurements, water-level recordings, discharge gaugings, groundwater monitoring or water quality sampling, is to provide a set of sufficiently good quality data that can be used in decision-making in all aspects of water resources management, in a wide range of operational applications as well as in research. Decisions may be made directly from raw data measurements or based on derived statistics or on the results of many stages of modelling beyond the raw data stage, but it is the collected data that form the basis for these decisions. Data sets are of great intrinsic value as they are collected through a huge commitment of human and financial resources and often during a long period of time, and they acquire additional and sometimes unexpected value when they are made available in an accessible format meeting users’ specific needs.

The management of hydrological data is therefore important work in itself and this work must be performed effectively in order to maximize the return on investments made in its collection and to realise its potential for effective and efficient use. WMO Hydrology and Water Resources Programme has always given high priority to issues related to data management, be it through the work of experts of the Commission for Hydrology (CHy), the publication of manuals and other guidance material, the implementation of projects for improving data collection and information production (such as WHYCOS) and data rescue, and the promotion of access to and exchange of hydrological data, through the implementation of Resolution 25 (Cg-XIII) - Exchange of Hydrological Data and Products. 2. Sharing of data and information

In May 1999, the thirteenth WMO Congress adopted Resolution 251, which deals with the exchange of hydrological data and products. Through Resolution 25, WMO Members are committed to broadening and enhancing, whenever possible, the free and unrestricted international exchange of hydrological data and products, in consonance with the requirements for WMO’s scientific and technical

programmes, thereby contributing to disaster risk reduction, human safety and well-being, and shared socio-economic benefits. Resolution 25 also urges WMO Members to make their best efforts to implement practices leading to international exchange of hydrological data and products and to assist other Members in developing their capacity in this area.

In 2005, a report from the Global Climate Observing System (GCOS) on data exchange problems in global hydrological and atmospheric networks identified standards as a key issue. The report stated that: “There are no established international standards on the acquisition of river data, the set of required metadata, data formats, and transmission modes"2. Volume 1 of the WMO Commission for Hydrology Guide to Hydrological Practices, updated in 2008, also states “There are currently no standards for data exchange formats for hydrological data”3.

In 2009, WMO and OGC (Open Geospatial Consortium) jointly formed a Hydrology Domain Working Group (HDWG) to address this issue. Through activities of the HDWG, in 2012, WaterML 2.0: Part 1- Time Series4 (referred to as WaterML 2.0 in this documentation) was adopted by the OGC as an official standard. WaterML 2.0 is a standard information model for the representation of water

1

WMO 13th Congress (Cg-XIII), Geneva, May 1999. Resolution 25 – Exchange of Hydrological Data and Products.https://www.wmo.int/pages/prog/hwrp/documents/Resolution_25.pdf

2_{GCOS-96 -}_{Analysis of Data Exchange Problems in Global Atmospheric and Hydrological Networks}_._February 2005, p.20. http://www.wmo.int/pages/prog/gcos/Publications/gcos-96.pdf

3_{WMO-No. 168 -}_{Guide to Hydrological Pactices, Volume I – Hydrology: From Measurement to Hydrological}

Information. 6th Edition, 2008, p. 10-26. http://www.whycos.org/chy/guide/168_Vol_I_en.pdf

4_OGC®_{WaterML 2.0: Part 1- Timeseries}_{. September 2012.}_{http://www.opengis.net/doc/IS/waterml/2.0}

2

(3)

observations data, developed to enable the exchange of hydrological data between information systems. Encoded in eXtensible Markup Language (XML) and based on existing OGC standards, WaterML 2.0 provides an interoperable hydrological exchange format that may be used to address a wide range of user needs. These include the exchange of data relating to:

• In situ observations at hydrological (gauges, reservoirs) or climatological stations;

• Forecast products (probabilistic or deterministic timeseries) at forecast locations;

• Emergency or operator-oriented alerts (of threshold exceedance) and reports;

• Timeseries of planned intake and release/discharge; and

• Groundwater observations of water level within wells.

Using WaterML 2.0, the linking of local, national, regional and global water information sources is possible as part of global water information networks.

In November 2012, at its 14th session, WMO Commission for Hydrology passed Resolution 3 recommending that WMO Members test, through pilot projects during 2013-16, the use of WaterML 2.0 for the exchange of hydrological data, with a view to the potential adoption of WaterML 2.0 as a joint WMO/ISO standard. The Commission also noted the importance of WaterML 2.0 and other emerging OGC standards to improve service delivery of key CHy programmes including the WMO's World Hydrological Cycle Observing System (WHYCOS) and the WMO Flood Forecasting Initiative. In September 2013 the WMO CHy released a WaterML 2.0 survey, whose responses highlighted initiatives in the exchange of hydrological data using OGC web services and the international standard for water data transfer: WaterML 2.0. Survey results were obtained from different organizations, including global data centres, national and regional hydrological services, research institutes and system vendors.

Through the Data Operations and Management activity initiated at CHy-14, the Commission is interested in engaging with projects that are using WaterML 2.0 for exchange of hydrological data at global, national or state/provincial levels.

There are many system choices, including commercial off the shelf and open source software, which can be used for hydrological data storage and management, some of which also now support OGC web services and data sharing using WaterML 2.0. WMO Members are encouraged to explore options for increased national, regional and global data sharing using WaterML 2.0 and to report their

experiences.

3. The importance of standards

Water observation data is fundamental to our understanding of water resources and their spatial and temporal variability. Water resource management within countries, regions and continents around the world is highly distributed with many organisations typically involved in the collection and management of water data, even within single countries. In addition geographic features such as river basins and aquifers generally do not align with the boundaries of nation states and 90% of people live in countries that share transboundary hydrological features with their neighbours5. As a result, to understand water resources within basins or aquifers, hydrological data sharing both within and between countries is usually required.

Content and format standards for water data exchange are a fundamental requirement to deal with the complexity which arises when disparate data from different organisations are brought together.

5

UN-Water (2008). Thematic Paper. Transboundary Waters: Sharing Benefits, Sharing Responsibilities.

http://www.unwater.org/downloads/UNW_TRANSBOUNDARY.pdf

3

(4)

Adherance to standards such as WaterML 2.0 allows data to be aggregated more easily without the complex task of understanding and translating multiple data formats. The need for exchange of water observations data operates at many different levels, from sharing across international borders to inter- and intra-agency sharing within individual countries.

Development of community-agreed consistent models and exchange formats for spatial and temporal data and metadata increases interoperability between information systems. It has many practical benefits including:

• Improved efficiency and quality of information models and systems;

• Wider use and re-use of information;

• Vendor and open source support at low or no cost to users; and

• New value to existing information through serendipitous uses.

The WaterML 2.0 standard for hydrological data exchange provides a solid foundation for improved interoperability between water information systems and increased data sharing between WMO Members.

4. Introduction to WaterML 2.0

WaterML 2.0 is an OGC® Encoding Standard for the representation of hydrological

observations data with a specific focus on timeseries structures. WaterML 2.0 is implemented as an application schema of the Geography Markup Language version 3.2.1, making use of the OGC Observations & Measurements standard.

WaterML 2.0 is designed to support a variety of exchange scenarios including: exchange of data for operational hydrological monitoring programs; supporting operation of infrastructure (e.g. dams, supply systems); crossborder exchange of observational data; release of data for public

dissemination; enhancing disaster management through data exchange; and exchange in support of national reporting6.

WaterML 2.0 was developed by members of the WMO/OGC Hydrology Domain Working Group which included staff from CSIRO and the Bureau of Meteorology in Australia; the Consortium for the

Advancement of Hydrological Sciences Inc (CUAHSI), the San Diego Supercomputer Center, USGS in the US; the Federal Institute of Hydrology, Federal Waterways Engineering and Research Institute (GmbH) and Kisters AG in Germany; the International Office for Water – Sandre in France; Deltares in the Netherlands; Environment Canada and Aquatic Informatics in Canada and various other national authorities and companies. Working group discussions focussed initially on understanding and harmonising existing water data exchange standards from countries around the world7. This was followed by development of a Unified Modelling Language (UML) information model and draft

eXtensible Markup Language (XML) exchange format which was tested through surface, groundwater and forecasting Interoperability Experiments, and formation of an OGC Standards Working Group to finalise the WaterML 2.0 standard.

The WaterML 2.0 information model conforms to the Observations and Measurements (O&M) standard8, which defines a conceptual schema and cross-domain vocabulary for description of

observational data and metadata, including features involved in sampling. Conformance to a common

6

OGC® WaterML 2.0: Part 1- Timeseries. September 2012. http://www.opengis.net/doc/IS/waterml/2.0

7

OGC® Harmonising Standards for Water Observation Data - Discussion Paper. June 2010.

http://portal.opengeospatial.org/files/?artifact_id=39090

8

ISO 19156:2011 Geographic information -- Observations and measurements. 2011.

http://www.iso.org/iso/catalogue_detail.htm?csnumber=32574

4

(5)

model such as O&M supports increased interoperability through standardisation of certain terms for observational metadata, including: feature-of-interest, observed-property, procedure, temporal metadata and result. This allows data from disparate sources to be mutually understood and reused, is applicable to many exchange scenarios and makes it easier to share data across subject domains9.

Figure 1: Observations and Measurements – Simplified OM_Observation type information model

In the O&M conceptual schema, an observation is an action whose result is an estimate of the value of some property of thefeature-of-interest, obtained using a specifiedprocedure. For example, a feature-of-interest (a river at a certain location) has a property (discharge) that is observed using a procedure (stream gauging), that produces a result (a timeseries of flow values).

9

Taylor, P., Cox, S., Walker, D., Valentine, D., Sheahan, P. (2014). WaterML 2.0: development of an open standard for hydrological time-series data exchange. Journal of Hydroinformatics vol 16.2 pp 425-446.

doi:10.2166/hydro.2013.174

5

(6)

WaterML 2.0 can be used to exchange many kinds of hydro-meteorological observations, encoded as a timeseries in XML, and provides the option to also store metadata including the quality, validity, interpolation type and other information to enable users to understand the data being provided. WaterML 2.0 files are written in eXtensible Markup Language (XML) and can be transported between systems via ftp, email, file-copy or http (web protocol). XML is a hierarchical text markup language, which is capable of being understood by software programs (machine readable), and can be self documenting. XML exchange formats for any domain of data can be developed based on community-agreed information models.

A key component of the WaterML 2.0 information model relates to the representation of timeseries observations and related metadata describing the observations. For example a timeseries of daily measurements of discharge at a river monitoring station for the period 1-6 January 2001 may include the values: Date Discharge 1 January 2001 2.03 2 January 2001 2.06 3 January 2001 2.42 4 January 2001 missing 5 January 2001 2.04 6 January 2001 2.23

Using WaterML 2.0, timeseries observations can be encoded in XML files for data exchange as either interleaved time-value pairs or as a domain-range encoding. These two encoding methods are shown in figures 2 and 3 respectively using the example discharge data for 1-6 January 2001.

(7)

Figure 2: WaterML 2.0 – XML example showing timeseries observations encoded as interleaved time-value pairs

Figure 3: WaterML 2.0 – XML example showing timeseries observations encoded as domain-range highlighting linked values

(8)

5. Case studies in the use of standards for hydrological data exchange

The following sections describe a number of case studies on the use of standards for data exchange provided by National Hydrological Services (NHS). The first two case studies involve fully

implemented national systems in Australia and USA respectively, while the other case studies represent pilot projects showing the benefit of data sharing using WaterML 2.0 within and between other countries and regions around the world.

5.1. Case Study #1: Australian Water Information Program

This case study relates to transfer of hydrological data using the file transfer protocol (ftp) from water agencies to the Bureau of Meteorology – Australia’s NHS. In 2007, the Australian Government gave the Bureau of Meteorology responsibility for compiling and delivering comprehensive national water information through its Improving Water Information Program. In Australia, water information is collected by over 200 organisations, using a variety of methods. The range of collection and reporting methods and arrangements for accessing water data made it difficult to monitor the status and use of Australia’s water resources and reliably forecast water availability, compromising effectiveness of water resources management and planning.

Section 126 of the Commonwealth Water Act 2007 (Australian Government legislation) specifies the Bureau’s water information functions and provides for the making of Regulations which prescribe:

• The organisations that must give the Bureau a copy of specified water information;

• Types of water information to be given to the Bureau; and

• Timeframes and format in which the information must be given.

The Regulations came into effect on 30 June 2008 and since then the Bureau has received 30 million files containing over 4 billion observations from about 200 data providers covering over 100

parameters across 10 data categories and thousands of hydrological monitoring sites. The Bureau continues to receive over 10,000 new files of water data per day via the file transfer protocol (ftp). A critical underpinning of the Bureau’s ability to receive, process, ingest and understand the water data being provided was development of an XML exchange standard for water information. This standard file format, known as the Water Data Transfer Format (WDTF)10, was developed in 2008 by CSIRO and the Bureau of Meteorology. Lessons learnt from development of this Australian standard, WDTF, were a key input into the international standardisation process to develop WaterML 2.0

through the WMO/OGC Hydrology Domain Working Group, which was led by CSIRO and the Bureau. Development of the Water Data Transfer Format and its adoption by Australian water data providers and water data management system vendors has enabled the Bureau of Meteorology to build a comprehensive national database used for critical analyses and improved water information reporting and forecasting in Australia.

5.2. Case Study #2: United States Geological Survey (USGS) data distribution

This case study illustrates data provision using WaterML 2.0 and other methods by the USGS, providing a freely available instantaneous values service providing access to streamflow and other real-time data from thousands of US monitoring stations, as well as a daily values service and

groundwater levels service over the Internet. In March 2014, support for provision of timeseries data in the WaterML 2.0 format was added to these services.

10

WDTF: Water Information: Bureau of Meteorology. http://www.bom.gov.au/water/standards/wdtf/

8

(9)

This data is used for water resource management, monitoring floods and droughts, bridge design and for many recreational activities. Through its National Water Information Service (NWIS)11, USGS provides summary reports, maps, statistics and other information via the Internet as well as the data services designed for machine-to-machine access via automated tools. These 24/7/365 operational services provide access to streamflow, water level and other data from over 50,000 surface water and 800,000 groundwater monitoring stations across the USA. More than 15,000 of the monitoring

stations report in near real-time.

As automated approaches to data access become more common, use of USGS water data services is increasing at a faster rate than more conventional web page access through as shown in figure 4.

Figure 4: Growth in use of USGS water data distribution methods (page requests per month)

Data provision by USGS represents the current state of the art in real-time access to discharge and water level data encoded in WaterML 2.0. Using the USGS service, it is easy to create a persistent URL (web address) that provides a link to the latest discharge data for any site. For example the WaterML2.0 service for the Colorado River at Austin, Texas is available at:

http://waterservices.usgs.gov/nwis/iv/?format=waterml,2.0&sites=08158000&period=P1D&parameterCd=00060.

5.3. Case Study #3: Within country data sharing pilots

5.3.1 Federated national data sharing in Italy

Over the last 100 years a “Hydrological Yearbook” has been published in Italy, divided into Part I (temperature, precipitation, snow pack) and Part II (river water levels, liquid and solid river discharges, groundwater levels, catchment areal precipitations and water balances). All these data are

11

USGS Water Data for the Nation. http://waterdata.usgs.gov/nwis

0..*

9

(10)

published after an accurate data validation and represent the official hydrological-hydraulic reference data available for Italy. In 1998 a national decree assigned the task in collecting, validating and publishing hydrological data to Italian Regions, which have to provide to the publication each year their collected hydrological data. Hydrological Yearbooks initially were available only in hard copy; in recent years they have been made available also via the Internet and every Italian Region has different systems to produce hydrological data.

Due to the different data formats and service types among Regions and Administrations, ISPRA12 (the governmental technical body established by the Italian Ministry of Environment) and ARPA Emilia Romagna13 (the regional environmental agency which inherited the Po Hydrographic Office) proposed the development of a national hydrological catalogue together with all the Italian Regions, applying an advanced architecture for standardized data sharing. The architecture is an Internet-based system that supports sharing of hydrologic data, including data discovery, access and dissemination.

A central catalogue was installed In Rome, at the premises of ISPRA, while all regional web services are registered into the catalogue14.

Figure 10 – Standardized hydrologic data exchange in Region Emilia Romagna

The Italian hydrologic monitoring is a federated network composed by nineteen Administrative Regions and two Autonomous Provinces. The portal provides access to the hydrological observations in Italy, commonly published as Hydrological Yearbooks. In particular, it provides additional operational capability, for in situ water observations, as a national registry of water data services

12

ISPRA. http://www.isprambiente.gov.it/en?set_language=en

13

Arpa Emilia-Romagna Idro-Meteo-Clima.

http://www.arpa.emr.it/dettaglio_generale.asp?id=2888&idlivello=1592

14

ISPRA Hydrologic Information System Central Web Service Registry.

http://www.hiscentral.isprambiente.gov.it/hiscentral/

10

(11)

catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

5.3.2 Standardized data distribution in New Zealand

Hydrologic monitoring in New Zealand is a federated network composed by 16 Regional Councils, together with National Institute of Water and Atmospheric Research (NIWA15), a Crown Research Institute established in 1992, which conducts leading environmental science to enable the sustainable management of natural resources for New Zealand and the planet.

A pilot project supported the development of a web portal providing access to the hydrological observations in New Zealand. In particular, the portal provides additional operational capability, for in situ water observations, as an international registry of water data services catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

Figure 8 – Hydrologic data exchange by standardized service types and data formats in New Zealand

15

National Institute of Water and Atmospheric Research (NIWA). http://www.niwa.co.nz

11

(12)

5.3.3 Standardised data dissemination in China

Hydrologic monitoring in China is a federated network composed by 22 Provinces, 4 Municipalities, 5 Autonomous Regions, 2 Special Administrative Regions and the Taiwan Province, together with the Ministry of Water Resources, the Chinese Government Department responsible for water administration.

A pilot project supported the development of a web portal providing access to the hydrological observations in China, commonly published as Hydrological Yearbooks. The portal demonstrated useful operational capability, for in situ water observations, as an international registry of water data services catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

Figure 6 – Hydrologic data exchange by standardized service types and data formats in China

(13)

5.4. Case Study #4: HYCOS data sharing pilots

5.4.1 Arctic-HYCOS

The Arctic-HYCOS16 program is being promoted through the World Hydrological Cycle Observing System (WHYCOS17). The main goal of the Arctic-HYCOS program is to improve monitoring, data accuracy, availability and dissemination of information in the pan-arctic drainage basin. This project is science-driven and is aimed at monitoring freshwater fluxes and pollutants into the Arctic Ocean with the objective of improving climate predictions in the Northern Hemisphere and assessing the pollution of the Arctic coastal areas and the open Arctic Ocean.

In this context a pilot web portal was developed to provide access to the hydrological observations in the Arctic monitoring network, composed of stations located in Canada, Denmark, Finland, Iceland, Kazakhstan, Mongolia, Norway, Russian Federation and United States of America. In particular, the portal provides additional operational capability, for in situ water observations, as an international registry of water data services catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

Figure 5 – Hydrologic data exchange by standardized service types and data formats in the Arctic Region

16

Arctic-HYCOS. http://www.whycos.org/whycos/projects/under-implementation/arctic-hycos

17

World Hydrological Cycle Observing System (WHYCOS). http://www.whycos.org

13

(14)

5.4.2 Niger-HYCOS

The Niger river basin hydrological observing system, designated Niger-HYCOS18, is a federated network composed by 9 States (Benin, Burkina Faso, Cameroon, Ivory Coast, Guinea, Mali, Niger, Nigeria and Chad), together with the Niger Basin Authority, an inter-governmental organization responsible for promoting cooperation amongst Member States and contributing to improve the living conditions of the basin populations through sustainable management of water resources and associated ecosystems.

In this context, a pilot web portal was developed to provide access to the hydrological observations in the Niger river basin, commonly published as Hydrological Bulletins. In particular, the portal provides additional operational capability, for in situ water observations, as an international registry of water data services catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

Figure 7 – Hydrologic data exchange by standardized service types and data formats in the Niger river basin.

18

Niger-HYCOS. http://www.whycos.org/whycos/projects/under-implementation/niger-hycos

14

(15)

5.4.3 SADC-HYCOS

The SADC-HYCOS19 program is a regional component of the World Hydrological Cycle Observing System. The main goal of the SADC-HYCOS program is to ensure that participating countries,

individually and collectively, attain the technical capability, effective management and equitable use of the freshwater resources of the sub-region and to establish a Water Resources Information System as an effective decision making tool.

In this context, a pilot web portal was developed to provides access to the hydrological observations in the Southern African monitoring network, composed of stations located in Angola, Botswana,

Democratic Republic of Congo, Lesotho, Malawi, Madagascar, Mauritius, Mozambique, Namibia, South Africa, Swaziland, Tanzania, Zambia and Zimbabwe. In particular, the portal provides additional operational capability, for in situ water observations, as an international registry of water data services catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

Figure 9 – Hydrologic data exchange by standardized service types and data formats in the Southern Africa

19

SADC-HYCOS Phase II. http://www.whycos.org/whycos/projects/under-implementation/sadc-hycos-phase-ii

15

(16)

5.5. Case Study #5: Regional data sharing pilots

5.5.1 Latin America

The Ibero-American hydrologic monitoring is a federated network composed by the Ibero-American National Meteorological and Hydrological Services, including the Conferencia de Directores de los

Servicios Metereologicos e Hidrologicos Iberoamericanos (CIMHET20), an international conference

focusing efforts on institutional development, training of human resources and consolidation of operational management in meteorology, hydrology and climatology.

A pilot project supported the development of a web portal providing access to the hydrological observations in the Ibero-American countries, commonly published as Hydrological Yearbooks. In particular, it provides additional operational capability, for in situ water observations, as a national registry of water data services catalogued using the standards and procedures of the Open Geospatial Consortium and the World Meteorological Organization.

5.5.2 WMO Regional Association IV

The main goal of the WMO RAIV21 program is to improve monitoring, data accuracy, availability and dissemination of information in North America, Central America and the Caribbean (WMO Members of Regional Association IV). This region stretches from the far north of Canada all the way to the equator. Also, it includes the rich economies of the USA and Canada, as well as the developing countries in the Carribean, such as Haiti. The main issues from weather and climate impacts are hurricanes, drought and flooding. The agricultural sector as well as transport and tourism are greatly impacted by severe weather and climate events. In support of these activities a demonstration of standardized data sharing was developed with the aim of providing additional operational capability, in particular for in situ water observations, as a federated resource in the regional association.

A pilot project supported the development of a web portal providing access to the hydrological observations in North America, Central America and the Caribbean monitoring network composed of stations located in Antigua and Barbuda, Bahamas, Barbados, Belize, British Caribbean Territories, Canada, Colombia, Costa Rica, Cuba, Curacao and Sint Maarten, Dominica, Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Saint Lucia, Trinidad and Tobago, United States of America, Venezuela.

20

CIMHET. http://www.meteo-iberoamericana.com

21

WMO Members of Regional Association IV. http://www.wmo.int/pages/members/region4_en.html

16

(17)

Figure 11 – Hydrologic data exchange by standardized service types and data formats in the WMO RAIV

6. A federated approach to water data sharing using INT-HIS

The following sections outline an example open source technical solution for in situ water observations, called INT-HIS (INTernational-Hydrologic Information System), which can be used to support the federated sharing of water data by National Hydrological Services using WaterML 2.0. The applied technology enables standardized publication of data and uniform access to distributed data repositories, allowing a national portrait of water resources at different spatial and temporal scales. The architecture originates from the pioneering works of CUAHSI22 and ESSI-Lab23, together with the technical and scientific contribution of ARPA SIMC24.

The solution has already been deployed in Italy, and tested through pilots using data from China and the Dominican Republic, river basins (Sava), regions (Latin America) and WHYCOS Components (Arctic, Niger and SADC). INT-HIS can be used to create a global registry of water data and map services catalogued using the standards of the OGC. Such a registry would be open to all users and institutions from any country or level of government, and could be applied to any type of water information. It is proposed that further testing of INT-HIS will be undertaken in collaboration with

22

Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI). http://www.cuahsi.org/

23

Florence Division of the Institute of Atmospheric Pollution Research (IIA) of the National Research Council of Italy (CNR). http://www.essi-lab.eu/

24

Hydro-Meteo-Climate Service of ARPA Emilia Romagna http://www.arpa.emr.it/sim/

17

(18)

Theme 2: “Data Operations and Management” of the WMO Commission for Hydrology, which is active for the period 2013 to 2016.

INT-HIS aims to develop infrastructure and services to support improved data exchange within in the international hydrological community. Hydrologic information science involves the description of hydrologic environments in a consistent way, using standardised data models for information integration and software tools that are either customized or based on commercial products to directly interact with the deployed data storage and communication components.

INT-HIS supports storage of water observations data in an open source relational database, publication in WaterML 2.0 via OGC web services on the internet, federation with water observations data published by other water agencies, and search across the various data holdings available on the network. These advanced data access and analysis capabilities are provided through the use of OGC web services (such as the Sensor Observation Service [SOS]) and a standardised data format (WaterML 2.0) for data transfer.

6.1. The HIS architecture

When sharing information across scientiﬁc communities, it becomes important to deﬁne a standard framework through which large quantities of multidisciplinary information can be shared, discovered and accessed.

Utilizing a collection of OGC Web Services, INT-HIS is designed as a “services stack framework” that shares catalogue data, metadata and data with the user as shown in figure 6. The services stack framework identiﬁes three types of services as essential to sharing water information across the Web: catalogue services, metadata services and data services. These three services work together to completely index, describe and provide access to water information (e.g. timeseries observations). Catalogue services provide users with an index of hydrologic metadata, metadata services identify collections of timeseries available over a domain of space and time, and data services provide the user with the raw data for a speciﬁed temporal period and spatial area. This framework is designed to publish and distribute point-based timeseries observations, but it can be extended to include grids and coverages as well.

(19)

Figure 12 – INT-HIS services stack framework showing catalogue, metadata and WaterML 2.0 data access

While other interoperability studies have focused on implementing custom data streams (e.g. bridges, adaptors, etc.) between client and server interfaces, INT-HIS focuses on a common data and metadata management model that leverages a suite of standard OGC Web Services which can be applied to multiple scientiﬁc communities – in particular hydrologic and atmospheric sciences. Furthermore, INT-HIS can be integrated within existing data discovery frameworks (e.g. portals, gateways, etc.) by leveraging mediation and brokering services.

6.2. The HIS components

As with any other Service Oriented Architecture, INT-HIS is built around two fundamental components: (1) service providers and (2) service consumers. Although service consumers directly connect to service providers to request and receive data, a third component, a service registry, is introduced to facilitate the discovery of different service providers; this can be done using various keywords, metadata and ﬁlters. As service providers introduce their services within INT-HIS, services are registered at the service registry. Service consumers can then search the registry to ﬁnd available services of interest.

INT-HIS can be deﬁned as a collection of components which work together to store, index, access and distribute hydrologic information. The system contains servers, catalogues and applications, which communicate with one another through a set of web services. Web services are the set of protocols and speciﬁed interfaces that can be used to exchange hydrologic metadata and data (i.e. timeseries observations) through the web encoded using a common standardised language such as WaterML 2.0.

The individual components of INT-HIS each serve an important role in the data discovery and access process. Data providers are the principal locations for storing large volumes of hydrologic data,

(20)

speciﬁcally timeseries observations. Within the provider itself, data and metadata are managed in a database and then exposed through a suite of web services so that remote users can then access the data through the web. In the example in Figure 13, INT-HIS was interfaced to the MCH (WMO

Meteorological, Climatological and Hydrological) database to support data sharing using OGC web services and WaterML 2.0 in a Latin American pilot project.

Figure 13 - Example data access in Latin America using INT-HIS and the MCH database

Another component of INT-HIS is the hydrologic metadata catalogue. The catalogue is the component of INT-HIS which facilitates the discovery of hydrologic data that has been published by the data providers.

Within INT-HIS, data providers are the primary repositories for hydrologic data, while the registry is the primary repository for hydrologic data services. The registry provides an interface where users can search registered data services by specifying keywords and metadata, which describe the hydrologic data of interest. The registry is like an Internet search engine for discovering hydrologic timeseries information. Data publishers can register their data on the registry and provide brief descriptions of the datasets they want to share. This is an important aspect of INT-HIS because it allows for data to be organized and discovered in an efﬁcient, structured and methodical way.

(21)

Figure 14 - Example web service registry for Italy

The third and ﬁnal component of INT-HIS is the web or desktop client. The client is the component of INT-HIS that allows for the harvesting of hydrologic information at the locality of one’s own computer or analytical system. The client is a platform located on the user’s machine and communicates with both data services and the registry. Users can directly download hydrologic information from data providers if they already know of their existence or can search the registry for data that they might not know about. Once the data of interest has been discovered, users can download the information onto their local databases.

(22)

Figure 15 - Example desktop client searching and downloading data via web services

6.3. Standardized service types and data formats

There are three components to the services stack framework which work together to provide a system in which data consumers can readily discover and access both timeseries and coverage data using spatial, temporal and semantic ﬁlters: Catalogue Services, Metadata Services and Data Services. At the core of the services stack framework lie the metadata services which act as middleware between the catalogue services and data services. Data services ultimately provide the user with the data they are searching for, encoded in WaterML 2.0, whereas catalogue services allow users to perform federated searches across multiple data providers. Metadata services link both these layers together by being registered at the catalogue level and providing all the information needed to access information at the data level.

Data services are at the bottom of the services stack framework and are responsible for providing users with the data sought after. These services conform to the Sensor Observation Service (SOS) and Web Coverage Service (WCS) specifications of the OGC. For hydrologic information from in-situ monitoring sites, timeseries data is obtained through a SOS implementation encoded in WaterML 2.0. Similarly, for atmospheric science information, coverages can be obtained through a WCS implementation as a netCDF file. Each service allows users to specify spatial, temporal and semantic filters to obtain subsets of the larger datasets supplied by the data service.

(23)

Metadata is an essential component of the data sharing process. Not only does it facilitate the search and discovery of information within one’s own research community but also helps foster interoperability between research communities. Discovery and access metadata is used to describe a set of data that shares common characteristics to others who are not directly familiar with the information, and is typically recorded conformant with the ISO-19115 standard for Geographic Information – Metadata25.

Catalogue services aid in the management, discovery and distribution of metadata describing geographic datasets and services. Within the INT-HIS services stack framework, catalogue services function as the interface through which data consumers discover indexed metadata services published as WFS. As part of the standard suite of OGC services, Catalogue Services for the Web (CSW) are the services that focus on the management and indexing of geographic metadata. The OGC designed CSW to help data consumers search through a set of matching resources. CSW allows data publishers to register and index a set of data sets and services with a variety of different metadata proﬁles conformant to ISO19115. Using a CSW interface, data publishers are able to share their catalogue of metadata with search clients as well as other catalogues.

6.4. WIS integration

The WMO Information system (WIS) is the single coordinated global infrastructure responsible for the telecommunications, data management functions, metadata and data exchange between WMO Members. It is a key component of the WMO strategy for management and exchange of weather, climate and water information in the 21st century. WIS provides an integrated approach suitable for all WMO Programmes to meet the requirements for routine collection and automated dissemination of observed data and products, as well as data discovery, access and retrieval services for weather, climate, water and related data produced by centres and Member countries through all WMO Programmes.

WIS was designed to dramatically extend WMO Members' ability to collect and disseminate data and products. Owned and operated by Members, it is the core information system utilised by the WMO community, providing linkages for all WMO and supported programmes associated with weather, climate, water, and related natural disasters.

In the context of its integration in the WMO Information system, INT-HIS can provide metadata to WIS interfaces to enable users to make searches of HIS via WIS. A potential future phase of the INT-HIS - WIS integration is to provide an extension of WIS functions and services tailored to the needs of the hydrological community including databases, simulation models, web-GIS and other tools.

6.5. Operational demonstrations of INT-HIS

INT-HIS provides a low cost approach to undertaking data sharing using this open source suite of tools, developed by a large international hydrological community. The architecture is going to be applied in pilot demonstrations of National Hydrological Services, which make request of support to the WMO CHy, with the aim of assuring standardized hydrological data exchange. It is anticipated that this activity will be undertaken in collaboration with Theme 2: “Data Operations and Management” of the WMO Commission for Hydrology, which is active from 2012 to 2016.

25

ISO 19115-1:2014 Geographic information -- Metadata -- Part 1: Fundamentals

http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=53798

23

1. Data collection and dissemination in hydrology

WMO Data Operations and Management

Global initiatives in hydrological data sharing

Table of Contents