Air Quality Data Management and Integration System
Scoping Study
Report to Defra and the Devolved Administrations
Unrestricted ED46602
Issue 1.2 AEAT/ENV/R/3005 August 2010
Title Air Quality Data Management and Integration System
Customer Defra and the Devolved Administrations Customer reference RMP 5603
Confidentiality, copyright and reproduction
This report is the Copyright of Defra and the Devolved Administrations and has been prepared by AEA Technology plc under contract to Defra and the Devolved Administrations dated January 2010. The contents of this
may not be reproduced in whole or in part, nor passed to any organisation or person without the specific prior written permission of Defra and the Devolved Administrations. AEA Technology plc accepts no liability whatsoever to any third party for a
interpretation or use of the information contained in this report, or reliance on any views expressed therein.
File reference
Reference number ED46602
AEA group
The Gemini Building Fermi Avenue Harwell Didcot Oxfordshire OX11 0QR t: 0870 190 6602 f: 0870 190 6377
AEA is a business name of AEA Technology plc AEA is certificated to ISO9001 and ISO14001
Author Name
Approved by Name
Signature
Date
Air Quality Data Management and Integration System - Scoping Study
Defra and the Devolved Administrations RMP 5603
This report is the Copyright of Defra and the Devolved Administrations and has been prepared by AEA Technology plc under contract to Defra and the Devolved Administrations dated January 2010. The contents of this
may not be reproduced in whole or in part, nor passed to any organisation or person without the specific prior written permission of Defra and the Devolved Administrations. AEA Technology plc accepts no liability whatsoever to any third party for any loss or damage arising from any interpretation or use of the information contained in this report, or reliance on any views expressed therein.
ED46602- Issue 1.2
AEA group
The Gemini Building Fermi Avenue
Oxfordshire OX11 0QR t: 0870 190 6602 f: 0870 190 6377
AEA is a business name of AEA Technology plc AEA is certificated to ISO9001 and ISO14001
Andrew Monteith, Ollie Cronk, Rachel Yardley, Paul Willis, Xingyu Xiao
Paul Willis Signature
August 2010
Scoping Study
This report is the Copyright of Defra and the Devolved Administrations and has been prepared by AEA Technology plc under contract to Defra and the Devolved Administrations dated January 2010. The contents of this report may not be reproduced in whole or in part, nor passed to any organisation or person without the specific prior written permission of Defra and the Devolved Administrations. AEA Technology plc accepts no liability
ny loss or damage arising from any interpretation or use of the information contained in this report, or reliance
Andrew Monteith, Ollie Cronk, Rachel Yardley, Paul
AEA iii
Executive summary
Fewer than half of the UK’s air quality dataset users have access to all of the air quality monitoring, modelling and emissions data that they require, and the majority do not have access to the associated information which is needed to provide context and relationships between the causes and impacts of air pollution, for instance, Met data, traffic and land use statistics, population and health data.
To overcome this barrier and to maximise the overall availability and use of the data it would be possible for the UK to integrate these datasets. This report summarises the findings of a scoping study undertaken in 2010 to investigate the feasibility of such an integration process, and makes
recommendations on how this could be achieved.
Benefits of the proposed integration system
The proposed integrated system will increase data processing efficiency and reduce operating costs.
Air quality and other related data will be more easily available and accessible to members of the public and the air quality community. Statutory data reporting procedures will be simplified. The amount of time spent searching for, manipulating and interpreting data will be greatly reduced. It will be possible to develop useful tools to aid policy-makers, making use of a wider pool of data than has previously been possible. Examples of data visualisation and analysis tools currently used in Europe are shown is Section 1.3.
The integration of the UK’s air quality data will help to meet our regulatory obligations, by allowing and promoting the reuse of public data, simplifying reporting procedures and standardising data formats.
The proposed system architecture is compliant with the requirements of INSPIRE and the UK Location Programme. The key regulatory drivers are discussed in Appendix 1.
Data considered in the study
This scoping study focuses on the top priority data sets of the National Atmospheric Emissions Inventory, Pollution Climate Mapping, Automatic Urban and Rural Network, London Air Quality Network and non-automatic network data (hydrocarbons, heavy metals, black smoke), and examines how maximum efficiency and benefit can be gained through the integration of these data.
Also key to the success of developing or improving the efficiency of air quality tools is the availability and accessibility of essential non-air quality data sets, in particular Met data and traffic data. Although Met data is currently not freely available, there are several publicly available traffic datasets, which could be made available in a straightforward manner to an integrated air quality platform, through liaison with the data providers. Both met and traffic data are priorities for integration.
Stakeholder feedback
Many stakeholders identified the same shortfalls and listed the same type of datasets that are
essential to get the most value out of the UK’s air quality data assets. The primary difficulty is knowing what data are available in the first place and then being able to find them. Better search and indexing tools are essential to allow users to see and search the datasets.
81% would welcome a single air quality portal with improvements to the availability, format and integrity of these data. 75% would welcome basic or complex online tools to help analyse these data, however, in making decisions on air quality we would caution against the setting of an automated response, which undermines the technical excellence of air quality expertise in the UK.
During the development of the proposed data integration structure, further stakeholder engagement will be required during the early stages to ensure a high level of commitment and to ensure the end results is focused on user requirements.
The current situation
Three key problems have been identified with the current situation. These are:
The lack of structure in the architecture of the overall system of air quality and other datasets
The disparity of the datasets in terms of standard formats
Lack of or inconsistent metadata
AEA v Development of tools to make better use of existing data
Section 4 of this report describes the complexity of the current situation, the structure of the datasets, the currently available tools and their limitations. In an integrated system the data inputs to the current models and tools could be fully automated, and the resulting data, reports, plans and maps would automatically feed into Local Air Quality Management and other tools, saving processing time and reducing the element of human error.
One of the benefits of a fully integrated system for air quality data would be the easier application of online tools to view, sort and analyse the data to give them meaning and make them applicable to real life. Suggestions are given in Section 4.8 under the following categories:
Local Air Quality Management or national air quality compliance tools
Presentational tools
Practical tools
Action planning and impact analysis tools
Emissions scenario testing tools
Analysis tools
Proposed Approach
The proposed approach provides a platform for tool development to allow the air quality community and members of the public to make the best use of existing data. It also provides a future-proof solution for standardising and integration air quality datasets in a way that is compliant with European legislation and initiatives.
There are costs associated with:
The creation of a detailed service architecture, development of data standards and other over- reaching activities which will impact all datasets, in the range of £60K to £100K depending on scope
The creation of good metadata, which is vital to make the data useful, is likely to be a major cost, in the range of £120K to £200K. It is advised that this is undertaken before any modifications occur on individual datasets.
The changes made to individual datasets. Costs per dataset have been illustrated in Section 5.1 as between £25K and £75K per dataset in addition to the costs that have occurred due to creation of good metadata.
Implementation of the web services themselves will also be a significant cost. This is dependent on how far standard tools can be used to meet the specification, and the level of appropriate skills the data providers have to implement them.
Through the use of industry standard formats and definitions of metadata the proposed architecture will allow for the system to be modified to meet changing requirements rather than fixed systems designed around specific requirements from a set era. The proposed approach will have the following benefits:
More automation and less manual intervention. Less time will be spent searching for data and manipulation of data will be quicker due to standardised formats and new tools
Reduced operating costs
Robust platform for developing new products and services to add value to existing data
Simpler and more automated reporting lines
Ability to link or overlay data from different datasets
Better, informed decision-making, using a wide range of datasets
Quick Wins
It would be possible to start to add value to the data and make them more useable in the short term, without the bottom-up standardisation approach which has been outlined above. A number of quick wins have been identified by this scoping study and are described in Section 6.4, this is a different approach to the proposed approach and the costs as such reflect different work that will be
undertaken. In some occasions the work will overlap with the approach stated above thus some of the costs defined below could be incorporated into the costs defined above:
vi AEA
Using this scoping study as a starting point, catalogue all useful air quality and non-air quality datasets and metadata, including those, but not exclusively those owned by Defra and the Devolved Administrations. Create appropriate web pages to list and provide links to these datasets. We anticipate the cost for this to be in the region of £25k.
Integration of the continuous air quality data is a prime candidate for early integration. The data is relatively simple, the measurements are important and we have a lot of them. One option for this integration is to make use of RDFa – this is where RDF (a variant of XML) is added to existing HTML pages designed for human consumption (invisibly to human users) that adds support for use of the data systematically. We anticipate the cost for this to be in the region of £50k per dataset.
Some data providers (KCL and BADC) hold local Met data. This could also be integrated with relatively little effort but with a great benefit (access to Met data has been identified by data users as a high priority). In particular, KCL have about 20 met sites in London and the South East whose data could be ready for integration in the short term. We anticipate the cost for this to be less than
£100k.
The collection and integration of metadata is viewed as a very high priority. Making these data available and easily accessible online is the first step to provide more meaning to the data.
Standardising the format of this metadata and ensuring its completeness is important, but a more difficult and longer term task. We anticipate the cost for this to be in the region of £120k.
If DfT can provide transport data in a consistent format these could be use to plug into LAQM tools. Further investigation is required to determine how DfT data could be readily transformed and automatically applied to currently available tools, but this is not expected to be a difficult nor expensive task. We anticipate the cost for this to be in the region of £25k.
Identify reporting processes which can be readily simplified and automated and develop simple tools to do this.
viii AEA
AEA ix
Table of contents
1 Introduction 11
1.1 Drivers 12
1.2 Objectives 13
1.3 Existing air quality data toolsets 14
2 Methodology 17
2.1 Workshops and data gathering 17
2.2 Steering Group 18
2.3 Acknowledgements 19
3 Data to be integrated 20
3.1 Top priority data 20
3.2 Mid priority data 22
3.3 Low priority 22
3.4 Non air quality data 22
4 Current Situation 26
4.1 Overview 26
4.2 Overview Diagram 27
4.3 Assessment of Air Quality Datasets 28
4.4 Current Architecture for Tools 30
4.5 Local Air Quality Management Tools 31
4.6 Tools for Public Information and Research 34
4.7 National Air Quality Assessment Tools 37
4.8 Stakeholder Survey 39
5 Proposed Future Approach & Architecture 44
5.1 Changes Required 44
5.2 Underlying Principles 45
5.3 Overview of proposed future Architecture 48
5.4 Platform for future tool development 50
5.5 Challenges 51
5.6 Implications for stakeholders 51
6 Roadmap and Proposed Transition 53
6.1 Proposed Strategy / Implementation Tasks 53
6.2 Costs 55
6.3 Gantt Chart 55
6.4 Quick Wins 55
6.5 Alternative Strategy 56
x AEA Appendices
Appendix 1 Regulatory Drivers Appendix 2 Workshop 1 Minutes Appendix 3 Data User Survey Appendix 4 Data Provider Survey Appendix 5 Workshop 2 Minutes
Appendix 6 Dataset Summary and Scoring Criteria Appendix 7 INSPIRE, SEIS and data.gov.uk
AEA 11
1 Introduction
Defra and the Devolved Administrations own a large amount of historic and current data on air quality and other environmental factors, which are managed by a small number of contractors. Each contractor, and in some cases each dataset, has a different system to capture and manage the data, which makes data analysis, reporting and decision making difficult.
The majority of these data are produced using public funds, yet many are not readily accessible to the general public.
Fewer than half of the UK’s air quality dataset users have access to all of the air quality monitoring, modelling and emissions data that they require, and the majority do not have access to the associated information which is needed to provide context and relationships between the causes and impacts of air pollution, for instance, Met data, traffic and land use statistics, population and health data.
To overcome this barrier and to maximise the overall availability and use of the data it would be possible for the UK to integrate these datasets. This report summarises the findings of a scoping study undertaken in 2010 to investigate the feasibility of such an integration process, and makes recommendations on how this could be achieved.
The aims of the proposed data integration process are to:
Maximise the overall availability and use of the data to support stakeholder objectives
Standardise definitions and data formats
Standardise data updates
Catalogue datasets and disseminate information about the data that are available
Allow different datasets and different systems to be interrogated as one standard system
Ensure that data produced with public money are made available to the public.
The proposed integrated system will increase data processing efficiency and reduce operating costs.
Air quality and other related data will be more easily available and accessible to members of the public and the air quality community. Statutory data reporting procedures will be simplified. The amount of time spent searching for, manipulating and interpreting data will be greatly reduced. It will be possible to develop useful tools to aid policy-makers, making use of a wider pool of data than has previously been possible.
We propose that a new data approach and architecture that incorporates a spatial data infrastructure are developed to enable access to the UK air quality data assets that are held by several contractors.
This data infrastructure will allow the current disparate geographic information systems better align and integrate. The proposed infrastructure will fully comply with requirements laid out in the INSPIRE Directive1, thus making the UK air quality data compatible with all other spatial data in the UK.
This report describes the current situation in the UK, with many data frameworks, data flows and architectures. It investigates the issues associated with these disparate datasets and identifies common difficulties encountered by researchers, consultants and other groups who need to access and use the data. From this the report lays out a vision for the future, considering options for the integration of the UK’s air quality data and recommending a solution. We outline the key considerations and steps towards achieving this ultimate integrated system and discuss the possible options for tool development based on the integrated data.
Where possible the report indicates approximate costs and timescales for the delivery of the integrated system.
1 Directive 2007/2/EC of the European Parliament and of the Council
12 AEA
1.1 Drivers
There are many reasons why Defra and the Devolved Administrations are considering the integration of all UK air quality data. Broadly these can be sorted into two groups:
1. The potential benefits for all stakeholders, including Government, the air quality community and members of the public
2. The regulatory requirements to standardise datasets and maximise availability and reuse of publicly funded data
1.1.1 Benefits
The benefits of integrating the UK’s air quality data assets are many, some of which are given in Table 1.1.
Table 1.1 Potential benefits of data integration
Monetary
Increased efficiency of data providers and data users due to more automation and less manual intervention. Less time will be spent searching for data
Quicker manipulation of data due to standardised formats and new tools
Reduced operating costs for data providers, data users and Defra and the Devolved Administrations
Potential for developing new products and services to add value to existing data
Growth and opportunities in the market for tools and services
A more cost effective and efficient approach to reporting at an International level (e.g.
obligations under EC directives) using automated procedures Process and quality improvements
Increased collaboration between stakeholders
Modernised workflows and future-proofing our data management system, The proposed integrated system will be INSPIRE compliant and use the latest technology
Quicker and easier analysis, linking or overlaying data from different datasets so the correct conclusions can be drawn more easily from the data, and more factors can be considered
Better, informed decision-making, using a wide range of datasets Public
Provision of access to data and reduced restrictions
Improved visualisation tools could be developed from the proposed platform
Increased transparency of UK Government
More effective emergency response as a result of data being more readily available
Possible raised public awareness and understanding of air quality issues
Possible improvements in public health
AEA 13 1.1.2 Local Air Quality Management Review
Defra’s In House Policy Consultancy (IHPC) has recently carried out a review of Local Air Quality Management procedures2. The final report is expected to highlight the following:
The need to improve links to other policy areas including health, transport, land-use planning and climate change, which would become more feasible with the integration of air quality and non air quality datasets
The need to streamline the Review and Assessment process and reduce the labour costs involved
The need to integrate local and national information and create a larger national pool of monitoring data
A recommendation for Government to routinely publish a fuller statistical overview of ambient air quality trends to match the published trend information on emissions, combined with better information on the health impacts. With an integrated system and associated toolset the statistical analyses could be generated quickly, easily and automatically, as frequently as is required.
These recommendations would all be addressed by the implementation of the proposed data integration system.
1.1.3 Regulatory Requirements
The key regulatory drivers of this scoping study and subsequent implementation of an integrated system for air quality and other environmental data, are listed below and described in more detail in Appendix 1:
EU Directive on the Reuse of Public Sector Information
CAFE Directive
INSPIRE Directive
UK Location Strategy
Shared Environmental Information System
The integration of the UK’s air quality data will help to meet our regulatory obligations, by allowing and promoting the reuse of public data, simplifying reporting procedures and standardising data formats.
The proposed system architecture is compliant with the requirements of INSPIRE and the UK Location Programme.
1.2 Objectives
The key objective of this scoping study is to investigate and define what needs to be done to create a data infrastructure that ensures the accessibility and re-usability of Defra’s and the Devolved Administrations’ key air quality data investments.
It has been agreed with Defra and the Devolved Administrations that any data infrastructure should:
Be compliant with the initiatives of the Defra INSPIRE Team
Be generally “future proof” by coordinating with any other required future data harmonisation plans
Set out a pathway for other data sets to be incorporated into, such as those outside of Air Quality
Readily permit inter-comparison of initiatives and data generated at local, national and EU level
Permit analysis of the impact / potential of measures.
Specific objectives of this scoping study are to:
Examine data presentation issues, in respect of geographic presentation and online tools to analyse the results
Discuss how it might be possible to move towards a common Data infrastructure for all Defra contractors
2 Review of Local Air Quality Management: final report. January 2010, IHPC
Consider options for tools to allow presentation and integration of past and current data, and projections where available, and other optional reporting or dashboard type tools including action planning and cost effectiveness tools
Identify and map key dependencies between the data sets and possible desired outcomes
Outline a new comprehensive data model that is compliant with the Defra INSPIRE plans, and a path towards that model
Analyse the cost of implementation of the new data model and features, the ease of implementation of each measure, and risk of failure
Lay out options for possible next steps.
1.3 Existing air quality data toolsets
This section highlights some high profile tools that have been developed for air quality data around the world, thus serving to demonstrate what could be achieved if the UK datasets were integrated, with standardised formats and architecture.
Eye on Earth
The European Environment Agency’s Eye on Earth is a communication tool that presents real time scientific water and air quality data alongside the observations of members of the public on an interactive map using a Microsoft platform. The tool has been developed so that, over time, additional datasets may be added to turn the Eye on Earth into a ‘global observatory for environmental change’.
It incorporates data from across Europe, from multiple data providers.
Figure 1.1 Eye on Earth screenshot (http://eyeonearth.cloudapp.net/) 04/03/2010
Ozoneweb
The European Environment Agency’s Ozoneweb provides the public with easy access to information about ground level ozone pollution across Europe via an interactive map and graphical tools. The information is based on near real-time data provided by national and regional organisations, whose datasets are compatible and are able to be interrogated by Ozoneweb software.
Figure 1.2 Ozoneweb screenshot (http://www.eea.europa.eu/maps/ozone/welcome ) 04/03/2010
Saneringstool
The Dutch Ministry of Housing, Spatial Planning and Environment’s Clean Air Policy Tool maps NO2
and PM10 concentrations along the entire Dutch road network. The tool allows policy makers to investigate the impact at any site along the roads of implementing regional and location-specific mitigation measures. The toolset includes analysis of the current data and scenario testing for the next ten years. This has been achieved through a co-operation programme between central government and the provincial and municipal authorities.
Figure 1.3 Saneringstool screenshot (http://www.saneringstool.nl/saneringstool_ENG.html) 04/03/2010
INSPIRE Geoportal
The European Commission’s INSPIRE Geoportal allows users to search for spatial data sets and spatial data services, and subject to access restrictions, view and download spatial data sets from the EU Member States. The available spatial data comes from a wide range of sectors governed by INSPIRE.
Figure 1.4 INSPIRE Geoportal screenshot (http://www.inspire-geoportal.eu/), 05/03/2010
2 Methodology
2.1 Workshops and data gathering
The UK air quality data assets are produced and managed by several private contractors, who will play a key role in any future data integration exercise. Currently each contractor manages a number of datasets, and between contractors there is no requirement for a consistent approach. In order to engage with the contractors and to gather the necessary information to conduct this study, two stakeholder workshops and other stakeholder involvement events were held between January and March 2010.
Figure 2.1 Overview of scoping study project flow chart
2.1.1 Introductory Workshop January 2010
An introductory workshop was held at Defra, Ergon House, for the contractors managing high priority datasets. The purpose of the workshop was to:
Introduce the concept of the scoping study
Gain support and engage with the data providers
Ensure a common understanding of the aims of the scoping study and the case for data integration
Present and discuss the current datasets, data management practices and barriers to integration
Discuss common problems with availability and analysis of air quality data
Brainstorm potential user tools which may be developed for use with an integrated dataset.
Minutes of the January 2010 workshop are attached in Appendix 2.
18 AEA 2.1.2 Briefing with Defra and the Devolved Administrations
February 2010
A briefing was held at Defra on the 10th February. The purpose of the meeting was to:
Introduce the scoping study and progress to date to the Devolved Administrations and the Defra INSPIRE team
Gain feedback on the study objectives and direction from the Devolved Administrations
Ensure that the objectives of the study and the proposed data infrastructure align with the INSPIRE Directive requirements and its implementation within the UK
Get input and an update from the INSPIRE team on the progress of the INSPIRE implementation within the UK
2.1.3 Stakeholder Questionnaires February 2010
It is very important that the data infrastructure specification and design are driven by the individuals and groups who use the data. Two questionnaires were released to gather more detailed information on the air quality data management systems currently used in the UK by Defra’s and the Devolved Administrations’ contractors, and the specific issues encountered by and requirements of the UK air quality data users. The Data Providers questionnaire was distributed to contractors involved with the production and management of the UK’s high priority datasets, as defined in Section 4. The Data Users questionnaire was distributed to a selection of Local Authorities, air quality consultants and researchers. Both questionnaires are attached in Appendices 3 and 4.
2.1.4 Stakeholder Review of Draft Report March 2010
The draft recommendations of this study have been reviewed by all stakeholders who participated, in particular the following groups:
Defra and the Devolved Administrations
Data Integration Scoping Study Steering Committee High priority data providers
2.1.5 Final Workshop March 2010
The final workshop was held at Defra in March 2010 to present the findings of the scoping study to the key stakeholders, to outline the next steps and to gain ongoing understanding and support for any future integration project.
Minutes of the March 2010 workshop are attached in Appendix 5.
2.2 Steering Group
An Ad-hoc steering group was formed to guide the study and review the project outcomes. Three experts in air quality were invited and agreed to participate, alongside representatives from Defra’s Atmospheric and Local Environment team, and Defra’s INSPIRE team:
David Carslaw
David Carslaw is a Principal Scientist at King's College London Environmental Research Group. His research interests are mostly related to understanding how transport systems affect air quality. Part of this interest is related to the development of analysis techniques to help better understand these linkages, particularly through the Openair project. David has been a member of AQEG since 2002.
AEA 19 Sue Grimmond, Kings College London
Professor Sue Grimmond joined King’s College London in January 2006 after being Assistant, Associate and Full Professor at Indiana University, Bloomington USA. She completed her undergraduate degree (BSc Hons) at the University of Otago, New Zealand, and graduate degrees (MSc and PhD) at The University of British Columbia. Sue is on the editorial boards of Journal of Applied Meteorology and Climatology; Agricultural and Forest Meteorology and Advances in Meteorology. She is the 2009 recipient of the Helmut E Landsberg Award from the American Meteorological Society 'for numerous important contributions that have greatly advanced urban meteorology and urban climate sciences, and for sustained and effective leadership that has energized the urban climate research community’.
Bryan Lawrence, STFC
Bryan Lawrence is the Director of the Centre for Environmental Data Archival at STFC, where he runs the NCAS/British Atmospheric Data Centre (BADC) and the NERC Earth Observation Data Centre (NEODC). One of Bryan’s areas of expertise is in information architecture for environmental data.
2.3 Acknowledgements
We would like to acknowledge the contributions of the following organisations in the production of this scoping study report:
AEA
Air Quality Consultants Bureau Veritas
British Atmospheric Data Centre CEH
Defra
Department of Environment Northern Ireland Kings College London
National Physical Laboratory Welsh Assembly Government TRL
20 AEA
3 Data to be integrated
The priorities for data integration have been defined as follows:
Top priority: NAEI; PCM; AURN; LAQN; non-automatic network data (hydrocarbons, heavy metals, black smoke); Devolved Administration archive data
Mid priority: Local Authority action plans, review & assessment information, Local Air Pollution Control (LAPC) data, Parts A&B process data, Noise & DEM maps, deposition monitoring and modelling data
Low priority: Regional or city scenario data as modelled through other research contracts
Non-air quality data
This scoping study focuses on the top priority data sets, and examines how maximum efficiency and benefit can be gained through the integration of these data. Mid- and lower priority data have also been considered in the development of the proposed solution and appropriate recommendations are made to ensure that the proposed infrastructure does not preclude the integration of these important but lower priority data sets into the framework.
Although not considered by this scoping study, any future integration should also include datasets from diffusion tube networks, other non automatic monitoring networks and local atmospheric emissions inventories (including the London Atmospheric Emissions Inventory and the London Energy and Greenhouse Gas Inventory) in the mid-priority category due to their widespread use and the current difficulties highlighted by the users in accessing these data.
3.1 Top priority data
3.1.1 NAEI
The National Atmospheric Emissions Inventory (NAEI) is funded by Defra and the Devolved Administrations, and compiles estimates of emissions to the atmosphere from UK sources such as cars, heavy goods vehicles, power stations and industrial plant. The programme also provides key information such as emission factors, and estimates of industrial greenhouse gas emissions, which helps participants of the UK Emissions Trading Scheme.
3.1.2 PCM
Pollution Climate Mapping (PCM) datasets include maps of roadside concentrations and background concentrations of numerous air pollutants across the UK and exceedence statistics for Air Quality Strategy3 objectives and EU limit values/target values. These are generated by models using monitoring data, emissions inventory data, point source data and meteorological data as inputs.
3.1.3 AURN
The Automatic Urban and Rural Network (AURN) consists of about 130 air quality monitoring stations located throughout the UK. All the stations use continuous automatic monitoring equipment to record concentrations of NOx, SO2, CO, O3, PM2.5 and PM10. The data from the network are available on a hour-by-hour basis on www.airquality.co.uk and are provided by UK Government annually to the European Commission in compliance with EU Air Quality Directives. A significant amount of metadata for the AURN is publicly available at www.bv-aurnsiteinfo.co.uk.
3 The Air Quality Strategy for England, Scotland, Wales and Northern Ireland, July 2007, Defra and Devolved Administrations
AEA 21 3.1.4 LAQN
The London Air Quality Network (LAQN) is a group of air quality monitoring stations in London, Essex, Kent and Surrey. Each borough funds the monitoring within its own area, with the exception of eight sites in London which are funded by Defra and are affiliated into the AURN.
3.1.5 Non-automatic networks
Black smoke, hydrocarbons and heavy metals are measured at sites across the UK using non- automatic analysis methods which produce daily, weekly or fortnightly average concentrations.
3.1.6 Devolved Administration data
Concentrations of NOx, SO2, CO, O3, PM2.5 and PM10 are measured at automaticmonitoring sites in Scotland, Wales and Northern Ireland, using equipment similar to that in the AURN. Although the AURN does include sites in these countries the full Devolved Administrations datasets are much larger.
3.1.7 Summary of Top Priority datasets
Table 3.1 Top priority dataset managers, location and format summary
Dataset Managed by Location Format
NAEI AEA AEA servers including
www.naei.org.uk
MS Excel and MS Access
PCM AEA AEA servers CSV files and MS
Excel, ESRI ArcGIS AURN raw and
validated data BV
http://www.bv-aurnsiteinfo.co.uk
(and Uploaded to www.airquality.co.uk)
From a Proprietary from the Indic AirViro software package.
AURN ratified data AEA AEA server
www.airquality.co.uk
Proprietary software running on Open Source Technology including MySQL Database
LAQN KCL www.londonair.org.uk SQL Server 2005
Non-automatic networks NPL www.airquality.co.uk MS Excel spreadsheets DA monitoring data
Scotland AEA AEA server
www.scottishairquality.co.uk
Proprietary software running on Open Source Technology including MySQL Database
Wales AEA AEA server
www.welshairquality.co.uk As above
Northern Ireland AEA AEA server
www.airqualityni.co.uk As Above
22 AEA
3.2 Mid priority data
CEH Deposition monitoring and modelling data have been considered as part of this study. These will be covered by the INSPIRE requirements and should therefore be readily integrated into any proposed air quality solution. However, it will be important and mutually beneficial for CEH to remain fully involved in the development of any future integration system to avoid unnecessary duplication of effort.
Outcomes from the review and assessment process are currently being developed into more standardised report formats which are uploaded to an electronic on-line register of submissions. In the future this could be further developed in order to include a directory of the underlying data files used to support the review and assessment process in each local authority. These data files would need to be in a prescribed format compatible with the Defra integrated air quality infrastructure but there is no reason why this could not be done.
Emissions data from LAPC, Parts A& B processes and so on are likely to require further work to integrate where they are not already covered in sufficient detail by emissions inventories. These data are often not easily accessible in electronic form.
Noise & DEM maps already exist and will come under the requirements of the INSPIRE Directive.
They should therefore be able to be readily integrated with any air quality assessment infrastructure in the future should they be considered to be important. Much of the data that underpin the online noise maps at www. defra.gov.uk/environment/quality/noise/environment/mapping/index.htm. would also be necessary inputs to future air quality tools, for example, traffic and land use spatial data.
Other non-automatic monitoring network data are managed by many of the high priority data contractors, and a horizontal approach to integration would be recommended, to prepare all datasets managed by each contractor at the same time, thus maximising efficiency and maximising the availability of data.
Local emissions inventories such as the London Atmospheric Emissions Inventory (LAEI), London Energy and Greenhouse Gas Inventory (LEGGI), and other Local Authority inventories are important datasets which would provide very valuable inputs into current and future models and tools including the PCM, and as such we recommend that they are considered for integration. The proposed solution detailed later in this report has been designed to allow for additions of datasets in the future, if the integration of these data were not possible immediately.
3.3 Low priority
No consideration of low priority data sets has been made so far as part of this study. These regional modelling studies will be investigated as to their value as part of any follow-on work.
3.4 Non air quality data
Key to the success of developing or improving the efficiency of on-line air quality tools is the availability and accessibility of essential non-air quality data sets. This includes information held by government departments, agencies and other organisations who have not been involved in this pilot study. In order to make progress these organisations needed to be included early in the next phases of the study, and the costs and barriers for integration of these data sets identified.
Some issues with key data sets have already been identified through discussions with the data providers and users:
3.4.1 Meteorological Data
Meteorological data for analysis purposes are mainly provided by the Met Office
Weather Service. This is a Trading Fund within the Ministry of Defence, operating on a commercial basis under set targets. Requests for meteorological data from air quality specialists are therefore currently treated on a commercial basis and may incur significant expense.
The most representative meteorological measurements for a particular study may not necessarily be from a measurement station since there is not always one in the vicinity
therefore be offered instead.
Within these constraints the Met Office
sets which are currently available for use by air quality specialists in the UK.
Assuming that there are no plans to make the UK national meteorological data sets freely available and immediately accessible to everyone, other data sets may be considered for the purposes of this study:
• Local authority air quality monitoring stations are often configured to make their own real measurements of wind speed, wind direction, temperature & relative
masts. Whilst these data are obviously specific to the local situation and not rigorously quality assured by meteorological experts, it is possible that they could be made freely available to an integrated data solution.
• Data from meteorological modelling funded by Defra or others through other UK contracts example the UK Air Quality Forecasting or Tropospheric ozone modelling contract. Forecast data fields are currently produced on a daily basis which could be made available t
Again these data will not be fully quality assured by meteorological experts, but the Open source WRF model which is used for air quality forecasting is widely accepted for use within the air quality community and can provide output fi
Figure 3.1 Realtime WRF model used for air quality forecasting
Meteorological data for analysis purposes are mainly provided by the Met Office - the UK’s National rading Fund within the Ministry of Defence, operating on a commercial basis under set targets. Requests for meteorological data from air quality specialists are therefore currently treated on a commercial basis and may incur significant expense.
representative meteorological measurements for a particular study may not necessarily be from a measurement station since there is not always one in the vicinity – modelled results may
Within these constraints the Met Office does offer the most comprehensive and quality assured data sets which are currently available for use by air quality specialists in the UK.
Assuming that there are no plans to make the UK national meteorological data sets freely available accessible to everyone, other data sets may be considered for the purposes of this
Local authority air quality monitoring stations are often configured to make their own real measurements of wind speed, wind direction, temperature & relative humidity on 3 metre met masts. Whilst these data are obviously specific to the local situation and not rigorously quality assured by meteorological experts, it is possible that they could be made freely available to an
eteorological modelling funded by Defra or others through other UK contracts example the UK Air Quality Forecasting or Tropospheric ozone modelling contract. Forecast data fields are currently produced on a daily basis which could be made available to other researchers.
Again these data will not be fully quality assured by meteorological experts, but the Open source WRF model which is used for air quality forecasting is widely accepted for use within the air quality community and can provide output fields of the type illustrated below.
Realtime WRF model used for air quality forecasting
the UK’s National rading Fund within the Ministry of Defence, operating on a commercial basis under set targets. Requests for meteorological data from air quality specialists are therefore
representative meteorological measurements for a particular study may not necessarily be modelled results may
does offer the most comprehensive and quality assured data
Assuming that there are no plans to make the UK national meteorological data sets freely available accessible to everyone, other data sets may be considered for the purposes of this
Local authority air quality monitoring stations are often configured to make their own real-time humidity on 3 metre met masts. Whilst these data are obviously specific to the local situation and not rigorously quality assured by meteorological experts, it is possible that they could be made freely available to an eteorological modelling funded by Defra or others through other UK contracts – for example the UK Air Quality Forecasting or Tropospheric ozone modelling contract. Forecast data o other researchers.
Again these data will not be fully quality assured by meteorological experts, but the Open source WRF model which is used for air quality forecasting is widely accepted for use within the air
3.4.2 Traffic Data
This is often considered as difficult to get hold of, however our initial research indicates that information from the Highways Agency is in fact freely available as a real-time data feed or through historical data files:
The real time information required on Traffic Flows on the Highways Agency's network can be obtained on a Datex II feed. More information concerning the Travel Information Highway (TIH) can be found at http://www.tih.org.uk/index.php/Home. This data provides traffic volumes and vehicle classifications for all links across the network.
Historic Traffic Data is also freely available and access to this can be obtained at the following website http://trads2.co.uk/.
Annual Average Daily Traffic Flows are also available as downloadable files and through interactive maps on the Department for Transport’s website at http://www.dft.gov.uk/matrix/.
Kings College also referenced the ANPR (Automatic Number Plate Recognition System) as a traffic data source.
The data cover all major roads i.e. motorways and A roads and excludes minor roads. The roads are broken up into a series of links with each link comprising a stretch of major road between 2 consecutive junctions with other major roads. A link may also start/end at a local authority boundary or an urban/rural area boundary. A traffic count takes place on each link of the major road network and is used to estimate the annual average daily traffic flows. These data and the associated road network information can be viewed and downloaded from the web site for each year from 1999 to 2008.
Figure 3.2 Daily Traffic Data screenshot, Department for Transport
AEA 25 Since these are publicly available datasets there should be no reason why through liaison with the data providers they could not be made available in a straightforward manner to an integrated air quality platform.
3.4.3 Data.gov.uk
As a result of one of the recommendations of the Power of Information Review4 the UK Cabinet Office has established a project to make public Government data more widely available. This initiative aims to embrace the Internet and open data standards to make information more widely available for organisations and individuals who wish to build tools with Government data. This project is currently in its infancy; it is largely a searchable catalogue of a range of different data formats (CSV, Excel, XML).
Ultimately the project aims to make use of semantic web technology which will make it possible to link data together, but this will require changes to the data (for example adding additional mark-up to HTML pages to make them re-usable by systems) which are currently available so that it takes advantage of Linked Data (see Appendix 7 for explanation of Linked Data). Many of the aims of data.gov.uk dovetail nicely with SEIS and INSPIRE and it is likely (but not yet confirmed) that the data.gov.uk website will be one of the ways in which the UK complies with its INSPIRE obligations.
The data.gov.uk website and project is an important development and one that needs to be considered for any future plans for air quality data and other Government datasets.
All datasets with a spatial element, regardless of whether they are included in the data.gov.uk website, will be required to comply with INSPIRE, and therefore issues of compatibility are likely to be less of a barrier than issues of availability, which currently cause significant problems for data users, as discussed in Section 4.
3.4.4 Other datasets
Other datasets which are publicly available, owned by Defra, another Government Department or the Devolved Administrations should be considered for integration in the future. This could also include, but should not be limited to:
RESTATS, Renewable Energy STATisticS database for the UK.
The RESTATs database is currently being improved and it is planned that it will offer XML data feeds on Renewable schemes. This data could be very useful for air quality modelling – for example should certain types of renewables schemes be operating in an area (e.g. biomass) this could have an impact on air quality in that area.
ETSWAP, Emissions Trading System Workflow Automation Project
The ETSWAP project is a project planned by the Environment Agency to capture the data around EU ETS – initially for Aviation but potentially being expanded to static sources and Marine in the future.
Various Data Frameworks including English Housing Survey and upcoming National Housing Model
These data frameworks contain data on the UK housing stock and energy usage by homes separated out by region.
4 The Power of Information: An independent review by Ed Mayo and Tom Steinberg, June 2007
26 AEA
4 Current Situation
4.1 Overview
Air quality data have been captured, quality assured and checked, processed and used for modelling in various formats and for a range of purposes for over 30 years. Over that time systems have been improved and tweaked to fit changing requirements but in some cases the underlying data management technology has not evolved and moved with the times. This is partly due to the fact that the capture of a lot of these data has to be done in real time on a 24/7 basis, making major changes potentially risky and costly unless carefully planned and managed.
Three key problems are:
The lack of structure in the architecture of the overall system of air quality and other datasets
The disparity of the datasets in terms of standard formats
Lack of or inconsistent metadata Architecture
The current situation around air quality data is one typical of many systems that evolve over time without a clear pre-defined technical architecture. It is tactical in nature; meeting the specific needs of an individual project or contract rather than allowing for a more strategic view and efficient reuse of the data assets. Each of the UK’s data providers, and in some cases each dataset, has a different system to capture and manage the data. This in itself is not a problem (in fact the INSPIRE Directive states that “Data should be collected once and maintained at the level where this can be done most effectively”) but the approach to the data is not standardised and the architecture of the different systems is typically not compatible with each other.
Disparity of Datasets
As a result the datasets are currently disparate and the access to them is limited to those individuals and groups who know where to look and how they are structured. The UK has numerous separate databases and Geographic Information Systems (GIS) for air quality and other data, many of which are incompatible. It often requires manual data export and import processes or analysis processes to use the data and make them meaningful. Data integration and manipulation can therefore be very difficult, if not near impossible.
Geographical information systems are valuable tools for interpreting data with a spatial element (referring to a geographic location) such as environmental air quality data. Some of the systems used currently to manage air quality data are not full GIS systems, but they offer a broad geographical view onto the data. Others offer far more detailed resolution of their spatial properties (e.g. Pollution Climate Mapping (PCM data)).
Metadata
Metadata (data that describes data) is either very basic or is specific to the database or dataset in use on a project. This makes it difficult to quickly know if data are compatible or comparable. In addition this makes comparing measured data (for example the Automatic Urban and Rural Network (AURN), CEH deposition data), catalogued data (for example the National Atmospheric Emissions Inventory (NAEI), PCM) and projected data (NAEI, PCM projections) extremely difficult. There are therefore limits to the capability to cross-analyse or overlay data from different sources.
Examples of this include:
Inconsistent naming of data attributes and identifiers.
Inconsistent descriptors (e.g. definition of ‘roadside’ is different in different datasets)
Spatial and temporal information can be stored differently across datasets
The mixture of dataset formats, metadata and platforms means that building new tools could be very costly. There is also a risk that tools developed in the current situation may not offer a robust picture of the state of UK air quality and the cost of developing such tools is difficult to estimate as a whole.
4.2 Overview Diagram
The diagram below illustrates an overview of the current situation with regards to air quality datasets.
Note that it does not illustrate all datasets or data flows and focuses on the priority datasets as defined in Section 3.
Figure 4.1 National and Regional web sites
In summary it is easy to draw from this diagram that building a holistic view of air quality systems and the inter-relating variables and external factors is currently difficult to achieve.
Evaluating the current situation also highlights:
Data duplication and out of date copies of data are being used
There is a lack of consistency in how data is managed or exchanged
Elements of “supplier tie in” in places as recreating some of these systems could be difficult for a new supplier
4.3 Assessment of Air Quality Datasets
Following the Introductory Workshop in January 2010, the contractors for high priority datasets were asked to provide further detailed information on these datasets through the Data Providers Questionnaire (Appendix 4) and/or one to one interviews. Once collated, this information was subject to review as part of the scoping study and each dataset was evaluated on a range of attributes using a scoring matrix. The criteria for the scoring and the summary results are given in Appendix 6. A graphical summary is shown in Figure 4.2 for the datasets.
This evaluation has assessed datasets against a new set of criteria that previously the datasets have not had to comply with. Therefore lower scores do not mean that the datasets are not fit for their current purpose, only that they will require more transformation to integrate them.
The focus is on the structure of the datasets in view of integration, rather than the potential for developing tools for analysing and reporting these data..
Figure 4.2 Scoring matrix summary results
0 1 2 3 4 5
Data Searchability Data Timeline Data Downloadability Data Historical Trendings Data Comprehensivenss Data Accuracy Data Consistency Data Processing MetaData Data Format and Standards INSPIRE
Scoring
Criteria
A Graph to show the Scoring Matrix Breakdown
NAEI PCM AURN LAQN NPL CEH Key:
AEA 29 The results from the data providers’ survey identified that the main issue with the current situation is the lack of structure. There are a high number of different manual processes related with the datasets throughout the entire data cycle, from the data entry phase, to the quality acceptance/quality control phase right through to the reports produced based on the data. This means that multiple formats of data are sent to the data host, the contractors of the UK Air Quality Archive (www.airquality.co.uk), who often must manually manipulate the data to ensure that the correct data are entered into the system.
Most of the datasets are not tagged to indicate what they contain, or have a consistent standard by which the datasets are tagged, and this results in the data user being unable to find the dataset that they are searching for. It is therefore apparent that data users will have to spend time to understand what the dataset actually contains, and this conflicts with the INSPIRE principle: It should be easy to discover which spatial data are available, to evaluate if they are fit for purpose and to know what conditions apply for its use.
This lack of structure across the various datasets is shown not only by the data searchability scores but also from the data downloadability scores. For some of the datasets the data was only shown in report format, for instance in PDFs which causes a problem if any user wishes to use these data for further analysis. This contradicts the INSPIRE principle: It should be possible to combine spatial data from different sources.
As many of the datasets are stored in dissimilar formats, a user wishing to do individual analysis on the dataset would have to download it and manipulate it to conform with their standard format, for each of the datasets which they wish to incorporate into their analysis. This clashes with the INSPIRE principle: Data should be collected once only and shared between all levels of government and all stakeholders.
Another issue identified by the survey is that data collection methodology has changed over time, so despite some of the datasets scoring highly on the data historical trending, all of these data would have to be manipulated if any data structure principles were to be produced.
As the graph shows, most of the datasets scored highly in the criterion which just inspected the dataset individually; such as data accuracy and data consistency. However when considering how they would integrate with different datasets, especially those held by a different data provider, the scores were lower, for example for the data searchability, data processing and INSPIRE compliance.
A summary of results from the scoring matrix is shown in Table 4.1. The scoring criteria and full set of results can be found in Appendix 6.
Table 4.1 Dataset integration suitability scores
Datasets
Criteria NAEI PCM AURN LAQN NPL CEH
Data Searchability 2 1 1 1 1 2
Data Timeline 2 2 3 5 2 1
Data Downloadability 1 3 2 2 2 2
Data Historical Trendings 5 3 5 2 1 1
Data Comprehensiveness 4 4 3 3 2 2
Data Accuracy 3 3 3 3 2 3
Data Consistency 3 4 4 2 5 5
Data Processing 2 2 1 4 2 2
Metadata 3 1 2 3 3 2
Data Format and Standards 2 2 2 2 2 1
INSPIRE 1 1 1 1 1 1
Total 28 26 27 28 23 22
4.4 Current Architecture for Tools
The below diagram illustrates the current challenge associated with building tools that require multiple data sources. A similar picture to that illustrated in the current summary in Section 4.2, in that there is not a consistent or efficient approach to re-using data from different providers and systems. As a result many of the current tools are based on processes that involve manual data manipulation or other inefficient data management techniques.
The below diagram only shows three users of the various datasets, one for each of: Map Based Tools, Individual Reporting and Visualisation Tools. Despite only showing three users on the above diagram, it is already appearing complicated with users duplicating data by downloading and manipulating it for their own tools, for example AQ Dataset A1 is used for both Map Based Tool Creator and the Data Analyst this means that there are three versions of this dataset including the original AQ Dataset A1.
The tool creators will have to download the datasets that they wish to incorporate into their tools. So for every tool that is produced the tool creator will have to find the desired datasets, manipulate each of them into the same structured so they can be integrated together and then produce a tool to use these modified datasets to produce the desired output.
Figure 4.3 Current Tool Summary
A comparable diagram showing the future approach to building tools can be found in section 5.4.
AEA 31
4.5 Local Air Quality Management Tools
The tasks involved in Local Air Quality Management range from highly structured and prescriptive screening assessments through to increasingly complex and less prescriptive methods. The following table outlines these LAQM tasks and the methods and tools currently available and commonly used to carry out the tasks.
Table 4.2 LAQM Tools Summary
Task Tools and data
Updating and Screening Assessment (USA). This is low-level assessment of monitoring data and the likely local air quality impacts of a wide range of activities in a Local Authority. Assessments that show concentrations above prescribed trigger levels activate the need for a more detailed assessment.
Prescribed assessment methods are set out in the Technical Guidance5 and are increasingly shifting towards online reporting. Assessment still depends to some extent on local data defining emissions source activity levels and monitoring.
Detailed assessment. This includes the use of monitoring, meteorological, source activity and emissions data in conjunction with dispersion models and GIS to assess whether, where and when one or more of the air quality objectives may be exceeded.
Overall assessments are completed by Local Authorities or their consultants using a wide range of commercial or Open source models in conjunction with local and national datasets and prescribed assessment methods or guidance.
Typical local datasets (held locally):
• Monitoring data at one or more sites
• Transport data (AADT flows, share of HDV traffic, average speed – sometime much more detailed to include fleet profiles and stationary traffic times) usually from some form of transport model
• Commercially available meteorological data from a representative site
• Dispersion model (from a range available) input and output files
• GIS data typically illustrating concentration isobars and the location of sensitive receptors
Typical national datasets (hosted on AQ Archive LAQM and NAEI websites):
• Emissions factors for emissions sources
• Fleet profiles
• Background pollutant concentration maps
• Emission projections
5 Local Air Quality Management Technical Guidance LAQM.TG(09), February 2009, Defra
32 AEA
Task Tools and data
Further Assessment. This is the re-assessment of those locations identified as at risk of exceeding the objectives. Typically this is a re-casting of a detailed assessment to include additional information.
Typically Further Assessments provide significant additional information on source apportionment and level of reductions required and many include useful scenario testing of action plan proposals.
In some cases surveys or other data are
available that disaggregate traffic flow by vehicle type or Euro standard (held locally).
Air Quality Action Plan (AQAP). This is the development of a strategy and action plan that is proportionate and cost-effective for the air quality issue. The plan should set out the adopted measures and when they would be implemented, project the improvements in air quality that the adopted actions may deliver and hence project when the air quality objectives may be achieved in the local Air Quality Management Area.
Guidance is much less prescriptive and the development of tools and data much less developed than for Review and Assessment tasks above.
Typical local tools and data (held locally):
• Qualitative assessment of abatement actions Current best local practice:
• Emissions inventories and dispersion models disaggregated sufficiently to undertake detailed cost-effectiveness and assessments (i.e. similar to those for detailed assessment but with the addition of abatement scenario analyses)
National data and tools (available from Defra website):
• Policy practice documents and associated data
• Cost discounting tools
• Damage cost accounting tools and guidance
• AQAP website which collects text based examples of good practice reports, case studies and links to other helpful websites.
The most data-intensive processes are therefore the Detailed Assessment, Further Assessment and Action Planning phases, which require all the data sets (& more) identified for the national assessment, perhaps at a higher resolution and with a much more local focus.
At the centre of these processes and data flows are dispersion modelling and MACC tools as illustrated in Figure 4.4.
Figure 4.4 Example data flows for local air quality management tasks.
