8.1. Conclusions
The Technical Working Group on Data Collection have agreed that the Standard Sample Description, defining variables and terminologies, and this Data Exchange Guidance, defining transmission details, are the first steps to harmonising the transmission of sample level data from Member States to EFSA. The guidance documents have been developed specifically to address transmission of chemical occurrence and pesticides data and, to date, have only been piloted in the pesticides domain (2008 and 2009 Annual Data Collection). Feedback from this experience has been incorporated into the documents. The system will evolve as further experience in its use is gained.
The Working Group considers it essential to plan for and put in place a process for further maintenance and development of these guidance documents to allow for improving and extending the overall process as it relates to the currently defined data collections (chemical contaminants and pesticides) to EFSA data collections supporting other risk assessment activities, e.g. zoonoses.
The group recognises that the ability of each Member State to transmit data to EFSA according to the Standard Data Model and via the automated transmission mechanisms will vary. Therefore the guidance documents are also intended to assist Member States for planning future developments and evolution of local, regional and national systems with the objective of harmonising data transmissions.
The working group acknowledges that further development work is required to support the functionalities described by EFSA in the document.
Harmonisation of data collections is recognised as a fundamental step in the development of an effective EFSA Data Warehouse. The establishment of the EFSA Data Warehouse is seen as a resource for Europe-wide risk assessment by EFSA and - with appropriate access policies - for Member States.
8.2. Recommendations
The Technical Working Group on Data Collection, after examining the details of the data transmission file formats, mechanisms and security considerations, makes the following recommendations which are aimed to harmonising the format and mechanism of transmission of data to EFSA for the Chemical Contaminants and Pesticides Residues domains. Common requirements for these domains are addressed in the guidance. In addition, during the development of this harmonised data exchange guidance, attention has been paid to commonality with the requirements for the collection of Zoonoses data, although domain specific adjustments have not been addressed.
The following recommendations are addressed to the European Food Safety Authority as the leading organisation and it is anticipated that the work necessary to achieve these will be undertaken in conjunction with Member States and the Commission.
Recommendation 1: XML Schema validation at source
Error and warning codes are defined within the guidance to assist data senders in identifying and rectifying data quality issues in advance of transmission. Errors in transmitted data involves unnecessary processing, flagging of „replaced‟ data in the EFSA database and iterative communications which should be minimised, especially once data transmissions move from pilot/test into operational modes.
The Working Group recommends that data senders implement the XML schema validation including business rules at data extraction time within their own system prior to transmission to EFSA.
Recommendation 2: Amendments to the XML schemas
Pilot use of the SSD XML Schemas for transmission of Pesticides data (2008 and 2009) and the ongoing work of the Article 36 beneficiaries in CFP/EFSA/DATEX/2009/01 were based on an earlier version of the schemas. As this work is ongoing, implementation of the revised schema as defined in this guidance needs to take account of these data collection project requirements. Mechanisms for implementing the changes to the XML schemas (as proposed in this guidance) in existing systems will need to be addressed by EFSA and Member States.
In order not to impact the operability of existing systems, the Working Group recommends that EFSA should support the use of both schemas defined in the ongoing projects and the new schema defined in this document until the end of 2011. Further, it recommends that all projects commencing after the publication of this document should use the new schema.
Recommendation 3: Human readability of acknowledgement messages
Error and warning codes are defined within the guidance and will be transmitted to data senders to indicate the status of receipt and processing of their submitted data. For data senders whose systems can import and interpret these codes, this is an efficient and effective mechanism.
For those data senders whose system may still be evolving, the Working Group recommends that each EFSA data collection team should consider issuing a style sheet (XSLT) to transform XML message codes describing warnings and errors into human readable format in order to assist the data sender in rectifying the issues identified.
Recommendation 4: Amendments to the update procedure
The current pattern of data collections at EFSA typically covers a particular dataset for a time period (e.g. pesticides annual data collection) or a particular group of parameters for a defined time period (e.g. biogenic amines over the past 5 years). It is anticipated that these relatively infrequent data collections may become more frequent under new legislative arrangements in the future. Should that be the case, the update (or replace) function may become more critical as partial records in early transmissions need to be replaced with completed records. E.g. where one test result is available at the end of a month and another test result for the same sample becomes available early in the following month.
The Working Group recommends that should more frequent transmissions be attempted, revision of the update (or replace) functionality be considered.
Recommendation 5: Move terminology validation to the MRN
The current guidance specifies that the MRN message (synchronous) will provide the sender with information regarding the validity of the transmitted XML file as compared with the defined schema. This does not include validation of the data provided for each element versus the relevant controlled terminology; that validation occurs with the business rules and is communicated back to the sender in the ACK message asynchronously.
The Working Group recommends that validation of the received data against controlled terminologies synchronously for inclusion in the MRN message would be a beneficial development. Further, it states that a future mechanism integrated with the controlled terminology maintenance tool to export the terms in XML would be useful.
Recommendation 6: Feasibility study of standard internet protocols
There was some discussion within the group regarding the potential use of commercially available internet protocols such as AS1 or AS2. These would have the advantage of addressing the non- repudiation requirement which our recommendations do not address. However, in the interests of simplicity and taking a pragmatic approach, the anticipated benefits did not warrant pursuing these immediately.
The Working Group recommends that this possibility be reviewed as availability of appropriate protocols evolves over time.
Recommendation 7: Support for Member States in migration to SSD and XML
There were some discussions within the group regarding the complexity of exporting the data in XML format. From the experience of the Working Group members, especially those already participating in the pilot transmissions, it was noticed that the major amount of work is the matching of the standard terminologies included in the SSD rather than the actual export of the data in XML format.
The Working Group therefore recommends that Member States plan carefully the time needed to solve the data management issues of matching the standard terminologies in the SSD rather than concentrating on the XML export. EFSA should also provide guidance to Member States on how to perform this planning.
Recommendation 8: Usage and maintenance of the data exchange guidance
Data file formats, transmission mechanisms and security considerations are addressed in this guidance based on current best practice and widespread availability of the technology advocated. Contact points in the different Member States should be established to coordinate feedback on the implementation of the data exchange guidance. The existing networking groups may provide many of the appropriate contacts.
As the industry standards for electronic data exchange evolve over time, the Working Group recommends planning for a maintenance process with a review group of experts including EFSA, Member states and network representatives to ensure ongoing harmonisation of this guidance. A first review is recommended to be organised late in 2011 involving appropriate EFSA and Member State stakeholders.
Recommendation 9: Usage and maintenance of transmission message codes
Transmission message codes for warnings and errors currently identified in the guidance are expected to cover most of the data quality issues. As the use of the SSD and Data Exchange Guidance extends beyond the pilot users, feedback on the implementation is expected to result in refinements and additions to these message codes.
The Working Group recommends that, similar to the above process for the data exchange guidance, a review group of experts including EFSA, Member states and network representatives should ensure ongoing harmonisation of these message codes. A first review is recommended to be organised late in 2011 involving appropriate EFSA and Member State stakeholders.
Recommendation 10: Specific business rules
Since the implementation of business rules involves time and resources on the Member State as well as the EFSA side, the Working Group recommends that data collection specific business rules are kept to a minimum and that related data collection areas should use the same specific business rules list wherever possible.
Recommendation 11: Consider application of the guidance for local/national workflow
The data exchange guidance is written to address the need to standardise the transmission of data from Member States to EFSA. In addition, it is anticipated that it might be applicable, with some modification, also to transmission of food safety data, if needed, to the European Commission, Eurostat or other EU organisations.
The Working Group recommends adaptation of the system to also assist in strengthening the data flows from local or regional organisations to national reporting organisations, which then collate and transmit national data to EFSA. This function is outside the scope of this Working Group and is only noted here as a potentially useful tool for some Member States to pursue themselves.