For citation purposes: European Food Safety Authority; Guidance on Data Exchange EFSA Journal 2010; 8(11):1895 [50 pp.]. doi:10.2903/j.efsa.2010.1895. Available online: www.efsa.europa.eu
GUIDANCE OF EFSA
Guidance on Data Exchange
1European Food Safety Authority
2, 3European Food Safety Authority (EFSA), Parma, Italy
This guidance, published on 11 November 2010, replaces the earlier version published on 5 November 2010.
A
BSTRACTData collection is an important task of EFSA and a fundamental component of many of its risk assessment activities. With the assistance of the Technical Working Group on Data Collection, EFSA has developed two guidance documents to facilitate the exchange of data between Member States and EFSA. The Standard Sample Description for Food and Feed, published in January 2010, aimed at harmonising the data content of submissions. The current document is complementary to the first guideline in that it addresses the transmission mechanisms, file formats, security requirements and message exchange protocols (including validation messages) to be used. These two guidance documents are intended to provide the basis for a general model to harmonise the collection and transmission of a wide range of measurements in the area of food and feed safety assessment.
KEY WORDS
Standard, Format, Chemical, Occurrence, Transmission, XML, Data, Security, Protocol.
1 On request of EFSA, Question No EFSA-Q-2009-00614, issued on 28 October 2010. 2 Correspondence: [email protected]
3 Acknowledgement: EFSA wishes to thank the members of the Technical Working Group on Data Collection for the preparation of this EFSA scientific output: Jens Hinge Andersen, Eileen O'Dea, Luisa Oliviera, Jean Cédric Reninger, Renata Del Rosario, Stijn Saevels, Lars Wiehle, Josef Wolf and Roger Wood and EFSA‟s staff members Fabrizio Abbinante, Stefano Cappè, Marco Leoni, and Jane Richardson for the support provided to this EFSA scientific output.
S
UMMARYData collection is an important task of EFSA and a fundamental component of many of its risk assessment activities. EFSA receives an increasing volume of data from different providers (e.g. Member States, the European Commission, industry etc.) in support of its scientific activities.
The Technical Working Group on Data Collection (TWG-DC) was mandated by EFSA to develop a proposal to harmonise the collection of analytical measurement data for the presence of harmful or beneficial chemical substances in food and feed.
The TWG-DC was requested to produce two guidance documents:
A „Guidance on Standard Sample Description for Food and Feed’ published on 25 January 2010 covering the harmonised description of analytical measurement data for food and feed samples including a list of standardised data elements (items describing characteristics of samples or analytical results such as country of origin, product, analytical method, limit of detection, result, etc.), controlled terminologies and validation rules to enhance data quality.
This ‘Guidance on Data Exchange’ prescribing procedures to efficiently transmit and exchange data between Member States and EFSA including specific file formats for data transmission (e.g. XML, Microsoft Excel etc.) and specific data transmission protocols to support electronic data exchange.
The TWG-DC aimed to develop the Guidance document on Data Exchange to be sufficiently generalised to be applicable to a wide range of data collection processes undertaken by EFSA, even if these data collections require a different data structure from that described in the Standard Sample Description for Food and Feed covering chemical and pesticide occurrence data.
The TWG-DC agreed that the establishment of these guidance documents is only the first step in supporting the harmonisation of data transmissions between Member States and EFSA. Another necessary element for a successful implementation of an efficient European data collection system is the construction of a maintenance and development program to ensure that the guidance documents continue to evolve in line with the changing needs of the data analysis and risk assessment stakeholders.
Additionally the standardisation effort should be extended to areas not currently in the scope of the guidance on Standard Sample Description to simplify the reporting requirements from the Member States‟ perspective.
The group finally recognises that, at the present time, the ability of each Member State to transmit data to EFSA according to the guidance will vary. Therefore this guidance should also be taken into account by member states when planning future developments and evolution of local, regional and national systems with the objective of harmonising data transmissions.
T
ABLE OF CONTENTSEuropean Food Safety Authority (EFSA), Parma, Italy ... 1
This guidance, published on 11 November 2010, replaces the earlier version published on 5 November 2010 ... 1
Abstract ... 1
Summary ... 2
Table of contents ... 3
Background as provided by EFSA ... 4
Terms of reference as provided by EFSA ... 4
Consideration ... 6
1. Introduction ... 6
2. Transmission Methods ... 7
3. File Formats ... 7
4. Security Requirements ... 8
5. Automated Messages Exchange Protocol ... 9
5.1. Type of operations supported ... 9
5.2. Protocol for message exchange ... 10
5.3. FTP implementation ... 13
5.4. Web services implementation ... 14
6. XML messaging ... 17
6.1. Data message ... 17
6.2. MRN message ... 21
6.3. Acknowledgment Message ... 22
7. Error Codes and Messages ... 27
7.1. General business rules ... 28
7.2. Specific business rules ... 30
8. Conclusions and Recommendations ... 31
8.1. Conclusions ... 31
8.2. Recommendations ... 31
9. References ... 35
Appendices ... 36
Appendix 1 – Description of the MRN and ACK as currently implemented in EFSA ... 36
A1.1 MRN message ... 36
A1.2 Acknowledgement message ... 37
A1.3 Transaction Identifier Synchronisation ... 40
Appendix 2 – Description of the business rules applied to the elements of the standard sample description ... 41
B
ACKGROUND AS PROVIDED BYEFSA
Data collection is an important task of EFSA and a fundamental component of risk assessment (Articles 22 and 23 of Regulation EC No 178/20024). EFSA receives from different providers an increasing volume of data in support of its scientific activities. Data may be submitted to EFSA by a variety of providers: national competent authorities, local authorities, laboratories, universities and others.
The increasing volume of data transmitted to EFSA involves a number of challenges:
Efficient use of human resources in data collection processes, both on the side of the data senders and on the EFSA side where data have to be collected, collated and analysed.
Quality of transmitted data, which may differ in origin, language, description and codification. A standard data quality facilitates the task of collating and preparing data for analysis.
Capacity of managing high volumes of data; many data collections contain a huge quantity of analytical results, such as data from control and monitoring programs and targeted surveys. The quantity of these data makes their manual processing unfeasible and introduces the need for automated methods of formatting, transmitting, processing and analysing the data.
Capacity to analyse the data and to produce valuable reports for the different stakeholders; the increased volume of data requires more advanced techniques for storing and analysing the data. Data should be, as far as possible, readily available for standardised analysis functions to harmonise processing among the different food data stakeholders.
EFSA requests that the Data Collection and Exposure Unit, through a self-tasking mandate, harmonise the collection of analytical measurement data for the presence of harmful or beneficial chemical substances in food and feed.
The Data Collection and Exposure Unit (DATEX) in coordination and cooperation with the Pesticide Risk Assessment Peer Review Unit (PRaPeR) will establish, for this purpose, a working group of technical experts (Technical Working Group on Data Collection).
The working group should report to the Chemical Occurrence Expert Group and to the Networking Group on Pesticide Residues. These networking groups will review and approve all deliverables of the Working Group.
T
ERMS OF REFERENCE AS PROVIDED BYEFSA
The working group should focus on both chemical occurrence and pesticide residue data, due to the close similarities between these areas. The working group should build on, expand further and finalise the effort of harmonising the analytical measurement data descriptions already started by the EFSA internal working group on controlled terminology and define the data exchange model format to be used.
The working group is requested to produce guidance documents on:
4
Regulation (EC) No 178/2002 of the European Parliament and of the Council of 28 January 2002 laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety.OJ L 31, 1.2.2002, p. 1–24.
the harmonised description of data on analytical measurements in food and feed samples (Guidance on Standard Sample Description for Food and Feed);
The Standard Sample Description contains a list of data elements that are standardised and can be conveniently used by both data senders and data recipients to fully describe samples and analytical parameters for evaluation purposes. This standard provides:
– name and structure of the defined data elements to be used;
– controlled terminologies, where needed, with exclusion of the food dictionary which will be addressed by a specific working group;
– validation rules to assess the validity of the information supplied, to ensure an adequate level of data quality in data export, transmission and storage.
The proposal for standardised sample description is based on the initiative developed by the EFSA internal Working Group on Controlled Terminology. The group elaborated a draft standard for the description of chemical contaminants and pesticide residues data. This document was presented at the IT Expert Meeting on 17-18 February 2009 and was also the basis for a pilot project on pesticide residues data transmission and for an EFSA grant on chemical contaminant data.
the methods to efficiently transmit and exchange data between Member States and EFSA (Guidance on Data Exchange);
The working group should propose a “Guidance on Data Exchange” to define methods for the transmission and the exchange of data which would be suitable for the data collected with the standard sample description for food and feed. The guidance should take into account the software tools already set up by the EFSA IT Unit and provide inputs for their enhancement. The guidance should promote the use of semi-automated or automated methods for the data transmission to lower the costs of manual human intervention.
The guidance documents should be presented for approval to the EFSA Chemical Occurrence Expert Group and to the EFSA Networking Group on Pesticide Residues.
C
ONSIDERATION1. Introduction
The Guidance on Data Exchange takes the logical structure defined in the Guidance on Standard Sample Description for Food and Feed (hereinafter referred to as SSD) and specifies:
the file formats for the transmission of data (e.g. Microsoft Excel, Comma Separated Values - CSV, Extensible Markup Language - XML5, etc.);
the protocol for exchanging data between the sender and the receiver; the internet protocol supported to transmit the data electronically; the file formats for the validation rules;
additional technical information and specifications to make the electronic transmission possible (error codes, security requirements, etc.)
Specifically, the Guidance on Data Exchange will describe the transmission of data between Member States and EFSA, but the methods described can be applied for transmissions between any data senders and any data receivers.
The data transmission system needs to meet the following requirements:
1) Simplicity: the system for both the sender and the receiver should be simple to ensure easy implementation. Simplicity should prevail, where possible, over other requirements such as memory occupation, disk occupation, bandwidth occupation;
2) Ensure secure transmissions; 3) Require minimum maintenance;
4) Flexibility: the system needs to be sufficiently flexible to ensure it can keep pace with scientific developments in the field of food and feed risk assessment. Changes should be predominantly managed through routine amendments to the controlled terminology dictionaries in consultation with relevant stakeholders. For more significant changes to the SSD elements, consultation with stakeholders and impact analysis would need to be made prior to implementation;
5) Involve minimal human resources: as far a possible the data collection process should be fully automated and ensure the most efficient use of human resources;
6) Cost effective: the implementation and maintenance costs for the sender organisation should be kept to the absolute minimum.
5 W3C Consortium - Extensible Markup Language (XML) 1.0 (fifth edition) available on line at http://www.w3.org/TR/REC-xml/ last accessed on 14/07/2010
2. Transmission Methods
The SSD defines the content for the transmission of data between data senders and data
receivers. This can be a manual process requiring user interactions or an automated process
using procedures developed within information systems.
In summary:
Manual transmission methods:
o
Manual posting of files on the
EFSA Data Collection Framework (
DCF
) Web
Application
;
Automatic transmission methods:
o
Automatic transmission via Secure File Transfer Protocol (FTPS
6);
o
Automatic transmission via SOAP
7(originally Simple Object Access
Protocol). This protocol relies on the eXtensible Markup Language (XML) as
its message format and other Application Layer protocols (mainly Remote
Procedure Call (RPC
8) and Hypertext Transfer Protocol (HTTPS
9,10) for
message negotiation and transmission).
Automation of transmissions is considered to be a practical way in which large numbers of files and volumes of data can effectively be transmitted without incurring significant human resource demands both for the data sender and data receiver. The Working Group recognises that full automation of the data transmission may not be feasible for all Member States in the short term; however, the publication of this guidance is intended to assist Member States in developing their systems to allow automation in the future. The Working Group acknowledges in addition that for annual monitoring programs, manual transmission is often sufficient. It is anticipated that, for large data collections, batching the data into multiple transmissions (e.g. maximum of 300MB each) may be a pragmatic way to minimise any difficulties in transmitting very large files.
3. File Formats
The WG recognises that the transmission format for the Standard Sample Description is XML. The automatic transmission with Web Services and FTPS will be supported in XML only.
Tools such as XSLT to format the XML files (data and error messages) into human readable format such as HTML will be provided by EFSA.
The main benefits of the XML file structure are:
6
IETF – RFC 4217 - Securing FTP with TLS available on line at “http://tools.ietf.org/html/rfc4217” based on FTP (RFC 959) and TSL (RFC 2246) last accessed on 17/03/2010.
7
World Wide Web Consortium - SOAP Version 1.2 Part 1: messaging Framework (Second edition) available on line at “http://www.w3.org/TR/2007/REC-soap12-part1-20070427/” last accessed on 17/03/2010.
8
IETF – RFC 707 A High-Level Framework for Network-Based Resource Sharing available on line at http://tools.ietf.org/html/rfc707 last accessed on 17/03/2010.
9
IETF – RCF 2068 - Hypertext Transfer Protocol -- HTTP/1.1 available on line http://tools.ietf.org/html/rfc2068 last accessed on 17/03/2010.
10
it is an open standard and based on text files, therefore files can be opened without the need of specific tools;
it is easy for machine interaction: information is structured, therefore it is easy for a computer program to read;
it is currently broadly supported: there are parsers and tools in several programming languages and technologies;
it supports UNICODE encoding therefore is natively appropriate to support a multi-language user community such as the EU;
it is easy to implement compared to other transmission standards and is platform independent;
each data element is assigned a specific data type through the XML schema, resulting in improved data quality;
basic file validation, which can be performed automatically on the sender side using an XML schema (XSD), prior to transmission to the receiver. This can reduce the number of errors and re-transmissions.
Since some senders may not be ready to implement XML transmissions, the WG recommends that EFSA supports manual transmission of other formats (e.g. CSV and Microsoft Excel) using the Data Collection Framework for a limited, defined period. However the WG recognises the following limitations of these transmission formats:
CSV: the presence of special characters in the data values (e.g. “double quote character”) can result in uploading failures. In addition, the inclusion of delimiter characters within the data values can result in import errors which can seriously compromise the quality of the submitted results;
Excel file: the wrong data type assignment (e.g. number formatted as text) can result in data values being converted so missing values during the upload process. Frequently the import tool provides no warning message and the missing values are only detected later in the analysis process.
In contrast with XML, for CSV and Excel formats, no automatic validation can be performed by the sender prior to transmission resulting in either an increase in the number of cycles of resubmission of data or potentially onerous manual validation by the sender or receiver or both.
The Working Group recommends also batching large transmission into files of approximately 300Mbyte or less (uncompressed) for efficient transmission and data management. In addition, it is recommended that the files are transmitted compressed using zip compression (with programs compliant with MIME type: application/zip or application/x-zip-compressed) in order to reduce the transmission time.
4. Security Requirements
The main security requirements connected to data transmission are identified below. It should be noted that whilst the first two requirements are critical, the third is currently seen by the working group as desirable.
Identification of the sender: the receiver can identify the sender performing the data transmissions.
Confidentiality of the transmission: the information involved in the data transmission is accessible only to the sender and to the receiver, even though a public network might have been used to deliver the message.
Non-repudiation: the receiver has proof of participation by the sender in the data transmission that cannot be denied.
There are Internet standards such as the AS1 and AS2 originally defined for Electronic Data Interchange (EDI), that can equally be used to transmit XML files and that address the three requirements. However the possible use of these standards would require difficult “in-house” development or the purchase of gateway software. It would also involve the implementation and the maintenance of client and server certificates to be used to digitally sign and encrypt the data transmissions.
In order to reduce the complexity of the implementation and the financial impact of specific software or services for the data senders, a simplification of the security requirements is suggested by the Working Group.
For the automatic transmission methods, the identification of the sender organisation is performed assigning a user identification and password to be used by the software performing the operation. The confidentiality is implemented using a standard Internet application protocol (FTP, HTTP, etc.) through SSL or TSL. The usage of these protocols implies the installation and the maintenance of server certificates on data receiver side.
For the manual transmission methods, the identification of the sender is performed with user identification (USERID) and password. Specific requirements for the length, format and duration of the password are defined. The confidentiality is implemented using the standard HTTPS internet protocol, which requires the installation and the maintenance of a server certificate on data receiver side (e.g. EFSA), but it does not require client certificates for the users.
The implementation of the non-repudiation requirement for both automatic and manual transmission methods is not seen as critical at this point, since the user community consists of competent authorities of Member States and EFSA. The implementation of this requirement would imply the usage and maintenance of digital certificates and the implementation of digital signatures by data senders which is currently seen as a significant work burden at the current stage of development of the data exchange through the Standard Sample Description.
The working group recommends that the security requirements are re-assessed periodically with the growth of the user community. In case the Member States and EFSA would envisage the need to fully support all the security requirements for data transmission, the technical working group recommends the usage of some gateway software and the implementation of existing Internet standards, which involve the usage of certificates for electronic signature and encryption of the electronic transmissions.
5. Automated Messages Exchange Protocol
5.1. Type of operations supported
The Message Exchange Protocol defines a “file transmission” as the unit of data to exchange between the sender and receiver. The file transmission is identified by the transmissionID (dataTrx:receiverTrxCode). This identifier allows the file transmission to be retrieved in the DCF and
should be included in all messages. Since the unit of data exchange is the file transmission, when it is necessary to change a single sample or result, all the samples and results belonging to the same file transmission should be retransmitted. This process, which at first implies bigger file sizes, allows much easier database management processes both at data sender and receiver side.
The protocol defines a message framework to perform the following operations: Insert: A new data transmission file is inserted in the data receiver‟s database.
Delete: A data transmission file included in the receiver‟s database is nullified. The data transmission file should never be deleted by the receiver‟s database, only flagged as non-current. Archiving procedures of nullified files should be set up on receiver‟s side.
Update (“Replace”): A new data transmission file, provided by the sender, replaces entirely a transmission file previously loaded in the receiver‟s database. The update operation is a more complex combination of the delete followed by the insert procedure11. The update operation uses a mechanism for tracking the IDs of updated transmissions, so that a link is kept with the old transmission. In addition the update operation should be performed in one transaction, so that in case of system errors at the end of transaction either the new transmission is loaded and the old is deleted or the old is active and the new transmission returned an error. During an update operation there should never be a case where two or more linked transmissions are current or flagged as deleted.
The message initiating the data transmission will trigger the different operations using the “dataTrx:opType” attribute. The “dataTrx:receiverTrxCode” must also be present for delete and update procedures.
5.2. Protocol for message exchange
The Message Exchange Protocol describes the exchange of “messages” between Member States and EFSA: the data message, the MRN message and the acknowledgment message.
Since the term “message” can be used in different contexts in information systems, it is fundamental to recognise that the term “message” refers to the EFSA XML message layer (i.e. the data message, the MRN message and the acknowledgment message) which is shown in Figure 1. This layer interfaces with EFSA‟s and Member States‟ applications for sending and receiving data. This layer is independent and transparent from the specific transport layer protocol defined for this protocol (e.g. SOAP and FTP), therefore the term “message” should not be confused with FTP messages or SOAP messages.
Figure 1: Protocol layers. 11
Currently EFSA Data Collection Framework does not implement an update procedure, therefore the tracking requirement between the different transmissions is not supported. Updates should therefore be downgraded to a delete plus an insert procedure.
Transport Layer: Transport protocol (SOAP, FTP)
EFSA XML message layer: Data message, MRN message, ACK message
Application layer: Chemical Occurrence Data/ Pesticide residues data
The main participants in the electronic transmission protocol are represented in Figure 2. The “Sender” represents the reporting organisation (e.g. a competent authority of a Member State). The “Receiver” represents the organisation receiving the data (e.g. EFSA).
Sender (Member State)
Receiver (EFSA) sendResumeData
Send Data Message Send MRN send Acknowledgment «extends» «extends» «extends» Protocol Validate XML Business rules validation * * * * * * Validate data message Validate MRN Validate ACK «extends» «extends» «extends»
*
* *
*
Figure 2: Participants in the protocol
The basic steps of the protocol are the following:
1. The “Data Sender” creates, validates against the agreed schema and transmits an XML file with data to the “Data Receiver”. Data will be wrapped in an XML message envelope designed to support the protocol which will be defined in the paragraph 6.1.
2. The “Data Receiver” will send back a Message Receipt Notification (MRN) to confirm the transmission. At this stage the “Data Receiver” verifies whether the XML message is a “well-formed” XML file and also whether it is valid against the provided standard XML schema. 3. The “Data Receiver” will validate the XML message against a standard set of validation rules
(e.g. validation rules published in the guidance on Standard Sample Description p. 39). On the basis of the result of the validation, an acknowledgment message is prepared12.
4. The “Data Receiver” will then transmit the acknowledgment message back to the “Data Sender”, reporting on the positive/negative result of the data transmission. 5. The “Data Sender” will confirm the receipt of the acknowledgment message through a
message receipt notification MRN. The sender should validate the acknowledgment before transmitting back the MRN since a positive MRN confirms also to the receiver that the acknowledgment was well-formed and valid XML. No business rules are applied prior to the
12 Currently the DCF only validate the XML messages against the defined controlled terminology, all the other validations specified in the standard sample description are implemented manually by the data receiver after the receipt of the message.
acknowledgment message. The MRN is sent back asynchronously calling the “Data Sender” Web Service. The sender will have to check for a well-formed XML.
In Figure 3 the term “XML validation” refers to both checking for a well-formed XML file and a valid XML file against the specified schema since the XML file cannot be valid without being well-formed first.
Figure 3: Message exchange protocol.
This definition of the message exchange protocol is only logical and independent from the specific application protocol used for the physical delivery of the message itself. These can be:
the File Transfer protocol (FTP); Web Services (WS).
There are cases where the MRN may be not generated by the system:
Network issues: the data transmission could be lost and the error will be indicated with a proper exception (it could be an HTTP error code for WS or a proper FTP error).
Sender (Member State) Receiver (EFSA)
Data message MRN XML validation Business rules validation Acknowledgment XML validation MRN
Security issues: In case of authentication failure the error will be indicated by the transmission protocol with the proper exception (an HTTP error code for WS or a proper FTP error in case of FTP usage).
XML validation: In case of successful transmission of an invalid XML document within the transmission file there is no guarantee about the content and delivery of the MRN. Therefore it is always recommended for the “Data Sender” to validate the XML files against the provided schemas prior to transmission to avoid this problem.
In these cases the sender should contact the receiver to find out the result of the transmission.
In transmissions performed using mechanisms which allow for some synchronous transmission (e.g. in case of Web Services), a synchronous transmission confirmation will be implemented in order to reduce the possibility of MRNs not being generated by the receiver system. This case will be further explained in the web service section.
5.3. FTP implementation
The sender opens an FTP connection on the receiver‟s FTP server. The Sender authenticates itself using the credentials provided and sends the file.
Each Sender organisation will have its own FTP folder on the EFSA system where their files will be saved. Accounts will be created and distributed to each Data Sender by the EFSA Data Manager. In order to signal the completion of FTP file sending, the sender shall put a file with the same name of the transmitted data file, plus the suffix “.go”.
For example:
data file = dataProvider1.xml trigger file = dataProvider1.xml.go
The content of the trigger file is irrelevant therefore does not need to contain data. It is necessary to put a go file for each transmitted XML file.
No further processing will start until the “.go” has been placed in the target directory of the FTP site.
The DCF checks the input file to verify that it is valid according to the defined XML schema. The system will send back asynchronously an MRN via e-mail using the information defined for the logged-in organisation.
The MRN contains detailed information about the uploaded data file and the result of the XML validation. MRN also contains the transaction identifier (contained in the “receiverTrxCode” of the MRN and of the acknowledgment) that can be used on the DCF web interface to check the complete validation of the message.
The Sender will be able to find the file under the “Data transmissions” table, accessible through the DCF web interface. The file will be removed from the FTP folder once uploaded by the DCF.
The DCF will execute further validations applying business rules on the original file asynchronously13. At the end of the process an acknowledgment will be sent to the “Sender” containing the validation result via e-mail. This FTP implementation contains some differences in respect to the Web service implementation proposed in the paragraph below. These differences are due to the fact that the FTP protocol is totally asynchronous while the WS protocol allows synchronous communication.
5.4. Web services implementation
Both the data senders and the data receivers will use WS technology to allow back and forward14 interaction (for MRN and acknowledgment exchange) during each transmission. The interface (WSDL) of the WS installed on the data sender‟s side is defined by EFSA and this interface is the same for all organisations involved in the electronic transmission. For simplicity also the interface installed on EFSA‟s side follows the same WSDL. The WSDL are therefore all the same with the exclusion of the URL address of the web service located in the element soap:address attribute location. This element must be modified manually by the sender in the WSDL obtained by the EFSA system.
A Web Service interface contract (WSDL) is available and can be downloaded from https://dcf-elect.efsa.europa.eu/elect?wsdl.
Each organisation has the responsibility to implement and maintain the WS of their organisation as reported in Figure 4.
The WSDL file includes three methods:
sendFileName: This method is for internal EFSA use. Senders are not requested to implement this method
ping: This method is for checking if the web service is up and running without the need of performing an actual data transmission. The synchronous response of this method is a text string reporting “TRXOK”.
sendResumeData: This method is the only method available for transmitting all the XML messages (data messages, MRN messages and acknowledgment messages). The synchronous response of this method is a text string reporting “TRXOK” if the receipt of the XML file was correct otherwise “TRXKO” if the transmission of the XML was not correct.
Therefore the Web Service transmission protocol proposed in 4 should be interpreted as described below.
13
Currently the DCF only validate the XML messages against the defined controlled terminology, all the other validations specified in the standard sample description are implemented manually by the EFSA data managers after the receipt of the message.
14 The transmission of each data message, MRN and ACK is a actually a one-way communication although in the WSDL the methods have been defined as two-ways communications: (this to overcome some library limitations)
Figure 4: Web service implementation.
1. The Sender (e.g. Member State) sends the data message to the Receiver side (e.g. EFSA) using a WS installed on the Receiver side, calling the method sendResumeData. The Data Receiver WS is protected and requires authentication: each Sender will be provided with its own credentials to be used to access the WS. The authentication will be based on the generic HTTP-basic mechanism but this may change to strengthen security. The receiver will verify that the essential information is present in the data message and contains valid information:
a. senderCode must exist and be registered in DCF and for the data collection described in the dcCode element;
b. A valid reference to an XML schema is reported in the document. c. receiverCode must be “EFSA”;
d. type must be a supported document type (e.g. “dcfmsg”);
e. version contains a supported version of the protocol (e.g. “1.0”); f. senderTrxCode contains an identifier
XML validation Business rules validation XML validation Minimum info in the header
Sender (Member State) Receiver (EFSA)
1.sendResumeData(Data message) 2.sendResumeData(MRN) 3.sendResumeData(ACK) 4.sendResumeData(MRN) TRXOK TRXOK TRXOK TRXOK
g. receiverTrxCode contains a valid and existent receiverTrxCode in case of opType of update or delete;
h. dcCode: contains a valid data collection code registered in the DCF system.
Where these checks are positive then the Receiver will confirm the transmission with a synchronous response “TRXOK” and the protocol will proceed with the MRN generation; otherwise the Receiver will generate a TRXKO and the transmission will be terminated.
2. The Receiver checks asynchronously whether the input file is well-formed and valid according to the XML schema. The Receiver will send back a MRN calling the sendResumeData method WS exposed function on the Data sender side. The Sender WS shall be security protected and the sender shall provide EFSA with a user ID and password to access their WS. The Receiver expects a positive response to the MRN transmission through a “TRXOK”. In case a “TRXKO” is received, the protocol is stopped and the data message is considered not to have been received. If the MRN was positive, the receiver moves to point 3; otherwise the protocol stops and the data message is considered not to have been received.
3. The Receiver will execute further validations applying business rules on the original file asynchronously. At the end of this process the Receiver will send back an Acknowledgment to the Sender containing the validation results calling the sendResumeData method of the WS of the Data Sender. The Receiver expects a positive response to the MRN transmission through a “TRXOK”. When a “TRXKO” is received, the protocol is stopped and the data message is considered not to have been received.
4. The Sender will validate the acknowledgment message and sends back an MRN to the Receiver calling again the sendresumeData method to notify the receipt of the acknowledgment message. The Receiver will validate the MRN and generate a positive response to the MRN transmission through a “TRXOK”. In case of errors in the MRN a “TRXKO” is sent back by the receiver, the protocol is stopped and the data message is considered not to have been received. In addition, the data message is considered not to have been received if the MRN transmitted by the sender is negative.
WebService Provider elect Internet SOAP / HTTPS MemberState EFSA WebService Provider MS_WSDL elect EFSA_WSDL WebService Client WebService Client Backend Backend IDs
Figure 5: Web services architecture diagram.
The Article 36 beneficiaries in CFP/EFSA/DATEX/2009/01 and EFSA IT will investigate possible enhancements in the implementation of the SOAP protocol particularly the synchronous MRN and the option of including information from the data message in the SOAP message.
6. XML messaging
EFSA has currently set up a system, supporting XML messages to pilot this type of data transmission during the article 36 projects CFP/EFSA/DATEX/2009/01 and Pesticide Residues Monitoring 2009. This system already includes a data message, an MRN message and an acknowledgment message. The Working group revised and modified the XML schemas for the data message, the MRN and the acknowledgment. Particularly the names of some data elements were changed. The main changes were performed to the acknowledgment message to support a better reporting of the detected errors and warnings to the sender. The acknowledgment now includes only an error summary, while the detailed error report has been moved to a separate file which can be downloaded by the sender after the acknowledgment has been received. While the EFSA system already supports the new version of the data message, the MRN and ACK are still aligned to their former versions, which, for completeness, are listed in Appendix 1.
The new versions of the data message, MRN and acknowledgment, as agreed by the Working Group, are described in the paragraph below.
6.1. Data message
The Data Message is comprised of three complex elements containing data fields:
The message header: contains information to allow the communication protocol to deliver the MRN message to the sender.
The data transmission (dataTrx): contains the information if the Sender is performing an insert (code 01), update (code 03) or a delete (code 02) operation. It defines also the data collection to which the data contained must be appended.
Dataset: contains the analytical measurement data as defined in the Standard Sample Description (SSD).
The “Data message diagram” reports the structure of the MRN message, which is then further detailed in “Data message elements”.
The element “dataTrx related attributes” is defined as an attribute in the data message and as an element in the MRN and ACK messages. This is only due to compatibility and historical reasons.
Table 1: Data message elements.
Element Name Type Description Mandatory
Message Element Message element Yes
Header Element Message header Yes
type xs:string (20) Values: dcfmsg
Type of transmission Yes
version xs:decimal (5, 2)
Values: 1.0
Message version supported in this transmission. Yes
code xs:string (100) Unique code for the file transmitted, the file name could be used if unique.
Yes receiverCode* xs:string (100) Code representing the organisation receiving the
data transmission
Yes senderCode* xs:string (100) Code representing the organisation sending the data. Yes sentDate xs:dateTime Date the message has been sent from the sender; in
case this date cannot be produced this element should include the date the data message was created.
Yes
dataTrx Element The data transmission part of the message contains the data for a specific data collection.
Yes receiverTrxCode xs:string (100) Identifier of the data message (identified by the
element “message”) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication.
No
senderTrxCode xs:string (100) Identifier of the data message (identified by the element “message”) inserted in the receiver database. Yes opType xs:string (2) Values: 01 02 03
Operation to be performed on the dataset: Insert=01, Delete=02, Update=0315
Yes
dcCode* xs:string (100) Identifier for the data collection. Yes dcName* xs:string (500) Name for the data collection. No trxComm xs:string (100) Text comment for this transmission No
* These codes are defined by EFSA and provided to the sender at the beginning of the reporting period.
15
6.2. MRN message
The message structure is made up of two elements containing data fields:
The message header: contains information to allow the communication protocol to deliver the MRN message to the sender
The message MRN: contains the information if the message has been accepted for processing by the receiver and whether the file is well formed. In addition the MRN body contains the transmissionID (receiverTrxCode) which allows the sender to find the transmitted dataset in the receiver‟s database.
Even if the message transmission receives a positive MRN, the actual acknowledgment message could be negative. The MRN contains only the information that the message is well-formed and valid according to the defined XML schema, and can be processed by the receiver. The MRN does not contain the actual result of the business validation of the content of the XML message. The actual result of the business rules validation process is delivered by the acknowledgment message.
The “MRN message diagram” reports the structure of the MRN message, which is then further detailed in “Message MRN elements”, where all elements included in the MRN message are listed.
Table 2: Message MRN elements.
Element Name Type Description Mandatory
header Element Message Receipt Notification (MRN) message header. The message header has the same elements as in the XML message
Yes
mrnBody Element Content of the MRN message Yes
senderTrxCode xs:string (100) Identifier of the data message (identified by the element ”message”) inserted in the receiver database.
Yes
receiverTrxCode xs:string (100) Identifier of the data message (identified by the element “message”) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication.
Yes
msgReceiveDate xs:dateTime This field specifies when the message has been received by the receiver's system.
Yes dcCode xs:string (100) Identifier for the data collection. Yes
dcName xs:string (500) Name for the data collection. No
mrnDate xs:dateTime This field specifies when the MRN message has been created by the receiver's system.
Yes mrnCode xs:string (2)
Values: 01 02
This field contains the MRN code. "01" means transmission accepted. "02" means transmission not accepted because of parsing errors in the message, or for errors at security level.
Yes
parsingError xs:string (4000) This element will contain the parsing error generated checking if the transmission file is a well-formed XML file and if it is valid according to XML schema.
No
To facilitate the reading of the acknowledgment message from users an XML Transformation Stylesheet to HTML will be produced by EFSA and an example can be found in the annexed zip file (XML.zip) with the name “formatMRN.xslt”.
6.3. Acknowledgment Message
The message structure is embodying three elements containing data fields:
The message header: Contains information to allow the communication protocol to deliver the acknowledgment message to the sender.
The message acknowledgment (msgAck): Contains information on the result of the entire message processing. If the transmission acknowledgment code (trxAckCode) contains the value:
o “01”: the message was loaded by the receiver system without errors and warnings. No further action is required by the sender of the message.
o “02”: the message was not loaded by the receiver system because of errors in the message. The sender has to correct the message and resubmit the data to the receiver. o “0316”: the message was loaded by the receiver system with warning, but with no errors. The sender may decide to correct the fields generating warnings but the action is not required.
In addition this entry contains the transmissionID which allows the sender to find the transmitted dataset in the receiver‟s database. It is paramount that the sender keeps track of the transmissionID held in the receiverTrxCode. The sender will have to use the transmissionID for any further operation of replacement or nullification (deletion) of the dataset.
The transmission acknowledgment (trxAck): This entry contains the result of the validation of the data contained in the data message in the form of error codes and error descriptions. The data fields in this entry provide to the sender information with the necessary information required to prepare a new data submission with the identified errors corrected. If no errors or warning have been detected by the receiver‟s system, then the entity transmission acknowledgment (trxAck) is not generated.
The “Acknowledgment diagram” reports the structure of the acknowledgment message, which is then further detailed in “Acknowledgment elements”, where all elements included in the acknowledgement are listed.
16 The element ackCode specifies a value “03” for warning to keep compatibility between the trxAckCode of the ACK message and the mrnCode of the MRN code.
Table 3: Acknowledgment elements.
Element Name Type Description Mandatory
ack element Acknowledgment element Yes
header element Message Acknowledgement (ACK) header. The message header has the same elements as in the XML message
Yes
ackBody element Message Acknowledgement Body. Yes
msgAck element This entity contains information on the overall validation of the message.
Yes msgReceiveDate xs:dateTime This field specifies when the message has been received by
the receiver's system.
Yes msgAckDate xs:dateTime This field specifies when the acknowledgment message has
been created by the receiver's system.
Yes trxAckCode xs:string Values: 01 02 03
This field contains the acknowledgment code. "01" means transmission successful. “02” means transmission not successful because of errors in the message. "03" means transmission successful with warning.
Yes
senderTrxCode xs:string (100)
Identifier of the data message (identified by the element ”message”) inserted in the receiver database.
Yes receiverTrxCode xs:string
(100)
Identifier of the data message (identified by the element “message”) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication.
Yes
dcCode xs:string
(100)
This field contains the code of the data collection which is involved in the data transmission.
Yes
dcName xs:string
(500)
This field contains the name of the data collection involved in the data transmission.
No trxAck Entity List of elements reporting the summary acknowledgment
information.
No errRef xs:string
(500)
This attribute contains the place where the detailed error list can be downloaded.
No ErrorInfo Entity This entity provides summarised information regarding
warnings and errors.
Yes errorType xs:string (1) The type of information the acknowledgment is returning.
"E" Error, "W" Warning, “I” information.
Yes ruleCode Xs:string
(10)
Code of the validation rule applied to the element. Yes errorCode xs:string
(10)
This field contains the error code generated by the receiver system.
Yes errorDescription xs:string
(500)
This field contains the text description of the error, warning generated by the receiver's system.
Yes
var xs:string
(1000)
The name/s of the elements/s tested in the validation rule (can contain $ separators).
No example xs:string
(4000)
An example of the combination of data message values that failed the validation rule (can contain $ separators).
No records xs:long Number of records (Results in case of Standard Sample
Description message) which failed this validation.
To facilitate the reading of the acknowledgment message from users an XML Transformation Stylesheet to HTML will be produced by EFSA and an example and an example can be found in the annexed zip file (XML.zip) with the name “formatACK.xslt”.
The acknowledgment message, as anticipated, contains only a summary of the errors detected in the sender‟s message. In case the sender wants to access the full list of errors, they can be downloaded following the URL link contained in the acknowledgment message, in the attribute errRef.
The XML schema for the detailed error file is displayed in “Figure 9 – Detailed error report”, which is then further detailed in “Acknowledgment elements”, where all elements included in the error report are listed.
Table 4: Error report elements.
Element Name Type Description Mandatory
errAckDetails element List of detailed errors encountered during the business rules validation process.
Yes
receiverTrxCode xs:string (100)
Identifier of the data message (identified by the element “message”) inserted in the receiver database. This Identifier is returned in MRN message by the receiver of the Data message (= EFSA) after the first insert operation and must be used for all further message based communication.
Yes
errAckDetail element This entity contains the output of an individual validation using the business rules.
Yes errorSeq xs:long Identification number of the error in the file. Yes errorType xs:string (2) The type of information the acknowledgment is
returning. "E" Error, "W" Warning, “I” information.
Yes ruleCode Xs:string (10) This field contains the code of the rule generating the
error.
Yes errorCode xs:string (10) This field contains the error code generated by the
receiver system.
Yes errorDescription xs:string (500) This field contains the text description of the error,
warning generated by the receiver's system.
Yes Var xs:string (250) List of the variables involved in the validation rule. No varVal xs:string (250) List of variable values that are involved in the
validation rule.
No recordCode xs:string (20) The "resultCode" from the data message file,
identifying the result which failed validation.
No
To facilitate the reading of the acknowledgment message from users an XML Transformation Stylesheet to HTML will be produced by EFSA and an example can be found in the annexed zip file (XML.zip) with the name “formatDetailedError.xslt”.
7. Error Codes and Messages
In general three steps of “validation” can be distinguished during the processing of an XML file. The steps are strongly consecutive. The three steps must be executed in sequence and the following step cannot be performed if the previous step was not successfully completed.
1. Check whether the XML is "well-formed": The file is tested to verify that it satisfies a list of general XML syntax rules, provided in the XML W3C specification, to verify if a generic file is an XML file. This control is performed automatically by any XML parser. The data message is verified “well-formed” upon arrival at EFSA and the output of this parsing process is part of the “MRN message”. The parsing error messages, if generated, will be inserted in the element “parsingError”.
2. Check whether the XML is "valid": The validation of an XML file is the process that checks if a generic XML file follows a specific template defined by an XML schema. Its elements and attributes are declared in the schema and follow the grammatical rules specified in the schema. This control is also performed automatically by an XML parser. The data message is verified “valid” at EFSA after the initial check for XML “well formed”. The output of the validation process is part of the “MRN message”. The validation error messages, if generated, will be inserted in the element “parsingError”.
3. XML is “checked against the business rules17”: Conformity to rules, which cannot be tested in step 2, but are necessary to verify the consistency and coherence of the data are checked in the last step. The business rules are verified at EFSA with custom software, which has yet to be developed. The output of the business rules check is reported in the acknowledgment message. Business rules describe the constraints used to accept data at the receiver side during the data transmissions. The receivers must always reply back to the sender through an acknowledgment transmission whether an SSD transmission is accepted or not. In case of failure of the transmission, the receiver should also describe the reasons which determined the failure of the message transmission.
Though the entire validation process is executed on the EFSA side, in order to reduce the possible occurrence of transmission failures, the sender is recommended to implement the entire validation process including the business rules check prior to sending the data transmissions to EFSA.
The business rules can generate an error or a warning message. Transmissions containing only warnings are still accepted by the receiver. Transmissions containing at least one error are rejected by the receiver.
The SSD document describes a generic set of business rules; in addition specific data collections may implement their own specific validation rules depending on legislative and scientific requirements.
7.1. General business rules
The following validations are always performed upon validation of the data file:
If the data element is mandatory, the element must be present and the value must be reported; If the value reported is compliant with the type defined in the SSD;
If the type requires a controlled terminology, then the code reported must be included in the controlled terminology specified.
In addition, the following validations must be performed. In cases where a data element is not mandatory, the rule will only be applied if the optional data element is provided (e.g. the rule for the data element S.05 “Area of sampling” will be applied only if the element S.05 is provided). In cases where the rule involves a comparison of more optional data elements, the rule will only be applied if all the data elements involved in the comparison are provided or can be built or calculated from the provided data.
In both previous cases, if at least one of the optional data elements involved in the comparison is missing, the rule is not verified and no errors or warnings concerning that particular business rule are generated.
17
In this guidance this type of check operations are called “business rules” in order not to confuse them with the XML validation. In the guidance on Standard Sample DescriptionError! Bookmark not defined. they were called “validation rules”.
Twelve validation rules have been identified as “basic validation rules”. These rules are generic and work as templates for the possible validation rules that can be applied to the data. These basic validation rules are listed in Table 5. Being generic template validation rules, they can be applied to different variables or with different parameters implementing all the current requirements of
validation of the general and detailed validation rules. These variables are indicated in the column “Parameter list”. In case multiple variable and parameters are necessary for a validation rule, a dollar “$” sign is used as separator. Some validation rules require lists of variables and parameters
(variableList and parameterList). The items of these lists are then separated by a space character.
Table 5: Basic business rules. Validation
rule code Validation rule name Parameter list Applies
BR01A When variable1 condition param1 then variableList is unique
variable1$condition$param1$variableList Data collection BR02A Variable1 is mandatory when at least
one of variableList is not null
variable1$variableList Result
BR03A Variable1 condition param1 variable1$condition$param1 Result BR04A Variable1 is null if variable2 condition
param1
variable1$variable2$condition$param1 Result BR05A Variable1 condition1 param1 when
variable2 condition2 param2
variable1$condition1$param1$variable2$ condition2$param2
Result BR06A VariableList are constant for variable1 variable1$variableList Data
collection BR07A Area variable1 contained in country
variable2
variable1$variable2 Result
BR08A Variable1 is mandatory if variable2 condition param1
variable1$variable2$condition$param1 Result BR09A Variable1 between param1 and param2 variable1$condition$param1$param2 Result BR10A Partial date
variableMonth1/variableYear1 condition paramMonth2/paramMonth2 variableMonth1$variableYear1$condition$v ariableMonth2$variableYear2 Result BR11A VariableDay/variableMonth/ variableYear is a valid date
variableDay$variableMonth$variableYear Result BR12A Date variableDay1/variableMonth1/variable Year1 condition paramDay2/paramMonth2/ paramMonth2 variableDay1$variableMonth1$ variableYear1$condition$ paramDay2$paramMonth2$paramMonth2 Result
Table 5 describes the variables, the conditions and the parameters applying to the rule:
Variable: elements of the Standard Sample Description or “implicit” values such as the current date (nowD,nowM,nowY) and the identifier of the organisation sending the data (orgID). Variables representing day, month, year are described as “variableDay”, “variableMonth”, “variableYear”
VariableList: list of variables as defined above. The elements of the list are separated by spaces.
Parameters: parameters can be variables or numeric constants (1, 2, 3.5, 700) or text constants (“RF-XXX-XXXX”).
Most of the business rules validate the correctness of related data element within a single record of transmitted data (e.g. if the result type indicates a result below LOD then the LOD must be reported). Two rules (BR01A and BR06A) validate instead the correctness of an entire data collection (e.g. that a specific parameter measured for a sample has been only transmitted once even across transmissions).
The basic validation rules must be associated with the relevant variables or parameters in order to create a validation rule instance and to provide the acknowledgment results. This association is described in Table 9 in Appendix 2.
Element: Referenceelement code to which the validation rule instance is applied. Rule code: Validation rule code as listed in Table 5.
Rule Name: Validation rule name as listed in Table 5.
Rule: Description of the specific rule obtained with this instance of the basic validation rule Variables: Variables and parameters used to create the instance of the validation rule. The “$” sign is used as separator for variable and parameters. When a variableList of parameterList is requested, the items of these lists are separated by a space character.
Error Type: If this instance of the validation rule is creating an error this field report “E” Error Code: Code for this instance of the validation rule.
Error Description: Additional description for the validation rule instance.
Scope: Defines if the scope of the validation rule is applicable to all the data collection (“All”) or only to a specific data collection.
7.2. Specific business rules
While general validation rules apply to all data collections utilising the standard sample description, specific validation rules apply only to a specific set of parameters defined by the “scope” variable presented in Appendix 2.
The specific validation rules are still instances of the twelve basic validation rules and are defined by each EFSA Data Collection Data Manager in conjunction with relevant stakeholders.
8. Conclusions and Recommendations
8.1. Conclusions
The Technical Working Group on Data Collection have agreed that the Standard Sample Description, defining variables and terminologies, and this Data Exchange Guidance, defining transmission details, are the first steps to harmonising the transmission of sample level data from Member States to EFSA. The guidance documents have been developed specifically to address transmission of chemical occurrence and pesticides data and, to date, have only been piloted in the pesticides domain (2008 and 2009 Annual Data Collection). Feedback from this experience has been incorporated into the documents. The system will evolve as further experience in its use is gained.
The Working Group considers it essential to plan for and put in place a process for further maintenance and development of these guidance documents to allow for improving and extending the overall process as it relates to the currently defined data collections (chemical contaminants and pesticides) to EFSA data collections supporting other risk assessment activities, e.g. zoonoses.
The group recognises that the ability of each Member State to transmit data to EFSA according to the Standard Data Model and via the automated transmission mechanisms will vary. Therefore the guidance documents are also intended to assist Member States for planning future developments and evolution of local, regional and national systems with the objective of harmonising data transmissions.
The working group acknowledges that further development work is required to support the functionalities described by EFSA in the document.
Harmonisation of data collections is recognised as a fundamental step in the development of an effective EFSA Data Warehouse. The establishment of the EFSA Data Warehouse is seen as a resource for Europe-wide risk assessment by EFSA and - with appropriate access policies - for Member States.
8.2. Recommendations
The Technical Working Group on Data Collection, after examining the details of the data transmission file formats, mechanisms and security considerations, makes the following recommendations which are aimed to harmonising the format and mechanism of transmission of data to EFSA for the Chemical Contaminants and Pesticides Residues domains. Common requirements for these domains are addressed in the guidance. In addition, during the development of this harmonised data exchange guidance, attention has been paid to commonality with the requirements for the collection of Zoonoses data, although domain specific adjustments have not been addressed.
The following recommendations are addressed to the European Food Safety Authority as the leading organisation and it is anticipated that the work necessary to achieve these will be undertaken in conjunction with Member States and the Commission.
Recommendation 1: XML Schema validation at source
Error and warning codes are defined within the guidance to assist data senders in identifying and rectifying data quality issues in advance of transmission. Errors in transmitted data involves unnecessary processing, flagging of „replaced‟ data in the EFSA database and iterative communications which should be minimised, especially once data transmissions move from pilot/test into operational modes.
