Sample N values for the input factors within their intervals of variation (Step

variation (Step 4.5)

The factors are considered as if they were statistical variables. In the use case example, the sampling strategy relies on the use of quasi-random sequences, and in particular the Sobol’ LP sequence [23], [24] which has remarkable low-discrepancy properties and allows the sample points to cover homogeneously the space of factors’ uncertainty (an illustration of the advantages of quasi-random sequences with respect to other sampling strategies is provided in Annex III. Moreover, in order to have an uniform coverage of the space of factors when using Sobol’ LP sequences, the theory suggests to use a sample size N equal to a power of two, i.e. either 8, 16, 32, etc.. The Matlab / Octave script for generating Sobol’ LP quasi-random sequences is available at:

https://ec.europa.eu/jrc/sites/jrcsh/files/LPTAU51.rar

Other sampling strategies could likewise be used to generate the sample, such as Latin Hypercube sampling (LHS) or simple random sampling. Computer programs to generate both LHS and simple random samples are available everywhere. Neither LHS nor simple random sampling recommends the use of specific values of sample size N. However, they have less uniform coverage properties with respect to the Sobol’ LP sequence using a sample size N equal to a power of two.

By using one of the sampling strategies proposed above, N values for each factor are sampled from the ranges and probability distributions defined in Step 4.4. The choice of N is made based on the experimental resources available. The N sampled values give rise to a numerical matrix (the ‘sample matrix’) with N rows and a number of columns equal to the number of input factors.

In the UC example, the sampling starts with N = 23 values using quasi-random sequences within the ranges and from the probability distributions defined in Step 4.4, generating a sample matrix of size 8x3 as in Table 5.

Table 5 Example of 8 x 3 sample matrix generated using quasi-random sequences Exp. 1 6.9000 5.9000 4.7500 Exp. 2 3.4500 8.8500 2.3750 Exp. 3 10.3500 2.9500 7.1250 Exp. 4 1.7250 7.3750 8.3125 Exp. 5 8.6250 1.4750 3.5625 Exp. 6 5.1750 4.4250 5.9375 Exp. 7 12.0750 10.3250 1.1875 Exp. 8 0.8625 11.0625 6.5313

Each row of the matrix corresponds to a Test Case to be executed. The i-th Test Case will use the values of the i-th row of the sample matrix as input factors. A template is proposed to summarise the the DoE procedure (Annex I). The next step is to execute the N Test Cases described in the next Section (Step 5).

5

Interoperability testing (Step 5)

Testing is the critical stage to verify interoperability. An important condition in achieving interoperability is the correct implementation of the standards and specifications. This can be verified by conformance testing: the concerning implementation/device is tested against a test tool or reference implementation of the standard. Conformance testing is also a prerequisite for interoperability testing. Conformance testing also verifies what part of the standard is implemented if it is not a full scope implementation.

The conformance or compliance testing determines to what extent the implementations under test are in accordance with the whole or partial implementation of a specified standard, or to what extent they do so with regards to specific requirements. The methodology developed here focuses on the interoperability testing, thus the implementations which will be selected to be tested should have as a precondition a pass mark in the conformance testing. Nevertheless, if discrepancies or specific problems will be found during the testing exercises, this will be reflected in the relative reports.

Interoperability testing is a procedure in which two or more implementations (systems, products) within a system are tested if they are able to exchange information according to the final defined functionalities. Implementations are tested in their final configuration together with other components of the total architecture, according to the same BAIOP. The aim is to verify that implementations do indeed work together effectively and interoperate, to deliver the expected results. It should be noted that interoperability testing is significantly different from conformance testing because it is possible for two implementations that individually comply with a standard (resulting in a positive conformance test) to be still unable to interoperate. This situation can arise for example when devices have implemented different or conflicting options or cover a different part of the standard(s).

The SGILab approach is towards creating a platform where quantitative assumptions, experimental options and simulations/emulations are systematically tested to assess the final outcome with reference to the stipulated UC, while identifying the factors jeopardising interoperability. Interoperability testing in the SGILab will be based on a systematic procedure carried out under control conditions. Testing will be based on the BAIOP and the Test Cases, either produced by the manufacturer/producer of the equipment under test, or in its absence developed internally. The IOP will clearly determine the requirements of the particular application, function, community, and/or context, to be tested.

The SGILab procedure aims at simulating or emulating real cases scenarios in order to discover potential functional or communication incompatibilities as well as to illustrate the effects or establish a hypothesis of any lack of conformance or compatibility.

The procedure for interoperability testing shall include:

Identification of candidate "Equipment Under Test" (EUT). An EUT is a physical implementation of an equipment that interacts with one or several other pieces of equipment. The specific functionality of the EUT to be tested is here identified

• Identification of the UC: The UC case is selected from the database and should describe the functionality of the EUT to be tested as defined above

• Identification of the BAP: The BAP is selected from the database and derives from the standard which has been used in the design of the EUT's functionality

• Selection of BAIOPs: Based on the UC and the selected above BAP, BAIOP is automatically produced

• Test cases are created based on the BAIOP and the DoE procedure. A companion document for each test case is also prepared

• Verdict/ Results: the pass/fail verdict and the results of the test cases are filed under the respective Test Case

The process is graphically presented below Figure 8 and it shows the procedure that the lab will follow with the use of a functional JRC database. The whole procedure will be automated with the use of the database which will provide the UC, BAP and BAIOP for the testing purposes based on the specific functionality. The Test Cases will be also created and tailored based on DoE.

Test Cases describe the exact sequences of actions required to be realised so that a verdict can be assigned. Steps should be clear and unambiguous. It is important that the steps can be followed exactly as described, so clarity and precision here plays an important role.

Each Test Case shall have a Test Description comprising the following information: - A Test Case identifier

- The reference UC upon which the test is based (with UC ID and BAIOP ID) - The interoperability layer on which the test is executed

- A summary of the test

- The purpose of the test: what is the objective of the specific Test Case - Description of the test sequence in discrete actions

- The Verdict of the Test Case after the execution of the Test Case is documented here Each Test Case shall also have a companion document. This document is to be filled in during the testing process and it is based on the EUT specifications (hereafter the process becomes device specific). The companion document is filled in once and applies to all the Test Cases. Thus after filling up the companion document, the Test Cases become Specific Device Test Cases and can be used for the IOP testing of the selected EUT. The companion document includes:

• Nameplate specifications

• Technical specifications of the EUT based on its Technical Manual

• Operational specifications of the EUT based on its Technical Manual

• Mounting/ Cabling/Connecting with the SUT of the EUT

• Protection/ Isolation/ Earthing of the EUT

• Specific safety issues

• Configuration/ Initial criteria

• Others (based on the EUT)

For the Test Cases and the Companion document a template has been created in Annex I.

With the use of the database outputs (UC, BAP, BAIOP and TC) the test can be realised then in the physical lab. The result is uploaded back to the database as an evidence for other experiments or further analysis.

Figure 8 Interoperability Testing Process