6 STEP conformance testing
6.4 Abstract test methods
In order to produce an executable test suite from an abstract test suite a choice of test method must be made. STEP describes two abstract test methods for use in conformance testing of STEP implementations; one for the generation of the exchange structure, pre-processor, and one for the interpretation of the exchange, post-processor.
“The basic approach of both test methods is that an IUT is supplied an ISO 10303 application protocol instance model in one format and is expected to translate the instance model into another format while observing the rules of the application protocol structures and
maintaining instance model semantics. “ [ISO10303-34]
Figure 17 – Test method actors
The Test Engine and the SUT operator are responsible for stimulating the IUT with input and observing the output.
The actors involved in the testing structure are the IUT, SUT operator and Test Engine as shown in figure 17. The IUT (Implementation Under Test) is the implementation to be tested. The IUT is located within an SUT that needs to be operated by an operator, giving access to the IUT. The SUT operator communicates directly with the IUT through a format defined by the IUT. This information exchange can be performed by a human operator in the role of the SUT operator.
The test engine is the system that executes and analyses the test cases. Communication is done with the IUT both directly through STEP defined format and indirectly through the SUT operator. The SUT operator is in that case an interface between the Test engine and the IUT. The test engine is the control and communication function in the test procedure.
Testing of STEP implementations are done on either ISO 10303-21 or ISO 10303-22 pre- processor, or ISO 10303-21 or ISO 10303-22 post-processor. ISO 10303-21 is the part 21 file defined in STEP, while ISO 10303-22 is Standard Data Interface Access (SDAI) defined in STEP.
Both test structures are executed after the test procedure explained later in this chapter. Preparation is performed before any test execution is undertaken.
6.4.1 Pre-processor test method
The pre-processor test model is depicted in figure 18. Pre-processor implementations produce an output in accordance with the STEP standard. The input format is implementation
dependent so the SUT operator is essential in order to ensure that the input is done correctly.
Figure 18 – Pre-processor test method
For Pre-processors the Test engine determines what input the SUT operator shall give the IUT, before observing the associated output.
Input to the pre-processor test is in the form of a human readable instance model from an Executable Test Case (ETC). The instance model can be either text, or text and graphics, in
The input entered into the IUT is monitored for accuracy control in the conformance log, deviations shall be analysed. If any data is incorrectly entered according to the ETC, the data shall be re-entered and the test case re-run, if this is for some reason not feasible it shall be reported as Inconclusive. The integrity of the input specification shall be maintained and documented, but STEP does not explicitly state how to do this.
When the input is completed, the IUT shall generate the specified output structure in the format of a P21 file or SDAI. “Pre-processor parameters identified in the ATC and resolved during testing preparations shall be applied to the IUT.” [ISO10303-34] The output structure from the IUT is the instance that is tested in the analysis phase, (3).
6.4.1.1 Analysis
The output structure from the pre-processor is analysed using a number of different methods. The analysis phase aims at discovering any inconsistencies between input and output, caused by the IUT. During the inconsistency checks, verdicts are assigned to the ETCs. If a single verdict criterion within an ETC fails the whole ETC fails and further testing can be
terminated.
6.4.1.1.1 Syntax analysis
The first part of the analysis process is to determine if the output structure is in accordance with the appropriate STEP structure. This is called the syntax analysis of the conformance test.
6.4.1.1.2 Structure analysis
Structure analysis examines the instance model in the exchange structure against the schema and other requirements of the associated application protocol. “The objective is to decide whether the instance elements and their structuring, represented in the instance model, are those permitted as defined in the Application Interpret Model and conformance classes of the AP. “[ISO10303-34]
Structure analysis tries to determine whether or not the instance model in the exchange structure is in conformance with the AP. The instance element structures and their
relationships are specified in the AP, and the instance model must follow the rules stated in the AP. Invalid relationships include violation of AP rules, informal propositions and conformance class constraints. [ISO10303-34]
6.4.1.1.3 Semantic analysis
The last form of analysis methods used for pre-processor testing is semantic analysis. Semantic analysis is concerned with the information maintained in the exchange structure instance model. The application protocol defines and supports specific semantics and this test checks whether or not these semantics have been conveyed accurately into the exchange format. The ETC makes use of the AP’s semantics and it is the ETC semantics that is under review, the ETC verdict criteria describe these semantics. A verdict criterion can be specified in the form of numerical values, classification categories, computations or operations, or other specific measurement techniques. [ISO10303-34] The ETC has some content (semantic meaning) that needs to be correctly conveyed in the STEP exchange structure.
Semantic errors can be either permutations of values or concept specific. Permutation errors occur when an input value is altered in the pre-processor and can not be identified in the output. A concept specific error occurs when an entity must be viewed in a specific concept and this concept is lost in the pre-processor. The ETC identifies the requirements for verdict assignment to the different test purposes within the ETC.
The list of semantic analysis methods is not exhaustive and other methods can be used. If other methods are used this should be documented in the conformance log.
6.4.1.2 Verdict assignment and conformance test report
Verdict assignment and conformance test report production is covered below. 6.4.2 Post-processor test methodAs for pre-processor test method, test preparation is a prerequisite to the test execution and analysis. The input to the post-processor is in the form of a pre-defined STEP format
containing an instance model, in either ISO 10303-21 or ISO 10303-22 format, (1). The input is assumed to be correct, and it is provided by the testing laboratory based on information in the ATS, PICS, PIXIT and the AP.
Figure 19 – Post-processor test method
For post-processor testing the Test engine send the input to the IUT and observes the output through the SUT operator.
“ATC parameters identified during the preparation for testing shall be applied to the IUT and the exchange structure input to the IUT.” [ISO10303-34] Deviations in the information communicated are sought by querying the IUT indirectly via the SUT operator, (2 and 3). This means that the queries come from the testing laboratory but are executed on the SUT by the SUT operator. Problems and deviations are documented in the conformance log.
6.4.2.1.1 Input handling analysis
Input handling analysis test the IUT by manipulating inputs to the system and observing the results. The input can be valid, invalid or syntactically invalid, and in each case the output is observed and compared to predefined expected output. With a valid input there shall not exist any errors, however with invalid elements or invalid syntax, the IUT shall generate an error response. Invalid elements and syntax are derived from the AP.
If the testing reveals any handling or error responses not documented in the AP, this should be identified in the PIXIT and added to the conformance report.
6.4.2.1.2 Semantic analysis
Semantic analysis as used for pre-processor is also partly applicable for post-processor testing, although with some adjustments. A set of queries are done on the IUT in order to determine whether or not expected semantics are found. The expected semantics are identified as verdict criteria. “These semantics are permitted by the information requirements clause of each AP and qualified by the conformance requirements clause.” [ISO10303-34]
The queries must be entered into the IUT by the SUT operator since post-processor output is not bound to any particular encoding. The test engine provides the SUT operator with the test queries. The queries comprise a structured and extensive part of the AP parts that are
applicable to the testing. Test response from SUT operator queries is sent to the test engine for analysis in a test engine defined format.