Agile Test Methodology for B2C/B2B Interoperability
A Dissertation
by
Jungyub Woo
Department of Industrial and Management Engineering
Pohang University of Science & Technology
Chair of Committee:
Dr. Hyunbo Cho
Committee Members:
Dr. Boonserm Kulvatunyou
Dr. Sooyoung Kim
Dr. Kwangsoo Kim
Dr. Jaekoo Joo
ABSTRACT
In the B2B/B2C environment, many applications and documents requires to be tested for conformance and interoperability. However because there are a lot of standards, solutions, specific environments, and test needs, it is difficult to develop all testing tools (i.e., testbeds) to cope with these test diversity. The testbed should be developed more rapidly and easier. In addition, the implemented testbed should be accommodated for various testing specifics. Thus we have proposed the agile test framework (ATF), which consists of test suite design and its execution model. The test suite is designed with a double layer: abstract and executable test suites. The execution model is also invented considering as reusability and plug-and-play concepts. This approach makes test engineer rapidly generate test suite and rapidly implement testbed. It also enables the implemented testbed to test various applications and documents based on diverse standard specifications. In other words, ATF is a test framework that has high reusability, high extensibility and high efficiency for the test. In conclusion, ATF makes B2B/B2C systems interoperable rapidly
Key words: Agile test framework (ATF), business-to-business (B2B) integration, conformance test,
interoperability test, standard, test automation, and test suite generation
TABLE OF CONTENTS
I. Introduction ...１ I.1. Rational for the Research ...１ I.2. Research Objectives ...２ I.3. Organization of Dissertation ...３ II. Literature Review ...５ II.1. Chapter Overview ...５ II.2. B2B Interoperability and Testing ...５ II.3. Requirements for B2B/B2C Testing...６ II.4. Existing Test Approaches ...７ II.4.1 ebXML Implementation, Interoperability and Conformance (IIC) Test Framework７ II.4.2 RosettaNet Self-Test-Kit ...１０ II.4.3 WS-I Test Tools ...１３ II.4.4 Testing Test Control Notation (TTCN-3) ...１４ II.4.5 Comparison Matrix among the Existing Test Frameworks ...１７ II.5. Chapter Summary...１７ III. Test Suite Design for Agile Testing ...１９ III.1. Chapter Overview...１９ III.2. Design Factors of Agile Test Suite ...１９ III.3. Overview of Agile Test Suite Design ...２１ III.3.1 Double-layers of Test Suite ...２１ III.4. Abstract Test Suite Design...２３ III.4.1 Test Assertion ...２３ III.4.2 Test Procedure ...２６ III.4.3 Test Environment ...２７ III.5. Executable Test Suite Design ...２８ III.5.1 Scripting Primitives...３０ III.6. Chapter Summary ...３１ IV. Execution Model for Agile Testing ...３２ IV.1. Chapter Overview...３２ IV.2. Considerations of Agile Execution Model Design ...３２ IV.3. Design of Agile Execution Model ...３３ IV.3.1 Conceptual Architecture of Agile Execution Model ...３３ IV.3.2 Definition of Test Components and Test Interfaces...３５ IV.4. A Reusable and Reconfigurable Testbed Development using Agile Execution Model３６ IV.4.1 Developing Test Components...３６ IV.4.2 Realization of Test Interface...４１ IV.5. Case Study ...４２ IV.5.1 RAMP Conformance Test ...４２ IV.5.2 Quality of Schema Design...４４ IV.6. Chapter Summary ...４７
V. Test Harness and Test Execution using Agile Test Framework ...４９ V.1. Chapter Overview ...４９ V.2. Conformance Testing vs. Interoperability Testing ...４９ V.2.1 Conformance Testing Harness...５０ V.2.2 Interoperability Testing Harness...５１ V.2.3 Test Harness Problems ...５２ V.3. ATF-based Conformance Test Harness ...５３ V.4. ATF-based Interoperability Test Harness ...５４ V.4.1 Test Harness Design ...５５ V.4.2 Development Test procedure ...５７ V.5. Case Study...５８ V.6. Chapter Summary...６０ VI. Case Study – Inventory Visibility and Interoperability...６１ VI.1. Chapter Overview...６１ VI.2. Inventory Visibility and Interoperability (IV&I) Project...６１ VI.2.1 AIAG Works...６１ VI.2.2 E-Kanban Business Scenario of IV&I Phase II...６２ VI.2.3 Testing Requirements ...６２ VI.3. Test Approach ...６３ VI.3.1 Information Mapping Test ...６３ VI.3.2 RAMP Messaging Test ...６５ VI.4. ATF Testbed Implementation...６７ VI.4.1 Test Case Development ...６７ VI.4.2 Testing Tool Implementation...７２ VI.5. Chapter Summary ...７２ VII. Concluding Remarks...７３
LIST OF FIGURES
Figure II-1 IIC Conformance Test Framework... ８ Figure II-2 IIC Interoperability Test Framework ... ９ Figure II-3 High-level system diagram of RNSTK ... １１ Figure II-4 Test flow diagram of RNSTK ... １２ Figure II-5 High-level system diagram of WS-I Test Tools ... １４ Figure II-6 Test procedure of the TTCN-3 test framework ... １６ Figure III-1 Typical test scenario from a standard to testing... ２０ Figure III-2 Agile Test Suites and their relationship ... ２２ Figure III-3 Design of Test Assertion ... ２５ Figure III-4 Example diagram of Test procedure ... ２６ Figure III-5 Design of Test Procedure ... ２７ Figure III-6 Design of Test Environment ... ２８ Figure III-7 General architecture and execution model of eTSL ... ３０ Figure IV-1 Conceptual architecture of agile execution model... ３４ Figure IV-2 Part of exemplary configuration information in an abstract test suite ... ３７ Figure IV-3 Sample code to retrieve a key value from XPATH validation properties ... ３８ Figure IV-4 Code to retrieve the key value of the WSDL URL ... ３８ Figure IV-5 Part of exemplary internal process script as WS-BPEL ... ４０ Figure IV-6 Part of a TVE WSDL service description ... ４２ Figure IV-7 (A) Sample RAMP executable test cases and (B) Internal Process Script... ４４ Figure IV-8 RAMP Testbed Implementation... ４４ Figure IV-9 (A) Sample QOD executable test cases and (B) Internal process script ... ４６ Figure IV-10 QOD Testbed Implementation ... ４７ Figure V-1 Conformance testing versus Inoperability testing ... ５０ Figure V-2 General conformance testing harness... ５１ Figure V-3 Accordare’s collaborative test harness ... ５２ Figure V-4 KorBIT interoperability test harness ... ５２ Figure V-5 Conformance testing harness for ATF ... ５４ Figure V-6 Test environment for conformance testing ... ５４ Figure V-7 Intelligent interoperability testing harness for ATF... ５６ Figure V-8 Test Environment for Interoperability Testing ... ５７ Figure V-9 Test procedure for the connecting mode ... ５７ Figure V-10 Test procedure for the debugging mode ... ５８ Figure V-11 Exemplary SOAP message encryption and decryption ... ５９ Figure VI-1 AIAG E-Kanban Interaction Diagram... ６２ Figure VI-2 Kanban authorization of IV&I Business Scenario... ６３ Figure VI-3 Proposed semi-automated input testing procedure ... ６４ Figure VI-4 Proposed semi-automated output testing procedure ... ６５ Figure VI-5 Sample of test assertion for Information Mapping Test ... ６８ Figure VI-6 Sample of test assertion for R1010 requirement in the RAMP test... ６８
Figure VI-7 Test procedure for IV&I Customer... ６９ Figure VI-8 Test procedure for IV&I Supplier... ７０ Figure VI-9 Test Environment for IV&I Scenario ... ７０ Figure VI-10 A part of ETSL test case to verify supplier... ７１ Figure VI-11 ATF-based IV&I testbed ... ７２
LIST OF TABLES
Table II-1 B2B interoperability layers... ５ Table II-2 Comparison matrix for conventional and proposed test frameworks ... １７ Table IV-1 Abstract definitions of Test Interfaces ... ４１ Table V-1 Example of WS-Security policy ... ５９ Table VI-1 Requirements of RAMP specification... ６５
I.
INTRODUCTION
I.1.
Rational for the Research
As the electronic collaboration through the Internet infrastructure, also called the business-to-business (B2B), has matured, the number of software solutions to support automated, secure, and reliable transactions has been growing. Because different businesses typically have different information requirements, processes, and protocols/interfaces to collaborate (or communicate) with others, the B2B/B2C requires several related standards as common references. There are several aspects of these standards such as those that specify regulation and conventions for common data format, communication protocols, and interfaces. There are a growing number of standards, potentially competing, addressing these aspects as requirements are discovered (Shaw et al. 2000).
Even though B2B software solutions are implemented based on the same standard, they cannot avoid interoperability problems because 1) the standard may contain ambiguous requirements which could be differently interpreted, assumed, or understood by developers, 2) the standard may have defects or omissions, and 3) developers occasionally partially implement the standard, and 4) the standard is designed to be flexible (intentionally left ambiguous) allowing different specializations and implementations (Shaw et al. 2000) (Moseley et al. 2004).
For these reasons a B2B/B2C solution must be tested in two different testing modes – conformance and interoperability. Other testing modes may present as different requirements arise. B2B/B2C standards typically contain a large number of regulations and conventions. A large number of solutions must also be pair wise tested in the interoperability testing mode. This nature calls for automated testing facilities – testbeds. Testbeds for different standards have different test components, different test materials, and different configurations because each of them focuses on different specific testing requirements even though they are similar and potentially related. This makes test case development expensive for domain experts because they have to learn different schemes when writing tests for different standards. Similarly, it makes test execution expensive for the solution provider, because they have to potentially develop different supporting test components and get familiar different artifacts when testing for different standards.
In addition, because standard specific testbeds are ad hoc and short-sighted, they commonly have the characteristics below:
- Low (re)usability: Testbeds are independently implemented without common scheme making their components useless in various standards and various environments.
- Low extensibility: Testbed implementations cannot scale to address other standards than the one they are specifically designed for.
- Low Efficiency: Most of automated testbeds provide uninformative test results. It leaves testing users with little clue when debugging their systems.
In summary, this section describes the issues associated with the current practice of building a testing
facility for B2B standards. These issues, if resolved or alleviated, present potential cost and time savings in the B2B testing. This research proposes an agile test framework to address these issues and realize these potential savings. The proposed test framework will allow components within the testbeds developed in compliance with the framework to be reused for various B2B/B2C solutions, various testing modes, various standards, and various computing environments. In addition, time saving is realized through automatic configuration of these test components. The proposed test framework also aims at enhancing testing efficiency by enriching test materials and test result reports.
I.2.
Research Objectives
The eventual objective of this research is to develop the Agile Test Framework (ATF) that is reusable, configurable, and efficient for the B2B solution testing. The detailed objectives are 1) to develop a specification necessary for designing and generating a modular test suite including multi-layer test cases, 2) to develop reusable and reconfigurable test components, 3) to develop guidelines for composing an ATF-based B2B test and for analyzing test results, and 4) to demonstrate the test framework’s agility by showing actual use-case studies in various B2B tests.
To achieve the first objective, following research issues are considered:
- Modular test suite: The test suite consists of test cases and related test data. A test case includes information about test procedure and assertions. These two pieces of information are tightly coupled in conventional test cases consequently making them less reusable. They also have no configuration information of testbeds. It makes test cases used in limited testing tools. Therefore, the proposed test suites should be designed with modular concepts. It means each module should be easily separated and its relationship with the other module should be well-defined. The proposed test suites will also include information about test environment to represent test harness.
- Double layer test case: The proposed test case consists of two layers, abstract and executable test case layers. The abstract test case allows test procedure to be captured in the domain/standard specific way. This enables a standard/domain expert to easily create test cases for its own tests because the expert does not need to learn the technical details about testbed execution, but to describe the test cases based on its own domain knowledge. The executable could be ready to be executed by testing tools. It should be generic for any B2B/B2C standard and environment.
The second objective addresses following research issues:
- Reusable test components: Test components are basic building blocks to form a testbed. We classify the test components into stationary ones and non-stationary ones. The stationary components are completely independent of the types of standards, solutions, testing modes, and test configurations of interest. Therefore, any testbed configurations re-use the stationary components as-is. On the other hand, even though the non-stationary components are re-used in any testbeds, they need to be flexibly re-defined or re-implemented according to specific
testing requirements.
- Plug-n-play test components: Interfaces between test components and with systems under test (SUTs) are defined well. To support the interfaces, we define plug-and-playable test components. These components make testbeds built on ATF reconfigurable, versatile, and agile to various tests.
Research issues related to the third objective are as follow:
- Test suite development guideline: Test suites, especially an abstract test case, will be created by a domain expert who has developed and, therefore, best knows about the standard specification of interest. The proposed ATF will provide a guideline for developing the abstract test case. This will help the domain expert reflect his/her test requirements in the test case. The guideline will also include specifications of a translator that transforms the abstract test case to executable test cases.
- Test execution guideline: The test case will be executed by a system developer (namely a testing user). The proposed test framework will provide a guideline for executing tests. This will help the testing user test target system.
- Test reports analysis guideline: After the testing, the testing user will receive test reports from the testbed. The testing user will modify his/her system against interoperability problems reported. The test framework will provide a framework for generating informative test reports. The forth objective has research issues as follow:
- ebMS and Web Services messaging testbeds implementation: The messaging testbed for ebXML messaging and Web Services standards will implemented based on the proposed ATF. These testbeds will demonstrate ATF’s agility. It will especially focus on the reusability of test components and test cases.
- Information mapping testbed implementation: The information mapping testbed will be implemented based on the proposed ATF. The testbed will demonstrate the agility and reusability by showing that the testbed is formed with several test components used in the ebMS and the Web Services testbeds.
I.3.
Organization of Dissertation
The remainder of this dissertation is organized as follows. Chapter II provides literature survey regarding conventional test frameworks for B2B interoperability. A comparison of existing test frameworks is also presented to provide research rationales. Chapter III provides a design of test suite specification for test cases/materials. In order to consider a reusability of test case/materials, test suite specification considers the modular and layered concepts. Chapter IV proposes an execution model of the test framework for an agile testbed implementation. In this chapter, formal definitions of test components and distinctive features of the agile test framework such as reconfigurability and reusability are discussed. Chapter V addresses a guideline of usage of testbed on the basis of proposed test framework. In this chapter, conformance and interoperability test harnesses are introduced and it is
explained how to realize them. Chapter VI demonstrates an agility of the proposed test framework when a testbed is developed and used in order to resolve interoperability problems in the real world. Finally, summary of this dissertation and concluding remarks are presented in Chapter VII.
II.
LITERATURE REVIEW
II.1.
Chapter Overview
The objective of this Chapter is to review and discuss relevant research works that are underlying test frameworks for the B2B/B2C testing. Many test frameworks and testing tools have emerged for B2B/B2C standards. In particular, we give a comparison matrix for existing test frameworks and derive requirements for our test framework proposal from them.
II.2.
B2B Interoperability and Testing
As the late 1990s saw the emergence of many standards for e-business, there are a number of papers discussing them. According to Hasselbring and Weigand (Hasselbring and Weigand 2001), the standardization for the exchange and processing of documents can be at the lexical level of character sets, at the syntactical level of document structures, and at a deeper semantic level of dictionary and integrity constraints. Respectively, Nurmilaakso and Kotinurmi (Nurmilaakso and Kotinurmi 2004) have argued that business partners have to know what information, when, and how to share before they can do e-business. Therefore, B2B standards prescribe business process, business information specification, and messaging infrastructure. Table II-1 shows B2B interoperability layers and typical ones of associated standards.
Table II-1 B2B interoperability layers
B2B layer Functionality Standards
Business Information
Information specification
Information constrains Usage
CC, UBL, OAGIS BODs, UNSPSC, VICS, PIDX, GISB, xCBL Business Process Process definition Process execution Transaction Choreography Orchestration
WSCI, BPML, ebXML BPSS, BPEL, WS-Transaction, ebXML Registry, ebXML CPP/CPA, UDDI Business Messaging Message packaging Reliable Security Addressing
ebXML Messaging, AS1, AS2, Authorization, SAML, Federation, Reliability, Reliable Messaging, WS-Security, WS-Trust, WS-Policy, XML Signature, XML Encryption, XACML
Whittaker and Jorgensen (Whittaker and Jorgensen 1999) have argued that software often fails because of 1) improperly constrained input, 2) improperly constrained stored data, 3) improperly constrained computation, and 4) improperly constrained output. A standard-based B2B software solution requires testing because it prevents late detection of errors (possibly by dissatisfied users), which could require complex and costly repairs. Testing can detect errors, and evaluate and approve the solution’s quality beforehand (Schieferdecker and Stepien 2003). An automated test approach helps in particular to efficiently repeat tests whenever needed for new system releases in order to assure the fulfillment of established system features in the new releases (Schieferdecker et al. 2001).
II.3.
Requirements for B2B/B2C Testing
A test framework is defined as a set of utilities, style-sheets and documentation that describe and facilitate development, documentation and use of the tests (W3C 2002). Before the test development work, W3C recommends the following actions:
- Understand the testing scope,
- Assess the testing work already done for the referenced specifications and facilitate reuse of test materials where applicable, and
- Assess the risks taken by using specification that has a lot of flaws such as contradictory statements or ambiguities.
B2B environments continuously require new and emerging standard specifications and software solutions in pursuit of progresses in information technology and diverse market. It is expensive and time consuming to implement a new testbed for every emerging standard. A B2B testbed should be developed such that it can scale to facilitate testing of various standards and software solutions. Therefore agility of test framework is urgent. In the B2B environments, the testing agility could be considered for:
- Testbed administrators: They implement a testbed based on a specific test framework for specific testing objectives. Therefore, they want the ability to quickly and easily construct a new testbed.
- Domain/standard experts: On the implemented testbed, they develop the test cases and materials for test executions. They want the ability to quickly and easily create a test suite. - Testing users (software venders): By the testbed and test suites, they test their solutions. They
want the ability to correctly resolve conformance and interoperability problems of their solutions
Required agility must be satisfied by the proposed agile test framework. That is, proposed agile test framework will have characteristics as follows:
- High reusability: Most of test materials (test cases and various test data) and test components can be reused. It minimizes the cost and efforts for developing a testbed.
- High extensibility: The testbed, which is implemented based on the proposed agile test framework, can test various applications and documents and be used for diverse standard
specifications using the reconfiguration approach.
- High Efficiency: The test suite document, test process, and test report should be user-friendly for various participants in the test. It assists them to catch interoperability problems of their applications/documents. As a result it minimizes the time and efforts necessary for debugging.
II.4.
Existing Test Approaches
For the purpose of test automation, testing systems (testbeds) and test material/case (test suite) have been designed and developed. Most of test frameworks are proposed to meet the requirements of testing for a specific corresponding standard. Therefore they generally have narrow scope and provide insufficient testing services. In this chapter, we discuss four representative test frameworks: ebXML IIC, RosettaNet Self-Test-Kit, WS-I tools, and TTCN.
II.4.1 ebXML Implementation, Interoperability and Conformance (IIC) Test Framework
As many vendors implement ebXML (Electronic Business using Extensible Markup Language) compliant systems, interoperability among them has been a key for seamless e-business environments. In January 2000, the OASIS chartered the ebXML Implementation, Interoperability and Conformance (IIC1) TC to provide a means for software vendors to create infrastructure and applications that adhere to the ebXML specifications and are interoperable. The OASIS IIC TC proposed a test framework, OASIS ebXML IIC Test Framework (IIC 2001), to verify interoperability among ebXML compliant e-business systems as well as their conformance to the standard. The initial scope of the IIC Test Framework covers both conformance and interoperability testing of ebXML Message Service Handlers (MSHs).
II.4.1.1 IIC Conformance Test Framework
The IIC test framework provides a conformance test framework to verify whether a candidate system under test (SUT) conforms to the ebMS specification. Figure II-1 depicts the conformance test harness in the IIC Test Framework.
1
ebXML Implementation, Interoperability and Conformance (IIC) Technical Committee, http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=ebxml-iic
Test Driver Transport Adapter Conformance Test Cases Test Reports Test Service SUT ebMS Message Testbed SUT
Figure II-1 IIC Conformance Test Framework
The framework consists of a System under Test (SUT), Test Driver, and Test Server. Test Driver is a main test component. Test Service is an auxiliary test component. On the SUT side, the SUT and test service are tightly coupled. If the SUT gets a message from the testbed, test service notifies the next step to the SUT by predefined rules. On the other side (testbed side), the test driver is a component to drive the execution of each step of a test case; test cases are execution instructions that contain the procedures and assertion scripts; and test reports contain test results generated while performing test cases.
The procedure to perform the conformance testing is as follows: first, the test driver generates a request message from a test case and sends it to the SUT. The message is sent directly to the SUT as a standard message (ebXML Messaging Services Specification Technical Committee 2001) through Hypertext Transfer Protocol (HTTP2) or Simple Mail Transfer Protocol (SMTP3). When the SUT receives the request message, the SUT notifies the test service of the received message. Second, based on the content in the received message, the test service requests that the SUT generates an ebXML message response back to the test driver. Third, the test driver verifies the received message by using the assertion scripts of the test case. These procedures continue until all test steps of the test case have been performed. Finally, the test driver generates test results.
II.4.1.2 IIC Interoperability Test Framework
The IIC test framework also provides an interoperability test framework to verify whether two or more candidate SUTs can correctly communicate with each other. Figure II-2 depicts the interoperability test
2
HTTP Hypertext Transfer Protocol, http://www.w3.org/Protocols/ 3
Send Mail Web Site, RFC 821 Simple Mail Transfer Protocol (SMTP), http://www.sendmail.org/rfc/0821.html
harness in the IIC Test Framework. Test Driver Transport Adapter Interoperability Test Case Test Reports Test Service SUT ebMS Message Driver Party Test Service SUT Responder P arty
Figure II-2 IIC Interoperability Test Framework
The framework consists of two SUTs, two test services, and test driver. Each SUT plays different roles: one party, the driver party, initiates the request of the test driver. The other party, the responder party, responds to the messages initiated by the driver party. The test driver is directly connected to the test service at the driver party.
The procedure to perform the interoperability testing is as follows: first, the test driver generates a request message from a test case and sends it to test service of the driver party. The message is sent to the SUT of the responder party through HTTP or SMTP. The SUT of the responder party notifies the test service of the received message. Second, the test service requests that the responder SUT generates a response message and send it to the driver party. Third, the SUT in the driver party receives the response message and notifies the test service. Fourth, the test service notifies the test driver of the received message. Fifth, the test driver verifies the received message by using the assertion scripts of the interoperability test case. These procedures continue until all test steps of the test case have been performed. Finally, the test driver generates test results.
II.4.1.3 Weaknesses of IIC Test Framework for the B2B Standard testing
Though the scope of the IIC Test Framework can be applied to all standard specifications (Durand et al. 2003) (Kulvatunyou et al. 2003), ebXML communities have primarily applied it to test ebMS solutions (KorBIT4, NIST5) because this test framework has some weaknesses as described below:
- Difficult to reuse the test case: The IIC TC already provides ebMS conformance test suite.
4
Korea B2B Interoperability Testbed (KorBIT) Web Site, http://www.korbit.org 5
NIST Information Technology Laboroty Web Site, ebXML Test Framework, http://xsun.sdct.itl.nist.gov/~mkass/ebxml_conformance_testing.
However, for another Simple Object Access Protocol (SOAP6) conformance testing, none of the ebMS test cases can be reused because 1) the test case is directly associated with the testing procedure which includes test steps and verifying assertions; and 2) each test step is directly associated with a specific message packaging method. The test case cannot be reused, for example for the Web Services SOAP test case, even though they have the same messaging steps and verifying rules as ebMS test case.
- Difficult to reuse the test component: The IIC test framework proposed the test driver and the test service as the test components. Except the ebMS testing, the test components should be redesigned because the major parts of functionalities are dependents on that specific standard. Unfortunately, the IIC test framework did not distinguish dependent components and independent components. This makes test components have low reusability. This problem makes standard organizations hesitate to adapt IIC test framework as their common platform for testbed implementations.
- Difficult to configure the test harness: The test harness is a deployment relationships between test components. The IIC test framework defines two test harnesses for the conformance and interoperability testings. However these test harnesses are fixed and test components and test case specification are designed upon these harnesses. It makes test components hard to adapt to a new a test harness.
- Hard to make and understand the test case: The IIC test framework provides a generic executable test case specification, which is independent of B2B standards. In other words, the IIC test case uses a low level language in order to represent all B2B functionality. The needs to understand and specify test cases using this low-level language make it difficult for the domain (standard) experts to develop test cases. These test cases are also difficult to understand for the same reason.
- Low testing efficiency: The IIC test framework provides a test report which only indicated whether each test step is passed or failed without any reason given. Even if there is a temporary network error, the test report outputs the same testing failure message. Therefore, testing user repeats the testing needlessly. Also testing uses do not get enough information from the testbed when they meet an interoperability problem during the test. This is one of the major reasons that a lot of software vendors disregard the B2B interoperability testing.
II.4.2 RosettaNet Self-Test-Kit
RosettaNet is a non-profit consortium aimed at establishing standard processes for the sharing of business information (B2B). RosettaNet is a consortium of major Computer and Consumer Electronics, Electronic Components, Semiconductor Manufacturing, Telecommunications and Logistics companies working to create and implement industry-wide, open e-business process standards. These standards form a common e-business language, aligning processes between supply chain partners on a global basis. The RosettaNet Self-Test-Kit (RNSTK) is a software package supplied by the RosettaNet
6
W3C Web Site, Simple Object Access Protocol, http://www.w3.org/TR/soap/ １０
consortium7. The RNSTK is an application developed by RosettaNet; it is a test system for compliance with the RNIF specification. The purpose of the RNSTK is to provide the ability to validate RNIF interfaces.
II.4.2.1 Test framework of the RosettaNet Self-Test-Kit The RNSTK consists of 8 test components as listed below:
- Controller: Overhead control of the test process. - Inbound: Handles incoming messages.
- Outbound: Handles outgoing messages.
- Repository: Storage for the Part Interchange Process (PIP) specifications and logs. - Results: Handles the result files and display pages.
- Secure message: Handles security issues. - Standards: Holds message standards.
- Validators: Handles validation against PIP and RNIF specification.
Figure II-3 is a high-level diagram of the system where collaboration between functionality in the 8 test components is visualized.
Apache Web Server RNSTK SUT Result Engine Status Engine Config Engine RN Listener RosettaNet Test Controller RNIF Message Result Logs Local Repository
Figure II-3 High-level system diagram of RNSTK
7
RosettaNet Web Site, RosettaNet Official Homepage, http://www.rosettanet.org/Rosettanet/Public/PublicHomePage
The RosettaNet solution that needs to be tested is in this system referred to as the system under test (SUT). The SUT sends business messages and action signals to the RNSTK and these messages and signals are validated by the RNSTK against a given XML-file containing the RNIF verification rules. The templates for the PIPs are stored in a repository on the servers local file system and every message sent to the RNSTK is validated against these templates.
Figure II-4 describes how incoming and outgoing messages are handled by the RNSTK. The process is quite complex and requires a lot of logging and validation. The test begins with the user filling out configuration values and sends them to the server. Then the server starts a listener that waits for a message from the SUT. When the message is received it is processed for unpacking and MIME validation. After this the test system adds references and the message get validated against a RN DTD. When this is done the process is finished and results get saved and the status web page gets updated. The outbound processes are very similar but are in a reverse direction. Each PIP action has its own validation class in the repository that validates the incoming RN message from the SUT. The validation is conducted using the DTD file for the current PIP action and it is tested in a child-parent manner. All results from the validation are stored in a log that is later on presented to the testing user using the status engine (RosettaNet 2004).
Test Handler Configure Receive Incoming MIME Unpack & Validation Apply References (From Standard) Validate Message Update Status Build Component Apply References (From Standard) Send Message Update Status Load Message Package Message (MIME) Outbound Process Inbound Process
Figure II-4 Test flow diagram of RNSTK
II.4.2.2 Weaknesses of RNSTK Test Framework for the B2B Standard testing
Though the scope of the RNSTK Test Framework applies to broad test requirements of RosettaNet B2B interoperability testing (Kjellin 2005), it is difficult to scale it testing of other standards due to the following reasons:
- Specific test suite: The test suite specification of a RNSTK is the RosettaNet specification itself (PIP and RNIF). That is, PIP describes the test procedure and RNIF defines the messages. Also all verification rules come from XML constraints documents of these standards. That is, RNSTK does not have a generic executable test script.
- Low testing efficiency: This weakness is similar to that of the IIC test framework. That is the test report is limited to the passed and failed feedback without further information.
II.4.3 WS-I Test Tools
The Web Services Interoperability organization (WS-I8) organization is dedicated to meeting the needs of Web Services developers, so as to enable them to develop and deploy interoperable web service on the computing platform and development language of their choosing. It is a primary work of WS-I to make profiles, which contains a list of named and versioned of Web Services specifications together with a set of implementation and interoperability guidelines. These profiles and guidelines recommend how the specifications should be used to develop interoperable web services.
WS-I is also developing a set of testing resources (WS-I Test Tools9). These testing tools and the configuration data used with them have helped Web Services developers ensure that their web services conform to the profiles and interoperability guidelines. The tools monitor the interactions with between web services, record those interactions, and analyze them to detect implementation errors. The web service itself is treated as a black box. The testing tools do not interact with the web service, nor do they have any view of the supporting code or infrastructure.
II.4.3.1 Test framework of WS-I Test Tools
The WS-I test tools consists of 2 test components and a test assertion specification as listed below: - Monitor: A tool used to intercept and log interactions with a web service. This tool generates a
log that is later processed by the analyzer to verify that the monitored interactions conform to a profile.
- Analyzer: A tool used to process the logs generated by the Monitor to verify that the intercepted web service interactions conform to the given profile.
- Test assertion: Test assertions are used by the Analyzer to verify that the observed behavior of a given web service conforms to the Profile.
8
WS-I Web Site, Web Service Interoperability Organization, http://www.ws-i.org/ 9
WS-I Web Site, Web Service Testing Tools, http://www.ws-i.org/deliverables/workinggroup.aspx?wg=testingtools
Figure II-5 is a high-level diagram of the system where collaboration between functionality in the 2 test components is visualized.
SUT (Provider) Monitor Analyzer SUT (Requester) Message log Test Assertion Repository Test results Capturing
Figure II-5 High-level system diagram of WS-I Test Tools
Two web service solutions (requester and provider) that need to be tested are in this system referred to as the system under test (SUT). The requester SUT sends to the provider SUT business message, which is captured by the Monitor. Then, this message is validated by the Analyzer against a given test assertion which is a set of verification rules for the target Web Services profile. Next, the repose message from provider is monitored and validated in the same procedure.
II.4.3.2 Weaknesses of WS-I Test Tools for the B2B Standard testing
Though WS-I provides useful tools, the major weakness is associated with its difficulty to adapt as a test framework for automated B2B testing. The details of this and other weaknesses are summarized below (Smythe 2006):
- No component driving the test: WS-I test tools consist of the Monitor and the Analyzer. In order to test a specific situation, testing users should directly operate their SUTs.
- Limited test coverage: This weakness stems from the first one. Since the tool has no control over the SUT, it means that some negative conditions (i.e. error detection) cannot be tested. - Low testing efficiency: This weakness is similar to that of previous test frameworks. That is
the test report is limited to the passed or failed feedback without further information.
II.4.4 Testing Test Control Notation (TTCN-3)
II.4.4.1 Test framework of TTCN-3
TTCN-3 is a language to define test procedures to be used for black-box testing of distributed systems (Grabowski et al. 2000). When stimuli are given to the system under test (SUT), its reactions are observed and compared with the expected ones. On the basis of this comparison, the subsequent test behavior is determined or the test verdict is assigned. If expected and observed responses differ, then a fault has been discovered which is indicated by a test verdict fail. A successful test is indicated by a test verdict pass.
TTCN-3 allows an easy and efficient description of complex distributed test behaviors in terms of sequences, alternatives, loops and parallel stimuli and responses. Stimuli and responses are exchanged at the interfaces of the system under test, which are defined as a collection of ports. The test system can use a number of test components to perform test procedures in parallel. Likewise to the interfaces of the system under test, the interfaces of the test components are described as ports (Baker et al. 2001) (Schieferdecker and Grabowski 2002).
TTCN-3 has a similar look and feel to a typical programming language. However, in addition to the typical programming constructs, it contains all the important features necessary to specify test procedures and campaigns for functional, conformance, interoperability, load and scalability tests like test verdicts, matching mechanisms to compare the reactions of the SUT with the expected range of values, timer handling, distributed test components, ability to specify encoding information, synchronous and asynchronous communication, and monitoring (Schieferdecker and Vassiliou-Gioles 2003).
A TTCN-3 test specification consists of four main parts: - Type definitions for test data structures
- Templates definitions for concrete test data - Function and test case definitions for test behavior - Control definitions for the test case execution
The data type definitions are generated from the corresponding XML schema of the Web Services to be tested. The templates are based on the corresponding data types and the behaviors of the service being tested that consist of sequences of requests and responses. Figure II-6 is a test procedure of the TTCN-3 test frameworks.
SUT Test Compon ents Abstract Test Case Test System Test Components Test Components Test Components Standard Specification 1) Generation of testdata structure A D A P T O R 2) Generation of testdata 3) Generation of Test behavior 4) Compilation to executable tests 5) Adaptor according
to the mapping rules
Figure II-6 Test procedure of the TTCN-3 test framework
An approach towards automated testing of system with TTCN-3 requires therefore the following steps. 1) The structure of the test data is derived from the XML definition.
2) Test data (i.e. the concrete values for test stimuli and observations) is created. 3) Test behavior (i.e. the sequences of test stimuli and observations) is created. 4) The resulting TTCN-3 module is compiled to executable code.
5) The tests are performed using a test adaptor, which follows the mapping rules for test data structure to encode and decode the standard messaging requests and replies.
II.4.4.2 Weaknesses of the TTCN-3 test framework
Though the scope of the TTCN-3 can be applied to all software testing, it has rarely used for B2B testbed implementation because this test framework has the below weaknesses:
- Hard to make and understand the test case: TTCN-3 provides abstract test case representation in which test behaviors are displayed in table and tree representation. However, every TTCN test case should contain all procedural information for testing realization which includes test components behavior, interface, and deployment. Even B2B testing shares a common test behavior, domain experts should define all information in every test case. It makes the process for creating each test case cumbersome and difficult. Also this problem makes TTCN-3 less reusability.
- Program language of test component: TTCN-3 test component looks like a pseudo code rather than component. Defined test component cannot work by itself without real implementation. To execute the testing, tester should develop compiler for TTCN-3. There is no official
complier for TTCN-3.
- No considering B2B/B2C natures: TTCN-3 has developed for general software testing. It means TTCN does not consider particular natures of B2B/B2C system. Especially, message components such as transport (messaging tool) and message store are absence.
II.4.5 Comparison Matrix among the Existing Test Frameworks
According to the characteristics above, the existing test frameworks are compared as shown in Table II-2.
Table II-2 Comparison matrix for conventional and proposed test frameworks
Characteristics IIC 1.0 IIC 1.1 RNSDK WS-I TTCN Proposed
ATF
Test component
scalability Partially Partially Y Y Y Y Test component
seamlessness Partially Partially Y Y N Y Reusability Test suite modularity N Partially Y N N Y Generic executable test suite Y Y N Y Y Y Extensibility Test component reconfigurability N N Y N N Y Power of
expression High High Low Low High High Testing diverse Partially Partially N N Y Y Efficiency
Intelligence of
Test report N N N N N Y
The IIC test framework has a rather low reusability because multiple functionalities are aggregated into a few test components and the test harness is not flexible. RNSDK shows high reusability, but its test scope is very narrow having specific test suites. WS-I testing tools have relatively higher reusability, while low test efficiency. The generic TTCN-3 does not address the reusability and test efficiency.
II.5.
Chapter Summary
This Chapter reviewed and discussed relevant research works, that is, test frameworks for the B2B/B2C testing. IIC, RNSDK, WS-I, and TTCN were surveyed, compared, and analyzed. Most of them are hard to develop test cases because their test suites and test components should be re-development rather than reused. The test suite of existing test frameworks is hard to understand, because it depends on a specific standard and its contents are mixed and less modular in a test case. In addition, the testbeds developed are quite solid. That means internal test components (modules) are fixed into the entire system, so that it is hard to be adapted for diverse test needs. After this survey, we have come to a conclusion that an agile test framework is urgent.
III. TEST SUITE DESIGN FOR AGILE TESTING
III.1. Chapter Overview
The objective of this Chapter is to show the test suite design for the agile testing. The test suite is a collection of test cases that are intended to be used as an input to a testing tool to verify that system under test has some specified set of behaviors and requirements (i.e., the behaviors and requirements listed in its specification). In particular, we have developed the test suite design as double-layer test case.
III.2. Design Factors of Agile Test Suite
Figure III-1 shows a typical test scenario: 1) a domain (standard) expert publishes a standard specification which includes requirements to be tested, 2) a software vender implements a system according to the specification, 3) simultaneously, a test engineer develops test cases to verify a system under test for these requirements, 4) the test engineer registers the test cases into the testing tool, and 5) the testing tool (testbed) executes the tests. However this scenario implies serious problems as following:
- Understanding gap: the software vender and the test engineer could differently understand contexts in the standard specification because it is not written in a formal language (context free or sensitized), but in a natural language. In the case that a system fails a test, the software vender and the test engineer could argue in a circle. It makes test execution consume much time and test analysis difficult.
- Test dependency: an existing test suite has been designed for a specific testing tool. The testing tool is also implemented for a specific standard specification. It means that once a test case is developed, it cannot be reused for any other testing tools and any other standard specifications. It makes test case development consume much time.
- Tightly coupled test information: a test case should include two major contents: test procedure and assertion. However the conventional test suite has been designed with tightly coupled contents. Even though these test cases are working well in a testing tool, it is difficult to understand them for a testing user (generally software vender). It makes both test generation and analysis consume much time.
Figure III-1 Typical test scenario from a standard to testing
In the conventional test suite designs, the problems above hamper an agility of the entire testing. To resolve these problems, we will take into consideration following factors to design a new test suite:
- Double-layer design: as discussed, the understanding gap occurs between a software vendor and a test engineering, both of which have not directly participated in standardization of the specification of interest. This gap may be minimized only when the domain expert who writes the specification intervenes them by means of, for example, providing test suites in a formal language. However, the expert is probably in ignorance of the test suite design and syntax. Therefore, we consider two-layered design of test suites: an abstract layer and an executable layer. The abstract layer is for human users, especially for the domain expert, to argument test suites for his/her own specification (and requirements). The format and syntax for the abstract test suite may vary from specifications or users. On the other hand, the executable layer, after automated transformation from the abstract test suites, will be fed into a testbed. The neutral executable test suite is tightly coupled with (and interpreted by) the testbed only, but in other words independent of the specifications. In a short, this separation resolves the test dependency problem by making the testbed read only one format of (executable) test suites; as well as allows the domain expert to provide his/her own (abstract) test suites, thereby minimizing the understanding gap between the software vendor and the test engineering. - Modular design: the test suite mainly consists of three separated parts – configuration data,
test procedure, and test assertions. The modularity relaxes the test suite to be loosely coupled, thereby allowing the test engineer to independently update each part of a test suite. For example, the test engineer may replace a new test assertion script only for verifying a different test requirement. This design factor will enhance the reusability of existing test suites, as well as easiness to create new test suites.
III.3. Overview of Agile Test Suite Design
In the B2B/B2C testing, a test suite is a collection of test cases. A test suite also contains detailed instructions for each collection of test cases and information on the system configuration to be used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.
The proposed agile test suite consists of an executable test suite and an abstract test suite. The executable test suite is a test suite that is ready to be executed by the testbed. This means that there exists a test harness that is integrated with the suite and such that the test suite and the test harness together can work on a sufficiently detailed level to correctly communicate with the system under test (SUT). On the other hand, the abstract test suite is more readable and understandable by human operators who compose or execute it, so that it has familiar structures and terminologies with domain experts. For example, users who want to test a messaging application such as ebMS or SOAP applications do not consider the Payload contents in the messages to be exchanged between messaging applications, but have interests only to represent the sequence of these messages. In this case, the test suite should focus to represent an order of the messages and header information of each message. In addition, it is better to omit content parts of the messages like using a default payload.
III.3.1 Double-layers of Test Suite
Figure II-1 illustrates the agile test suites and their relationships. Once an organization publishes a standard specification, many applications are developed according to it. Hereby there are three kinds of operators — standard expert, testing user, and test suite developer. The standard expert has knowledge about detailed regulations and constraints of a standard. He/she should establish the test requirement document that declares objectives and purposes of tests. The testing user has an application/document to be tested. In order to consider his/her specifics, the testing user should compose the user requirement document that indicates limitations and constraints of the application/document under test. The user requirement could sometimes contain specific test requirements that are not mentioned in the standard specification, but required to invoke him/her own applicaton/document. The test and user requirements are commonly called as test metadata for the test suite.
Test Assertion
Test Sequence Verification Script
Abstract Test Case
Test Harness Standard Specification
Test Procedure Test Environment
Executable Test Case User Requirement User Environments Transform to composed of’ Implement Composed of
Abstract Test Suite Derived from Derived from
Test Requirement
Executable Test Suite Test Metadata
Referred to
Figure III-2 Agile Test Suites and their relationship
The domain expert makes abstract test cases in reference of the test metadata. The abstract test case could be read and understood by both the domain expert and the testing user, so that it is made up of familiar terminologies and syntaxes with them. Also the abstract test case will be transformed into executable test cases to be used by testbed. Therefore the abstract test suite generally has the following features:
- Dependency on a standard specification and domain environment of concern. - Independency from a particular test execution model.
The abstract test case consists of a test procedure, test assertions, and configuration data for test environment. The test procedure represents a series of test steps; it should contain all possible states and transitions from each state to others. Test assertions are definite statements that could be used as principles by testing to determine whether the test is success or fail. The test environment describes participants and their roles of testing. It also includes specific requirements to execute the test case. The abstract test case minimizes understanding gap between domain experts and software vender because they are described by a formal language with the modular design.
The test engineer implements executable test cases from the abstract test cases. The executable test case could be ready to be executed by testing tools. It means all scripts in the executable test case should be machine-readable. Also the executable test case should reflect specifics of a testing tool used. Therefore the executable test suite generally has the following features:
- Independency from the standard specification and domain environment. - Dependency on the test execution model.
The verification script, inherited from the test assertions in the abstract test case, is a machine-readable script that could be executed as an identifier by the testing tool to determine whether the test is success or fail. Eventually the test case has one or more verification scripts to make a test result. On the other hand, the test sequence, implementing the test procedure of the abstract test case, is directions for the test. That is, the test sequence includes an order of test steps from test initiation to making a final test report. The test sequence has references: test harness and test data. The test harness defines required configurations of testing tools when the test case is executed.
Both of test suites are designed with modular concept. In the abstract test suite, each module like a test assertion, test procedure, and test environment is separated with each other and their relationship is well-defined. In the executable test suite, verification script is encapsulated in the test sequence, so that it could be easily replaced with another script.
III.4. Abstract Test Suite Design
The abstract test suite is a combination of abstract test cases and test documentation. It is used to verify conformance of an implementation to the requirements in the standard specification. Generally the domain expert should compose the abstract test suite because he/she exactly knows purposes and objectives of requirements which the implementation must satisfy. Therefore the abstract test suite should be designed in the viewpoint of domain expert. In addition, it is recommended that the abstract test case is presented for relevant requirement in the standard specification. The abstract test case is primarily drawn out for two reasons:
- To clarify the meaning of a requirement for a software vender who implements it.
- To present for test engineers how to make sure whether an implementation satisfies the requirement.
The abstract test suite design includes three parts of test assertions, test procedure and test environment. These parts could be combined with each other. Each document refers to related requirements of standard and the other document in the abstract test suite.
III.4.1 Test Assertion
The Test Assertion is a formal statement of behavior for an implementation: action or condition that can be measured or tested. It is derived from the specification’s requirements and bridges the gap between the narrative of the specification and the machine-interpretability of test cases. Each test assertion is an independent, complete, testable statement on how to verify that an implementation satisfies a conformance requirement. Typically, it describes the expected behavior of an implementation, when subjected to a particular situation or context. Generally assertions take one of the following forms:
- Bold assertion form: (e.g. a Boolean statement)
- Conditional assertion form: If <condition> : then <bold assertion>
- Cause/effect behavior form: When <cause event> occurs, then <effect event> results.
The bold assertion could be determined for a system under test, while the conditional assertion could be determined only when a condition is satisfied. If no condition, condition assertion can be considered as bold assertion. Namely the condition assertion is a broader concept than the bold assertion. The cause/effect behavior form means a cause event makes outcome effect events. Because all conditions and situations could be explained as an event, all test assertions could be represented as a cause/effect behavior form. Therefore the test assertion consists of cause-and-effect components and each component consists of a condition and an event.
Test assertions are classified according to two factors:
- Is it possible and practical to test the behavior? [T=testable, U=untested] - Is the requirement mandatory or optional? [R=required, O=optional] Assertions are thus classified as following:
- TR: Testable assertion of required feature - UR: Untested assertion of required feature - TO: Testable assertion of optional feature - UO: Untested assertion of optional feature
Figure III-3 shows a design of the test assertion. The test assertion consists of two parts: Header and Cause/Effect form. The five main parts of a test assertion are:
- A Metadata contains a description of the content, the author, the publication date and the physical location.
- A Reference to the specification requirement to which this Test Assertion is related. Section and sub-section numbers should be sufficient, possibly the line number in the specification. An abstract test case may also decide to explicitly give an identifier to each conformance statement, within or outside the body of the specification. Test Assertions would then refer to such identifiers.
- An Assertion Classifier takes one of assertion types: TR/UR/TO/UO.
- A Cause describes the situation in which the test assertion may be verified. More precisely, there are two aspects to a Cause, either of which could be absent, depending on the situation at hand:
• A Cause-condition: The cause-condition is a predicate (e.g. a Boolean statement) that qualifies the context or situation in which the requirement verified by the Test Assertion applies. To verify this test assertion in a context that does not satisfy the cause-condition would not make sense: the outcome would be “non-applicable”. The test assertion does not have to describe how to make this situation occur: this would be the role of a test case, and it may happen that a test environment and a test procedure will have little control on the production of this situation.
• A Cause-event: The cause-event defines an operation or event that causes a change in state of the system under test, which is a prerequisite for verifying this test assertion. The cause-event is usually under control of the test environment and the test procedure. - An Effect describes the expected behavior of an implementation, or the expected state of the
system under test, after the Cause was observed. There are two aspects to an Effect, either of which could be absent, depending on the situation at hand:
• An Effect-condition: The effect-condition is a predicate (e.g. a Boolean statement) that qualifies the context or situation after the condition was satisfied and/or the cause-event occurred. To verify this test assertion in a context that does not satisfy the effect-condition would not make sense: the outcome would be “fail”. If satisfied, it is “success”. • An Effect-event: This is an operation or event that is expected to take place when the
specified Cause is observed, it usually describes the expected reaction of the system under test, when the cause-condition was satisfied and/or the cause-event occurred.
Figure III-3 Design of Test Assertion
III.4.2 Test Procedure
A test procedure is to make a situation that a test tool could verify the system under test. In the test procedure, testbed may interact with SUTs or try to verify the SUT. Figure III-4 is an example of test procedure. The test procedure shows, as parallel vertical lines, different service under test that live simultaneously, and, as horizontal arrows, the messages exchanged between them, in the order in which they occur. This allows the specification of simple runtime scenarios in a graphical manner. A test procedure is defined in three main phases:
- Service definition in the system under test - Message exchanges
- Service Lifetime completion
Figure III-4 Example diagram of Test procedure
The diagram above could be represented in a programmable code. The following calls can be used to define a test procedure. Using calls below, Figure III-4 could be represented in a code style as shown Figure III-5.
- service(name, label): Defines an service with the given name, labeled on the diagram as specified.
- placeholder_service(name): Defines a place where the named service will later be created. - active(service): Changes the service’s status to active, and changes its lifeline drawing style
correspondingly.
- inactive(object): Changes the object's status to inactive, and changes its lifeline drawing style correspondingly.
- message(from_service, to_service, label): Draws a message between two service, with the given label. Self-messages (where a service sends a message to itself) are supported.
- create_message(from_service, to_service, service_label): Has from_service create the to_service, labeled with service_label. The message is labeled with the «create» stereotype. - destroy_message(from_service, to_service): Sends a message labeled with the «destroy»
stereotype from the from_service to the to_service. The service to_service is marked as destroyed, with an X at the end of its lifeline.
- async(): All subsequent messages are asynchronous and will be drawn correspondingly. - sync(): All subsequent messages are synchronous and will be drawn correspondingly.
Figure III-5 Design of Test Procedure
III.4.3 Test Environment
The test environment defines partners (system) presented at a test. Each partner has various services that are used or tested in a test. The test environment also includes specific configuration information
for test harness when it is tested at a testing tool. Figure III-6 shows a design of the test environment. The test environment consists of two parts: Header and body. The environment body could present multiple partners. Each partner has required services. Service defines an interface of messages: in/out message structure.
Figure III-6 Design of Test Environment
III.5. Executable Test Suite Design
The executable test case could be ready to be executed by testing tools. In the agile test framework to be proposed, eTSL (event-driven Test Script Language (Durand 2007)) is used as an executable test suite design. The eTSL execution model is based on the concepts of event and workflow. The main notions are described below, some of which are inherited from the IIC TestFramework.
- Monitor. A monitor is a single-threaded workflow script controlling the execution of test steps, with event capabilities (listening and generating). A test case is represented by one or more monitors that may execute and synchronize concurrently.
- Trigger. A trigger is a control of a test case execution which defines under which conditions and events, a monitor execution takes place. A trigger reacts either to events in the event board (event–watching) or to a scheduled date/time (clock-watching). The general execution flow and timing of several test cases is then controlled by triggers, the set of which represents a test suite.
- Event Board. Event management, central to eTSL, is done by the event board, which supports event logging, search, and correlation. The event board normally suffices for mediating all test case inputs as well as outputs.
- Event-Adapters. The message traffic, expected in e-business applications, is mapped to and from events. The event-adapters perform these mappings, allowing for abstraction of test cases from specific communication protocol details.
- Evaluation-Adapters. eTSL is designed to delegate some test result evaluations to external modules because different protocols need different evaluation engines. For example, evaluating a messaging protocol typically requires a simple pattern matching. Conversely, evaluating a business document or business process may need a ruled-based or reasoning engine.
eTSL underlying design principles are:
- XML-centric. The test script and expression rely on XML syntaxes and related technologies such as Xpath10 and XQuery11. This allows the scripting to work naturally with today’s specifications, many of which are XML-based.
- Simplicity. The goal is to make eTSL implementations easy to validate. For example, a minimal set of control features sufficient for e-business testing has been selected, rather than replicating the full range of conventional workflow operators.
- Event-centric. Every type of event - from low-level protocol signals (e.g. Acknowledgements) to business documents - may be captured by the event board, and wrapped into a standard XML event envelope, as opposed to just business messages as in the IIC test framework. Capturing all traffics in the form of events allows for using the same test script in either a deferred or live validation context. The coordination of test case executions within a test suite is also event-driven, e.g. a trigger may start a new test case by watching for the final status event of the previous test case. Event orientation provides the necessary flexibility for interrupting or recovering a test suite execution - especially during a long e-business monitoring session – in contrast with a conventional workflow approach.
- E-business protocol agnostic: Test script logic and control are abstracted from e-business protocols which makes it versatile for messaging, business process and business content testing regardless of technologies (e.g., applicable to either ebXML and/or other Web Services message (SOAP) profiles).
The general architecture and execution model for eTSL script execution is illustrated in Figure III-7. Generally, the test case starts its execution via a trigger that in turn initiates the (root) monitor. The trigger is designed to listen to a particular event (e.g. a time point or a message) when activated. The monitor proceeds through test steps by looking for events of interest on the event board. The event board gets events from both the monitor (internally generated) and the event adapter (intercepted communication traffic). The monitor may start child monitors synchronously or asynchronously to
10
XML Path Language (XPath), http://www.w3.org/TR/xpath 11
XQuery 1.0: An XML Query Language, http://www.w3.org/TR/xquery/ ２９
look for related events. After all child and root monitors exit, the latter generates a test result.
Figure III-7 General architecture and execution model of eTSL
III.5.1 Scripting Primitives
Triggers and monitors are scripted in XML. Each of the test steps executed by a monitor workflow is an atomic unit of work. Eleven types of test steps are defined. They are briefly described below. Event operations:
- post: generate an event from testing or monitoring material (e.g. business message) and post to the Event Board.
- find: select event(s) from the Event Board. The selector component is based on XPath expressions, though allows other XML processing tools (e.g. XQuery). The operation may wait for the event(s), acting as a synchronizing control.
- mask: mask or unmask some past events to a monitor instance. Monitor flow control:
- start: start a new instance of a monitor, either synchronously to the current instance or asynchronously (concurrent to the current instance).
- set: assign a value or an XML infoset (e.g. a business document or a message header) to a variable or to a constant.
- sleep : the monitor execution is suspended either until a specified date or for a specified duration.
- cad: check-and-do operation that verifies a predicate on the result of one or more previous test steps (e.g. status variable) and does a single operation in case the verification is positive – e.g. exit the test case. It allows for optional branching to another labelled step after the “do” part. - jump: pursue the execution thread at another (labelled) test step in the monitor.
External resources:
- call: invoke either an adapter or an evaluation-adapter. The invocation of an event-adapter will result in a message generation/sending from an event or from input argument(s). Invoking an evaluation-adapter will produce an XML infoset e.g. for reporting on a document validation.
