Conformance, Interoperability and Certification

Standard based solutions have to be tested to ensure interoperability. The most commonly used testing mechanisms to ensure interoperability are: 1) conformance testing; and 2) interoperability testing (Kindrick et al 1996; Ray 2009; Shah and Kesan 2008).

Kindrick et al (1996) summarise advantages and disadvantages of conformance testing as well as interoperability testing. The ideal scenario is to perform both kinds of tests, however it is impractical in most cases due to associated cost.

The following sub-sections give a detailed overview of conformance testing and interoperability testing.

2.7.1 Conformance Testing

Any consumer knows that just because a product passes a test or conforms to a standard does not mean that the product will satisfy his/her needs.

Frenkel 1990 (cited in Arnold 1994)

For Arnold (1994) ‘conformance’ means “that an implementation produces effects which the standard permits.” Conformance testing ensures whether a product meets all requirements of a standard and it behaves as expected by a standard (Arnold 1994;

APEC-SCSC 2010, p.8). Kindrick et al (1996) give another definition to conformance testing as “the assessment of a product to determine if it behaves as expected while interoperating with a reference system.” We prefer the latter definition as it involves comparison of a product with a reference system in addition to testing against the requirements of the standard.

The history of conformance testing goes back to the early 1970s, for testing computer programming languages (Ainsworth 1989). Ainsworth (1989), Asano (1990) and Kindrick et al (1996) have all stressed the significance of conformance testing for achieving interoperability. Benefits of conformance testing include (Kindrick et al 1996; Arnold 1994):

 early detection of interoperability problems in a product’s life-cycle;

 the use of carefully developed test suites that maximise the coverage while minimising redundant tests;

 the strong connection with standards results in faster problem resolution and feedback to the standard development;

 promotion of confidence and momentum for product development in early stages of the standard’s life cycle;

 provision of widely available stable working references in the middle-to-latter stages of standard’s life cycle;

 provision of recognisable status for products;

 easy recognition of problems of products, due to focus on one implementation at a time;

 need to perform limited number of tests;

 consumers get a known set of functionalities, implemented and tested for conformance;

 make the functionalities independent from the suppliers or products.

Conformance test suites are being used to evaluate implementations and they usually consist of a set of test cases specifying precise objectives, operating conditions, inputs and expected outputs (Kindrick et al 1996; Shah and Kesan 2009). However, conformance test suite development is one of the most challenging tasks as they involve a significant budget, technical expertise and administrative overheads (Ainsworth 1989;

Arnold 1994; Kindrick et al 1996; Shah and Kesan 2009). Therefore the development of own conformance test suites might be uneconomical and impractical for individual organisations (Ainsworth 1989). Realising these issues, there are initiatives to pool resources in order to develop conformance testing suites.

Garguilo et al (2007) discuss the importance of accurately defined conformance criteria as well as implementation statements of conformance in standards. A product that shows a successful level of conformance to a published standard implies successful interoperability among other products (Garguilo et al 2007). However, Arnold (1994) presents a contradictory view and mentions conformance testing does not guarantee interoperability. It is likely that non-conforming products will not interoperate effectively (Asano 1990). Shah and Kesan (2009; 2008) assert that “the best method to enhance interoperability is conformance testing.” Unlike interoperability testing, conformance testing is a “one-off” operation for each product (Ainsworth 1989; Asano 1990). Therefore, needless to say, the conformance testing is economical.

Ainsworth (1989) as well as Arnold (1994) highlight that the availability of recognised conformance testing suites have significant positive impact on the conformance level of products of a market. Kindrick et al (1996) mention that the availability of common testing environments and tools results in better products and reduced development costs.

Asano (1990) gives examples of successful scenarios that have proven the effectiveness of making available conformance testing suites.

When conformance testing suites are published, vendors can buy access to those test suites and carry out their own tests prior to product certification or launch (Asano 1990;

Kindrick et al 1996). Moreover, vendors themselves can apply for establishment of accredited conformance testing laboratories.

2.7.2 Interoperability Testing

Kindrick et al (1996) define interoperability testing as “The assessment of a product to determine if it behaves as expected while interoperating with another product.” Unlike conformance testing, interoperability testing does not attempt to cover all aspects of a standard. Aside from all the benefits of conformance testing, interoperability testing also has its own advantages. As mentioned above, interoperability testing is not confined to the requirements of standards. It goes beyond the standards to assess the implementations of important interoperability requirements of users. Therefore, interoperability testing helps to review the standards as well as to improve the standards by identifying important interoperability requirements that are not covered by standards (Kindrick et al 1996; Anastasiou 2011).

In contrast to conformance testing, interoperability testing takes every possible scenario into consideration and testing has to be repeated among every two products (Shah and Kesan 2008). Therefore the associated testing cost very is high compared to conformance testing (Ainsworth 1989).

Interoperability tests help to identify drawbacks and the effectiveness of standards. Test results can feedback into the further development of standards (Asano 1990). Errors of

the testing methods have to be identified and corrected as otherwise it will hinder the interoperability (Asano 1990).

In the following, we review an empirical methodology proposed to evaluate the interoperability of document formats. It is highly important in the context of this research due to the similar objectives of both. It provides us with valuable insights into a systematic methodology towards evaluating the interoperability of standards in the localisation domain.

A Framework to Evaluate Interoperability of Document Formats

The interoperability issues between document format standards such as Office Open XML (OOXML) and Open Document Format (ODF) have been widely discussed. Shah and Kesan (2009; 2008) propose a systematic methodology for assessing the level of interoperability among different software implementations of open document formats, particularly the ODF and OOXML formats. They assert the necessity for improved interoperability testing in order to prevent vendor lock-in. The methodology involves comparing different implementations against a chosen reference implementation. Test suites have been prepared to assess the level of tool support for the most frequently used features. Then, a scoring mechanism has been proposed to assess tools’ ability to: 1) read the documents; 2) read and write; and 3) handle metadata. The research presents significant findings related to interoperability issues and lack of solutions for the selected document format standards.

According to this research, a valid reason for tool developers not to offer full interoperability is the lack of need to support all the features in their tools. However, Shah and Kesan state that users expect 100% interoperability among implementations for various reasons including actual requirements as well as avoiding potential problems. In this research, they only focus on frequently used features of the standard as their goal is to assess whether implementations are "good enough for most users.”

These routinely used features have been identified by “examining various instructional

materials for using office productivity software” (Shah and Kesan 2008). Unfortunately, the details of their procedure for identifying these features are not given.

2.7.3 Certification

Conformance testing provides a basis for product certification (Ainsworth 1989).

Product certification by an approved testing organisation will increase the reliability of the product (Asano 1990) and confidence that ‘approved’ products will interoperate (Ainsworth 1989). However, Arnold (1994) warns that “a false sense of security arising from a blind expectation that a certificate of conformance guarantees a particular level of performance is equally dangerous.”

Although certification can be carried out by any organisation, in order to make the certification process meaningful the certifying organisation should possess certain characteristics (Asano 1990; Ainsworth 1989):

1) technical capability to carry out tests and reporting capability;

2) maintain quality of the testing procedures;

3) wide recognition (international recognition);

4) in case of multiple laboratories, the use of the same tools and procedures;

5) trustworthiness and unbiasedness.

ISO/IEC GUIDE 2 (ISO/IEC 2004) defines accreditation of a test laboratory as “the formal recognition that a test laboratory is competent to carry out specific tests or specific types of test.” Accreditation is the process that assures credibility of a testing laboratory. It ensures the above characteristics of a testing laboratory are at a satisfactory level. Usually, accreditation is carried out by the main operating body of a system or a third party with adequate authority (Asano 1990). The main concern of accreditation is on administrative and procedural issues associated with certifying organisations or testing laboratories and to ensure that they are consistent and unbiased with regards to the clients, tools and procedures (Ainsworth 1989). Accreditation is awarded after successful assessment and surveillance of a laboratory (Asano 1990).

Ainsworth (1989) discusses two special considerations that need to be taken into account when certifying products: 1) whether implementation needs to pass all the tests (including other concerns like who decides the number of tests to be passed); and 2) time to carry out retesting (e.g. due to product updates, new releases).

In the next section, we review literature relevant to the localisation domain. This is the third major area considered in our literature review and certainly the most important area of our research.