Methodology

6.3.3.1 Modelling the Architecture

A key challenge facing PANDArch evaluation is the lack of open source Java projects having well- documented, up to date and sufficiently detailed architectures useful for checking conformance. Due to this reason, an architecture recovery approach is taken to build an architecture specification of the subject software using a commercial product named Structure101 [68]. This tool is capable of generating a component view of a Java application by statically analysing its bytecode. The tool is primarily targeted towards detecting undesirable dependencies such as cyclic references among components in the implemented architecture. However, Structure101 is also useful for identifying the implemented component architecture, i.e. component implementations and their interactions as they appear in the compiled Java code. The steps involved in extracting the component architecture of a Java application using Structure101 and adapting this architecture for checking conformance with PANDArch are explained next.

1. Initialise the static analyser. Provide the compiled Java code of the target application to Structure101 to begin its static analysis process. This can be in the form of a single jar file, a folder containing a number of jar files, or the folder hierarchy of the application class files. 2. Analyse the Java bytecode. Set up Structure101 to perform a detailed analysis of the Java

bytecode that also includes exploring method invocations among Java types. Although the tool is capable of detecting only static (i.e. non-reflective) method invocations, they are helpful for understanding runtime interactions among components.

3. Demarcate implementation artefacts for architecture recovery. Select the root names- pace of the Java implementation for which conformance checking should be performed. A typical non-trivial Java application may have a number of top-level namespaces that map to external libraries, test harnesses, etc. These are generally not part of the software architecture of the application and therefore should be omitted when checking for conformance.

4. Recover static architecture structures. Request Structure101 to generate a component view of the architecture for the selected namespace. The tool uses package, type and method relationships it discovers in Java class files to build this view. This architecture view serves as the basis for building a corresponding Grasp architecture specification of the software.

6.3. TESTING CONFORMANCE DETECTION AND PERFORMANCE 111

5. Recover architecture relationships. Request Structure101 to generate the dependency matrix for components identified in the previous step. A dependency matrix produced by this tool shows various relationships among a set of components in the implementation. These relationships are derived from the underlying classes in each component. For example a class in componentAreferring to a class in componentBis shown as a “references” relationship

fromA to B in the dependency matrix. In this case A is considered to be dependent on

B. Similarly, if a method in B invokes a method in A, the dependency matrix shows this as a “calls” relationship fromB toA. Besides these relationships, the two components in this example may simultaneously have other relationships between them. Each of these relationships can either makeAdependent onBor vice versa.

6. Describe the recovered architecture in Grasp. Use the component view and the de- pendency matrix produced by Structure101 to manually construct a Grasp model of the implemented architecture. While every component in the recovered architecture is included in the Grasp specification, the “calls” relationships in the dependency matrix are modelled as

runtime interactions usinglinkstatements in Grasp. Alink element in Grasp connects a

requiresinterface of a component to a matchingprovidesinterface of another component. The recovered component structure, however, lacks sufficient information to reliably deduce the two interfaces at either end of an interaction. Therefore, to overcome this limitation, every interaction is modelled as a generic, non-interface method call. Grasp supports this

scenario by equipping every component with an intrinsicoutinterface to model outgoing

calls and an equivalentin interface to model incoming calls. Moreover, components in

real-world software do not always communicate through strict interfaces, and PANDArch has been designed to accommodate these practices.

Other component relationships, such as “refers to” dependencies, are not modelled in Grasp. 7. Map components in the Grasp architecture specification to Java code structures. The

component view generated by Structure101 shows the Java package names from which the tool inferred those components. This information is used to associate the components in the Grasp model back to the Java implementation.

8. Set up directives for the conformance checker within the Grasp specification. The conformance checker is provided with directives such as namespaces that should be excluded from the evaluation process. These directives embedded within the Grasp specification along with implementation mappings. The required directives are decided upon by manually examining the the component view generated by Structure101.

6.3.3.2 Inducing Architecture Violations

The notion of architecture conformance checking centres around the ability to compare software implementations against their intended architectures. As described in the previous section, architectures employed for evaluating the framework are implemented architectures derived from the Java bytecode of their corresponding applications. Conformance in this case is guaranteed, since the extracted architecture reflects the implementation. On the other hand, the implemented architecture of mature software often differs from the intended architecture of the software, which leads to architecture erosion. Nevertheless, in the absence of accurate architecture documentation or input from an authoritative source, building a useful Grasp model of the intended architecture is practically not feasible.

In light of this limitation, the approach taken to evaluate the framework against the chosen open source software is to artificially implant conformance violations. Two methods are employed in this regard. In both cases, the baseline architecture is the implemented architecture recovered from the code at the beginning of the experiment.

1. Make changes in the Java code to deliberately breach the architecture. Selected points in the Java implementation are altered in a manner that breaks conformance with the baseline architecture. Particular attention is given to making code changes that result in conformance violations detectable only at runtime. However, due to the lack of detailed design documentation, these modifications purposely do not alter the functionality of the software in order to avoid introducing defects.

2. Change the architecture to intentionally diverge from the Java implementation. The baseline architecture is altered to cause a discrepancy with the implementation.

Evaluation results are discussed under both these categories for each application tested with the framework.

6.3.3.3 Evaluating Performance Impact

To evaluate the effects of the framework on their performance, the applications are executed in three modes.

1. Application executes outwith the framework.In this scenario the test cases are executed without coupling the application to the PANDArch framework. Hence the application executes under normal conditions.

6.3. TESTING CONFORMANCE DETECTION AND PERFORMANCE 113

2. Application executes with the framework plugged in, which has been configured to use only the JDI.The tests are performed with the application coupled with the framework. In this case, the framework is configured listen to runtime events through the JDI alone. 3. Application executes with the framework plugged in, which has been configured to use

both probes and the JDI.This scenario is almost identical to the above, except that the framework is set up to use both instrumentation probes and the JDI. In this mode, the probes

supply the Java monitor components with interface-load, class-load, and method-entry

events, while rest of the events come from the JDI.

In all three cases, performance measurements are presented in both tabular and graphical forms to understand the variation of performance impact with the number of architecture violations, as well as illustrate the difference between the two Java event gathering mechanisms.

6.3.3.4 Verifying Non-intrusiveness

Testing whether PANDArch satisfies the non-intrusive principle is included in this evaluation. While the source code of the subject application is not made available to the framework, the compiled Java classes are tested whether they have been modified during conformance checking by comparing the MD5 digests (i.e. hashes) [156] of the JAR packages taken before and after the experiments. These digests are reported with the rest of the results.