Results of Deployment Evaluation

To measure the performance of deployment, the harness of the SPEC JVM98 benchmark suite has been extended to alternately deploy and undeploy an aspect. The modified benchmark harness executes in the scheme presented in Figure 5.1, where the number n of iterations in each phase and the number m of complete cycles are chosen such that the benchmark application respec- tively the execution environment’s deployment subsystem reach a steady- state.

The prototype of envelope-based weaving based on Class Redefinition, presented in Section 2.4 is included in this evaluation. It is called Envelope- Unaware and HotSpot 1.5.0 is its reference execution environment.

The extended benchmark measures different properties of the execution environments. First, the time needed for the very first run during initial phase (step 1) is measured. During this run class loading and, if applicable to the AOP implementation, preparation of loaded classes takes place. Further, the time required to perform deployment and undeployment of an aspect, i.e., steps 2 and 4 is measured2_{. The aspect deployed by the harness is presented in}

AspectJ syntax in Listing 5.1. It advises all calls to methods or constructors within any class in any sub-package of spec.benchmarks. The advice simply increments a counter.

The median of the times for the n benchmark iterations in the initial- phase determines the steady-state performance of the execution environment when no aspects are deployed. The median is used to rule out start-up

2_{To abstract from differences in how to deploy an aspect in different AOP environments,}

the extended benchmark harness defines an interfaceDeployer with the methodsdeploy andundeploy. This interface is implemented for each AOP environment.

Evaluating Optimized Dynamic Deployment

1 aspect Aspect { 2 static long counter; 3 before() : 4 call(∗ spec.benchmarks..∗.∗(..)) || 5 call(∗ spec.benchmarks..∗.new(..)) { 6 counter++; 7 } 8 }

Listing 5.1: Aspect used by the modified SPEC JVM98 benchmark suite.

performance as well as to level out peak runs. The median of the performance figures in the deployed-phase shows how much performance is influenced by advice calls. The median of the figures from the undeployed phase shows whether an aspect that was once deployed is removed completely from the application at undeployment or whether it leaves some footprint.

The invalidation mechanism used to undo inlining of envelopes in this thesis, ensures that invalid code is not executed after deployment. However, re-compilation does not necessarily occur at deployment, but only when the methods whose machine code has been invalidated are executed the next time—similar to the application start-up. To measure the effect of deferred re-compilation, the performance of the first benchmark application run after deployment, respectively undeployment is measured and compared to the steady-state performance in this phase.

For the performance values of deployment, undeployment, and start-up after deployment respectively undeployment, the median of the measured times of all m cycles is calculated.

Tables 5.2 through 5.5 show the results of the benchmarks for the ap- proaches discussed in Section 5.2.1 and the Envelope-Aware prototype with lazy envelopes and OSR.

Table 5.2 shows how many milliseconds are spent for deploying and un- deploying the aspect in the modified SPEC JVM98 benchmark harness, pre- sented above. For every benchmark application Envelope-Aware provides the best performance with an average of 3 ms and a maximum of 16 ms. While Steamloom₁ performs well for some benchmark applications, the deployment time goes as high as 4,977 milliseconds and averages out at 154 ms for un- deployment and 941 ms for deployment. With AspectWerkz the time for undeploying the aspect even goes up to 20,230 ms in the jess application.

Evaluating Optimized Dynamic Deployment

Approach Envelope- Envelope- Aspect- JAsCo Steam-

Aware Unaware Werkz loom₁

Operation depl. und. depl. und. depl. und. depl. und. depl. und. compress 1 0 63 126 349 442 331 – 102 3 jess 4 2 538 635 16,833 20,230 3,042 – 1,211 188 db 0 0 19 103 215 195 251 – 4 2 javac 14 16 672 1,152 – – 4,472 – 4,977 805 mpegaudio 2 1 109 236 869 864 1,228 – 55 17 mtrt 2 1 58 152 3,784 4,247 729 – 195 53 jack 1 1 149 380 1,470 1,476 1,446 – 45 16 average 3 3 229 397 3,360 3,922 2,685 – 941 154

Table 5.2: Time in milliseconds for deploying and undeploying the benchmark aspect on SPEC JVM98 benchmarks.

For JAsCo, no undeployment times are presented because it failed during undeployment. As a result, also only one deployment was performed and the time required for this is reported in the table. This also made it impossible to determine the steady-state performance of JAsCo’s deployment subsys- tem. Consequently, JAsCo’s deployment figures may actually be better than what is presented here. However, the figures cannot be expected to change significantly in the steady-state.

Table 5.3 shows the steady-state performance of the different execution environments. For each AOP implementation the overheads in the initial (ini.), deployed (depl.) and undeployed (und.) phase are shown as compared to their reference virtual machine.

The Envelope-Aware prototype imposes no overhead to the steady-state performance of an application when no aspects are used (initial phase). JAsCo also provides a good steady-state performance in the initial phase with only 1.7% overhead. For the other approaches the overhead in the ini- tial phase reaches from 8.6% (Steamloom₁) to 121% (AspectWerkz).

Like in the initial phase, Envelope-Aware imposes no overhead to the undeployed phase; for AspectWerkz the overhead drops from 121% in the initial phase to 58.5%. All other approaches perform slightly worse in the undeployed phase than in the intitial phase. For JAsCo no discussion of the performance in the undeployed phase can be given here since it crashed during undeployment in the benchmark.

Evaluating Control Flow Quantification

Please note that the overhead presented for the deployed phase also con- tains the overhead for executing the additional advice functionality. The approaches have different schemes of invoking advice, ranging from calling a static method to looking-up a receiver object and calling a virtual method. Consequently, the overhead of different execution environments imposed in the deployed phase is not directly comparable and, thus, not discussed here. AOP implementations may also imply a degradation of the start-up per- formance. As presented in Table 5.4, the Envelope-Aware prototype on av- erage performs start-up up to 5.6% slower than the unmodified Jikes RVM. The impact of the other benchmarked approaches on start-up performance, though, is much higher. Of these approaches, Steamloom₁’s impact is the smallest with 8.9%; AspectWerkz even imposes 215.9% overhead.

When an aspect is deployed, affected methods are invalidated and re- compiled the next time they are invoked. As a consequence, during the first run after deployment just-in-time compilation is performed, similar to the very first run. Table 5.5 shows the performance of the first run after de- ployment and undeployment compared to the steady-state performance of the deployed respectively undeployed phase. Envelope-Aware performs fairly well with an average overhead of 51.5% after deployment and 56.4% after un- deployment. The Envelope-Unaware prototype performs best. Steamloom₁ performs roughly the same as Envelope-Aware after deployment and better after undeployment. AspectWerkz and JAsCo perform worse than Envelope- Aware.