The tools used by performance analysts range from load generators for supplying the workload to a system under test, to monitors for gathering data as the system executes (Woodside et al., 2007). Monitoring itself can be performed through hardware, software or a combination of the two. Monitoring can be broken down into two broad categories:
Instrumentation –
it is the insertion of probes into a system to measure some sort of events. Some instrumentation is usually built into the host operating system and minimally indicates the utilization of the various devices including the CPU. Other instrumentations are added manually to applications. Frameworks such as the Application Response Measurement (ARM) (Johnson, M. W., 1998) are beneficial as they form a common platform to which disparate programs can gather performance information. Instrumentation can be added automatically as well. Aspect-Oriented programming can be used to automatically insert instrumentation code into applications (Debusmann et al., 2003). Quantify (IBM, 2002) adds probes at the beginning and end of basic blocks of object code to count the number of cycles executed. The Paradyn tool (Merson et al., 2005) carries this one step further by instrumenting the actual executables dynamically. Monitoring introduces system overheads. However, to reduce the overhead of monitoring system requests, two orthogonal concepts exist: (i) quantitativethrottling: throttling the number of requests that are actually monitored (Gilly et al., 2009) and (ii) qualitative throttling: throttling the level of details for the requests that are being monitored (Ehlers et
34
Developers and testers use instrumentation tools to help them find problems with systems. However, users depend on experience to use the results. Better methods and tools are a future requirement for interpreting the results and diagnosing performance problems (Woodside et al., 2007). This requirement is one of the aspects addressed by this thesis.
Profiling –
A program profile is a histogram of the execution of a program (Knuth, D. E., 1971). Profiles can be generated using instrumentation through statistical sampling or by running the program on a virtual machine and counting the execution of the actual instructions (Nethercote et al., 2003). One key problem with these performance tools is that these are not well established at earlier stages in the software life cycle. Some of the obstacles to the adoption of these tools are (Malony et al., 2001):a) A lack of theoretical justification for the methods for improving performance that work and why they do so. The measurement data collected requires expert interpretation to fix the problems.
b) A conflict between automation and adaptability. Systems which are highly automated but are difficult to change and vice versa. As a result no tool does the job the user needs. So the user invents one. Also, various tools have various forms of output which makes interoperability quite challenging.
This thesis attempts to alleviate the problem of leaving the collected measurement data to subsequent expert interpretation by attempting to formulate an approach applicable during the implementation phase of the development life cycle.
A combination of model configuration, automated measurements, statistical inference and model based performance prediction in order to support performance evaluations during the evolution of the software system is proposed in (Westermann et al., 2010). The run-time monitoring approach uses systematic measurements to build mathematical models or models obtained with genetic optimization. The models serve as interpolation of the measurements. The main idea of the approach is to abstract from system internals by applying a combination of systematic goal-oriented measurements, statistical model inference and model integration. The measured functional dependencies are integrated in the PCM. However, this approach intrinsically has a few drawbacks. One of the major problems in the area of software system measurement is a lack of discussing the meaning of the numbers produced by measures and the inappropriate statistical operations on those systematic measures (Zuse, 1998).
35
Intuitive interpretation of the measure values is applied. If there do not exist more information about how the numbers are produced, it is difficult to interpret them. Behind every software measure (resource, process or product measures), a qualitative model is hidden. Doing software measurement we have to quantify qualitative properties of objects in reality. Behind every measure, models of qualitative attributes are hidden (Zuse, 1998). In a software application it can be quite challenging to ascertain how the various processing activities are contributing to the measured systematic data. It takes a lot of effort, time and most importantly the right technical insight to identify these hidden attributes. The approach does not consider the internal structure of the underlying system but focuses on the observable data. A “black-box” specification of performance characteristics is used i.e. the performance of the system is captured by a function of its usage. These black-box performance models do not contain any information about the system‟s internal structure. The other significant limitation of this approach is that it integrates into the PCM. Depending on the approach, it may not follow an industry standard and therefore widespread use may be a long term goal. Existing UML tools cannot be used to create PCM instances. The learning curve for developers familiar with UML is potentially higher. Existing UML models are not supported by the PCM analysis tools and have to be transformed to PCM instances to carry out performance predictions. Implementing such transformations is complicated (Becker et al., 2009).