Methods of Obtaining Program Measurements

There have been several different methods or combination of methods used to collect program measurements. The factors that may influence the choice of a method include:

1. implementation effort 2. accuracy of measurements 3. types of measurements required

4. computational requirements for obtaining measurements

Several techniques for collecting program measurements are described below.

Program simulation imitates a machine by interpreting the machine instructions [AlW75, BSG77].

Dynamic architectural-level measurements are updated as the simulator interprets each instruction. The efforts to construct a simulator and a compiler are comparable. The total execution time of a simulated program is typically hundreds of times slower than if the program is executed directly [HLT87]. The main advantage of program simulation is its portability. Since a simulator for a specific architecture can be written in a high-level language, it can be executed on any machine that supports that language. Thus, measurements can be obtained for a machine that does not yet exist.

Program tracing records a sequence of events which occur during the execution of a program.

These events may include the sequence of instructions executed and the memory addresses referenced.

Program tracing can be accomplished by several mechanisms and is sometimes available as an option on a machine. One way to implement tracing is to have the machine interrupted after the execution of each instruction [DDD81]. The interrupt handler then invokes a user-defined handler that records the trace information associated with the instruction, which can produce large volumes of output. A filter, a pro-gram that reads input data, performs some operation on that input, and writes output data, can process the trace output to obtain the program measurements. The execution of a program with tracing enabled may run 1000 times slower than one without tracing [HLT87]. An advantage of using tracing is the ability to collect dynamic architectural-level measurements that require knowledge of the order of the execution of instructions [Lun77, PeS77, Wie82].

Program sampling periodically interrupts a program for a very short interval of time in order to

inspect its state. Measurements are collected during each of these intervals. Sampling incurs little over-head since the program being measured is only interrupted for a small percentage of its execution time.

Sampling measurements typically vary with different executions of the same program and thus can only be used as estimates of complete dynamic measurements. If the state of the program is sampled more frequently, the accuracy of the measurements improve and the overhead of obtaining measurements increases. Profiling is the dynamic measurement of the frequency or time spent at each place in a specific program. The profiling facility of UNIX uses sampling to estimate the execution time required by each routine in a program [GKM82].

Program monitoring collects dynamic architectural-level program measurements without

perturba-tion of a machine. This method is only possible with a hardware device known as a hardware monitor.

The frequency and time taken by each instruction can be measured with this method [ClL82]. The advantage of this method is that program measurements can be collected with no overhead in execution time. The disadvantages include limited measurements and the requirement of a specialized, expensive, hardware device.

Program analysis, a simple and fast method, extracts static measurements directly from the text of

a program. Filters can easily be constructed by modifying the parser of a compiler or an assembler to obtain the measurements. Since this method just collects static measurements, more realistic information on performance can only be obtained using other methods that collect dynamic measurements [Kat83].

This method has been used to extract static measurements from both source and machine code [Coo89, Dit80, Knu71, SwS82].

Program instrumentation modifies a program to increment counters during execution. Static

infor-mation, collected by program analysis, is associated with each counter. After the termination of the exe-cution of the program, each set of static information is weighted by the value of its associated counter to produce the dynamic measurements. This method has been used to modify the source code of a program to obtain dynamic source-level measurements [Knu71]. It has also been used to modify assembly or machine code to obtain both dynamic source-level measurements, if optimizations are not applied across source-level statement boundaries [Tan78], and dynamic architectural-level measurements [HLT87].

Modification of programs has been accomplished through the use of filters or as an option in compilation.

Since a basic block has only one entry and one exit point, the instructions within a basic block are always executed the same number of times. Counting the number of times that each basic block is executed to measure the frequency of executed instructions was proposed by Weinberger [Wei84]. Modification cap-tures program measurements with little overhead in execution time. One must ensure, however, that the modifications made to collect measurements do not change the execution behavior of the program.

An emulator has been defined to be a complete set of microprograms which, when embedded in a control store, define a machine [Ros69]. Emulator instrumentation modifies the microcode of a machine to update dynamic architectural-level measurements after the execution of each macroinstruction. This method does not require much runtime overhead as compared to tracing or simulation. For example, modification to collect instruction frequency information slowed a Mesa emulator by about a factor of six [McD82]. Another problem is that the microcode of a machine is not always accessible to one desiring to collect program measurements. Even if it can be accessed, microcode is also difficult to modify and maintain which would discourage experimentation.