We use the TimeTrial performance monitor to make empirical measurements on the execut-ing system. To calibrate the queuexecut-ing model, we use TimeTrial to measure the input arrival rate (λin), the service rates of stages 1a, 1b, and 2 (µ1a, µ1b, and µ2), and the branching probabilities (p1a1b, p1b2, and p23). To validate the queuing model, we use TimeTrial to measure the server utilizations of stages 1a, 1b, and 2 (ρ1a, ρ1b, and ρ2) and the queue
occupancies of the queues into stages 1b and 2 (NQ,1b and NQ,2). To complete the valida-tion, we compare the empirically measured server utilizations and queue occupancies to the predictions made by the model.
Each of the above measurements is made for 2 distinct test cases. The runs are as follows:
• Run 1: The first data set is the human chromosome 22 (from build 19 of the human genome) divided into 1,134 65,400-base segments as the query. The database consists of the 9th build of the mouse genome (2.7 GBases).
• Run 2: The second data set consists of comparing all the non-mammal vertebrate mRNA split into 8,608 65,400-base segments as the query. The queries were searched against all the mammal mRNA in the NCBI RefSeq repository (791 Mbases) as the database.
Table 7.1 gives the model inputs, while Table 7.2 provides a comparison between model predictions and empirical measurements. Note that the units for the service rates vary as one moves down the pipeline. Where needed, appropriate units conversions are incorporated into the model (e.g., DNA bases are encoded as 4 bits per base, so one byte of data transfer across the PCI-X bus delivers 2 bases to the input of stage 1a). In addition, the service rate for stage 1b is a non-linear function of whether or not the external memory port is saturated. Additional details of the model are described by Dor in [34].
Table 7.1: Input parameters to queuing model.
Parameter Value for Value for
Run 1 Run 2
λin 900 MB/s 720 MB/s
µP CI 1 GB/s 1 GB/s
µ1a 2.1 Gbases/s 2.1 Gbases/s µ1b 130 Mseeds/s 50 Mseeds/s µ2 133 Maligns/s 133 Maligns/s
p1a1b 0.018 0.035
p1b2 0.88 0.76
p23 0.0002 0.0004
Starting with the server utilization results, we observe that there is a close match in virtually every case between the model predictions and the empirical measurements. This is not surprising, as the server utilizations are generally based primarily on mean job flow rates and are fairly insensitive to the distributions. This implies that the distributional assumptions present in the M/M/1 queuing models will not inordinately impact the quality of the model.
Table 7.2: Model predictions vs. empirical measurements.
Parameter Model Empirical Error Prediction Measurement
Run 1
ρ1a 0.84 0.85 0.01
ρ1b 0.25 0.23 0.02
ρ2 0.21 0.21 0
NQ,1b 0.08 approx. 0 0.07
NQ,2 0.06 1.2 1.1
Run 2
ρ1a 0.68 0.68 0
ρ1b 0.999 0.93 0.07
ρ2 .29 .29 0
NQ,1b 7500 580 6900
NQ,2 0.12 1.7 1.6
Turning to the queue occupancies, for 3 of 4 cases we again have a very close match between the model predictions and the empirical measurements. The one significant discrepancy is for the queue associated with stage 1b in run 2. Here, the high server utilization indicates that this server is the performance limiting bottleneck in the application. The physical queue is of length 600 entries, so the empirical queue occupancy cannot grow larger than that. The model predicts a much larger queue occupancy. Both the model and the empirical results are indicating that the queue will fill; however, the infinite queue capacity in the model is not capped by the length of the physical queue.
7.5 Chapter Summary
Here we have illustrated the use of straightforward queuing models to describe the perfor-mance of high perforperfor-mance streaming data applications and used TimeTrial to calibrate and validate this model. In general, they do surprisingly well at predicting the performance properties of the real application. Where there are discrepancies, the model can assist in understanding those discrepancies. For example, in the actual system, there is backpressure being asserted upstream of stage 1b due to the fact that the queue is full. While the notion of backpressure doesn’t exist explicitly in the current queuing model, we can estimate it by asking the model for the probability that the queue occupancy is greater than the actual
capacity of the queue. For run 2 this gives us P1b[N ≥ 600] = 0.92, a likely event. Addi-tional details on this proposed paradigm as well as its suitability for an M/M/1/K queuing model can be found in [34].
Chapter 8
Conclusions and Future Research
In this dissertation we have enabled developers to measure the performance of streaming, heterogeneous systems through the design and implementation our performance debug-ging tool, TimeTrial. We showed how TimeTrial can enable low-impact measurements across FPGA and multi-core processors. The TimeTrial measurement language provides a straightforward mechanism for developers to specify which aspects of a program are to be monitored and in what way they are to be monitored. Through integration with the Auto-Pipe system, developers can specify desired measurements and these measurements are automatically added to the resulting program. TimeTrial was demonstrated measur-ing two applications and providmeasur-ing helpful guidance where optimization efforts should be focused.
We now return to the research questions posed in Chapter 1 and describe how the work pre-sented in this dissertation addresses those questions. Questions 1 and 2 asked the following:
How can the performance of a distributed, streaming application be measured?
What aspects of performance should be measured?
How can the performance of an streaming application distributed across different FPGA accelerators and processor cores be measured?
TimeTrial’s profiles of streaming applications focus on performance metrics that enable a developer to find the location of the bottleneck in the system. These metrics include throughput, latency or utilization. TimeTrial in its current form is able to measure proper-ties of communication links and ports. Instead of deciding the specifics of what data should be collected for the developer, TimeTrial enables the developer to choose from a broad class of measurements to be included in the profile. To support heterogeneous application measurements, TimeTrial deploys a monitoring agent that is responsible for collecting data on each computation resource. The FPGA agents communicate the measurements back
to the software agent where a profile of the whole application is developed. Techniques to measure the performance of such an application deployed on FPGAs and processor cores were presented in Chapter 3.
Question 3 related to the ability to collect the desired information:
What data needs to be collected in order to answer the above questions? Can this data be collected in a low-impact manner and still satisfactorily answer those questions? What techniques should be used?
TimeTrial monitors events on communication channels. By time-stamping and counting these events, the TimeTrial agents are able to transform these event streams into perfor-mance metrics. TimeTrial supports metrics that are both relevant to perforperfor-mance and relatively easy to compute. To further reduce impact, the developer can choose which met-rics, how they are summarized and what portions of the application to measure. By letting the developer select the metrics of interest, overhead and impact can be reduced compared to measuring all metrics all the time. The results from profiling micro-benchmarks and real heterogeneous applications show that this approach has minimal impact on the executing application.
Dealing with pieces of the system that are not directly measurable, question 4 asked:
How does one build a profile of resources that have limited inherent visibility (e.g. system buses)?
A specific example of such a resource, a PCI-X bus, was shown to be able to be profiled with TimeTrial. The approach taken was to measure the pieces of the system that were available, measure properties of the traffic on the ingest and egress of the bus, and use a stochastic simulation of the bus to recreate the queue occupancy. This technique was validated against a micro-benchmark that allowed the true queue occupancy to be replayed. This approach proved to give good qualitative insight into the performance of that resource. Finally, a mechanism was described that allowed the developer to test when the model assumptions had failed and the simulation result should be discarded.
The final question relates to whether TimeTrial can be used to support modeling:
How does one use the collected performance data to calibrate or validate per-formance models?
We addressed this question by developing a strategy to use TimeTrial to calibrate a per-formance model and then validate its perper-formance on an application. We illustrated its effectiveness validating a straightforward queueing model. More robust models could be handled in a similar manner.
In conclusion, TimeTrial provides an automated measurement system for understanding the runtime performance of streaming, heterogeneous applications with little impact on those applications.