The back-end portion of the model was validated aga inst measurement resu lts from numerous DebitCredit benchmarks (Digital's precursor of the TPC Benchmark A) on many VAX computers wit h the VMS operating system, runni ng VAX ACMS and VAX Rdb/VMS software.1 With sufficient deta iled parameters ava i l able (such as transaction instruc t ion count, i nstruction cycle t ime, bus/memory access t ime, cache hit ratio), the model correctly estimated the MQTh and many intermediate results for several multiprocessor VAX systems. The model was t hen extended to i nclude the front-end sys tems. In this section, we discuss applying this com plete end-to-end model to the TPC Benchmark A on two VAX platforms, the VAX 9000 Model 210 and the VAX 4000 Model 300 systems, and then compare the results. The benchmark environment and imple mentation are described in t he TPC Benchmark A
Implementation section of this paper.
MEMORY 1 SINK �- - - -
�
I I I I � - - - I KEY: CLOSED CHAIN OPEN CHAIN SOURCE I I I I I - r - - - - - 1 I I I I I I I MEMORY m 1 I I I • _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ !Figure 11 CPU-cache-bus-memory Submodel
Transaction Processing, Databases, and Fault-tolerant Systems I N I T I A L I Z E : T x n P L , M Q T h , B u s y W a i t P L , C p u U t i l i z a t i o n ; L O D - s u b m o d e l ( i n p u t : M Q T h ; o u t p u t : L O D ) R E P E A T I / 0 - G r o u p i n g - s u b m o d e l ( i n p u t : M Q T h ; o u t p u t : D i o P e r T x n , T x n P L ) ; R E P E A T R E P E A T B u s y W a i t - s u b m o d e l ( i n p u t : T x n P L , B u s y W a i t P L , C p u U t i l i z a t i o n , D i o P e r T x n ; o u t p u t : B u s y W a i t P L ) ; U N T I L ( B u s y W a i t P L c o n v e r g e s ) ; C P U - C a c h e - B u s - M e m o r y - s u b m o d e l ( i n p u t : T x n P L , B u s y W a i t P L ; o u t p u t : C p u U t i l i z a t i o n , A v g C p u S v c T i m e ) ; U N T I L ( C p u U t i L i z a t i o n c o n v e r g e s ) ; R E P E A T M Q T h - m o d e l ( i n p u t : A v g C p u S v c T i m e , L O D ; o u t p u t : M Q T h , C p u U t i l i z a t i o n ) ; L O D - s u b m o d e l ( i n p u t : M Q T h ; o u t p u t : L O D ) ; U N T I L ( M Q T h c o n v e r g e s ) ; U N T I L ( M Q T h c o n v e r g e s ) ;
Figure 12 Tbe Iterative Procedure to Integrating Submodels
Because both the VAX 9000 Model 210 and the VAX 4000 Model 300 systems are uniprocessor systems, there is no other processor contending for the processor-memory interconnect and mem ory subsystems. Such contention effects can there fore be ignored w hen model i ng a uniprocessor system. The transaction process i ng performance prediction for the VAX 9000 Model 210 system is a successful example of the application of our analyt ical model.
We needed an accurate estimate of TPC Bench mark A p erformance on the VAX 9000 Model 210
system before a VAX 9000 system was actually ava i l able fo r rest ing. The high-level (MQTh) model was used with estimated values for the input parame ters, LOD and transaction CPU service time. The est imated LOD was based on previous measure ment observations from the VAX 6000 systems. The other parameter, back-end transact ion CPU service time, was derived from t he
• Tim i ng information of the VAX 9000 CPU
• Memory access t ime and cache m iss penalty of the VA.'\ 9000 CPU
• Pred iction of cache hit ratio of the VAX 9000 sys tem under the TPC Benchmark A workload
• Transaction path length of the TPC Benchmark A
imp lementation
• Instruction profi le of the TPC Benchmark A
implementation
56
The high- level model predicted a range of MQTh, with a high end of 70 TPS and w i t h a strong proba bility that the high-end performance was achievable.
Add i t ional pred ictions were made l ater, when an early prototype version of the VAX 9000 Model 210 system was ava i l able for testing. A variant of the Debi tCredi t benchmark, much smaller i n scale and easier to run, was pe rformed on the p rototype system, with the emphasis on measuring the CPU performance in a transaction process i ng environ ment. The result was used to extrapolate the CPU service time of the TPC Benchmark A t ransactions on the VAX 9000 Model 210 system and to refi n e t h e early estimate. T h e results of these m od ifica t ions supported the previous h igh-end estimate of pe rformance of 70 TPS and refined the low-end pe rformance to be 62 TPS. The final, audited TPC Benchmark A measurement result of the VAX 9000 Model 210 system showed 69.4 TPS, which closely matches the p red i ction . Tab le 8 compares the results fro m benchmark m easurement and the analytical model outputs.
Table 8 Measurement Compared to Model Predictions System VAX 9000 Model 21 0 VAX 4000 Model 300 Measured MOTh 69.4 21 .5 Modeled MOTh 70.0 20.8
The VAX 4000 Model 300 TPC Benchmark A
results were also used as a validation case. VAX 4000
Model 300 systems use the same CMOS chip as
the VAX 6000 Model 400 series and the same
28-nanosecond (ns) CPU cycle time. However, i n
the VAX 4000 series, the CPU-memory interconnect
is not the XMI bus but a direct primary memory
i nterconnect. This d irect memory interconnect results in fast main memory access. The processor, cache, and main memory subsystems are otherwise
the same as in the VAX 6000 Model 400 systems.
Therefore, the detai led-level model and associated
parameters for the VAX 6000 Model 410 system
can be used by ignoring the bus access time. The TPC Benchmark A measurement results are within 7 percent of the model p rediction, which means that our assumption on the memory access time is acceptable.
Conclusion
Performance is one of the most important attrib utes in evaluating a transaction processing system. However, because of the complex nature of trans action processing systems, a universal assessment of transaction processing system performance is impossible. The performance of a transaction pro cessing system is workload dependent, configura tion dependent, and implementation dependent. A standard benchmark, like TPC Benchmark A, is a step toward a fai r comparison of transaction pro cessing performance by different vendors. But it is only one transaction processing benchmark that represents a l imited class of applications. When evaluating transaction processing systems perfor mance, a good understanding of the targeted appli cation environment and requirements is essential before using any ava i lable benchmark result. Additional benchmarks that represent a broader range of commercial applications are expected to be standardized by the Transaction Processing Performance Counci l (TPC) in the com ing years. Performance modeling is an attractive alterna tive to benchmark measurement because i t is less expens ive to perform and results can be compiled more quickly. Modeling provides more insight into the behavior of system components that are treated as black boxes in most measurement exper iments. Modeling helps system designers to better understand performance issues and to d iscover existing or potential performance problems. Model i ng also provides solut ions fo r improving perfor mance by modeling d ifferent tuni ng or des ign alternatives. The analytical model presented in this
Digital Technical journal Vol. 3 No. I Winter 1991
paper was val idated and used extensively in many engineering performance studies. The model also helped the benchmark process to size the hard ware during preparat ion (e .g., the number of RTE and front-end systems needed , the size of the database) and to provide an MQTh goal as a sanity check and a tu n i ng a id. The model could be extended to represent ad ditional d istributed configurations, such as shared-d isk and "shared nothing" back-end transaction processing systems, and could be applied to additional transaction pro cessing workloads.
Acknowledgments
The Digital TPC Benchmark A implementation and measurements are the resu lt of work by many individuals within Digital. The authors would like especially to thank Jim McKenzie, Martha Ryan, Hwan Shen, and Bob Tanski for their work in the TPC Benchmark A measurement experiments; and Per Gyl lstrom and Rabah Mediouni for their con tribut ions to the analytical model and val idation.
References
1 . Transaction Processing Performance Council,
TPC Benchmark A Standard Specification
(Menlo Park, CA: Waterside Associates, November 1989).
2. Transaction Processing Systems Handbook
(Maynard: D igital Equipment Corporat ion, Order No. EC-H0650-57, 1990).
3. TPC Benchmark: A Report for the VAX 9000 Model 2 1 0 System (Maynard : Digital Equipment Corporation, Order No. EC-N0302-57, 1990). 4. TPC Benchmark: A Report for the VAX 4000
Model 300 System (Maynard: Digital Equipment Corporation, Order No. EC-N0301-57, 1990). 5. L. Wright, W Kohler, and W Zahavi, "The Digital
DebitCredit Benchmark: Methodology and
Results," Proceedings of the International
Conference on Management and Performance Evaluation of Computer Systems (December 1989): 84-92.