A strea m of wel l -known benchmarks was se lected that represented each of the above- mentioned Engineeri ng/Sc ientific , Com merc i a l , and General Timeshari ng markers.
• The engineering stream consists of typical
programs used in electrical circ u i t simu lation ,
Digital Technical journal
No. 7 A ugust I 'J88
oil reservoir s i m u l at ion , fl ight simulation . and l i near equation solvers.
• The sc ient i fi c stream conta ins s i mu lation programs that use Monte Carlo techniq ues to track particle movement, along with com monly used routines from national labora tories.
• The commerc ial stream contains the activit ies
done by a personnel department ro support salary planning
• The general ti mesharing stream represents the
activities done in a software development or education environ men t .
M u l t i p l e copies of t h i s stream were run simulta neously to take advantage of multi processor com pure resources (Figure 1 ) . To capture the maxi mum throughput , we ensured that a l l of the
processors were 1 00 percent busy whi le the mul
tiple streams were ru nning on the system .
Multiuser Wo rkload Development
The overall process of workload development is shown in Figure 2. Our goa l was to represent typ ica l timesharing environments for the d i fferenr ta rget markets . The entire strategy consisted of• Identifying typical real s i tes
• Col lecting data on resource u t i l i zat ion and i mage usage patterns
• Deriving a packaged workload ro represent the
real s i te environ ment
• Va lida t i ng the workl oads by comparing the
resource u t i l i zation of the workload aga inst the resource u t i l i zation at various cusromer sires and modifyi ng the workloads as req u i red
Performance Evaluatio n of the VA X 6200 .�)'stems R EA L SYSTEM R ESOU R C E U T I L I ZATION DATA
l
R ES O U R C E U T I L I ZATION DATA-TER M I NAL ACTIVITY
-USER CHARACTER ISTICS
-USER M I X
1
-VAXRTE SCRI PTS - U S E R CHA RACT E R I STICS - U S E R MIXSTANDALONE SYSTEM
Figure 2 Interactive Multiuser Workload Development
In the fol l owing sections, we describe how we used this strategy to develop two mul t i user workloads: the engineeri n g workload , which rep· resents an Electronic Computer-Aided Engineer i ng environment (ECAE) ; and the Software Deve l opment Environment Workload (SDEW) .
Data Collection
Two Digital sires were chosen to represen t the ECAE and SDEW environments . I n ternal sites were chosen i n i t i a l l y to faci l itate the data col lec tion process. Both s i res had c lustered environ ments that consisted of a variety of VAX systems along with some workstations.
We collected i n formation on these c lustered systems to capture thei r behavior u nder the l oad generated by the environment over a period of one week . VAX SPM software was used to col lect resource util ization data (CPU, 1/0. and memory uti l i zation) on a l l the systems at both user level and system leve l . VMS I mage Accou n t i ng was used to obtai n resource u t i l i za tion data on a n i mage basis. Usi n g t h e SET HOST/LOG Digital Command Language ( DCL) command , we col lected log files of user sessions to study user habi ts . Other user characteristics, such as think t i me and type rates, were obtai ned through inter views and observa tions .
Data A nalysis
The performance team studied t he c luster-wide resource util ization profil es in order ro select the t i me when the i n teractive activities were pre dom i nant. We compared resource u t i li zation profi l es of i nd ividual systems agai nst r he c l uster-
66
w ide average over a week's accumu lation of data . Based on t h is comparison , we selected a typical day and a typical syste m . O ne hour was c hosen from the typical system on a typical day d u ri ng the period of peak i nteractive use to c haracterize t he system at fu l l load .
Further, based on the user profiles, we classified users accordi n g to computer usage, that is, heavy or l ight computing (for ECAE workload) and heavy, med iu m , or l ight comput· ing (for SDEW workload) . We then used the i mage accoun t i ng data and user log files to c l as sify users according to the type of activity t hey performed .
Once several user c lasses were identified, the number of users in each class, or user m i x , was determi ned. We defined the user m i x by l ooki ng at ( 1 ) t he n u m ber of users i n each c lass at the
Table 3 ECAE and SDEW User Mix
Type of User
Eng ineer: H eavy Eng ineer: Light
Type of User
ECAE User Mix
SDEW User Mix
Heavy software development Light software development Secretary Technical writer No. of Users 3 3 No. of Users 1 3
Digital Technical journal No. 7
one-hour pea k , and ( 2 ) the organization struc ture at t he rea l sites. Tab l e 3 shows the user m i x for ECAE a n d SDEW workloads . I n addition to interactive users, t hese workloads a lso have batch jobs runn i ng in the background.
Developing the Workload
Having identified the user c lasses and activities, we then deve l oped an imermediate workload using DCL command p rocedures. This i nter mediate step a l l owed easier translation to the final workload , which was based on VAXRTE (VAXfVMS Remote Terminal E mu lator) scripts. I ndividual user scripts were developed and val i dated . We then packaged t h e entire workload by i ntegrat i ng a l l of the user scripts and the batch jobs. Once development was complete, the workload was val idated at both system a nd user levels against the rea l i nternal site. Further val i dation was done at the user l evel agai nst Digita l 's customer sites.
Workload Validation
This section describes the workload val idation process using the ECAE work load as an example of t he validation methodology.
Val idat ion aga i nst " real " i nternal site - The workload was tested using the same hardware configuration as t he rea l system . For the ECAE workload , a VAX- 1 1 /780 system with 3 2 mega bytes (MB) of memory, RA8 1 disks, a nd six inter active users was rested. The purpose of this test was to compare the resource u ti l ization of the workload in an hour-long experi ment to the resource uti l i zation of the real system during the typical hour. System - and process-level resource u t i l izat ion data of severa l different resources were compared.
User- l evel val idation - To val idate t he work load at the user leve l , we compared the average CPU and d i rect 1/0 (DIO) uti l i zations computed for 1 hour for the d i fferent user c lasses. The resu l ts are shown in Table 4 .
CPU u t i l ization for a l l t h ree user c lasses val i dated to within approxi mately I 0 percent, which was considered tO be wel l within accept able l i m i ts. Val idation of the 010 rare was made somewhat d ifficu l t because ( 1 ) the 010 rare on a per-user basis was very low (0 . 3 010 per second for t he heavy user) , a nd ( 2 ) measurement of the 010 rate is only accurate to 0 . 1 010 per second . For a l l three user classes, the workload came to
Digital Technical journal
No. 7 A ug ust I YBB
Table 4 User Resource Utilization for Real Internal System and ECAE Workload
CPU
minutes/hour DIOfsecond User Class Real ECAE Real ECAE
Heavy 1 .6 1 .5 0.3 0.4
Light 0.5 0.5 0.2 0 . 1
Batch 42.8 48.5 0.0 0 . 1
within 0 . 1 0 1 0 per second of t he va lues mea sured from the real site.
System-level validation - For system - l evel val idation , we compared t h e system-level usage of CPU, disk 1/0. and memory for t he 1 -hour ECAE test experiment to the peak hour of t he real system . Figu re 3 shows that t he CPU was used 1 00 percent of t he t i me on the real system during the I hour; whereas t he CPU u t i l i zation i n t he workload tended tO vary s lightly more , but was a lways between 90 percent and 1 00 percent sat urated . The average CPU u t i lizations of t he real system and the ECAE workload are very c lose at
1 00 percent and 9 3 percent , respectively.
The DJO util ization over a 1 -hour period for the rwo systems is compared i n Figure 4 . For borh systems there is significant variabi l i ty i n the 010 rare over the 1 hour period . The ECAE workload was s lightly more bursty, but t he average DIO rates for the rea l system a nd t he ECAE workload were very close at 3 . 3 and 3 . 0 010 operations per second, respectively.
Memory u t i lization on the two systems d id not vary substantial ly over the 1 -hour period. How ever, total average memory usage with the workload , 2 3 MB, was less than on the rea l sys tem , 2 9 M B , as depicted i n Figure 5 .
Al though the workload val idated very wel l for CPU and 010 resource u t i lization , t he workload used 20 percent l ess memory than was used at the real site. This was in parr due to the fact that during the develop ment of the workload t he CPU and d isk 1/0 u t i li za t ion of subprocesses was added tO the resource u t i l ization of t he parent process. Although the workload represents the work done by t hose subprocesses and the load p laced on CPU and disk 1/0 resources, the workload does not represent the additional mem ory requ i red by those subprocesses . As w i l l be described in subsequent sections, the lower memory u t i l i zation of the workload d id not con stitute a problem.
1 00% 90% 80% 70% 60% 50% 40% 30% 20% 1 0% u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u U U U U U U E U U U U U U U U U U U E U U U U U U U U U U S E U U U U U U U U S U S E U U U U U U U U S U E E U U U U U E U U E U E K U S U U U E U S E U K K U E S U U K U E E U K K U E E E U K U E K U K K U K E E U K U K K E K K U K K K U K S K K K K K S K K K U K S K K K K K E K K K U K E K K K K K K K K K U K K K K K K I K K K K U I K K K K I I K K I K S I K I I I I I I I I I E I I I I I I I I I I I K I I I I I I I I I I I I 1 0 1 4 1 0 39 1 1 04 REAL SYSTEM CPU UTILIZATION ( P E R C E NT) V E R S U S TIME OF DAY FROM 6-NOV- 1 986 1 0 1 4 : 06.45 TO 6-NOV-1 986 1 1 1 4 39.91
EACH COLU M N� 300 SECONDS (5 M I N UTES)
CPU I DLE
TOTAL PAGE SWAP PAGE & SWAP
I D L E WAIT WAIT WAIT
0 . 1 % 0.0% 0.0% 0.0% CPU BUSY INTER STACK 6.4% EXECUTIVE 4.0% USER 76.6% SYSTEM 2 1 .5% K E Y : I - I NT E R R U PT E - EXECUTIVE U - USER K - K E R N EL S - S U PERVISOR K E R N E L 1 1 .2 % S U P E RVISOR 1 .9% COMPAT I B I L ITY 0 . 0% TASK 78.4% 1 00% u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u 90% u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u 80% u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u 70% u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u W% U U U U U U U U U U U U U u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u 50% u u u u u u u u u u u u u u u u u u u u u u u u u u U U U E U U U U U U U U U U U U E U U U U U U U U U U U U E U U U U E U U U U 40% U U U E U U U U E U U U U U U U K U U U U E U U U U U U U K U U U U E U U U U U U U K U U U U E U U U U U U U K U U U U K U U U U 30% U U U K U U U U K U U U U U U U K U U U U K U U U U U E U K U U U U K U U U E U E U K U U U U K U U U E U K U K U U U U K U U U K 20% U K U K U U U U K U U U K U K U K U E U U K U E U K U K U K U K U E K U K U K