out the J\1VAX logical verification effort and thus helped to steer the team's work. Hugs were tracked carefu lly with an on-line system and ana lyzed each week to consider trends, successfu l and unsuccess ful bug-findi ng techniques, and bug hot spots, which requi red additional attention. The bug detec tion rate was fairly constant throughout the project at abou t 22 per month, with the excep tion of the last month in which the rate dropped to nearly zero. An analysis of the bug-detecting effectiveness of each testing tech nique shows that all test tech niques were effective and seemed to complement each other. Table 1 shows the percentage of bugs detected by each technique. This table includes data on the ever-valuable, nonsim ulation verifica tion tec hnique of simply reviewing, inspecting, and discussing the design and its many representations. The decision to release the design for fabrication of first-pass chips was a consensus decision made by the verification, archi tecture, and design teams. From a verification perspective, the design was ready for release when the bug detection rate remained at zero for several weeks and the majority of the planned tests had been i mplemented . The verification of some areas of the design was defe rred until after the release of the first-pass design . The development team decided that any bugs that m ight be founcl in these areas wo uld not h ave a significant negative impact on the system develop ment schedule, whereas additio nal delay in releasing the design wou ld.
Ta ble 1 Bug Detection Using Va rious Techn iques
Percent of
Techn ique Total Bugs Found
Focused tests 28
Directed, pseudorandom
exercisers 23
Review, i nspection,
observation, thought 20
Detection technique unknown 1 2
AXE 8
MAX 6
HCORE 2
Other
Results and Conclusions
Only 15 logical bugs were fo und i n the first- pass NVAX CPU chip design, a ll of which were either eas i ly worked aroun d or did not impact normal system operation . The n an1re of the bugs fo u nd in the first pass design ranged from stra igh tforward bugs that escaped detection tor clear-cut reasons to extremely complex bugs that required h o u rs or weeks of rig ormJs prototype system test ing to u ncover. Some of the bugs escaped detection during s i m u l ated verifi cation for cl assic reasons such as:
• Tes t i ng of the function was pe rformed just before release, in a hurried man ner.
• Sim u l ation performance proh i b ited r u n n i ng a cert a i n type of test case.
• A test was n o t run i n a certa i n mode due to the difficu l ty of running it i n all possible modes.
• It took an exerciser r u n n i ng on a simu l ator a
long time to encou n ter the conditions that wou ld evoke the bug.
• A test was i nadvert e ntl y dropped from the set of exercisers that were run contin uously
Det a i l s abou t five of the more i nterest i ng hugs fou nd in the first -pass design fol low. Included is i n fo rmation about how the hug was detected, a hypothesis on why the bug e luded detection before first-pass c h ips were fabricated , and les sons lea rned from the detection and e l i m i n ation of the bug.
Digital Tecbuica/ journal liJI. ·I .Vo ..i S111nml!r l')')l
Logical Veri(ica tiun of the NVAX CPU Ch ip /Jeslj{ll
1 . One simple bug was detected by r u n n i ng the HCORE test su ite o n the pro totYpe system with the floati ng-point unit ( F- box) ll isable d . This bug could have been fo u nd i n the same way through simu latio n , but the test suite was n o t run as a fin a l regression test w i th the F-box d isabled . I n ge nera l , focused tests l i ke
HCORE
were n o t run wit h varied chip/system configu rati ons. The ver ification team concl uded that a l l foc used tests sho u ld be r u n with d ifferent chip/system con fig u rat i o ns. At the m i n i m u m , a configuration that disables all possible fu nctions s hou ld be H::sted.2. Another hug was discovered because the Cl' l l
c h i p generated spurious writes to memory in the protot ype system. The exercisers prolx1bly did generate the conditions necessary to evoke thi s bug; however, the spurious wri tes wen t un no ticed . It is extremely diffi cult to verify that a machine does everyt h i ng i t is supposed to do and noth ing more. Ad ditional assertion checkers or m o n i tors in t h e models might detect such bugs in the fu t ure.
3. A third bug was evoked when a prototype system exerciser executed a translation bu ffer inva l idate a l l ( TBIA) i nstruct i o n u nder certa i n con d i t i o ns. On a rea l system, the TBI A instruc tion is used o n ly by the operating system. I n our verificat ion effort, the TIIIA instruction was l i ttle used by the exercisers that were simu lated. Operations that are performed o n ly by the oper ating system should not be underem phasized in exercisers.
4. One first -pass bug was related to the halt in ter
rupt, which i s used o n l y d u ring debugging oper ations. The h a l t i nterrupt receivecl m i n i m a l testing a n d was not tested a t a l l i n a n y t y p e of exerciser. Discovering this bug was especia l l y ann oying beca use a similar bug h a d escaped detection by t he i nitial logical verification effort
fo r a previous VAX i m rlementation. This turn of
events reinforces the be.l ief th at there is va l u e i n reviewing t h e escaped b u g l ists from other proj ects. Also, dur i ng the veri ficat i o n effort, there
seemed to be a natural, but erro neous, tendency
to u mlertest fu n ctions used i n frequent l y or not at a l l during normal system operation. Such fu nctions sometimes require extra ::tttent i o n , because they may b e qu ite complex a n d may have been given less carefu l thought d ur ing the design process.
NVAX-rnicroprocessor VAX Systems
5. A state bit that needed to be i ni t ia l ized on power up was not. This problem was not iced during i n i t ia l ization simu lation but erroneously ratio nalized as being acceptable. Design assu mptions and assertions about i n itial ization should be ver ified through simu lation or other means.
Overa l l , the 1\f\/AX Cl'll chip logical verification effort was a su ccess. The pseudorandom testing strategy detected several complex and subtle logi cal bugs that otherwise probably wou ld not have been detected by simu lation. The extensive simula tion performed on the schematics-derive(! model of the chip prov ided a h igh degree of confidence in the design .
The goals of producing h igh ly functional first pass chips and bug-free, second-pass chips were both met. Neither the bugs in first-pass chips nor their work-arounds impeded prototype system debugging in any sign i ficant way. and first-pass chips with work-a rounds were used in prepro d uction, field-test systems. The verification team corrected the 15 first-pass design bugs fo r second pass chips, which were shipped to customers i n revenue-producing systems.
Acknowledgments
The NVAX logical verificati o n effort was performed by a team of engineers from the SEG microproces sor verificati o n group . Members of this tea m included Wa l ker Anderson, Rick Calcagn i , Sanjay Chopra, and Joh n St. Laurent. 1\f\/A.,.'( archi tects M i ke Uhler and Debra Bernstein provided extensive tech n ical d irec t ion and assistance to the verification rea m . The SEG
CAD
group helped by i ts cont inual46
development and support of too ls. 'l'he CHA GO simu lator wou l d not have been possible without the significant co ntribu ti o ns of Kev i n Lack l . Will Sherwood provided h igh-qua l itv, top- level gu id ance th rough a l l phases of the project. 'fhe system development groups performed system - level simu lations and rigorous p rototype test i ng. The VAX
Arch itecture G ro u p A..'<:E/�LAX team, once again, prov ided a nd supported a n effect ive verification tool. Lastly, the success of the project and the final qual i ty of the NVAX chip l ogical design are as much a tribute to the work of the '\f\IAX arch i tecture a nd design teams as they are to the work of the verifica tion team.