This section is designed to cause you to think “outside of the box” regarding tools available for software testing. When the concept of the software testing tool is dis- cussed, many testers think of automated tools provided by vendors of testing software. However, there are many manual tools available that can aid significantly in testing software (for example, code inspections).
The objective of this discussion is to categorize the tools used by testers into generic categories. A test script, for example, is a means for accomplishing some aspect of soft- ware testing. There are both manual tools to help you create scripts, such as building use cases, as well as automated tools that can both generate and execute a test script.
Testing tools are the aids used by individuals with testing responsibility to fulfill that responsibility. The tools cover a wide range of activities and are applicable for use in all phases of the systems development life cycle. Some of the techniques are manual, some automated; some perform static tests, others dynamic; some evaluate the system structure, and others, the system function.
The skill required to use the tools and the cost of executing the tools vary signifi- cantly. Some of the skills are highly technical and involve in-depth knowledge of com- puter programming and the system being tested. Other tools are general in nature and are useful to almost anyone with testing responsibilities. Some techniques involve only a short expenditure of man-hours, whereas others must be conducted by a team and make heavy use of computer resources in the test process.
The following is a list of the more common testing tools:
■■ Boundary value analysis.A method of dividing application systems into seg- ments so that testing can occur within the boundaries of those segments. The concept complements top-down system design.
■■ Capture/playback.A technique that enables you to capture the data and results of testing, and then play it back for future tests.
■■ Cause-effect graphing.Attempts to show the effect of each test event processed. The purpose is to categorize tests by the effect that will occur as a result of test- ing. This should reduce the number of test conditions by eliminating the need for multiple test events that all produce the same effects.
■■ Checklist.A series of probing questions designed to review a predetermined area or function.
■■ Code comparison.Identifies differences between two versions of the same pro- gram. You can use this tool with either object or source code.
■■ Compiler-based analysis.Utilizes the diagnostics produced by a compiler or diagnostic routines added to a compiler to identify program defects during the compilation of the program.
■■ Confirmation/examination.Verifies the correctness of many aspects of the sys- tem by contacting third parties, such as users, or examining a document to ver- ify that it exists.
■■ Control flow analysis.Requires the development of a graphic representation of a program to analyze the branch logic within the program to identify logic problems.
■■ Correctness proof.Involves developing a set of statements or hypotheses that define the correctness of processing. These hypotheses are then tested to deter- mine whether the application system performs processing in accordance with these statements.
■■ Data dictionary.The documentation tool for recording data elements and the attributes of the data elements that, under some implementations, can produce test data to validate the system’s data edits.
■■ Data flow analysis.A method of ensuring that the data used by the program has been properly defined, and that the defined data is properly used.
■■ Database.A repository of data collected for testing or about testing that can be summarized, re-sequenced, and analyzed for test purposes.
■■ Design-based functional testing.Recognizes that functions within an applica- tion system are necessary to support the requirements. This process identifies those design-based functions for test purposes.
■■ Design reviews.Reviews conducted during the systems development process, normally in accordance with systems development methodology. The primary objective of design reviews is to ensure compliance to the design methodology.
■■ Desk checking.Reviews by the originator of the requirements, design, or pro- gram as a check on the work performed by that individual.
■■ Disaster test.A procedure that predetermines a disaster as a basis for testing the recovery process. The test group then causes or simulates the disaster as a basis for testing the procedures and training for the recovery process.
■■ Error guessing.Uses the experience or judgment of people to predict what the most probable errors will be and then test to ensure that the system can handle those test conditions.
■■ Executable specs.Requires a computer system for writing system specifica- tions so that those specifications can be compiled into a testable program. The compiled specs have less detail and precision than will the final implemented programs, but they are sufficient to evaluate the completeness and proper func- tioning of the specifications.
■■ Fact finding.Information needed to conduct a test or to ensure the correctness of a document’s information, achieved through an investigative process requir- ing obtaining information or searching for the facts about a predetermined condition.
■■ Flowchart.Graphically represents the system and/or program flow in order to evaluate the completeness of the requirements, design, or program specifications. ■■ Inspections.A highly structured step-by-step review of the deliverables pro-
duced by each phase of the systems development life cycle in order to identify potential defects.
■■ Instrumentation.The use of monitors and/or counters to determine the fre- quency with which predetermined events occur.
■■ Integrated test facility.A concept that permits the introduction of test data into a production environment so that applications can be tested at the same time they are running in production. The concept permits testing the accumula- tion of data over many iterations of the process, and facilitates intersystem testing.
■■ Mapping.A process that analyzes which parts of a computer program are exer- cised during the test and how frequently each statement or routine in a program is executed. This can be used to detect system flaws, determine how much of a program is executed during testing, and identify areas where more efficient code may reduce execution time.
■■ Modeling.A method of simulating the functioning of the application system and/or its environment to determine if the design specifications will achieve the system objectives.
■■ Parallel operation.Runs both the old and new version within the same time frame in order to identify differences between the two processes. The tool is most effective when there is minimal change between the old and new process- ing versions of the system.
■■ Parallel simulation.Develops a less precise version of a segment of a com- puter system in order to determine whether the results produced by the test are reasonable. This tool is effective when used with large volumes of data to automatically determine the correctness of the results of processing. Normally, this tool approximates only actual processing
■■ Peer review.A review process that uses peers to review that aspect of the sys- tems development life cycle with which they are most familiar. Typically, the peer review offers compliance to standards, procedures, guidelines, and the use of good practices, as opposed to efficiency, effectiveness, and economy of the design and implementation.
■■ Ratios/relationships. Quantitative analysis that enables testers to draw conclu- sions about some aspect of the software to validate the reasonableness of the software. For example, in test planning, they may want to compare the pro- posed test budget to the number of function points being tested.
■■ Risk matrix.Tests the adequacy of controls through the identification of risks and the controls implemented in each part of the application system to reduce those risks to a level acceptable to the user.
■■ Scoring.A method to determine which aspects of the application system should be tested by determining the applicability of problem criteria to the application being tested. The process can be used to determine the degree of testing (for example, high-risk systems would be subject to more tests than low-risk systems) or to identify areas within the application system to deter- mine the amount of testing needed.
■■ Snapshot.A method of printing the status of computer memory at predeter- mined points during processing. Computer memory can be printed when specific instructions are executed or when data with specific attributes are processed.
■■ Symbolic execution.Permits the testing of programs without test data. The symbolic execution of a program results in an expression that can be used to evaluate the completeness of the programming logic.
■■ System logs.Uses information collected during the operation of a computer system to analyze how well the system performed. System logs are produced by operating software such as database management systems, operating sys- tems, and job accounting systems.
■■ Test data.System transactions that are created for the purpose of testing the application system.
■■ Test data generator.Software systems that can be used to automatically gener- ate test data for test purposes. Frequently, these generators require only para- meters of the data element values in order to generate large amounts of test transactions.
■■ Test scripts.A sequential series of actions that a user of an automated system would enter to validate the correctness of software processing.
■■ Tracing.A representation of the paths followed by computer programs as they process data or the paths followed in a database to locate one or more pieces of data used to produce a logical record for processing.
■■ Use cases. Test transactions that focus on how users will use the software in an operational environment.
■■ Utility programs.A general-purpose software package that can be used to test an application system. The most valuable utilities are those that analyze or list data files.
■■ Walkthroughs.A process that asks the programmer or analyst to explain the application system to a test team, typically by using a simulation of the execu- tion of the application system. The objective of the walkthrough is to provide a basis for questioning by the test team to identify defects.