A SMOKE TEST (13) is quick but not exhaustive. For it to be effective, you need to do the hard work of exhaustive testing as well. If you want to establish release candidates, you need to be sure that the code base is robust. This pattern explains how to gener-ate builds that are no worse than the last build.
How do you ensure that existing code doesn’t get worse as you make other improvements?
Software systems are complex and with each change or enhancement to a system comes the possibility of breaking something seemingly unrelated to your changes.
Fixing a defect has a substantial chance of introducing another(Frederick P. Brooks
1995). Without change, you can’t make progress, but the impact of a change in hard to measure, especially in terms of how the a unit of code interacts with the rest of the system.
You can exhaustively test your system after each build. Exhaustive testing takes time, but if you don’t do this testing you waste developer, and perhaps, customer time. If everyone runs exhaustive tests all of the time, they will not be able to spend much time coding.
Even if you decide to do exhaustive system-level testing periodically on the code base, you are left with the problem of how to structure the tests. You can write tests from first principles by doing an analysis of the inputs and outputs, but the payoff for writing tests like this on a system level may be small relative to the amount of effort that they take to write.You can also make intelligent guesses about what to test, but a software system is always full of surprises, especially in the ways that it can fail.
When you solve the problem of which system tests to execute, you are still left with the problem of when to run them. Some integration tests may need resources that are not on every development machine. What is a good development environment may not always be the best test of the system. Some problems may require large data sets to reproduce, or multiple clients in a client-server system. You may not have the resources to run these sorts of tests all of the time.
When the system does break, you want to identify the point in time when something broke. If you run the tests on every build or check in to source control, you will be able to identify when something failed. But your testing may not keep up with your check in and build process if the tests take long enough.
If your exhaustive tests find a problem, and you fix it, you want to be sure that you can identify when this problem happens again, since you don’t want to waste time in known issues. A problem that happens once can happen again (this is what we mean by regression, after all). This means that we should accumulate a set of test cases as we discover problems. Especially since we may not be able to guess all of the prob-lems ahead of time. These test cases can add up over time.
We need to be able to check for these recurring failure modes.
Two Steps Back
I worked for a small software product company that had a code base com-bined of newer, cleaner code, and also code that evolved. On any given day, it was not clear whether you could get an update from source control and have a working system, or whether you could would have to spend the day getting the system to a point where you could do your work. The problem was that there was no automated testing of the core APIs. People would avoid moving to a current code base in fear of wasting a day, but this eventually caused other problems.
The lack of a way to check for old problems recurring caused many easily pre-ventable quality issues.
Test for Changes
Run regression tests on the system whenever you want to ensure stability of the codeline, such as before you release a build, or before a particularly risky change.
Create the regression tests from test cases that the system has failed in the past.
Regression tests are end to end black box tests that cover actual past or anticipated failure modes. A Regression test can identify a system level failure in the code base, but may not necessarily identify what broke. When a regression test fails, debugging and unit tests may be necessary to determine what low-level component or interface broke.
Regression Tests test changes in integration behavior. They are large grained, and test for unexpected consequences of integrating software components. Unit tests can be thought through fairly easily. As you add component interactions it is harder to write tests based on ‘first principles.’
Build regression test cases out of:
• Problems that you find in the pre-release QA process
• Customer and user reported problems
• System level tests based on requirements.
As you discover problems, write a test that reproduces the problem and add that sce-nario to the test. Over time you will end up with a large suite of tests that cover your
most likely problem areas. Each problem may involve more than one test case. You can include running all unit tests in your Regression testing, but it is better if the tests involve system input.
Regression Testing is designed to make sure that the software has not taken a step backwards (or regressed) Always run the same tests for each regression cycle. Add tests as you find more conditions or problematic items to test. Always add test cases to the regression test suite; If a problem happened once, it is likely to happen again, so remove test cases only for very well thought through reasons.
Since regression tests can take a long time to run, you don’t want to run them before every check-in, or even after every build (unless resources permit). There are advan-tages, however to having an automated procedure to run the regression test after each change, so that you can identify the point at which the system regressed. Run the regression tests as part of the nightly build. Developers should also run a regres-sion test before any significant sweeping change. If something breaks, you can always run the unit tests to localize the change. You also have to investigate if the unit test inputs no longer match the system. Institute a policy of automated regres-sion testing tied to each release. (Booch 1996)
Further Reading
• Steve McConnell has a lot of information about testing of all kinds in Code Complete (McConnell 1993)
• The Art of Software Testing (Myers 1979)by Glen Meyers is a classic.
Software Configuration Management Patterns: Effective Teamwork, Practical Integration by Steve Berczuk with Brad Appleton. Copyright 2002 Addison-Wesley, Boston, MA. All rights reserved