Whether they come from user stories or from direct translation of capabili-ties, Google TEs create a large number of test cases that prescribe, in terms ranging from general to precise, inputs and data to test an application.
Unlike code and automated test artifacts that are managed by common infrastructure, test case management is still a group-by-group thing, but a new tool is beginning to change all that.
Spreadsheets and documents have often been the tool of choice for stor-ing test cases. Teams with rapid feature development and fast ship cycles aren’t too concerned with keeping test cases around for long. As new fea-tures arrive, they often invalidate the scripts, requiring all the tests to be rewritten. In such cases, documents that can be shared and then trashed are as suitable as any other format. Documents are also a suitable mechanism to describe the context of a test session as opposed to specific test case actions.
Such tests are less prescriptive and more general suggestions about which feature areas to explore.
Of course, some teams have rather elaborate spreadsheets for storing test procedures and data and some teams even document ACC tables in spreadsheets because they are more flexible than Google Test Analytics (GTA). But this takes discipline and some continuity in the TE team as one TE’s procedure will be another TE’s shackles. Large team turnover requires a more structured approach, a structure that can outlast any individual team member.
Spreadsheets are preferred over documents because they provide con-venient columns for procedure, data, and pass or fail tags, and they are easy to customize. Google Sites and other types of online wikis are often used to display the test information to other stakeholders. They can be easily shared and edited by anyone on the team.
ptg7759704 As Google grew, many teams had a growing set of prescriptive test
cases and regression tests that needed better management. Indeed, the test cases documented grew so large, they became burdensome to search and to share. Another solution was needed and some enterprising testers built a system called Test Scribe that was loosely based on any number of commer-cial tools and other homegrown test case management systems our testers were familiar with from previous employers.
Test Scribe stored test cases in a rigid schema and had the capability to include or exclude test cases from a specific test pass. It was a basic imple-mentation and enthusiasm over using it and maintaining it was waning;
however, many teams had taken dependencies on it and after nursing it along for a few quarters, it was trashed and a new tool was written in 2010 by a Senior SET named Jordanna Chord. Google Test Case Manager (GTCM) was born.
The idea behind GTCM was to make it simple to write tests, provide a flexible tagging format that can be tailored to any project, make the tests easily searchable and reusable, and, most importantly, integrate GTCM with the rest of Google’s infrastructure so that test results could become a first-class citizen. Figures 3.10 through 3.14 show various screenshots of GTCM. Figure 3.11 shows the page for creating test cases. Test cases can have arbitrary sections or labels. This allows GTCM to support everything from classic test and validation steps, to exploratory tours, cukes,7and user-story descriptions, and some test teams even store code or data snippets in the GTCM test case itself. GTCM had to support a wide variety of test teams and their varied test case representations.
7Cukes are a behavioral test case definition. See http://cukes.info.
FIGURE 3.10 The GTCM home page is focused on the search experience.
ptg7759704 FIGURE 3.11 Creating a project in GTCM.
FIGURE 3.12 Creating a test in GTCM.
ptg7759704 FIGURE 3.13 View of test cases when searching for Chrome in GTCM.
FIGURE 3.14 Simple test case for Chrome’s About dialog.
ptg7759704 The metrics around GTCM are interesting to get a feel for what testers
are doing with test cases in aggregate. The total number of tests and tests result trends are interesting, as shown in Figures 3.15 and 3.16. The total number of tests is reaching an asymptote. Basic analysis of why this is so is that Google is also deprecating older, more manual-regression focused proj-ects, along with their tests. Also, GTCM largely holds manual tests, and many teams are replacing their manual testing with automation or external crowd-sourcing and exploratory testing, which is putting downward pres-sure on total test case counts in our internal TCM—even while coverage is going up. The numbers of tests logged are increasing as these numbers are dominated by several large and necessarily manually focused teams such as Android.
FIGURE 3.15 Test case counts over time in GTCM.
The total number of manual test results logged is increasing, as one would generally expect (see Figure 3.16).
FIGURE 3.16 Test result counts over time in GTCM.
ptg7759704 Taking a look of a plot of the number of bugs associated with GTCM, in
Figure 3.17, is interesting to look at, but it also doesn’t tell the whole story.
Google is bottom-up, so some teams are particular about tracking which bugs came from which test cases; others are much looser with their data, as they don’t find that information too valuable for the project. Also, some of these logged bugs are filed via automation; not all of these are from manual test execution.
FIGURE 3.17 Total number of bugs logged during GTCM test execution over time.
A primary requirement of GTCM from day one was to have a clean and simple API. TestScribe technically had an API, but was SOAP-like and the authentication scheme was so painful that few people used it. And, as secu-rity tightened internally, the original authentication mode used made it too awkward to use. To resolve all these issues, GTCM now has a restful JSON API.
The team intends to open GTCM up for general external usage soon.
We also hope to open source this test case database for the world to collec-tively maintain. GTCM was also designed with external re-use in mind. It is built on a Google App Engine for scalability and also to allow other testers outside of Google to host their own instance if they prefer. GTCM’s inter-nals are also designed to abstract much of the logic and UI from Google App Engine, so it can be ported to other stacks if people prefer. Watch the Google Testing Blog for any updates on these efforts.