The oldlest bug filed and still in existence: May 18, 2001, 15:33. The title is Test Bug. The body of the documentation is “First Bug!.” Interestingly, this bug is often used accidentally when developers are asked to provide a bug that their CL fixes.
The oldest active bug was filed in March of 1999. This bug suggests a performance investigation to reduce latency for serving for ads based on
8We interviewed Ted Mao in Chapter 2.
ptg7759704 geographic region. The last activity was in 2009. The last edit said that this
could be investigated, but that it would require architectural work, and the latency metrics are just fine.
Here are some bug charts of overall bug activity at Google. Some of these are filed automatically and some manually; these are aggregate infor-mation. Some of the automation dominates these bug trends, and we don’t highlight any single teams here, but it is interesting nonetheless.
As you can see in Figure 3.18, there are many P29bugs, far fewer P1, and even fewer P0 bugs. Correlation is not causation, but it can be a sign that the engineering methodology described in this book works. It might also be that no one bothers to file P1 issues, but that isn’t what we see anec-dotally. P3 and P4 bugs are often not filed as they are rarely looked at.
9Like many bug-filing systems, we use PX, where X is an integer, to define how high a priority the bug is. P0 bugs are the worst, P1 less awful, and so on.
FIGURE 3.18 Number of bugs in Buganizer by priority.
The average age of bugs is also generally what would be expected (see Figure 3.19). The anomaly seems to be with P0 bugs. In practice, though, P0 bugs are often more difficult to fix as they represent severe design or deployment issues, which are often complex to debug and resolve. The rest of the bugs on average take longer to fix, with increasing priority numbers, as they are less important.
ptg7759704
The chart for the number of bugs found over time in Figure 3.20 shows a slight uptick in bugs found each month. We don’t systematically under-stand why this is increasing. The obvious explanation is that we have more code and more coders over time, but the bug rate increases at a rate lower than the increase in number of testers and developers. Perhaps either our code is getting better with quality controls or we aren’t finding as many of them.
FIGURE 3.19 Average age of bugs in Buganizer.
ptg7759704
Our fix rate in Figure 3.21 shows that teams generally have a handle on the bug rate. Many teams simply stop adding features when the incoming bug rate exceeds the team’s ability to fix them. This practice is highly rec-ommended versus focusing on the feature or code-complete milestones.
Focus on small bits of tested code, incrementally testing, and dogfooding help keep bug rates in check.
FIGURE 3.20 Trend of bugs found.
ptg7759704
As Google’s products have moved more into the open, such as Chrome and Chrome OS, it’s no longer possible to maintain a single bug database.
These projects also deal with externally visible bug databases such as Mozilla’s Bugzilla for WebKit issues and issue tracker for chromium.org issues. Googlers are encouraged to file issues in any product they see, including those from competitive products. The primary goal is making the Web as a whole better.
Issue Tracker is the core repository for all Chrome and Chrome OS bugs. Its bug database is public. This means anyone can look at the bug activity, even the press. Security bugs are sometimes hidden between the time they are found and fixed to avoid tipping off hackers, but other than that, it is a public bug database. External users are free to file bugs, and they are a valuable source of bug information. Figures 3.22 and 3.23 show searching and finding a bug related to the Chrome logo in the About box.
FIGURE 3.21 Bugs fixed over time. It’s a good thing this looks similar to the trend of bugs found!
ptg7759704 FIGURE 3.22 Issue tracker search.
FIGURE 3.23 Open issue in Chromium Issue Tracker.
ptg7759704 However, the longest-lived and most widely used piece of testing
infra-structure at Google is Buganizer and it deserves further discussion. For the most part, it is a typical bug database, but it also supports the core quality cycle of tracking problems in our software, from discovery to resolution, and in building regression test cases. Buganizer is also built on top of Google’s latest core storage technology for scale and speed.
Bugs are filed with a subset of the following fields, very few of which are required, and their definition is deliberately not well defined so individ-ual teams can decide how they want to manage the information to fit their own workflow.
• Assigned to (Assignee)
[Optional] LDAP name of one person who should take the next step in handling this issue. This person is automatically emailed when the issue is created and whenever anyone changes the value of any field in the issue. Buganizer administrators specify a default assignee for every component.
• CC
[Optional] Zero or more LDAP names of people to email when an issue is created or modified. Names are LDAP or mailing list names only, no
@google, so only Google mailing lists or employees are valid entries.
This is a comma-delimited list. Do not include the “Assign to” name when filing an issue, because this person is emailed by default.
• Attachments
[Optional] Zero or more attachments to the bug. Any file type is accepted. There is no limit to the number of attachments for an issue, but the maximum total size of each attachment is 100MB.
• Blocking
[Optional] The IDs of bugs that this bug prevents from being resolved.
This is a comma-delimited list. Updating this list will automatically update the Depends On field in the listed bugs.
• Depends On
[Optional] The IDs of bugs that must be fixed before this bug can be fixed. Updating this list will automatically update the Blocking field of all listed bugs. This is a comma-delimited list.
• Changed
[Read-only] The date and time when any value in the issue was last changed.
• Changelists
[Optional] Zero or more change list (CL) numbers for CLs dealing with this issue in some way. Only specify CLs that have been submitted; do not specify pending CLs.
ptg7759704
• Component
[Required] The thing that has the bug or feature request if it is known.
When filing an issue, this should be a complete path to the component.
An infinitely long path is now supported. When filing an issue, you do not need to assign the issue to a leaf component (that is, one without children).
Additional components can be created only by project and engineering managers.
• Created
[Read-only] The date when the bug was created.
• Found In
[Optional] Use for versioning by entering the number of the software version where you found the issue, such as 1.1.
• Last modified
[Read-only] The date that any field in the issue was last modified.
• Notes
[Optional] Detailed description of the problem and running comments as the issue is handled over time. When filing the issue, describe the steps required to reproduce a bug or how to get to a screen involving a feature request. The more information you put here, the less likely future issue handlers will need to contact you. You cannot edit previous entries in the Notes field, not even if you added them; you can add only new values to the Notes field.
• Priority
[Required] The importance of a bug, where P0 is the highest priority.
This indicates how soon it should be fixed and how many resources should be allocated to fix it. For example, misspelling “Google” in the search page logo would be low severity (the page function would not be affected), but high priority (it would be a Very Bad Thing). Setting both fields lets the bug fix team allocate its time more wisely. See also the Severity description.
• Reported by (Reporter)
[Read-only] The Google login of the person who originally reported the bug. The default value is assigned to the person creating the bug, but this can be modified to give credit where credit is due.
• Resolution
[Optional, entered by Buganizer] The final action chosen by the verifier.
Values include Not feasible, Works as intended, Not repeatable, Obsolete, Duplicate, and Fixed.
ptg7759704
• Severity
[Required] How much the bug affects the use of the product, where S0 is the most severe. Setting both priority and severity can help prioritize the importance of this bug to bug fixers. For example, misspelling
“Google” in the search page logo would be low severity (the page func-tion would not be affected), but high priority (it would be a Very Bad Thing). Setting both fields lets the bug fix team allocate its time more wisely. Severity values have the following text equivalents:
— s0 = System unusable
— s1 = High
— s2 = Medium
— s3 = Low
— s4 = No effect on system
• Status
[Required] The current state of the bug. See Life of an issue (see Figure 3.24) for details on how these values are set in the issue. Available sta-tuses include the following:
— New: The issue has just been created and not assigned yet.
— Assigned: An assignee has been specified.
— Accepted: The assignee has accepted the issue.
— Fix later: The assignee has decided that the issue will be fixed in the future.
— Will not fix: The assignee has decided that the issue will not be fixed for some reason.
— Fixed: The issue has been fixed, but the fix has not been verified.
— Verifier assigned: A verifier has been assigned to the issue.
— Verified: The fix has been verified by the verifier.
• Summary
[Required] A descriptive summary of this issue. Be sure to make this as descriptive as possible; when scrolling through a list of issues in a search result, this is what helps the user decide whether or not to exam-ine the issue further.
• Targeted To
[Optional] Use for versioning by entering the number of the software version in which the issue should be fixed, such as 1.2.
ptg7759704
• Type
[Required] What type of issue:
— Bug: Something that causes the program not to work as expected
— Feature request: Something you would like to see added to the program
— Customer issue: A training issue or general discussion
— Internal cleanup: Something requiring maintenance
— Process: Something tracked automatically via the API
• Verified In
[Optional] Use for versioning by entering the number of the software version where the issue fix was verified, such as 1.2.
• Verifier
[Required before issue can be resolved] Each issue is assigned one per-son who has the right to mark the issue as resolved. This perper-son need not be assigned until the issue is ready to be resolved, but the verifier is the only one who can change the status to “Verified” (the issue is closed). The verifier can be the same person as the assignee.
Figure 3.24 summarizes the life of an issue.
FIGURE 3.24 The basic workflow for bugs in a Buganizer world.
Life of an Issue
ptg7759704 A few key differences between bugs at Google and elsewhere:
• For the most part, the bug database is completely open. Any Googler can see any bug in any project.
• Everyone files bugs, including engineering directors and (Senior Vice Presidents (SVPs). Googlers file bugs against products they use even if they are not part of that product team. Groups of testers often “dog pile” on a product just to be helpful.
• There are no formal top-down bug triage processes. How bugs are triaged10varies widely from team to team. Sometimes it’s an individual task or a TE and SWE do it together informally at one or the other’s desk. Sometimes triage is part of a weekly or daily standup meeting.
The team figures out who should be there and what works best. There are no formal methods or dashboards for triaging or big brothers check-ing in on a team. Google leaves this up to the individual teams to decide.
Google projects are often in one of two states: new and rapid develop-ment swamped with a continual stream of issues or established infrastruc-ture released incrementally and with so much upfront unit testing and fault tolerance that bugs aren’t very common and kept to a small and manage-able number.
The dream of a unified dashboard has haunted Googlers, much like it has at many other companies. Every year or so, a new effort tried to take root to build a centralized bug or project dashboard for all projects at Google. It makes sense, even if confined only to Buganizer data. But, as each group has different ideas of what metrics are important for shipping and for day-to-day health of the projects, these projects have consistently not gotten far off the ground. This idea may be useful at more uniform com-panies, but at Google, well-intentioned folks often keep ending up frus-trated due to the variety of projects and engineering at Google.
A Google-scale approach to filing bugs was started by the Google Feedback team11(http://www.google.com/tools/feedback/intl/en/
index.html). Google Feedback enables end users to file bugs against select Google services. The general idea is that external users don’t have full con-text on what bugs have already been filed, or even already fixed, and we want to make it quick and easy to get feedback from users. So, Google testers and software engineers have made it point-and-click simple for users to file a bug. The engineers on this team have gone to great lengths to ensure that users can intuitively omit parts of the page that might contain private data when filing bugs back to Google (see Figure 3.25).
10Triage is the process of reviewing incoming bugs and deciding what order and who should fix them. It is much like triage in a hospital emergency room.
11An interview with Brad Green, engineering manager for Google Feedback appears in Chapter 4.
ptg7759704 The Google Feedback team has done some very cool work to avoid
blindly dropping these bugs from the wild into our bug databases, as dupli-cate issues can overwhelm the bug triage process. They use clustering algo-rithms automatically to de-dupe and identify the top issues coming in.
Google Feedback has a helpful dashboard showing this processed data of user-reported issues to the product team for analysis. It is not unusual for many tens of thousands of pieces of feedback to pour in the week after a major application launch—and have them boil down to just 10 or so major and common issues. This saves a lot of engineering time and makes it possi-ble for Google to actually listen to end users without having to process each individual piece of feedback that would be prohibitive. The team is beta testing this with a small set of Google properties, but aims to provide this for all Google properties in the future.
FIGURE 3.25 Google Feedback shown with privacy filtering.
ptg7759704