When we find the right mix of skills, we bring a candidate in for an inter-view. We are often asked how we interview TEs; indeed, it ranks as the most common question we receive on our blog and when we speak at pub-lic events. We’re not willing to give up the entire set of questions we ask, but here is a sampling (which will be deprecated now that they are public!) to give you insight into our thinking.
To begin, we probe for test aptitude. Our intent is to find out whether the candidate is more than just bright and creative, and whether he has a natural knack for testing. We are looking for innate curiosity about how things are built and what combinations of variables and configurations are possible and interesting to test. We are looking for a strong sense of how things should work and the ability to articulate it. We are also looking for a strong personality.
The idea here is to give a testing problem that requires a variety of input and environmental conditions and then to ask the candidate to enu-merate the most interesting ones. On a simple level, we might ask a candi-date to test a web page (see Figure 3.26) with single text input box and a button labeled count which, when pressed, computes the number of As in the text string. The question: Come up with a list of input strings you would want to test.
FIGURE 3.26 Sample UI for test question.
Some candidates just dive right in and start listing test cases. This is often a dangerous signal because they haven’t thought about the problem enough. The tendency to focus on quantity over quality is something we have learned to interpret as a negative (mostly through experience) because it is an inefficient way to work. You can learn a lot about a candi-date by just looking at how they approach a problem before they even get to their solution.
Better is the candidate who asks clarifying questions: Capital or lower-case? Only English? Is the text cleared after the answer is computed? What about repeated button presses? And so on.
ptg7759704 When the problem is clarified, candidates then start listing test cases. It
is important to see whether they have some method to their madness. Are they just looking to break the software or also looking to validate that it works? Do they know when they are doing the former or the latter? Do they start with the obvious simple stuff first so they can find big bugs as quickly as possible? Can they figure out how to clearly represent their test plan/data? Random ordering of strings on a white board doesn’t indicate clarity of thought, and their test plans are likely to be ragged if they bother to plan at all. A typical list might look something like this:
• “banana”: 3 (An actual word.)
• “A” and “a”: 1 (A simple legal case with a positive result.)
• “”: 0 (A simple legal case with a zero result.)
• null: 0 (Simple error case.)
• “AA” and “aa”: 2 (A case where the count is > 1 and all As.)
• “b”: 0 (A simple, nonblank legal case with a negative result.)
• “aba”: 2 (Target character at the beginning and end to look for an off-by-one loops bug.)
• “bab”: 1 (Target character in the middle of the string.)
• space/tabs/etc.: N (Whitespace characters mixed with N As.)
• long string without As: N, where N > 0
• long string with As: N, where N is equal to the number of As
• X\nX in the string: N, where N is equal to the number of As (Format characters.)
• {java/C/HTML/JavaScript}: N, where N is equal to the number of As (Executable characters, or errors or accidental interpretation of code.) Missing several of the previous tests is a bad indicator.
Better candidates discuss more advanced testing issues and rise above the specifics of input selection. They might
• Question the look and feel, color palette, and contrast. Is it consistent with related applications? Is it accessible for the visually impaired, and so on?
• Worry that the text box is too small/suggest that it be long enough to accommodate the longer strings that can be entered.
• Wonder about multiple instances of this application on the same server.
Is there a chance for crosstalk between users?
• Ask the question, “Is the data recorded?” It might contain addresses or other personally identifiable information.
ptg7759704
• Suggest automation with some real-world data, such as drawing from a dictionary of words or text selections from books.
• Ask the questions, “Is it fast enough? Will it be fast enough under load?”
• Ask the questions, “Is it discoverable? How do users find this page?”
• Enter HTML and JavaScript. Does it break the page rendering?
• Ask whether it should count capital or lowercase As, or both.
• Try copying and pasting strings.
Some concepts are even more advanced and indicate an experienced and valuable testing mind willing to look past only the problem presented.
They might
• Realize that if the count is passed to the server via a URL-encoded HTTP GET request, the string can be clipped as it bounces across the Net. Therefore, there is no guarantee how long the supported URL can be.
• Suggest the application be parameterized. Why count only As?
• Consider counting other As in other languages (such as angstrom or umlaut).
• Consider whether this application can be internationalized.
• Think about writing scripts or manually sampling string lengths, say by powers of 2, to find the limits and ensure that string lengths in between would work.
• Consider the implementation and code behind this. There might be a counter to walk the string and another to keep track of how many As have been encountered (accumulator). So, it’s interesting to vary both the total number of As and the length of a string of As around interest-ing boundary values.
• Ask the questions, “Can the HTTP POST method and parameters be hacked? Perhaps there is security vulnerability?”
• Generate test input and validation with script to create interesting per-mutations and combinations of string properties such as length, num-ber of As, and so on.
Digging into the issue of what length strings the candidate uses as test cases often reveals a lot about how well they will do on the job. If candi-dates cannot be more technically specific than “long strings,” which is an all too frequent answer, then that is a bad sign. More technical candidates will ask what the specification for the string is and then provide boundary tests around these limits. For example, if the limit is 1,000 characters, they will try 999, 1,000, and 1,001. The best candidates will also try 2^32, and many interesting values in between, such as powers of two and ten. It is
impor-ptg7759704 tant that candidates show an understanding of which values are important
versus just random numbers—they need to have some understanding of the underlying algorithms, language, runtime, and hardware because this is where the faults most often live. They will also try lengths based on possi-ble implementation details, and think of counters and pointers and off-by-one possibilities. The very best off-by-ones will realize the system might be
stateful and that tests must take into account previous values entered. Thus, trying the same string multiple times or a zero length after a 1,000-length string become important cases.
Another key characteristic often looked for in interviewing is the ability for the TE to deal with ambiguity and push back on goofy ideas. We often change specifications or specify a behavior that doesn’t make sense when candidates ask clarifying questions. How they deal with this ambiguity can reveal a lot about how well they will do on the job. At Google, specifica-tions are often moving targets and open to interpretation and modification given the pace of our release cycles. Suggesting that the maximum length of five characters is odd and would likely frustrate users is a great indicator that they are thinking of the user. Candidates who accept this blindly and move on might do the same on the job and end up validating goofy behav-ior. Candidates who push back or question specifications will often do won-ders on the job, as long as they can do it diplomatically.
The final part of our TE interviews is probing for what we call
“Googliness.” We want curious, passionate engineers who don’t just do what they are told but investigate options and do things outside the direction job description. Job responsibilities must get done, but life and work should be about maximum impact on the world around you. We want people who are connected to the world around them and the larger com-puter science community. People who file bugs on open-source projects are one example or those who generalize their work for reuse are another. We want to hire people who are enjoyable to work with, get along well with others, and who add to our culture here. We want engineers who want to continue to learn and grow. We also want people who we can learn from as well—people with new ideas and experiences that will add to our collective talent pool. With a corporate motto of “do no evil,” we want people who will call out evil if they see it!
Interviewing at big technology companies is intimidating. We know because it was intimidating to us! Many don’t make it through the first time around and require a practice round to get it right. We’re not trying to be harsh; we are trying to ensure that we find people who can contribute and when they become Google TEs, can grow in their role. It has to be good for the company and the candidate. Google, like most companies intent on keeping a small-company feel despite the big company numbers, tends to err on the side of caution. We want Google to be a place we want to con-tinue to work for many years to come. Hiring the right people is the best way to ensure that!
ptg7759704