If your company is like most, there’s some amount of natural tension between marketing and product. One often controversial topic is the appropriate role in product discovery of market research tools and techniques such as focus groups, customer surveys, site analytics, site visits, usability testing/field testing and competitive analysis.
Unfortunately, I think this is an area of significant confusion, fueled in part by the various camps— those from a marketing background that may see the benefits of these tools, and those from product that see the limitations. The result is that some product teams miss out because they don’t take advantage of the information these tools and techniques can offer. Yet other teams go astray because they depend on these techniques to answer questions the tools are incapable of answering.
This is a big topic, but I’d like to discuss the major market research tools and consider how they can help you—and where they can’t.
The tools for market research have improved dramatically in the past decade. Many of the concerns of the past—which I’ll discuss shortly—are addressed by new technologies for easily reaching out to large numbers of users and customers. These technologies can also help you analyze your user’s activity and behavior—who they are and what they do with your product. That said, there are still some very fundamental, inherent limitations to market research tools, so it’s important to understand that too.
Capabilities
Let’s begin with a summary of the main tools and techniques:
Customer surveys. The Web has made this approach easy and powerful. Combined with techniques
such as conjoint analysis (to help users rank order their preferences), customer surveys are so easy and so inexpensive, that they’re a must-do for any product. However, there are two important things to note. First, there is an art to coming up with the survey questions themselves—it’s not as easy as it sounds. Think hard about the questions and context, otherwise you’ll find that people in your company will discount the results. They’ll argue “garbage in, garbage out,” which may very well be true if the questions are unclear or biased in their phrasing. Second, set expectations in your company that this data is but one input to the answer—it isn’t the answer itself. You may very well have every user come back and say “I want X” and it still may make more sense for your company to instead give them “Y.”
your users are using it. You’ll have to do a little work to make sure your site is instrumented appropriately, but it’s well worth it. Get the site analytics in place early and continually watch and learn—and adjust. If your product is not a Web site, you can usually instrument your product so that it records valuable information about how the product is used and sends the data to you. You may have to be clear to your customers that you’re sending aggregated data and nothing personally identifiable, but it’s worth getting it.
Data mining. You’ll collect data from many sources, such as the site analytics I’ve mentioned above,
billing and user account information, and your own product’s data. Today there are better tools than ever for analyzing and harvesting that data. Want to know the gender breakdown of people that use some combination of your services? Or the activity level tiers and distribution of a specific persona? You can usually answer these and thousands of other questions easily and quickly with the new breed of data analysis tools.
Site visits. There is no real substitute for visiting with your users in their native habitat—home,
office, mall—wherever they will use your product. It can be expensive and time-consuming yet, whenever I do a site visit, I realize something I wouldn’t have known any other way. Site visits are extremely valuable, but for cost and time considerations you’ll want to pick them carefully.
Personas. I like personas, especially for product definition and design. Market researchers use
personas too, although not for the same purposes. It’s essential to realize that there is no single “user” and your job is to deeply understand the major types of users out there—those who are your current customers, and those you want to have in the future. See the chapter Personas for Product
Management.
Usability testing. I am a huge fan of usability testing, and advocate its use early and often (see the
chapter Prototype Testing ). You can also use this tool with existing products to better understand what users really think. Essentially, it’s a way to see through their eyes while they use your product— you can gauge enthusiasm or frustration, and watch actions (and not just words). There are tools for doing this remotely, and for recording and analyzing what exactly people do, but this is all just icing on the cake.
Competitive analysis. Too frequently product teams write off competitors as clueless, but in my
experience every product has at least some things that the product does well, and it’s your job to find these things. Learn from their successes and their mistakes.
With these tools and techniques you can get some very real help answering the following important product questions:
Do you understand who your users really are? How are users using your product?
Can users figure out how to use your product? Where do they stumble? Why do users use your product?
What do users like about your product?
Limitations
Notice that while these questions are critically important, they do not directly address the fundamental question for most product people: What product to build? This information certainly is an input to the product creation process, but you’re in trouble if you try to steer your product with market research. The product discovery process is about answering these questions:
What technologies can I apply to solve this problem in a better way? What should the user experience be?
As useful as market research tools and techniques are, I know of no winning product that was created by market research. Not Google, not eBay, not the iPod or iPhone, not FaceBook or MySpace. None. Winning products come from the deep understanding of the user’s needs combined with an equally deep understanding of what’s just now possible.
I wish we could simply ask customers what they want, but if you do that you’ll end up with incremental and evolutionary improvements to what they already have (at best) or—more likely—a random collection of band-aid features, and not the new and dramatically better solution that you’re looking for.
If you’ve already launched your product, and if you have a set of active customers, you can learn a great deal from talking to them about what parts they like—and what parts they don’t—and get their views on incremental features. The key is to understand the limitations of each, and that this data is about refining an existing product rather than conceiving a new one.
So by all means use market research tools to help refine your product and make it as good as it can possibly be. Just don’t expect the techniques to produce the idea for the next Facebook, Flickr, or YouTube.