In product management, you’re always on the hunt for fresh user feedback. You want to answer your most pressing questions and inform the way you shape your product.
For example, it could be that your digital product is acquiring users, but they’re not staying with you.
Maybe engagement is low, and you aren’t sure why.
Engagement may be great, and you’re wondering how to leverage that success in a future product.
Whatever the situation, you know you need input from a group of users or potential users. It seems logical to run a survey to find out the answers to your questions. After all, it’s quick, easy, and provides a lot of data — right?
Not so fast. Easier doesn't equal better data.
The risk of surveys: Bad data is worse than no data
Surveys are easy to put together, but they often provide an incomplete picture.
One company discovered this the hard way
After conducting a phone survey, they concluded that 40% of their target market wanted the company’s digital product to go down a particular path.
The company successfully pitched that route to investors and raised several hundred thousand dollars.
They then spent a huge sum of money licensing technology and building the product their survey indicated the market wanted.
Their product launched — to crickets. Why?
Because the survey data they relied on was incomplete. Instead of speaking directly to customers to learn the nuances of what they wanted, their survey forced respondents’ answers into a series of boxes. Those predefined multiple choice answers didn’t accurately represent their desires.
Dive deeper into this discussion
Listen to a discussion on the risk of using quantitative surveys too soon in your customer research process on this episode from the Exploring Product podcast.
The role of bias: why surveys are difficult to do well
People are not computers. We want to think that by administering a survey, we’ll be able to easily categorize people and their behaviors.
Unfortunately, people don’t always fit into neat categories.
Think about it like this: What did you have for lunch yesterday? The day before that? What about the day before that? You might answer, “Oh, I had a sandwich today, a bowl of soup yesterday, and a casserole the day before that.” It’s likely you’d have to think quite a bit to recall what you ate any longer ago than that.
Past a certain point, your answers would be nothing more than guesswork. And it would be your best guess, based on the years of lunches you’ve eaten. There would be no malice in the fact you likely provided incorrect information.
People do not aggregate their daily actions into a running total that lets you know you’ve had soup for lunch 33% of the time, sandwiches 40%, and so on. While that may be the information the survey is designed to collect, the data will still be inaccurate.
People are not wired to automatically tabulate data, and we also don’t re-adjust the scale. Most survey participants aren’t going to read the entire question set first. As a result, a survey may introduce bias based on the order in which questions are presented. Respondents will often answer subsequent questions relative to the first question they answered. Therefore, the first question creates a baseline for responses.
As an example, if the answer set ranges from 1-10, and respondents answer the first question with a 4, they will be inclined to respond to future questions based on their first response.
Surveys reflect the biases of the people who write them
The answers and scales you create may not align with how your participants think. if you don't provide a possible response that matches their situation, they may be forced to select something less than accurate. And that means bad data.
As an example, consider the common survey question, “Do you want to receive discounts for products you use?” Who isn’t going to say yes to that question? Almost no one.
But a more revealing question may be to ask, “How often do you use discounts?”
It may come as a surprise to find that not everyone uses discount codes even when they are provided, but the answer to that question could result in an unexpected twist to the survey results.
Unexpected twists lead to a different set of conclusions, and thus different decisions about the product.
Key considerations before running a qualitative survey
Quantitative surveys are designed to measure quantity, while qualitative surveys measure quality.
As an example, a survey designed to determine how frequently a user interacts with your product would be a quantitative survey, while a qualitative survey would measure how the user feels about interacting with your product.
Ask yourself what you are really hoping to learn from the survey.
If, returning to an earlier example, you’re struggling with low engagement and hope to run a survey to figure out how to increase it, take a step back. Your first step is to figure out why user engagement with your product is low in the first place. And a quantitative survey isn’t going to provide that information.
Instead, start by looking at your product analytics. That will help you zoom in to find where users are running into stumbling blocks using your product. Some users will be very successful, but others might hit a brick wall at predictable points and drop off. You’ll want to sample those two populations in different ways.
Once you’ve defined the groups from which you need information, conduct a few customer interviews. Even as few as five will begin to shine a light on the real questions you should be asking. Start by asking about their experience with your product. “When's the last time you used the product? Why didn't you pick it back up again? What are you using now instead?”
Once you have initial answers to those questions, you can begin considering whether a quantitative survey would be appropriate. Surveys are useful for validating what you already know. You can use them to confirm the data you gathered from qualitative research or to help quantify insights you already have.
If customer interviews are difficult to arrange, you can run a qualitative survey with open-ended questions, but be aware that the results may not be consistent.
Prioritize qualitative data in digital product research
Survey design is the subject of doctoral-level courses, so it’s no surprise that running a survey well is going to require more than stringing together a list of questions you want answered.
Once you’ve had initial customer interviews and know what questions to ask, you can begin constructing a survey.
Let's say you identify a trend via customer feedback but aren't sure how prevalent it is. A survey can help you quantify the problem by telling you how big or common it is. But it can't tell you the why behind the problem.
For that critical information, only qualitative research will do.
Prioritize qualitative data through open-ended questions and customer interviews. Once you’ve collected an initial data set from those interviews, you may find a survey isn’t even necessary.
When searching for insight from your customers, surveys are an incredibly valuable tool — in the right situation, and with the right questions. Before defaulting to a survey, think carefully about what information you’re hoping to learn.
Talk to customers before doing surveys
Prioritize qualitative research to make your product teams customer-centric rather than product-centric.
And remember: People are complicated, and so are the reasons behind our actions.
More resources on customer research
Understanding Your Customers' Journey: Improve the Way You Help Your Customers
Customer Research Methods for Startups
Continuous Discovery Habits with Teresa Torres
Understanding Your Customers When You Don't Have Any Yet
