Most of the clients we talk to have had some negative experiences with consumer research before. A number of them just put up with them because there are few alternatives and “that’s just the way it is”, and others have stopped doing research altogether because they did not feel like the value they got for their money was high enough to justify the expense it the hassle, the money and the lack of reliability of the results. In this post, I’d like to go through some of the pains that clients have mentioned, and explain why those exist. Maybe you are suffering from the same pains, and it’s worth understanding why they exist and what to do about it.
In general, there are three main problems with consumer research.
- It’s very expensive
- It’s very slow when it comes to delivery of results
- It’s often lacking in terms of data and reliability (statistical significance)
Let’s go through all of these and explain the underlying problems.
It’s very expensive
Do me a favor: Next time you talk to your consumer research agency, ask them where the respondents for your study come from and if they know who they are. Chances are, one of two things is true: They either a) have recruited via telephone or b) have sourced respondents from so-called sample providers. In both cases, they will have not idea who the respondents actually are. This matters, because both cases drive up the price.
If recruitment has been done via telephone (still the main method to get respondents), in most countries that means that an army of call center agents has dialed random numbers and asked for 10 minutes of their time to go answer your survey. Most of these calls, still today, happen on landlines. Now think about what this means in terms of reliability of data: Your survey is answered by people who a) have a landline phone and b) are willing to give up 10 minutes of their time to answer questions they have no incentive to answer. If I ask a random person on the street to go fetch me a sandwich (“it will only take 10 minutes”), chances are they will look at me weirdly and just walk away. It takes a special kind of person to agree to do a stranger a favor without any incentive to do so.
If respondents are sourced through a sample provider (e.g. a company that provides respondents to market research companies), this opens up a new set of questions: Where did they get the respondents? Is the data reliable? What stops a sample provider to just make up random answers? How are respondents incentivized? All those questions are difficult to answer when you work with external suppliers and do not know exactly who each respondent is.
The reason why both cases bring up the price is that in the first case it’s extremely difficult to actually find people who are willing to answer a survey without any incentive to do so. In fact, the response rate for those surveys has been declining for years.
With sample providers, you simply add middle-men into the mix who also wants to earn money. In fact, in many cases, a research company will hire a sample provider who hires a sample provider who hires a sample provider and so on, in which case you add several middle-men who all take a cut. Often time market research companies also work together and “share” respondents, meaning that company A provides certain respondents to company B and vice-versa, and every time additional companies take a cut of the money.
Lastly, the way that most consumer research companies work is incredibly manual. There are many people involved in gathering, cleaning, and interpreting the responses to a survey before someone stands in your office and clicks you through a PowerPoint. Afterwards, many times the results are made so difficult to act on that you are inclined to pay for additional consultancy services.
So, next time you talk to your research company or interview companies that you are evaluating, ask them the following questions:
- How do you recruit respondents, and how are they incentivized?
- Did you recruit them yourself? How did you recruit them?
- Do you know the real identity of each respondent?
You will be surprised by the answers you’ll get.
It’s very slow when it comes to delivery of results
You might know already what’s coming: This is a direct result of what I mention above. Because it is difficult to recruit responses in the traditional way via phone, it takes a lot longer to recruit a certain number of respondents. Because handling the resulting data in a manual way takes a long time, it takes longer for responses to turn into insights and be delivered to you.
There is an additional component to this: When you actually see the data, most of the time you will want to dig deeper into it. It’s natural that any research report you get sparks questions and makes you want to test certain assumptions by breaking the data down into certain segments, correlate different answers, and so on. In the traditional way, this means briefing the research consultant on what you want to know, and them finding the answers to your questions. This adds another layer of time to the whole process (in addition to additional consultancy fees) and is repeated the next time you have a question.
At Trybe, we believe that allowing you to play with data yourself in an intuitive way is the much better way to do it. You already know what questions you have, so we think giving you the ability to answer it yourself in seconds by providing you with an intuitive dashboard with access to all your data is the much faster and convenient approach. You know your brand and product better than anyone, so why add a middle-man who you need to explain your question to when you should be able to answer questions anytime, anywhere, yourself.
It’s often lacking in terms of data and reliability (statistical significance)
You guessed it: This is, again, a problem caused by the difficulty to recruit and not knowing the identity and characteristics of each respondent.
Most of the time, research companies will tell you that getting 1000 respondents for your study is enough. And it is if all you want is look at a representative set of the total population. The problem is that looking this kind of data is not actionable. If a metric, for example, brand awareness, goes up across the population, what exactly are you supposed to do with that? What you WANT to know is a) “why did it go up?”, b) “does it differ between different segments of the population? Why?” and c) “What can I do to bring it up even more?”. To answer these questions you need to look into certain sets of the population rather than the population as a whole. The problem with 1000 respondents is that the moment you start breaking down the respondents into groups, statistical significance goes out of the window. You want to look at women between the age of 18 and 24? If you had 1000 respondents at the start, you are now left with 50. Good luck making any reliable discoveries given those numbers.
So why do companies stick to 1000 respondents? Two reasons: First, it’s a number that everyone thinks about when they think about “representativeness”, mostly driven by political polls (which most of the time do not break down that pool further). Secondly: More money and less hassle for them. Since it’s so hard to recruit respondents it’s just a lot easier to stick to a smaller number.
So, next time you get your research results back and someone wants to sell you a study with 1000 responses, ask them why they picked 1000.
Every problem with consumer research can be traced back to two things: a) The problem of recruiting respondents and the way it is currently done most of the time, and b) the fact that research is very manual in how it’s being conducted and delivered. We started Trybe to fix exactly those problems to make consumer research cheaper, faster and better. First, we know every respondent because we own our own panel of users who are the ones answering the surveys. Secondly, we believe in recruiting with interesting incentives and with a global blog and community partnerships. Everyone should have an incentive to answer questions. And thirdly, we automate the research process so that incoming data is automatically turned into insights, allowing you to answer your own questions through an intuitive and easy to use dashboard. Of course, we help, but we will never charge you for it. Our job is only done when all your questions are answered.
I should mention that there are good research companies out there doing great work. Not all of them are stuck in the 80s. But it’s important to ask the right questions when you are evaluating a research agency so you aren’t stuck with one that is.
FUE’s (“Frequently used excuses”)
“We use telephone surveys because it’s the only way respondents are not self-recruiting, which makes it much more reliable”
“Self-Recruiting” means that a respondent is volunteering to answer a survey by themselves rather than being prompted to do so. A frequent argument for telephone surveys is that people are randomly called, so the answers to the survey also provide a random sample of the population. Unfortunately, this is not true. There is a huge element of bias in telephone recruiting: Firstly, most of the time and in most countries, you can only recruit people with a landline phone since there is no public directory of mobile numbers. Less than half of the US population live in a household with a landline nowadays, which means you are cutting out 50% of your possible respondents, largely skewed towards the younger generations. Secondly, as I mentioned above phone recruiting relies on someone giving up their time without any incentive to do so. THAT is self-recruiting in its own right and skews the data in a big way.
“It’s important that actual people look at the data when it is turned into metrics. A computer can’t do that”
A computer can crunch any amount of data into any metric you would like to look at because crunching raw data is what a computer was made to do. This data is the basis for metrics and our job is to give you the ability to make sense of them. That doesn’t take away the need for people to look at it and decide on actions based on the metrics, but there is nobody better suited to come up with appropriate actions than you. Research companies will never know your brand or product well enough to make good recommendations. Our job is to provide you with good, fast and reliable information so you can make an informed decision based on your knowledge.
Related Blog Posts
Consumer Insights: An Introduction to The Basics of What It Takes
Increase Sales And Grow Your Portfolio With Consumer Research