What’s the most misleading insight you’ve collected from your market research studies? There’s a long track record of marketers misinterpreting market research results and throwing themselves into a new product without looking back on the conclusions they reach. Honourable mentions of this brashness in action are the launches of Crystal Pepsi, “New” Coke, and Colgate’s puzzling foray into frozen meals… Yeah, that was our reaction too.
So we took a look at the common mistakes we make when carrying out research and interpreting our results. Sometimes, insights can lead us down a path paved with good intentions that ends in commercial hell. As for avoiding these mistakes, we’ve written a useful little guide to writing better surveys, the first step to making sure your insights are accurate and precise.
It all starts with a survey
Just because it’s quantitative research, doesn’t mean it’s easy to separate the wheat from the chaff. In real life, hardly anything is black and white. Collecting actionable insights starts with writing surveys that don’t skew your data. It’s an art in itself: you need to consider the comfort of your respondents, so they won’t abandon your survey, while making sure you don’t encode mistakes into the datasets you collect.
The most notorious example of this in action, malicious or not is up to you to judge, can be seen in the Trump administration’s Official Presidential Job Performance Poll, released late 2017. Their survey broke the basic rule of scale questions, by not giving equal options for positive and negative answers. This meant that their final dataset wouldn’t record criticisms of the administration. The controversy even spawned a populated Twitter hashtag, boasting contributions up to one year after the first reports.
Even assuming your survey is perfect (after reading our 10 Tips for pointed survey writing of course), how we interpret results and the weight we give to certain data can also lead us down the wrong path.
Bias: ours, yours, theirs. It’s everywhere
We all want to consider ourselves unbiased and objective judges. We hold ourselves to the humanistic standard and assume we’re, above all, objective. We staunchly believe in our capacity to make fair judgements, consider all the evidence, and use our rational minds to reach rational conclusions.
Unfortunately, human brains are just not built like that. There’s a very well populated body of research into all the cognitive processes that make us biased in our view of the world. What’s more, the World Bank’s 2015 World Development Report looked to test its own researchers and figure out how far cognitive bias affected their work. Turns out, our post-enlightenment view of ourselves might need some updating.
They tested workers by presenting them with identical datasets on two different issues: gun control and skin cream. They found that depending on their opinion on the subject, they would interpret the data differently, despite being the exact same datasets. They controlled for data literacy skills, which didn’t seem to quench this tendency to find confirmation to one’s previously held beliefs.
Researchers themselves tend to highlight conclusions they agree with, which means that cognitive bias plays a major role in how we judge and evaluate “hard” data. Be careful not to make that mistake when reading through your market research report. In fact, the World Banks’ recommendation is to get a special team of people who are uninvolved in a project to read through research designs, surveys and their conclusions. We take that into account too, and it’s the reason our research team is on hand at any stage of the research. With the help of an independent perspective, you can be sure you’re not seeing a new consumer market that’s not actually there.
One of the key skills of survey writing, which we expand on in our guide, is putting yourself in the shoes of who is answering, and trying to see the world as they would. Not doing this has a real effect on the insights you can gather. For example, by not understanding the local vocabulary you’ll create a blind spot in your data, as researchers found when trying to figure out how many people used the internet in Indonesia. Turns out, nobody uses the internet, but a lot of people use Facebook. Another example is how you’ll evaluate a scale: a score of 7/10 for the quality of a product in Germany is actually a really good result, whereas in the UK you’d be left wondering what went wrong.
Being aware of cultural habits, and remaining sensitive to culture and individuals is essential to creating marketing campaigns that jive with the consumer target. What’s more, this knowledge is incredibly important in designing research and surveys that don’t misrepresent your target consumer. As always, diversity within your research team itself is an important way to get diverse voices and perspectives into your projects.
It’s important to do Market Research right because it can help you get an overview of society as a whole, and take your company, campaigns, and product to another level.
December 12, 2018
Consumer’s grocery lists: are we what we eat?
Our grocery lists are changing: avocados travel halfway around the world to UK supermarkets (from Mexico, for the most part), we enjoy juicy fruit and veg year-round, even when they’re not in season, and a trend towards healthy eating is changing the way we think of nutrition. So how, exactly, are consumers’ grocery lists changing?…READ ARTICLE READ ARTICLE
January 14, 2019
You should know about your CRM’s blind spots
Your CRM will help you understand your client. But Agile Market Research will help you understand the consumer, the market and your position in it. If you truly want to innovate, look beyond the Consumer Relationship Management horizon. Companies always mine their data for insights. We piece together how well campaigns are received…READ ARTICLE READ ARTICLE