Lately, some people asked me how to conduct a “representative” research among their target customers. I believe that this is one of the most widely known “buzz word” of consumer research, yet the true meaning of the term seem to hide somewhere under a thick layer of statistical mumbo-jumbo.
Firstly, I would like to state that in the first half of the blog post, I will mainly talk about “quantitative” research methods when we ask a great number of people in a structured way and assume that responses will reveal proportions, ratios, rankings, preferences that we can at least partly project to our wider target group.
The first thing to note before daydreaming of getting absolutely flawless results: it is almost possible to reach the perfect level of representativeness. See the following quote:
“Life is never simple. We never have perfectly representative samples; in fact, it’s
impossible to select a perfectly representative sample. So we do our best to pick good
samples, and we use probability theory to work out a predication of how confident we can be that the statistics from our sample are representative of the entire population.”
This doesn’t mean that we should bin such methods! Not at all! It is very useful to ask members of your target groups to get a sense of preferences. Look for tendencies and patterns that can guide you – this is what you need, not (only) the exact results on a chart.
I just wanted to point out that sometimes you have to accept this special beauty of statistics and don’t sweat it too much. It’s still better doing it than not doing it at all.
Of course, once you do it, try to do it right.:) There are some important guidelines that you should follow if you would like to get some reliable research done.
1. Let’s start with identifying your target group. It can actually be pretty much anything, starting from the niche category of red-haired city dwellers that skateboard to work (and remember, only 4% of the humanity is naturally red-haired and they surely can’t all do skateboarding) through to all the people that use the internet in English (+500 millions). Next, you should try to find sub-groups within the big group, e.g. red-haired males and females, just to name a simple example.
The more sub-groups you can identify, the more precise your recruitment technique can be, because you won’t forget to capture this real-life diversity in your sample. You should do this as accurately as possible, doing some secondary research, in one word, “doing your homework”.
2. Now, how can I create a (relatively) representative study among these people? Is the truth in the numbers? Not necessarily. It is not enough to figure out a sample size that seems big enough in the context of your target group. Obviously, setting the optimal (minimum) number of respondents is an important step, but a big sample on its own doesn’t guarantee success and we will soon discuss why.
Still talking about the magic numbers themselves, there are some basic guidelines and calculators that help you set an approximately relevant minimum size as a rule of thumb.
I definitely recommend SurveyMonkey’s cheat sheet on the topic where they explain the main attributes of a quantitative study in a very simple language (!) and also suggest sample sizes.
However, if you take a closer look, you can see that no matter if your target group comprises 100K people or over one million, about 400 respondents are suggested for an online survey.
In fact, even if you have a really huge target segment, it would just be crazy to expand your sample size to that proportion. It would cost waaaay too much time and money. Obviously, big companies and public polls usually ask around 2K people or even more, but it is not a shame if you don’t have the same budget and capabilities.:) Remember, it is much better to do some research than not doing any at all.
In order to be able to interpret such size calculators, we have to decrypt two scary words appearing next to the sample sizes, namely “confidence interval” and “margin of error”.
As a rule of thumb, a rationalistic confidence interval is usually set at 95%. SurveyMonkey explains it the following way: setting a 95% confidence interval means that you would get the same results 95% of the time (imagining that you conducted the research multiple times, not just once). It means that your research and your sampling method can be considered pretty reliable (from my point of view).
Margin of error is usually considered 5% as a rule of thumb. This means that if you conducted a research among red-haired respondents and found out that 90% of your red-haired respondents use a longboard, a 5% margin of error would add 5% on either side of that number, meaning that actually 85-95% of your sample uses a longboard (and not another type of skateboard). It sort of relates to the level of accuracy.
A longboarder girl just about to get goin’
Final note regarding numbers: I don’t want to disappoint anyone, but you have to consider that not everyone receiving the link will respond to your survey invitation, so you might want to send the invitation to a much higher number of people than your preferred sample size. (20-30% response rate is usually considered very lucky!)
Also: don’t worry if you don’t have that much time or resources, just try to look on the bright side and be happy with whatever response rate you got and try to make the most of it!
3. After taking a look at the numbers, remember I said that a sample big enough does not guarantee success on its own. Why? Because everything depends on getting the right people!
At this part, we both talk about quantitative and “qualitative” research. This latter is about “quality” (hence the name) rather than quantity of responses, and it is much deeper, much more intuitive, in many case it is like a session at the psychologist. It reveals a lot deeper insights about your target group members when it is done the right way.
“The right way” not only means asking the right questions and being able to see the patterns even if you only talked to a few people, but also about finding the right respondents. The right respondents are such people who truly represent your target population – preferably true to the level of real-life diversity of people we talked about – and are as little biased as possible.
What does that mean? Even if it seems truly enticing, try not to limit your qual sample to your friends’ circle. Nor to the students of one university, to visitors of just one external page etc., in general, don’t limit your recruitment to one source.
I know it seems easy and low-cost, but they will probably provide less versatile opinions as they are under the same influences. Friends might also be much more biased about you and your ideas than strangers; the phenomenon of adapting to the imaginary expectations of the interviewer is quite a general risk in research anyway, although it can be reduced by smart interpersonal skills and formulating questions the right way.
If you only rely on people that are the easiest to reach, it is called “selection bias”. It’s advised to mainly include people from different environments who are relevant to you yet are probably less biased. Involving respondents from your sign-up list, sending out invitations on professional forums, involving bloggers, walking into stores, going out to retail sites or streets, recruiting on Facebook pages or even just asking friends to share the link with others can all increase the chance of getting more realistic feedback. (This may both refer to quant and qual type of research.)
Obviously, it is still worth asking your friends about their opinion and suggestions, especially if they know much about the topic, but try to enrich, “target and randomize” your sample as much as you can, by addressing the right type of people (this can also be ensured with some initial screening questions) and sampling from different sub-groups instead of just one little community.
How to choose the right type of people in terms of relevant characteristics? Remember the part when we talked about knowing more of your target audience? We have to use those discoveries now, identifying attributes that our main target population or its sub-groups have. They act like a fingerprint that helps us find the right people to ask.
One website describes the following example for such a process:
„The match between the sample and the universe must be strong for all attributes anticipated to be influential on survey outcomes. One example of a sample-to-universe match could be the selection of consumers for a perfume designed by a young, female celebrity. In this instance, attributes anticipated to be influential would be: Female, 18-28 years old, entertainment-savvy. A secondary set of attributes might be: Urban-dwelling, enrolled in college, residing on the east coast or west coast, discretionary income (income levels).”
Look out for people with the necessary characteristics and get rollin’.
4. Final words:
As a summary, the main points are: don’t get lost in numbers too much; doing your homework and getting to know your target population and its degree of diversity as much as possible is the most important thing. Don’t get too lazy and try to devote energy to crucial factors such as involving representative people from various resources, asking the right questions and trying to identify patterns, tendencies.
I hope this post wasn’t too intimidating to you and helped you get a more complex picture on some research “buzz words” and what’s really important.
If you would like some help regarding such research methods or have any questions, please feel free to contact me as I would be happy to assist you.
Articles for those who were sitting with hungry eyes in the front row at the Stats class:
For those who didn’t: