A couple of months ago one of our blog posts mentioning our view on the (non existing) value of demographic targeting for user testing got featured on Designer News. Man, we had to face harsh criticism for that piece. But this happened already a couple of months ago, so I thought it’s time to fan the flames.

What is demographic targeting?

Demographic targeting in user testing means to select very specific people based on demographic variables like age, gender, income, job description, education or ethnicity.

What do people expect from targeting specific users?

People want to select test participants and hope to get more qualified feedback from people matching exactly their target audience.

The common understanding seems to be that people selected from the precise target audience provide better and more valuable insights in user tests than people outside the user group.

This misconception is fuelled by the marketing machine of big players in the remote user testing industry, who charge more money for „more granular filtering based on demographics“:

Would you rather like to do your testing with the 34-year old Hannah W., who earns $25-50k a year or with 46-year-old female computer user Susan C., who makes $45-60k (Image source: UserTesting.com)

While we offer the selection of preferred devices or the language of the testers, we don’t offer any other form of demographic targeting. With Userbrain you can target by language and preferred device type, and we make really sure NOT to include ANY demographic targeting at all.

Why don’t we give you candy for lunch?

One of the most frequent support requests we get at Userbrain involves targeting specific demographics for our user tests. Here are some examples from support tickets:

Is there a way to target the videos to specific demographics or qualify the users in any way?
– Samantha

We were hoping to target a specific demographic. Would we be able to work with you to select testers who meet our criteria?
– Jennifer

In terms of screening users, what other things can you screen for? Age, industry etc?
– Loretta

Our usual answer to that question is to reefer to the previous demographics blog post, explaining that we’re not offering targeting for our user tests. Damn, this is really hard. I’m sure that we could sign up a lot more customers by providing some kind of demographic targeting (in fact many people even tell us this story). We even send people to our biggest competitors who insist that they need this feature. Why? Because we really want to give you what you need, and not what you want.

It’s like when a kid wants candy for lunch, but their parents insist on a healthier meal because that’s what the kid needs.
– Michael Sueoka

We’ve been working in the field of UX for over 10 years now, and we have planned, conducted, observed and analyzed thousands of user tests. With just a very few exceptions (I’ll write about them in a future post), testing with the exact target audience it made ABSOLUTELY NO DIFFERENCE AT ALL.

But of course – as one reader of our post on Designer News noted correctly – as a user testing service NOT offering any demographic targeting, it’s quite clear that we make a point of adhering to our won opinion, as it is evident that we have to defend our own set of features.

Therefore, I thought it might be time to introduce you to the experts:

It’s not only us – hear the experts!

Good news, we’re not the only ones sharing this view – here are some opinions by other well-known usability experts:

“When faced with a new product or website, a user’s past behaviour will predict their performance much more accurately than their demographic profile will. Demographic factors like gender may be important segmentation variables for marketers but they have very little impact on the way someone actually uses a product.”
– David Travis

David further advises to focus on behavioural variables like Internet knowledge and task knowledge (the user’s knowledge of the domain) to find representative users.

This is what Craig Tomlin from Useful Usability has to say on the topic of demographics and usability test recruiting:

“Demographic clustering is helpful when creating advertising or marketing communications that seek to reach a specific target audience, with a specific target message. Recruiting usability participants however has little to do with age, or profession, or even geographic location. Instead, usability testing participants should be recruited based on matching the behaviours, needs relative to specific critical tasks, and a base of knowledge about the topic the user is expected to have.”
– Craig Tomlin

Similar to David Travis, Craig also recommends recruiting based on behavioural variables. So far so good. Let’s head to the next UXpert (sorry, I could not resist…):

Jeff Sauro is a mastermind in statistics, qualitative and quantitative user research. He is well know for backing up his arguments with statistical proof. Wanna know what turns up as Number #5 in his blog post Five User Research Mistakes to Avoid? Have you guessed? Well, I’m gonna tell you: Obsessing over demographics:

“When it comes to usability testing, we’ve consistently found that the biggest differentiator in usability metrics is not demographics differences, but whether users have prior experience or are more knowledgeable about a domain or industry.”
– Jeff Sauro

Jeff is in line with the other experts stating that prior experience and domain knowledge are more important for usability testing than obsessing over demographics.

Let’s move on to Christine from Perfetti Media:

“When thinking about the right participants to recruit for a study, many teams start by focusing on demographics, such as age, gender, or ethnicity. Unfortunately, in most cases, recruiting for demographics will be one of the least effective ways to find the most appropriate users for your tests.”
– Christine Perfetti

I totally share Christine’s opinion, that worrying too much over demographics is very ineffective. The reason is that people tend to test not at all, because recruiting the “perfect participant” seems to be a very time-consuming and daunting task (it actually is).

Demographic targeting can have exactly the opposite effect: instead of hoping to improve the quality of the test by recruiting the perfect participants, you actually test less (if at all) because of the time-consuming task, and thus you spot fewer problems and the quality of your test actually decreases.

But the best comes last. Let’s see what Steve Krug, one of the best-known names in UX and usability has to say:

“The best-kept secret of usability testing is the extent to which it doesn’t much matter who you test. For most sites, all you really need are people who have used the Web enough to know the basics.”
– Steve Krug, Author of Don’t Make Me Think

Summing up: It’s all about behaviour and not demographics

Perfect demographics are really helpful when your try to reach a specific target audience with a specifically targeted message. Or in short: When you try to sell a very specific product to very specific people.

You don’t want to sell something while you are doing user testing. In fact, one of the worst things you could do in user tests is trying to sell your product to the participants in the test.

Instead, usability testing participants should be recruited based on matching their behaviour and prior experience and knowledge about the topic.

And if you’re just starting with user testing, don’t worry much about demographics at all. You don’t want to find the love of your life – you just want to observe behaviour and detect errors. 🙂