The only really big mistake you can make in user testing is not to start testing. However, this is something that often happens, when people wish to start their first tests and frantically try to find participants who are exactly the target group, because they fear that otherwise they’ll get worthless test results due to having the wrong test participants. This plan ultimately fails, because in practice it’s actually difficult to get the perfect test participant, so that many people not even start testing. This shouldn’t strike us as something surprising, when requirements for test participants sound like this:
We are looking for people who meet our exact target group, i.e. women between the ages of 18 and 29, single, living in the met- ropolitan area of Graz or Vienna, with a gross income of at least 50,000 € a year and with a degree in business administration.
These exact requirements concerning gender, age, income, occupation or place of residence are often determined by marketing managers. In marketing, the selection of the right demographic characteristics plays a major role – reaching out to the right people with the advertising message ultimately determines the success of entire campaigns. But when doing user testing, we don’t want to sell anything. What we do want is to discover usability problems by observing people’s behavior. Thus, such demographic characteristics actually only play a very small role in our user tests:
“The importance of recruiting representative users is overrated.”
– Steve Krug in “Don’t Make Me Think!”
exact demographics ≠ better results
We won’t necessarily get better results just because we’ve managed to find testers who have the exact demographic characteristics of our target group. Because, unlike, for example when doing surveys, we’re not interested in the opinions of individuals; we’re trying to understand their behavior to draw conclusions about usability problems. And the remarkable thing about human behavior is that different people behave similarly in similar situations. If the checkout button on your page is hidden, this will confuse somebody who fits your customer profile perfectly, just the same as it will somebody who isn’t a potential customer, but has been given the task of shopping on your page.
The test participants in you user tests doesn’t need to match your exact target group. But there is an exception: products or services that require a high level of experience with a certain topic or a specific expertise. You’ll find it difficult to test a UI for controlling a construction crane with somebody who has never operated a crane. But if you’re not in the business of developing software for rocket scientists, something which requires that your users have specific knowledge of the subject and also some experience, it doesn’t make much difference if the people you’re testing with are part of your target group or not. Especially if your digital product is accessible to everybody on the Internet.
Stop searching for the perfect test participant
The search for test participants who perfectly match the profile only causes a greater effort and higher costs. Because even if we find a few participants who match our exact demographic characteristics, we’ll always need new usability testers, as we’re continually conducting tests. And finding new test participants from a very small target group is pure gambling and often ends with people stopping testing for lack of new participants. The perfect test participant doesn’t exist – so you shouldn’t wait to find one or you’ll never start testing.
