The impact of investments in user experience projects is measurable in dollars and cents and can be directly linked to revenues. It’s very easy to do the math, and this post will show you how improvements in the user experience saved one of our clients thousands of dollars every month and how we came up with our return on investment calculation.
The following example is taken out of our real-life experience – we just changed the name of the company.
Megacorp’s Usability Issue
Let’s call our client Megacorp. They are running a quite successful e-commerce site and approached us to do a usability check on their site and listen to what their customers had to say. Megacorp assumed there were issues on mobile devices on the basis of quantitative data from Google Analytics (Low bounce and conversion rate on mobile devices).
To understand the reason for the bad bounce and conversion rate on mobile devices, we suggested running a remote usability study with 5 participants, taking into account the analysis of existing quantitative data from Google Analytics.
And, yes, we also needed to explain to Megacorp why 5 users are sufficient to uncover usability issues if you’re testing on an ongoing basis.
Creating and pilot testing the tasks
The next step was to create a couple of representative tasks for the e-commerce site. We focused on the checkout experience to learn more about the relative low bounce and conversion rates on mobile devices.
These are part of the 11 tasks we used in our study:
- Imagine that your baby sister is getting married next Sunday. To make a good impression, you want to get some new clothes for the occasion.
- Find an outfit suitable for the wedding and choose a combination of at least 4 products in your size, matching your personal style.
- You want to order this outfit. Will you receive your order in time for the wedding on next Sunday? Try to order the outfit now.
As you can see, we always try to formulate tasks participants can easily relate to, so that they behave in a more natural way. It’s also important to first test the task yourself and only then have somebody else do it.
After testing our tasks and making a couple of tweaks, we put the test into our own remote usability service Userbrain and waited for the results from first-time users to come in.
After a couple of hours, we got a notification that our first 2 user test videos were ready.
Finding the elephant in the room after just 2 user tests
When we started with Megacorp to improve their site’s usability, at first everything looked fine. A great, responsive site, happy customers and a great marketing agency who was running the strings in the background. But after running our unmoderated usability session we noticed a weird thing on the page, just before submitting the order. As the first two user tests came in, we noticed that two of our participants on mobile devices hadn’t been able to select a dropdown menu, something necessary to place the order. Thus, they were not able to check out – a worst case scenario for an e-Commerce site.
They were both using an iOS device – iPhone and iPad, and the follow-up analysis showed that this issue only occurred when a certain payment method was selected and only on certain mobile devices – a very hard thing to spot with only raw quantitive data and without a real user testing the website.
When issues such as these arise, we attempt to find the cause immediately. Thus, we got back to Megacorp as soon as possible without waiting for the end of the study and the complete results.
Spotting patterns with the help of task completion
Keeping track of all the usability issues found can very often be daunting. This is why it’s important to capture task completion – did the participant manage do complete the given task? In the case of the Megacorp study, we kept track of task completion using a rainbow spreadsheet, displaying individual testers in rows and the specific task in columns.
The checkout issue occurred with Task 7 (A7), i.e. the checkout process:
You can see at first glance that 3 participants were not able to complete the task at all (marked in red), one had minor problems (yellow) and only one managed to get through without a hitch (green). Once again, this supports the 5-tester theory – you can really spot the big usability issues with just 5 testers. Once the bug with the dropdown spotted, it was fixed within a day and the dropdown menu could be selected on all mobile devices.
What about the ROI?
Yes, I promised some numbers at the beginning. So, what’s the return of investment of this usability study? To answer this question, we first need to find answers to these 2 questions:
- How many people are affected by this bug?
- What’s the impact on the conversion rate?
How many people are affected by this bug?
Let’s assume that 50% of Megacorp’s webshop visitors use mobile and that 50% of mobile orders are coming from an iPhone or iPad (You can look these numbers up for your own site in Google Analytics -> Mobile Overview and Devices). This leaves us with 25% of all orders coming from iPhones or iPads – our problem devices.
As I mentioned before, the issue only occurred when selecting a certain payment method. This payment method is used in 15% of all orders. (You can find this data in your webshop system)
| Megacorp’s eCommerce site | |
| Orders from iOS devices | 25% |
| Orders using this payment method | 15% |
| Affected customers | 3.75% |
The math: 15% of 25% is 3.75%. Now we know that this issue affects around 3.75% of our customers.
What’s the impact on the conversion rate?
This is an easy one. Since our testers were not able to proceed at all and had to cancel the task, the impact was 100%. All 3.75% of our customers run into this issue which causes a complete loss of the transaction.
Bring the money in
Now it starts to get interesting. As Megacorp’s eCommerce site is a quite successful one in their niche, let’s say that they’re making $100,000 per month. As the issue with the non-working dropdown affects 3.75% of all orders this would result in:
| Megacorp’s eCommerce site | |
| Revenue per month | $100,000 |
| Magnitude of the issue | 3.75% of all orders |
| Loss per month | $3.750,– |
The bottom line: Megacorp is losing almost $4k due to this bug in the checkout process EVERY SINGLE MONTH.
Of course, a more sophisticated calculation can be done considering:
- The average conversion rate for iOS users choosing the affected payment method
- The average order size for iOS users choosing the affected payment method
But for our example, the simple calculation above is clear enough to do the job.
Remember that this is just the loss of revenue caused by a single usability issue – the most severe one, but only one of more than 80 issues found and reported with the help of this study. Hidden revenue loss due to frustrated customers not ordering again or telling their friends about their shitty online experience with Megacorp might be a lot higher.
The ROI calculation
We now have everything we need to run our ROI calculation. Let’s assume that we’ve outsourced the user testing study to an agency and our cost to run this study has been $10k. Fixing the major checkout bug on mobile devices improved our conversion rate by 3,75%.
Once again: I’m doing this ROI calculation on the basis of only fixing the major usability issue found – the checkout bug on mobile devices. In reality, usability improvements could increase the conversion rate by up to 135% – I’m therefore very conservative with the numbers used.
Speaking of numbers, let’s get back to business. Let’s assume that Megacorp’s online shop gets 150,000 unique visitors per month and that the average order value is $33.00.
Here’s the breakdown before and after usability testing:
| Before | After | |
| Unique Monthly Visitors | 150k | 150k |
| Average Order Value | $33 | $33 |
| Conversion Rate | 2% | 2.075% |
| Monthly Revenue | $99,000 | $102,713 |
Fixing just this single issue brought in an additional $3,713 PER MONTH. By fixing this one single bug (which took the development team of Megacorp just an hour), we made them bring in almost $4k of new revenue per month, something that will increase significantly when Megacorp increases traffic or the average order value.
It therefore took just 3 months to break even with the $10k investment in usability improvements.
Summing up
Studies show that normal websites can increase desired metrics by 135% on average following a usability redesign. You can easily calculate the return of investment for your own usability activities if you keep track of about just a couple of your business metrics.
You might be surprised about the outcome.
