How to test your system with real users
February 14th, 2008Aroxo recently turned a major corner in its development, we moved from closed functional testing to using real live alpha testers, people who’d never seen Aroxo before.
Without doubt, this is one of the most revealing, painful, and valuable stages in creating your start-up, so I thought I’d blog about it.
When launching anything you want ensure that not only does it work, but people find it easy and natural to use. Your own functional testing should cover the first objective, your alpha testing should cover the UI.
At Aroxo we’ve employed four different techniques to get the system ready for launch:
|
Test type |
Description |
How many people |
| Over-the-shoulder | The main sticking points in the system. Where the system confuses users. Start when functional testing at 80% readiness. Earlier with mock-ups also possible. |
10-15 |
| Task-driven testing | How well the system stands up on its own. Start when the major usability holes uncovered in OTS testing have been fixed. |
Start with 20-30 keep growing invites to 100 or so |
| Goal-driven testing | End-to-end flaws across the system. Start when functional testing at 95% system readiness with a slick UI. |
150-200 |
| Beta testing | The marketing points for the site, highlights future developments. If there are enough users it may also reveal performance issues Start when the system is 99% ready. |
250+ including members of the public |
Not only is each stage different but you get different learnings from it. I discuss each stage below.
Over-the-shoulder tests
This test phase is the most fascinating for the founders, as you get to watch people actually interacting with your system. This test should be done independently, as those close to the system are inclined to guide the user towards positive responses, rather than getting genuine feedback.
But if you are bootstrapping or operating on a tight budget, you may well be doing it. So, here’s what you do:
- Write out a list of scenarios or tasks which you are going to as people to do
- Find 5-15 people who represent your expected customer base
- Sit them in front of a machine and get going
Your first task should be to get them to look at the homepage for a few seconds (no more than 30 seconds), then ask them what they think the site does. People should find it easy to understand the site from the homepage. If not, probe a little to find out what you can change to make it clearer. Then get started on the scenarios.
Your job is to generally keep quiet and take notes as they use the system. You are particularly looking for times when something confuses them, you may notice the person lean forward and start to look confused. At these points feel free to interrupt and probe, ask them what they are looking for and what’s missing.
When probing, be very careful to use words like “what” (open) rather than “why” (implies fault). Try not to reveal any emotion, you’re not asking people to tell you how great your system is, you’re after honest feedback. Therefore it is also important not to make people think they are stupid for not being able to use the system, frame it as “we want to get some open and honest feedback”.
Expect some excellent and easy improvements to come out of this to really make the system substantially easier to use.
Task-driven testing
Task-driven is quite different. For a start, you won’t actually be watching people, they’ll be running through tasks on their own. This is particularly interesting because for the first time people will be using the system without anyone to ask for help. This means you can get more people involved.
To perform this sort of testing there are a few things you need:
- around 20 particpants to start with (expect about half to actually use it and 2-3 who give you really good feedback)
- a well defined list of tasks for them to perform
- a way to communicate the tasks to them and
- a way for them to communicate their findings back to you.
As you push these changes through to the alpha platform you should start adding more and more people onto the trial. You want fresh eyes to test your UI improvements, people who’ve already seen it, won’t test it as well, if at all.
Your objective here is the test how well the system performs with a small number of people performing defined tasks, so each task you ask people to do should come with some prompts to help them give you relevant feedback.
Goal-driven testing
Goal-driven testing differs from task-driven testing in two ways. Firstly there are a lot more people involved in the trial, and secondly instead of having tasks which people have to perform on the system (e.g Register on the site). You set wide goals for the participants. In context of Aroxo this is something like “buy something” or “sell something” the UI and the website itself should then guide the users through the system, rather than the tasks.
Start your goal-driven testing when you’ve reached the point that you believe the UI of the system is strong enough that people know what they should be doing and how to use the system.
Beta testing
Beta testing isn’t really testing in the same way as the alpha testing which you’ve already performed is. You want a large number of people get see the system and many of them will have registered on your site to do it, so you don’t know them.
This makes beta testing public. If it is public, it’s part of your marketing. Therefore it is essential that it shows your site running well and easy to use. What you may get is some useful learnings about how the system scales, or you may also be able to learn something from some clickstream work.
However, what is most important about beta testing is that it should not be part of your functional and UI testing, it is too late to make any significant changes to the system as you’ll be moving towards a full launch within weeks of starting your beta.
Closing thoughts
One thing which I’ve not discussed in this article is the emotional impact which testing has on the founders. If you’ve been careful with your IP whilst you’ve been building your system it’ll be the first time which you’ve had large numbers of people looking at your system. This means you are going to get a lot of feedback.
It is important not to take this feedback personally. It should help make the system stronger.
You need to remember that you are asking people to look at your system and tell you the flaws and problems with it. You are not asking them to look at it and tell you how clever it is! This means you are going to get a lot of (hopefully) constructive criticism. You need to take this on the nose and think of creative ways to improve the system.
Finally, in the case of Aroxo something which we learned is that it just isn’t possible to test the UI of your own stuff: you are way too close to it. So it is important to have lots of fresh eyes to test each new release.
Good luck!
Email This Post
Posted in Aroxo, development, entrepreneur, entrepreneurship, how to start, start-up, testing, web 2.0 | Trackback | del.icio.us | Top Of Page
February 20th, 2008 at 4:26 am
Great article, I’m about to release my company’s public beta (online video platform) and I agree that getting as many outside opinions as possible is a great thing. No offense taken on any hardcore “negative” feedback is very important. Keep up the great work Aroxo!
April 7th, 2008 at 11:12 am
Great article! The most painful part of software development (usually the last 20%
split into four test phases and analyzed. It’s all very true and I’m glad somebody shared his view.
I learned (the hard way) the usefulness and demand of unittests and regression tests. Especially when you’re a single person wearing all the hats, including the development and web design. Each to-do list of mine is sectioned with all regression tests appearing at the bottom. I cannot move to the next batch of tasks until I finish all tests in automatic fashion. I said, the hard way.
Keep up the great work!
– gil
July 2nd, 2008 at 1:54 am
(comment resubmitted with a correction)
Matt
Thanks for such an articulate article. You have broken down this complex task into some logical steps.
I must say that if your clarity of thought also pervades the aroxo experience, it is time for eBay to start trembling in its socks.
I particularly agree with Jeremy’s insight that not taking offence during OTS testing is vital. I admit it is also difficult. I have been flabbergasted by some of the approaches users take to navigating the web - it beggars belief. But, they are users, and what they do is what is most important. Otherwise, we could just load our sites onto a local server and play with them ourselves.
All the best.
July 2nd, 2008 at 8:52 am
Hey guys - thanks for your comments!