How to test your system with real users
Thursday, February 14th, 2008Aroxo recently turned a major corner in its development, we moved from closed functional testing to using real live alpha testers, people who’d never seen Aroxo before.
Without doubt, this is one of the most revealing, painful, and valuable stages in creating your start-up, so I thought I’d blog about it.
When launching anything you want ensure that not only does it work, but people find it easy and natural to use. Your own functional testing should cover the first objective, your alpha testing should cover the UI.
At Aroxo we’ve employed four different techniques to get the system ready for launch:
|
Test type |
Description |
How many people |
| Over-the-shoulder | The main sticking points in the system. Where the system confuses users. Start when functional testing at 80% readiness. Earlier with mock-ups also possible. |
10-15 |
| Task-driven testing | How well the system stands up on its own. Start when the major usability holes uncovered in OTS testing have been fixed. |
Start with 20-30 keep growing invites to 100 or so |
| Goal-driven testing | End-to-end flaws across the system. Start when functional testing at 95% system readiness with a slick UI. |
150-200 |
| Beta testing | The marketing points for the site, highlights future developments. If there are enough users it may also reveal performance issues Start when the system is 99% ready. |
250+ including members of the public |
Not only is each stage different but you get different learnings from it. I discuss each stage below.

