Overall I was a combination of slightly bermused, impressed by the technology bowled over by the real-time goodness and the slick UI, but fundamentally unsure of what I was supposed to use Google Wave for.

But what actually went wrong here? I’ve got a theory and it’s not what you might expect – valid though those criticisms are. Google Wave went wrong, not because of what it is, but because it didn’t harness any viral power:

1. Invites are not sent fast enough
2. Wave is closed
3. Wave has no pointers

Google Wave is first first and foremost a communications technology, which means for people to really see the benefits they need to be communicating with people they know.

When you join Google Wave you receive 8 invites, when you’re through those you get another 12. But when you send the invite they don’t arrive for several days. Meaning you’ve got practically no-one of significance to communicate with when you’re first into it*.

So your first experience of Wave is with strangers.

The second problem with Wave is that you don’t know when you’ve got an unread Wave. You signed in first time around, played around a bit with a few people you vaguely know online, and sign out. If someone now communicates with you whilst you’re signed out you’re not informed.

No email, no RSS feed, no Tweet, nada.

So you don’t sign in and respond and you don’t slowly fall away from being a Wave user.

The final thing which they got wrong is that there are no easy to follow pointers to get your started, clues for intentional uses, a quick “first action”.

When people first use a technology they are in a state of mild stress, in short they don’t want to publicly demonstrate that they don’t know what they’re doing and therefore look stupid. So they want something easy to do, figuring out the complex stuff afterwards. This is as true of techie geeks as it is of our non-techie friends.

Google Wave doesn’t supply these pointers.

What’s important about what I’ve said here is that all of this is entirely fixable, and there are plenty more people who desperately want to try out Google Wave. With any luck the feedback from the 2nd and 3rd round of adopters will be a lot more positive!

* I totally acknowledge that there maybe good technical scalability reasons for this, but it doesn’t change the affect.

Without doubt, this is one of the most revealing, painful, and valuable stages in creating your start-up, so I thought I’d blog about it.

When launching anything you want ensure that not only does it work, but people find it easy and natural to use. Your own functional testing should cover the first objective, your alpha testing should cover the UI.

At Aroxo we’ve employed four different techniques to get the system ready for launch:

Not only is each stage different but you get different learnings from it. I discuss each stage below.

Over-the-shoulder tests

This test phase is the most fascinating for the founders, as you get to watch people actually interacting with your system. This test should be done independently, as those close to the system are inclined to guide the user towards positive responses, rather than getting genuine feedback.

But if you are bootstrapping or operating on a tight budget, you may well be doing it. So, here’s what you do:

1. Write out a list of scenarios or tasks which you are going to as people to do
2. Find 5-15 people who represent your expected customer base
3. Sit them in front of a machine and get going

Your first task should be to get them to look at the homepage for a few seconds (no more than 30 seconds), then ask them what they think the site does. People should find it easy to understand the site from the homepage. If not, probe a little to find out what you can change to make it clearer. Then get started on the scenarios.

Your job is to generally keep quiet and take notes as they use the system. You are particularly looking for times when something confuses them, you may notice the person lean forward and start to look confused. At these points feel free to interrupt and probe, ask them what they are looking for and what’s missing.

When probing, be very careful to use words like “what” (open) rather than “why” (implies fault). Try not to reveal any emotion, you’re not asking people to tell you how great your system is, you’re after honest feedback. Therefore it is also important not to make people think they are stupid for not being able to use the system, frame it as “we want to get some open and honest feedback”.

Expect some excellent and easy improvements to come out of this to really make the system substantially easier to use.

Task-driven testing

Task-driven is quite different. For a start, you won’t actually be watching people, they’ll be running through tasks on their own. This is particularly interesting because for the first time people will be using the system without anyone to ask for help. This means you can get more people involved.

To perform this sort of testing there are a few things you need:

1. around 20 particpants to start with (expect about half to actually use it and 2-3 who give you really good feedback)
2. a well defined list of tasks for them to perform
3. a way to communicate the tasks to them and
4. a way for them to communicate their findings back to you.

As you push these changes through to the alpha platform you should start adding more and more people onto the trial. You want fresh eyes to test your UI improvements, people who’ve already seen it, won’t test it as well, if at all.

Your objective here is the test how well the system performs with a small number of people performing defined tasks, so each task you ask people to do should come with some prompts to help them give you relevant feedback.

Goal-driven testing

Goal-driven testing differs from task-driven testing in two ways. Firstly there are a lot more people involved in the trial, and secondly instead of having tasks which people have to perform on the system (e.g Register on the site). You set wide goals for the participants. In context of Aroxo this is something like “buy something” or “sell something” the UI and the website itself should then guide the users through the system, rather than the tasks.

Start your goal-driven testing when you’ve reached the point that you believe the UI of the system is strong enough that people know what they should be doing and how to use the system.

Beta testing

Beta testing isn’t really testing in the same way as the alpha testing which you’ve already performed is. You want a large number of people get see the system and many of them will have registered on your site to do it, so you don’t know them.

This makes beta testing public. If it is public, it’s part of your marketing. Therefore it is essential that it shows your site running well and easy to use. What you may get is some useful learnings about how the system scales, or you may also be able to learn something from some clickstream work.

However, what is most important about beta testing is that it should not be part of your functional and UI testing, it is too late to make any significant changes to the system as you’ll be moving towards a full launch within weeks of starting your beta.

Closing thoughts

One thing which I’ve not discussed in this article is the emotional impact which testing has on the founders. If you’ve been careful with your IP whilst you’ve been building your system it’ll be the first time which you’ve had large numbers of people looking at your system. This means you are going to get a lot of feedback.

It is important not to take this feedback personally. It should help make the system stronger.

You need to remember that you are asking people to look at your system and tell you the flaws and problems with it. You are not asking them to look at it and tell you how clever it is! This means you are going to get a lot of (hopefully) constructive criticism. You need to take this on the nose and think of creative ways to improve the system.

Finally, in the case of Aroxo something which we learned is that it just isn’t possible to test the UI of your own stuff: you are way too close to it. So it is important to have lots of fresh eyes to test each new release.

Good luck!