Multi-variate testing ready to burst forth....oh reeeaallly!?

Reading this just now and I'm subsequently bracing myself for a spate of useless statistical analysis from the field of Web analytics. My experience with the application of multi-variate testing goes back a decade and includes the fields of Statistics, archaeology, marketing and more recently, information architecture. Time and time again I see multi-variate testing wasted through a complete lack of multi-variate analysis.

Folks, it isn't enough to calculate the mean of several variables and pat yourself on the back for your multi-dimensional approach to research. Unless you're going to perform analysis that creates a correlation between variables you are wasting your time. Even something as simple as cross-tabulation will provide you with insights not available through standard summary statistics on a single variable - despite calculating them for a series of variables.

For example:
Out of 100 users...
65 found the interface easy to understand, 25 found it confusing, 10 found in frustrating
50 were able to locate the information they required, 35 were unable to find the information, 15 found the information but didn't recognise it.

So, does that mean that a majority of users find the interface easy to understand and were able to locate the information they required?...

What if I told you that, of the 50 users able to locate their information, 25 of them were the ones that found the interface confusing? How about if, of the 65 that found the interface easy to understand, 35 of them were unable to locate their information?

Anyway, it bugs the bejeesus out of me when I see this sort of thing.

And if you're thinking this guy may not be representative of the standard within the Web analytics fraternity, pick up a book - any book - on the subject, and I challenge you to locate the analysis that goes beyond this style of simplistic, superficial level.

If you find one let me know. I'll even buy a copy.

MMORPG - the role of the tutorial and self-help in complex systems

I recently started playing a new computer game - Eve Online - a massively multi-player online roll-playing game, or MMORPG for short. The game is a spaced-based mixture of adventure, commerce, pirate-hunts, and character development, set in a galaxy far, far away. The game is rich, complex, and involves interacting with real players around the world to achieve common goals.

The game is FANTASTIC! I love this style of game. But that's not why I'm writing about it...

The complexity of the environment and the rules of engagement make it almost impossible to simply document in a user manual. The item database itself - the things you can buy, find, build, install etc - runs into the hundreds of pages, and a lot of the contents won't be relevant until months after you start playing.

The problem for the game designers, and new players, is that there's so much to know and learn and yet you can't force people to spend a couple of weeks poring over a user manual before they can start playing; you have to provide an 'in' to the early levels of the game.

The game designers (and I don't think this is unique to this particular game) have tackled the problem of how to introduce new players to such complexity in two interesting ways:
i) A fairly extensive tutorial that leads new players step-by-step into the environment. From how to configure a ship, to trading & commerce, to combat and moving through space.
ii) A rich online chat built into the game that provides general how-to support for new players, and the opportunity to communicate with fellow players in real time.

With the growing prominence of rich internet applications - now in three flavours - and the increasing richness (ha) that derives from these interaction environments - I'm starting to see the need for a similar approach (i.e. an introductory tutorial) to Web applications. Whilst user research will uncover the primary tasks and objectives of the audience; and usability testing will uncover barriers to use; sometimes it will be necessary to provide a step-by-step run-through of the complex processes before users will 'get it'.

Unlike computer games, however, Web applications lack the initial commitment from the user that would make such a personal investment likely prior to actual use. So is there a limit to the complexity we can introduce into the interaction design of our Web applications before the up-front investment in time will be prohibitive to use?

More about product design...

Just to balance my karma a little after my last gripe about product designers, I have to make mention of the FujiXerox printer that recently came into my 'possession'. A3, double-sided, colour laser printer. Networked (via Ethernet), three trays... all the things you want in a printer.

Installation and configuration took under 10 minutes from opening the box to first printing. That includes time spent exclaiming over the size!!

For those interested in such things, it's a DocuPrint C2428.

Lovely; easy; efficient.