Jakob's latest Alertbox - How many users to test?

Jakob Nielsen's latest Alertbox, titled "Quantitative Studies: How Many Users to Test?" looks at the improvement in confidence intervals and margins of error to be had through increasing the number of users tested when measuring usability metrics. [Note: this is in contrast to his recommendation to test 5 users when looking qualitatively at usability.]

The article draws on large numbers of usability tests carried out by the NNg and provides some interesting points of note:
i) Usability metrics tend to follow a Normal (or Guassian) distribution - which makes the statistical analysis that much more convenient;
ii) User time-on-task performance tends to show a standard deviation of 52% of the mean;
iii) 20 users offers a reasonable test sample size for most usability metrics.

A couple of counter-points worth keeping in mind:
i) Although the finding that usability metrics tend to follow a Normal distribution is useful, most statistical analysis techniques include methods whereby the Normal distribution is not a requirement. This is particularly the case when performing quantitative analysis on non-parametric data (e.g. ranks, categorisations or counts);
ii) Always calculate the standard deviation and margin of error based on the data that you've collected. Whilst NNg's insight provides a useful starting point for deciding the number of test subjects, you need to go through the process of calculation sd and e for your data set;
iii) NNg use a 90% confidence interval as their baseline for determining the recommended number of test subjects: sometimes this level of confidence is insufficient, and so a greater number of test subjects would be required. (I tend to use 95% CI myself, particularly if the implementation cost is high.)
iv) Confidence intervals form part of the general set of summary statistics about a data set (along with mean, variance etc). They describe a characteristic of a particular sample, which allows some inference as to the nature of the general population - they don't provide a comparison between two populations. For example, is a time-on-task CI of 3 mins +/- 30secs better or worse than a time-on-task CI of 3:10mins +/- 15 secs?

Finally, this article feels like a response to the JUS article cited here previously. In particular, a response to Lewis and Sauro's references to use of 5 test subjects for usability studies, which may itself have been influenced by a previous Alertbox article.

Statistics without tears

That's the title of an interesting introductory book on Statistics that I'm reading through at the moment. Written by Derek Rowntree and published by Penguin (ISBN: 0-14-013632-0), the book provides an introduction to the theory and practice of statistical analysis without (much) recourse to formulae, calculations, charts, graphs or tables of figures.

Aimed at people that have to deal with statistics and statistical analysis, but who could go their lives seeing without a Gaussian distribution and not view it a loss, the 184-page text takes the reader on a fairly painless journey through descriptive and inferential statistcs; their meaning, use, and calculation.

If you've ever found yourself struggling to make headway into the topic of statistics then this book may offer you an olive branch.

Alternate titles for the book were: Statistics without calculations; Statistics for the innumerate; Statistics in words & pictures; The underlying ideas of statistcs; or How to think statistically.

PS: I'm reading this in preparation for a presentation on statistical analysis of usability and user research data to a largely non-mathematical audience later this year.

Australian football comes of age?

Un-f^%#ing-believable!!

I spent my childhood surrounded by kids of European and South American backgrounds, being taunted by them about Australia's (and, by implication, Australians') complete absence from the soccer World Cup. Whenever it rolled around the Italians, Croatians, Brazilians, Uruguayans, Chileans etc would cheer on their national sides, and laugh at our failure (again) to even qualify.

Earlier this morning Australia's Socceroos played their first World Cup game since '74. Behind for a good portion of the match they put in a tremendous effort in the last 20 mins, finally scoring in the 84th minute of the game. A second goal in the 89th minute put us in front; and a sealer in the 92nd minute made it comfortable.

It may be an overstatement to say we've come of age, but we've certainly proven we belong.