What's wrong with the following...

Using the old form design, users took an average of 4mins 31secs to complete Task 6. In the re-designed form task completion time was reduced, with users taking as little as 48secs on the task.

Can you spot the problem? Answers welcome...

---------------------------------------------------------------------------------------------------------------------------

OK, I understand. Better things to do on a weekend than think about something like this...

The above is an example of statistical sleight of hand. In reading through the above you've probably come away with the impression that:

• Two form designs have been tested;
• The task completion time on the old form is longer than for the new form;
• Based on the numbers given - 4mins 31secs & 48secs - that improvement from old to new is substantial.

Well, the second and third points are completely unsupported. Why?

The first part of the statement provides us with the average task completion time for the old design - 4mins 31secs. This is followed by an assertion: "In the re-designed form, task completion time was reduced". The only supporting evidence provided for this assertion is the low-end extreme measure of task completion time for the new form design.

But how does this compare to the low-end extreme measure of task completion for the old design? What about the mean task completion time for the new design. What about the variation in each sample?

We're simply not comparing the same thing in each case, but the 'discussion of results' clearly wants us to think that we are. This is a common type of analytical two-step seen in many forms of research, and one for which you should look out.

User research smoke & mirrors - a 5-part series by Christopher Fahey

Over the past week I've really enjoyed reading (and re-reading) Christopher Fahey's 5-part series of blog posts on user research. This thought-provoking and provocative series sets out Christopher's issues with much of the research carried out in the interaction design/user experience space, providing examples, caveats, pros & cons.

The series is, I believe, a much-needed dose of scepticism and critical analysis of our discipline's love-hate relationship with research, and the results derived.

An absolute must-read for anyone involved in the design of Web sites and applications based around user research.

Make sure you take the time to read through the various comments & Christopher's responses. They contain some interesting perspectives on the issue from other practitioners (including myself).

Luke W on granular bucket testing

Carrying on the user research theme...

Just reading through an article on bucket testing over at Luke Wroblewski's blog - Functioning Form (thanks to Pabini @ UXmatters for the link). Bucket testing is the process wherein you test two versions of the same thing in parallel (page, form, design element etc) by channeling part of your site traffic through one version or the other and analysing the results (note).

Luke raises the issue that, with advances in the ease with which bucket testing can be undertaken, organisations are performing tests on increasingly isolated design elements. For example, the use of text colours in particular areas of a page. However, looking at the results of a very granular test, and then adjusting the design accordingly, does not result in an optimised design.

Any student of mathematics, finance, economics etc will tell you that the 'optimal' solution to a set of equations or conditions is rarely the combination of the optimal solution of each equation. This carries directly into the design of a page or form or screen of a Web site/application. It is invariably pointless carrying out a test only on a specific element and then adjusting the design based on those results without also verifying that the change produces a more optimal solution to the whole.

Luke puts it like this:

"A cohesive integration and layout of all the elements within an interface design is what enables effective communication with end users." [emphasis mine]

In other words (not that he needs my help), the optimal solution to a design problem is not the stitching together of individually-optimised components.

Note: I'm in the process of writing a post/article on how you can carry out this style of analysis with some degree of statistical rigour.