What's wrong with the following...

Using the old form design, users took an average of 4mins 31secs to complete Task 6. In the re-designed form task completion time was reduced, with users taking as little as 48secs on the task.

Can you spot the problem? Answers welcome...

---------------------------------------------------------------------------------------------------------------------------

OK, I understand. Better things to do on a weekend than think about something like this...

The above is an example of statistical sleight of hand. In reading through the above you've probably come away with the impression that:

• Two form designs have been tested;
• The task completion time on the old form is longer than for the new form;
• Based on the numbers given - 4mins 31secs & 48secs - that improvement from old to new is substantial.

Well, the second and third points are completely unsupported. Why?

The first part of the statement provides us with the average task completion time for the old design - 4mins 31secs. This is followed by an assertion: "In the re-designed form, task completion time was reduced". The only supporting evidence provided for this assertion is the low-end extreme measure of task completion time for the new form design.

But how does this compare to the low-end extreme measure of task completion for the old design? What about the mean task completion time for the new design. What about the variation in each sample?

We're simply not comparing the same thing in each case, but the 'discussion of results' clearly wants us to think that we are. This is a common type of analytical two-step seen in many forms of research, and one for which you should look out.