Why usability testing might end up dumbing down your product


Recently, I was involved in an argument at work regarding which UI pattern we should follow with regards to a specific problem. We wanted to  present some data in a table format, and have the user decide how to process each row. There were two alternative solutions…

A: Either we’d put a checkbox on each row, as well as a checkbox with a “select all” capability, then put a toolbar with the different commands like “delete”, “send”, “edit”, etc. on top, or
B: put a button for each possible command on each row.

Solution A: Using checkboxes and a toolbar.

Solution B: Using buttons.

Usability testing showed – or so I was told – that users preferred solution B. While the test result itself might be formally correct, one starts to wonder how the testing was conducted.

Simplicity vs. Efficiency

While keeping things simple is a good thing, there is a tendency to “dumb down” things too much.

A lot of people would probably choose B in a simple ‘at-a-glance’ A/B testing because of its simplicity. The efficiency in the long run is actually not tested, and is poor at best. To be able to get a good result from usability tests, the test subjects would have to repeat the task several times, almost to some kind of ‘breaking point’. Only then will the actual efficiency of the UI be fully understood.

Scalability

When constructing a page for usability testing, one might start out with a simple layout and a small amount of data. The problem with that approach is of course that the result might be misleading. If we start adding a couple of possible actions, the amount of buttons to add will use more and more space, invading the space used to present the actual data. It will waste valuable screen estate. Imagine image B above, with three or four more action buttons added. The scalability of this design pattern turns out to be subpar. If a similar page elsewhere on the site need a similar pattern, solution B might turn out to be a dead end, design-wise.  This would lead to having several, conflicting design patterns on different parts of the web site – breaking the consistency of the user experience. The same holds true on the other axes. As the amount of data rows increases, usability will decrease and come to a grinding halt.

What others do

Obviously, we are not alone. It turns out that GMail as well as Hotmail have done some homework and are using checkboxes and a toolbar.

It turns out that GMail have chosen option A…

…as has Hotmail.

There are other sites that use this model as well.  It’s worth noting that the general concensus seem to be that the checkbox column should be placed to the left, not to the right. I guess the rationale behind this is that it will always be visible, even when the table itself is cropped to the right by reduced screen widths. Another thing worth noticing is that a toolbar is a better foundation if you need to start using submenus etc. With the button design pattern, you need one button for each action. With a toolbar, you could actually swap the toolbar for a fold-out menu or basically any other navigational pattern you want to implement.

Conclusions

I am not an opponent of usability testing. Done right, it is a valuable asset in the toolbox of understanding UI design. However, as with all tools, it need to be fully understood and used with common sense. In the end, an expert need to evaluate the data and have final say.

You should never let the tests themselves make decisions for you. They can never replace creative vision or expertise – only verify a hypothesis.

 

To generate the dummy data, I used Behind the name‘s Random Name Generator by Mike Campbell, and Daily Mail-o-matic by Chris Applegate.