What to ask your agency about device testing

Every audit we run turns up some version of this. If you are briefing an agency or a freelancer on device testing, these questions are worth asking early.

You do not control the device, the network, or the operating system version, and the design has to accept that. It is the sort of thing that looks like polish right up until it costs you an enquiry.

Questions worth asking

• Who will actually do this work, and have they done it before?
• How will we know afterwards whether it worked?
• What happens if it needs changing in a year?
• What are you assuming that we have not confirmed?

What a good answer sounds like

The newest phone is not the phone your customers own. There is a version of this that is over-engineered, and it is worth avoiding. The version that survives contact with a real deadline is the simple one.

Automate the regression path, test the rest by hand. The cost of getting this wrong is rarely visible on the day it happens. Budget a little time for it every quarter and it never becomes a project of its own.

In practice

Mobile releases are slower to correct than web ones, so the cost of shipping a mistake is higher. Three things worth confirming about device testing before you move on:

• Someone can say what the current setup is without going to look
• Automate the regression path, test the rest by hand — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.