Device testing: a practical guide

Every audit we run turns up some version of this. This guide covers what device testing actually involves, where it usually goes wrong, and how to tell whether yours is in reasonable shape.

You do not control the device, the network, or the operating system version, and the design has to accept that. Nothing below assumes a large team or a large budget — most of it is a decision somebody has to make and then write down.

What it costs to ignore

The newest phone is not the phone your customers own. The cost of getting this wrong is rarely visible on the day it happens. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

For most businesses the question is not whether this matters but how much of it is worth doing right now. That depends on what you are trying to achieve in the next few months, not on best practice in the abstract. The cost of getting this wrong is rarely visible on the day it happens.

Where to start

Test on mid-range hardware and poor networks. The cost of getting this wrong is rarely visible on the day it happens. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.

The version of the app your customers are running is rarely the one you just shipped. The version that works in practice is usually less elaborate than the version described in the guides.

Automate the regression path, test the rest by hand. In practice this is a scheduling problem more than a technical one. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

A working checklist

If you want a quick read on where you stand, work through this. Anything you cannot answer confidently is where to start.

• The newest phone is not the phone your customers own
• Test on mid-range hardware and poor networks
• Automate the regression path, test the rest by hand
• Someone is named as the owner, not just assumed to be
• There is a date in the calendar to review it again
• The decision and the reasoning behind it are written down somewhere findable
• You could explain the current setup to a new hire in five minutes

Common failure modes

The most common failure is not doing this badly. It is doing it once, during a launch, and never revisiting it. Circumstances move, the setup does not, and the gap widens quietly until something breaks or somebody notices the numbers.

• It was configured during a launch and has not been touched since
• Different people in the business believe different things are true about it
• There is no way to tell whether the last change helped or hurt
• The only person who understands it has left, or is about to

Mobile releases are slower to correct than web ones, so the cost of shipping a mistake is higher. None of that requires a large budget, only a decision and someone to own it.

How we approach it

On our projects this gets handled during the build rather than added afterwards, because retrofitting it costs several times more than including it. We write down what was decided and why, so the next person to touch it is not guessing.

If you are working with someone else, the questions worth asking are simple: who owns this, how will we know it is working, and what happens when it needs to change?

Where to go from here

Pick the single item from the checklist above that would cause the most trouble if it turned out to be wrong. Fix that one, confirm it worked, then move on. Worth checking on your own setup before it becomes someone else's problem to fix.