How to get data quality right
The advice here is unglamorous, which is probably why it gets skipped. The short answer to data quality is that it is mostly a sequence of small decisions, not one big one.
Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.
What it costs to ignore
Bad data spreads faster than anyone corrects it. Where this goes wrong is almost never a lack of knowledge. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.
The steps
- Establish what you have today before changing anything
- Automated checks catch drift that eyeballs miss
- Someone must own each dataset or nobody does
- Write down the decision so the next person does not re-litigate it
Someone must own each dataset or nobody does. That sounds obvious written down. It is still the thing most often skipped. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.
Where to go from here
How to tell if yours is fine
Most data problems are ownership problems that turned into technical ones. Three things worth confirming about data quality before you move on:
- Someone can say what the current setup is without going to look
- Automated checks catch drift that eyeballs miss — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If you are not sure where your systems currently stand on this, it takes us about an hour to find out.