What is bad data? 5 things you need to know

Your data has errors. That’s inevitable. But there are ways you can manage bad data. 

The first important thing to realize is ‘don’t pretend your bad data isn’t there’. Instead, if you design bad data into your data architecture from the outset, you can avoid problems later on. And it’s not all bad news. Bad data can actually be a good thing in some situations, if you learn to treat it as an indicator of problematic areas in your business, and a driver for improvement.

In this article, we'll be covering what bad data is, why it can be a genuine asset rather than just a liability, why error management and the right ownership matter, and why auditing your data errors is worth taking seriously.

Key takeaways

• Bad data is an inaccurate set of information, including missing data, wrong information, non-conforming data, duplicate data, and poor entries such as misspellings or inconsistent formatting.

• Bad data isn't just a problem to eliminate, treated properly, it's a diagnostic signal that can reveal systemic issues elsewhere in your business, from training gaps to revenue leaks.

• Business users, not IT alone, should be deciding how bad data gets fixed, since they understand the context and own the permissions the data requires.

• Auditing bad data and rejected records reveals hidden inconsistencies between systems, and can uncover fraud or process issues well beyond the original data error.

• The goal isn't eliminating bad data entirely, since that's not realistic, it's building systems that expect it, catch it early, and route it to the people who can fix it.

What is bad data?

There’s many reasons data can be rejected going through a process. From a typo or a missing reference during input validation, to a violation of business logic at some point along the pipeline, all the way through to an issue with pushing data to its target - any of these reasons and more can cause records to be rejected.

The impact of bad data on your data quality management process can vary depending on how many records get rejected. Missing records can affect downstream processes or analysis, or delay crucial operations such as deliveries or payments. In the worst case, bad data can cause the entire process to fail, leaving mess and inconsistencies behind in the systems involved.

The more efficiently your error handling system can deal with these rejected records and return them to the processing pipeline, the better your data (and therefore your business insight) becomes.

How can bad data be a good thing for your business?

Proactively building for and managing bad data, rather than trying to pretend it doesn’t exist, means that you can not only minimise the negative impact on your business, but can potentially pave the way for important business improvements too.

Proper visibility into the data correction process helps to understand the causes of bad data, and help to reveal other systemic problems that need to be addressed. Problems with your data can indicate the need for changes elsewhere in the process.

Changes in how data is sourced or processed, or how staff are trained for example, can often not only improve your data quality, but also improve efficiency, turnaround times and so directly impact business health.

Data errors can also provide insight into revenue leaks, showing you where you need to focus attention to fix problems that affect the bottom line.

Why is an error management process important? 

A good error management process keeps corrected data flowing back into your system quickly, provides consistency and transparency through standardized reporting, and gives you an early warning when something larger is going wrong.

We've covered the mechanics of this in depth, from profiling and business rules validation to how to handle the specific technical sources of bad data that tend to slip through a pipeline, in our dedicated guides on the subject.

The short version: automating as much of this as possible, rather than manually fixing and re-trying rejected records one at a time, frees up resources for higher-value work and gives you aggregate reporting that can highlight lost revenue or pinpoint problematic sources.

Who should be fixing bad data?

It's easy to think of data errors as the IT department's job to fix. But IT doesn't own the data, and often lacks both the permissions and the business context to know what the correct fix is. The people who understand the data, usually business users, are the ones who should be assessing errors and deciding what the fix should be, which is exactly why self-service data correction matters so much to getting this right at scale.

Why is the auditing of bad data important?

Tracking and auditing bad data and rejected records can sometimes be a requirement, for instance in financial companies subject to regulation, but is important for all data processes.

Auditing can help reveal hidden inconsistencies between systems, which can then be addressed and the subsequent data improved, potentially leading to better insight and analysis. Audit trails can also uncover internal fraud or issues within a particular business area, and identify trends that can be used for improvements in business processes across the organization.

Read more about the best practices for designing automated data processing pipelines to take account of bad data. Our whitepaper outlines:

• How to create an effective and sustainable data validation and correction loop.
• Tools and practices that enable business users to effectively identify, correct and manage bad data
• The best ways to effectively control errors
• The importance of reporting in your error handling process

Final thoughts: Build for bad data, don't fight it

Bad data will always exist. The organizations that handle it well aren't the ones chasing a perfect, error-free dataset, they're the ones who've built systems that expect errors, catch them early, and route them to whoever has the context to fix them.

Managing for bad data, and architecting systems to handle data errors effectively, helps eliminate unexpected downtime, prevent data loss, and avoid operational delays. Let's talk about how CloverDX can support you.