Ensuring data quality throughout your data pipelines can be challenging. Most teams know bad data is a problem in the abstract, but far fewer can name exactly where it comes from, which makes it much harder to design against.
By understanding the sources of bad data, and considering data quality at each pipeline step, you can mitigate the risks associated with poor data quality before they turn into bigger problems.
In this article, we'll be covering the specific technical sources bad data most often comes from, including schema drift and data decay, two patterns that quietly cause more damage than most teams realize, and the practices that help you catch problems closer to where they start.
Bad data most often originates from a handful of specific, recurring technical sources: human data entry, format inconsistencies, schema drift, and simple programmer error.
Schema drift, when a column is added, removed, or renamed upstream without warning, is one of the most common and disruptive causes of bad data reaching downstream systems.
Data decay is a distinct cause worth watching separately from entry errors: data that was accurate when captured can become wrong simply because the real world changed.
String-based APIs and evolving user interfaces can silently introduce inconsistent data types into a system that wasn't designed to expect that variation.
Treating each pipeline stage as an independently testable component, like unit testing in software development, catches problems closer to their source rather than downstream.
The patterns in your data errors, not just the individual errors themselves, can reveal bigger issues like a whole region using the wrong timestamp format.
Data quality refers to the assessment of data to ensure its suitability for the intended purpose. Poor data quality, characterized by issues like accuracy, completeness, and consistency, can adversely affect the trustworthiness of the data your users are receiving, and result in bad business decisions.
It's crucial to prevent bad data from entering systems and propagating further, as repairing the damage caused by poor data can be costly and even harm the reputation of your whole organization.
Bad data can originate from a number of sources, including human errors during data entry and software-related issues like schema drift. For example, point-of-origin errors commonly occur when data is entered or recorded by individuals, leading to inaccuracies. Additionally, software-related factors such as data structure inconsistencies or format mismatches can contribute to bad data.
It's important to recognize these sources and be aware of how easily poor data can infiltrate systems.
To illustrate how easily poor data can enter systems, let's consider a few examples.
Date formats are a classic source of error, with discrepancies in formats causing confusion and misinterpretation. String-based APIs, which often return data as strings, can introduce data quality issues if not handled correctly.
User interfaces can also contribute to poor data quality, particularly when interfaces evolve while underlying data structures remain unchanged, for instance, a free text field being changed to a multiple choice dropdown. We end up with both types in our database, and data users might not be aware of this variation, so aren't designing for it.
String-based APIs, which often return data as strings, can introduce data quality issues if not handled correctly.
Programmers themselves can inadvertently introduce data errors through simple mistakes like transposing data or inserting incorrect values. Finally, case sensitivity can lead to duplicate files and data inconsistencies if not managed properly.
The UI example above is really one specific case of a broader, and increasingly common, pattern: schema drift.
Schema drift happens when the structure of incoming data changes, a column gets added, removed, or renamed upstream, without any coordination with the systems downstream that depend on it staying the same. A field that used to always be populated starts arriving empty. A value that was always numeric starts arriving as text. None of this looks dramatic at the moment it happens, but it's exactly the kind of change that breaks a pipeline silently, rather than with an obvious error message.
Schema drift is particularly disruptive because it tends to originate somewhere you don't control, a source system's own upgrade cycle, a vendor's API update, a client changing how they export their own data. The systems most exposed to it are the ones ingesting data from many different, independently evolving sources, exactly the kind of pipeline most teams are building more of, not fewer.
What makes schema drift particularly worth naming explicitly, rather than just treating as one more thing that occasionally goes wrong, is that it's detectable before it causes damage. A pipeline that expects a fixed set of fields and fails loudly the moment one goes missing is in a much better position than one that quietly accepts whatever arrives and only surfaces a problem three steps later, once the bad value has already propagated into a report or a downstream system.
Not all bad data starts out wrong. Some of it was perfectly accurate when it was captured, and has simply gone stale since.
Addresses, phone numbers, job titles, and other time-sensitive fields decay naturally as the real world changes around them. A customer's address is correct right up until they move, and nothing about your pipeline necessarily knows that's happened. This is worth watching as a distinct category from entry errors, since the fix isn't better validation at the point of entry, the data was fine then, it's an ongoing need to refresh and re-verify data that ages over time.
The practical challenge with decay is that it doesn't announce itself the way a validation failure does. A record with a missing field or an invalid date gets flagged and rejected immediately. A record with a perfectly well-formatted, perfectly plausible address that happens to be six years out of date sails through every check you have, because nothing about its structure is wrong. Catching decay usually means periodic re-verification against a source of truth, or accepting that some percentage of any dataset is quietly going stale at any given time, and building processes that assume that rather than treating every field as permanently settled once it's captured.
Most quality issues arise when you put your data in motion, an integration, a data ingestion process, or reshaping for use in a warehouse and so on. These data processes typically involve a data pipeline with multiple stages, each of which can allow errors to creep in. And they snowball, a seemingly small error can cause further errors throughout the pipeline and result in improper results for the end user.
Here's what you can do in your data pipelines to catch these issues closer to where they start.
Incorporate explicit data quality checks in your pipeline, such as validation and profiling stages. Validation applies business rules to each record to confirm it meets specific criteria, while profiling builds a statistical picture of the whole dataset to catch decay and drift before they become critical. We've covered exactly how these work together, including a real example
Recognize that poor data quality can emerge from any stage in the data pipeline.
Implement checks at each step to capture and address data quality issues as soon as possible, reducing the chances of propagating errors further downstream. Similar to unit testing in software development, treating each pipeline step as a testable component ensures early detection and resolution of data quality problems.
Instead of merely capturing data errors, consider establishing a data error management process, a central hub for collecting rejected data from the pipeline so it can be corrected and reprocessed. It involves a combination of people, processes, and technology, and standardizing how errors get reported matters as much as catching them in the first place. We go into this in more depth, including who should own the fix, in our guide to what is bad data.
Implementing these stages into your pipelines to identify and handle bad data can not only improve your immediate data quality, but it can also give you a valuable opportunity to identify larger business issues such as flawed processes or training requirements (e.g. 'All of our West region POS devices are using the wrong timestamp') or uncover and reconcile differences in interpretations or definitions (e.g. 'Is the project closed because the Status field says Closed, or because the Close Date is in the past?').
Every source of bad data covered above shares the same underlying challenge: catching it requires visibility into the pipeline itself, not just the data sitting at either end of it. CloverDX is built around exactly this, giving you visual, inspectable pipelines where every stage, ingest, transform, validate, load, can carry its own quality checks, rather than treating validation as a single bolt-on step at the start or end.
Schema drift and format inconsistencies are handled through configurable, reusable validation rules that flag or adapt to structural changes rather than failing silently, and profiling surfaces decay and drift trends before they cross a critical threshold. When something does need a human decision, business users can review, correct, and reprocess rejected records directly, without waiting on a developer to become available.
For teams onboarding data from many independently evolving sources, exactly where schema drift tends to originate, CloverDX's approach to faster data onboarding means a new source or a changed format doesn't mean building a new pipeline from scratch.
Most bad data isn't random, it follows patterns. Schema drift, data decay, format mismatches, and simple human error account for the overwhelming majority of what goes wrong in a pipeline. Once you can name them, you can design for them, rather than treating every new instance as a fresh surprise.
Let's talk about what that could look like for your pipelines.
This post is based on a webinar of the same title. You can watch the full video, presented by CloverDX Solutions Architect Kevin Scott, below: