Data cleaning is rarely the part anyone looks forward to. But leave it for long enough and eventually it gets in the way of everything else.
Susan Walsh knows that problem better than most. She is the founder and Managing Director of The Classification Guru and author of Between the Spreadsheets: Classifying and Fixing Dirty Data and Optimizing Sales and Marketing Data. Her route into data was unconventional, beginning in sales and retail before a data classification role turned into a career spent working with messy, real-world datasets.
In a recent episode of Behind the Data, Susan joined host Matthew Stibbe to discuss what organizations get wrong about clean data, why AI still needs human oversight and why some of the most valuable data work is also the easiest to postpone. The underlying problem is not necessarily a lack of technology, but recognizing that cleaning data is skilled work in the first place.
One of Susan’s earliest observations about data education is that datasets used in training often bear little resemblance to what people encounter inside organizations. “You are working with sanitized data on a course,” she says. In the real world, information is rarely so accommodating.
Records are incomplete, naming conventions vary, descriptions lack context and fields contain values they were never designed to hold. Even when a technical process works exactly as intended, the result can still be wrong.
Susan has encountered this disconnect when working with technically skilled teams. “I can’t tell you the number of arguments I’ve had with data scientists, analysts who are like, ‘Well, my code is right.’ And I’m like, ‘Your code might be right, but the output in what I’m seeing is not right. So something’s wrong somewhere.’”
Data classification is not always a binary exercise. “There’s a lot of context involved, and there’s often more than one right answer,” Susan says. The goal is not necessarily to discover one mathematically perfect classification, but to apply judgment consistently enough that the resulting data remains useful.
Technical ability matters, but so do critical thinking, business understanding and the ability to recognize when an apparently correct output does not make sense.
AI has created obvious opportunities to accelerate some of this work, and Susan is already using it herself, including building agents to support data cleaning. But she is quick to put a boundary around what that means.
“You cannot just use agents and build agents to clean your data,” she says. “You still need to know and understand your data first.” That means knowing what correct data should look like, understanding what is wrong with the information you already have and checking what the AI gives you in return.
“It should be used as an aid and an assistance,” Susan says, rather than assuming “it’s gonna do everything for me.”
Her own experience shows why. Susan’s team has developed Samification, a tool that standardizes and normalizes company names. She explains that variations such as Dell, Dell Computers and Dell Inc. can be matched against a large master list and normalized to a consistent name.
But once AI is asked to handle records it has not seen before, pattern recognition can create new errors. Susan gives the example of businesses with names like Walsh Hairdressers, Walsh Lawyers and Walsh Cleaners. The system would see the repeated name and normalize them all to “Walsh Group”.
“We didn’t tell it to do that,” she says. “It just saw the name Walsh and decided that it was all the same thing.” It is a good illustration of the difference between recognizing a pattern and understanding what the data actually represents. As Susan puts it, “There’s a lot of work to be done. It’s not magic.”
It can be tempting to respond to imperfect AI output by adding another algorithmic layer, whether that means fuzzy matching, confidence scoring or additional rules. Susan takes a more cautious approach because her priority is accuracy.
When her team uses AI to normalize names, the result is still manually checked and cleaned afterwards. “That’s the best guaranteed way that you can get it right,” she says. “Have someone check it afterwards.”
The right balance between automation and human involvement also depends heavily on the type of data being processed. AI may be extremely effective when classifying standardized part numbers and descriptions, for example, but introduce ambiguous abbreviations or incomplete descriptions and human judgment becomes much more important.
“This is where we have to stop talking about data as one blanket thing,” Susan says. There is no single ratio of machine to human work that applies to every dataset. The appropriate level of automation depends on how standardized the information is, how much context is required and what happens if a classification is wrong.
AI should also become more effective as organizations build cleaner reference data it can learn from. But that creates something of a dependency, because better automation often requires cleaner data to begin with.
Many organizations already know they have a data quality problem, but that does not mean they deal with it quickly.
Anyone who has opened a CRM and been presented with hundreds or thousands of possible duplicates will recognize the problem. The software can identify that something might be wrong, but deciding whether two records actually represent the same person or organization often requires context. As Susan says, “people don’t do it because it is hard.”
There is also a longstanding reluctance to invest in cleaning. “People don’t want to pay to clean data,” she says. “They think it’s something they can do themselves or their team can do.”
Technically, they often can. But there is a difference between being capable of cleaning data and doing it repeatedly enough to recognize the patterns, exceptions and less obvious problems hidden inside it. “Everybody can clean data,” Susan says, “but when you do it day in, day out, you’ve got the right tools, you can do it far quicker, far more efficiently and far more accurately.”
Experience changes what people know to look for. When consolidating records, for example, Susan’s team will not simply deduplicate based on a company name. They might also examine postal codes, email addresses, telephone numbers and other reference points.
“If you’ve never done that before, then you’re not gonna know to look for it,” she says. “We’re looking for things that you don’t even know exist.”
That is part of what makes clean data a craft rather than simply a mechanical task.
The same issues appear across procurement, finance, sales and marketing data, and for Susan there is a simple reason. “Ultimately data problems are people problems,” she says. “It’s whoever’s putting that information in that’s causing the problem.”
Sometimes that person is an employee. Sometimes it is a customer or member of the public. Either way, errors introduced at the point of entry can travel through systems and become much more difficult to fix later.
Organizations can reduce the problem by introducing sensible controls, such as preventing text from being entered into numeric fields, but technology can only go so far. Good data management therefore starts long before somebody opens a spreadsheet to clean up the mess.
It requires thinking about how information is entered, what standards people are expected to follow and whether those expectations are clear enough for non-data specialists to understand.
The payoff becomes clearer when clean data begins exposing things an organization could not previously see.
Susan frequently works with companies that lack even basic visibility into their spending because information is spread across countries, systems and formats. The process begins by bringing those different sources together, standardizing suppliers and classifying spending into categories that reflect what the business actually needs to understand.
The insights can be surprisingly simple. A business might discover it is spending millions on a category it had never properly quantified, or that it has 500 suppliers providing stationery when a much smaller number would suffice.
“There are cost-saving opportunities, there are supplier rationalization opportunities,” Susan says.
In one example, a client had invoices with an average value of £23, while processing each invoice cost more than the invoice itself. That creates an obvious operational question: could purchases be consolidated so that the business orders once a month or once a quarter rather than repeatedly processing low-value transactions?
The benefits then extend beyond the purchase price to shipping, processing costs and employee time. None of these insights requires a futuristic algorithm. They require data that has been brought together and organized well enough for the business to see what is actually happening.
Classification presents another challenge for automation because words do not always mean what they appear to mean.
Susan’s team might search supplier names and invoice descriptions for terms such as “lawyer”, “solicitor”, “hotel”, “inn” or “lodge” to find clues about how something should be categorized. But clues are not conclusions.
“You can’t just do a blanket keyword search and paste stuff,” she says. “You still need to check.”
A company called Walsh Lodge might sound like a hotel but turn out to be a dental practice. “That’s where you’ve got to be careful with AI,” Susan says, because the model is looking for patterns and words. If “lodge” is most frequently associated with travel in its experience, that can become the default assumption.
“You need that human aspect to be able to check it.”
The same principle appeared when Susan tested someone for a data classification role. LinkedIn appeared as a supplier and the accompanying description contained the word “restaurant”. The candidate classified it as a restaurant because they followed the description without considering the record in context.
What she is looking for is not simply whether someone can apply a rule, but whether they demonstrate “common sense” and use context to determine whether the answer makes sense.
Every taxonomy eventually encounters something awkward. A category works for almost everything, then a record appears that does not quite fit, and the tempting solution is to create a catch-all “miscellaneous” category.
Susan’s reaction is unequivocal. “That is the worst thing,” she says, because people will naturally take the shortcut and begin putting more and more information into it.
If something is valuable or occurs frequently, it deserves an appropriate category. If it appears once and is worth very little, Susan recommends placing it in the closest sensible category rather than creating a dumping ground.
“If it’s a high value or an important thing, it should have its own category somewhere,” she says. “But if it’s only occurring once and the value’s like five pounds, don’t worry about it. Just put it in the nearest thing.”
The important point is that the classification should still relate to what the item actually is. The problem with miscellaneous categories is behavioral as much as structural because, as Susan puts it, “people will go for the quickest, easiest route so they don’t have to think about it. If you give them the lazy option, they will use it.”
That can quickly destroy the usefulness of a taxonomy. If a marketing database contains a large “miscellaneous” segment, for example, it becomes much harder to target those customers meaningfully. The data technically has a classification, but that classification provides little value.
AI introduces another complication because incorrect classifications can still look highly convincing.
Susan encountered this while helping to quality-assure an AI classification system. Two records had exactly the same name and description but appeared in different places within the dataset, and the AI assigned them different classifications.
Both answers could be justified, but that was precisely the problem. The model was treating each line independently rather than considering the dataset as a whole and maintaining a consistent decision.
“That same exact query could come in on a different day and you might get a different result,” Susan says.
Adding a confidence score does not necessarily solve the problem either. “Confidence based on what?” she asks. “I’ve seen a hundred percent confidence accurate and it’s not accurate.”
As Susan puts it, “just because it’s confident doesn’t mean it’s right.” For organizations building AI into data workflows, that makes validation essential. Confidence can be a useful signal, but it is not a substitute for checking whether an answer is actually correct.
Susan’s approach to preventing these problems is summarized in her COAT framework, which stands for Consistent, Organized, Accurate and Trustworthy.
The starting point is consistency. “Make sure it’s consistent so everyone’s using the same processes, the same naming conventions, terminology, like units of measure, date formats,” Susan says.
Next comes organization. Data should be categorized and structured so that people can find what they need without having to reconstruct its meaning every time.
Accuracy also needs to be defined rather than assumed. Teams should agree on what accurate data means in their particular context and make sure everyone working with it understands the standard.
Once data becomes consistent, organized and accurate, Susan argues that the final outcome is trust. “We always hear, ‘We don’t trust the data,’” she says. With trustworthy information, people can make business decisions with much greater confidence.
But putting a COAT on the data is not a one-off exercise. “You can’t just clean your data and leave it,” Susan says. “You have to maintain it and manage it.” She recommends revisiting it monthly or quarterly rather than waiting for the problems to become overwhelming.
That may be the least glamorous lesson in data management, but it is also one of the most important. Clean data does not stay clean by itself because people introduce errors, businesses change, new records appear and categories evolve. AI finds new patterns too, including some that organizations never intended it to find.
The dishes keep coming back, and the organizations that get the most value from their data are unlikely to be the ones that find a magical way to avoid the work. They will be the ones that build the processes, skills and habits required to keep doing it well.
To hear more from Susan Walsh about clean data, classification and where human judgment still matters in an AI-enabled world, listen to the full episode of Behind the Data.
If you're looking for a better way to integrate, transform and manage complex data across your organization, get in touch with the CloverDX team for a demo.