Data problems have a habit of surviving technology changes. Organizations replace applications, move workloads to the cloud, adopt Agile ways of working and introduce AI, yet many of the same underlying problems remain. Legacy systems accumulate, data quality issues get pushed downstream and teams work around problems nobody has addressed at source.
Scott Ambler has spent decades watching that pattern repeat. A consulting methodologist at Ambysoft, he co-created Disciplined Agile, created the Agile Modeling and Agile Data methods, and coined the term data technical debt. He is also the author of more than 30 books, including Refactoring Databases: Evolutionary Database Design and his latest book, Not Just Data.
In a recent episode of Behind the Data, Scott joined host Matthew Stibbe to discuss why so many data projects fail for familiar reasons, how organizations can make the financial case for fixing dirty data upstream, and what AI changes for enterprise data teams.
Enterprise data environments are messy almost by definition. Large organizations may have hundreds or thousands of databases built at different times, using different technologies, naming conventions and assumptions. Even if every one were well designed, Scott says the combined estate would still be “a dog’s breakfast” because “the semantics are different, the naming conventions are different, it’s different technologies, different people at different times with different priorities.”
Many of the problems are not new. Organizations are still dealing with technical debt, limited data literacy, reluctance to invest in data quality and short-term decisions that create long-term problems. Scott says leaders can make choices that “make a lot of sense to them” but “really shoot them in the foot for the long term, particularly on the data side of things.”
That is why he is skeptical when data failures are treated primarily as technology failures. “99 times out of a hundred, it’s a people problem, like a management problem,” he says. In his career, Scott can identify only one project he would confidently describe as a genuine technology failure, and even there, “the decision to use that technology was a people problem.”
Scott argues that enterprise data is continuous in two senses. The data itself is always changing, arriving through real-time feeds, batch processes and hundreds of other sources. At the same time, data teams should be capable of responding continuously rather than relying on long delivery cycles.
“Even working in two week sprints is just breathtakingly slow in my world,” he says. If the business needs a new report, dashboard element or AI use case, he questions why providing the necessary data should routinely take weeks or months rather than hours or days.
DataOps has helped bring practices such as automated testing, integration and deployment into data work, but Scott still sees cultural resistance. He repeatedly encountered data teams that considered themselves exceptions to wider Agile or engineering practices and became “almost always some sort of bottleneck.”
Legacy databases are a good example. Scott accepts that careless schema changes can break dependent applications, but argues that this is a reason to learn how to make changes safely rather than avoid them indefinitely.
“If you’re scared to evolve a legacy production database, you’ve got a serious problem there,” he says. “And it’s you. Look in the mirror; you’re the problem.”
Scott and Pramod Sadalage documented 65 common database refactorings more than 20 years ago in Refactoring Databases. Once teams know how to evolve databases safely, he argues, there is less justification for months of speculative design before implementation. “Days, yeah, you bet. Months? Hell no.”
Scott makes his strongest case for addressing data technical debt through simple economics.
Organizations often allow dirty data to leave an upstream system and ask every downstream team to fix it independently. Reporting cleans it one way, analytics another, and an AI team may build yet another cleansing process.
“Cleansing data downstream, at the point where it’s used, is financially indefensible,” Scott says. “The numbers just don’t work out.”
His preferred conversation is with the CFO. Ask how many teams are using the same data, then establish how many are writing cleansing code, maintaining tests and updating those processes whenever something changes.
There is also no guarantee they are cleaning it consistently. “And let me guess, they’re probably not consistent, are they?” Scott says. Different approaches can leave different teams producing different answers from the same underlying information.
Once the duplication is visible, the argument becomes straightforward. “We know for a fact that you’re cleansing it six times,” Scott says. “I’m proposing it gets cleansed once and once only. Run the numbers. The conversation is now over.”
The goal is to fix the cause as far upstream as possible. If application code is producing bad data, fix the application. If manual entry is creating errors, fix the process. In Scott’s words, “pay down your technical debt” instead of repeatedly paying other teams to manage its symptoms.
The financial argument may be simple, but changing the organization is not. Moving responsibility upstream means changing funding, processes and established ways of working.
Scott says organizations need to stop treating this as a conventional project. “We gotta stick with it for five or ten years,” he says, “because you’re not gonna change this overnight and it’s not a project.”
Maintaining attention for that long is difficult because the work is important without being especially exciting. As Scott puts it, “it’s a high priority, but it’s not sexy.” That helps explain why technical debt survives even when the cost of leaving it alone is well understood.
AI adds another consumer of enterprise data, but Scott cautions against expecting LLMs to introduce certainty where the underlying technology is probabilistic.
“The fundamental nature of the architecture of LLMs is that they’re making guesses,” he says. “They’re getting better, but they’re still making guesses.”
Scott describes LLMs as hallucinating “100% of the time” because every output is probabilistically generated. His concern is less about the label than whether people understand the nature of the answer they are receiving.
“The issue is, how good is the quality of that hallucination?” he says. In some situations, a generated answer can be perfectly adequate. In others, being wrong carries far greater consequences.
“LLMs are not deterministic. And that’s okay,” Scott says. “Just know that that’s the case.” Whether an output is good enough remains “a judgment call for the humans”.
He also warns against treating every AI problem as an LLM problem. “Have some real data scientists on your team,” he says, meaning people who understand the range of algorithms available and can choose the right approach for the business problem.
An insurance fraud system, for example, may use a traditional machine learning model for detection and an LLM later to explain or translate the result. “You might use a combination of algorithms or a combination of tools,” Scott says. “And so it’s not just an LLM play.”
Data technology will continue to change, but many of the choices facing organizations remain familiar. They can clean the same data repeatedly downstream or invest in fixing its source. They can work around legacy systems indefinitely or build the capability to change them safely. They can add AI on top of existing problems or address the data debt underneath them.
Scott’s argument ultimately comes back to cost and organizational behavior. Cleaning the same data six times may be politically easier than persuading an upstream team to change, but it is harder to defend financially. If bad data keeps moving downstream, preventing the problem at source reduces duplicated effort, inconsistency and maintenance.
AI gives organizations another reason to tackle the work, but it does not change the basic lesson. Good tools cannot compensate indefinitely for poor data and the decisions that allowed it to become poor in the first place.
To hear more from Scott Ambler about data technical debt, DataOps, Agile data practices and AI, listen to the full episode of Behind the Data: Data debt is holding your AI hostage with Scott Ambler
If you're looking for a better way to integrate, transform and manage complex enterprise data, get in touch with the CloverDX team for a demo.