200,000 XML files with constantly changing schemas, 3000 fields that needed to be mapped into 150 database tables. Just two months until the migration had to go live.
When a software company realized that their team of developers writing custom scripts wasn’t going to be able to meet their deadline, they called us.
Complex data migration, little time
The project involved working with a manufacturing company who needed to migrate thousands of critical manufacturing formulae from a legacy system into a new management platform in just a few weeks. The data was complex, was changing frequently throughout the project and there was a lot of it. While the client had had a team of developers creating separate scripts to migrate individual data structures, it quickly became apparent that the approach couldn’t scale quickly enough to handle the 200,000 files that needed to be migrated.
A more scalable data migration approach
Rather than trying to migrate so many different data structures manually, we took a different, more scalable approach and created a system that could automate the migration. The tool recognizes XML elements and attributes and automatically creates mapping documents, all without needing consultants’ input.
Despite the XML structures changing multiple times over the course of the project, the automated migration tool detected and adapted to structure changes and could successfully process all the files generically, automatically analyzing and mapping 3,000 fields to the new database. The project delivered on time and accurately and with a smaller team than originally planned.
Reusable for the future
The benefits of the automated data migration didn’t end there. A data migration is always an iterative process that continues beyond the go-live, so the fact that the framework is completely reusable and configurable means that the client can continue tackling further changes in data or structures for as long as they need.
Data integration software and ETL tools provided by the CloverDX platform (formerly known as CloverETL) offer solutions for data management tasks such as data integration, data migration, or data quality. CloverDX is a vital part of enterprise solutions such as data warehousing, business intelligence (BI) or master data management (MDM). CloverDX Designer (formerly known as CloverETL Designer) is a visual data transformation designer that helps define data flows and transformations in a quick, visual, and intuitive way. CloverDX Server formerly known as CloverETL Server) is an enterprise ETL and data integration runtime environment. It offers a set of enterprise features such as automation, monitoring, user management, real-time ETL, data API services, clustering, or cloud data integration.