When it comes to data migrations, the actual moving of the data is often the easiest part. What’s difficult is the parts around the actual data movement - the data discovery, quality and cleansing parts - as well as managing the process at scale. Specialist data software can help enormously with the entire data migration process.
CloverDX, which at its foundation is an ETL (Extract, Transform and Load) tool, is ideally suited to data migrations. After all, what is a migration but Extracting data from one system and Loading it into another, usually with some Transformations in between.
CloverDX’s approach of creating workflows using a visual interface, with plenty of ready-made components, makes building the data processes you need fast and flexible.
Building a repeatable workflow that handles as many of the steps as possible automatically means you save time by not having to re-do work every time something in your data or requirements changes.
A data migration is an iterative process. By creating a process that you can easily tweak and repeat you’re saving a huge amount of time and effort, as well as making your ‘go-live’ more predictable.
CloverDX workflows are built to be repeatable, so you can iterate and review changes quickly on your whole data set, not just an artificial test set. So when you get to your switch over point, you should already know exactly how your data will look.
The importance of repeatability for data migrations
Trying to manage a data migration manually can cause problems when there’s a small change in your data or your target system (which there inevitably will be). With a manual process you might need to start all over again and re-do all your work. With a system built for repeatability, you can easily go back and forth, make changes, and re-run the whole process automatically.
Automating as much of the data migration process as possible saves time, energy and frustration - and results in better delivery.
Not only does automation eliminate bottlenecks that can come from manual work, but it also reduces human error. Being able to perform as many of the migration steps as possible at the click of a button speeds up anything from a series of transformations to a full-scale automated mapping framework.How 200,000 files with constantly changing schemas were migrated in just two months, using a tool that automatically creates mapping documents without human input
And automation enables you to test your workflows more effectively by making it easier to test the whole of your data set, rather than a sample.
CloverDX has built in data quality features to clean your data before you move it to your new system. Perform basic checks such as validating emails or phone numbers, flagging missing values or checking data conforms to defined rules is simple with built-in components that are fully customizable.
Eliminating dirty data as part of your migration process not only improves your overall data quality, but also helps reduce storage costs (as you no longer have to pay to store redundant or duplicate data), and can speed up your processing times.Case Study: Migrating Legacy Data
Automate your error handling by exporting any values that don’t conform to your rules, and define repeatable processes to fix errors and return cleaner data to your systems.Managing Bad Data: 5 Things You Need To Know
You can’t usually get away with just exporting data from one place and importing it to another. There’s generally some transformations required to get the data into the target system in the way that you need it.
CloverDX is built for data transformations.
Out of the box components that you can define in a visual editor, or dive into and amend in code, make transformations such as these transparent and flexible:
And of course because you can work right in the code with everything in CloverDX, you can configure or build whatever you need to.
Fine-tuning your rules can take time, but with CloverDX it’s easy to adjust the rules and re-run the process, without touching the source data itself. So you can do it as many times as you need, so your business and technical teams agree and understand what’s happening and what the results will be.
Manual massaging (in Excel or some data wrangling tool like OpenRefine) may be clear to everyone at the time, but what’s been done will get lost over the coming months. Manual data migration processes mean there’s no way to keep track of changes other than writing (a lot of) documentation - and keeping it up to date.
Because CloverDX shows each step of the workflow, and automatically records what is happening, the whole process is easy to see and track back.
Whether to manage your data migration in-house or contract external resource to help - or to manage the entire project - depends on a number of factors.
Are you working to a hard deadline? The more time pressure you are under, the more you might be able to benefit from extra resource. Not only can an external party provide more manpower, but they might also be able to propose a faster or more scalable solution than you could implement.
Do you have enough people in-house with the skills to manage your migration? Do they know enough to cover the technical aspects (including up to date best practice and future-proofing) but also how to get the best out of the business users who will need to be closely involved in the process?
Even if you can answer yes to #2 above, is handling your data migration the best use of your team’s time? For instance, if your business is built on providing a product or service to your clients, taking engineers’ time away from that core work to invest in an internal project could be a bad decision.
If you do decide to outsource some or all of your data migration, make sure you ask the right questions of your potential partner:
Taking a data migration project as an iterative piece of work with several small steps means that any changes that come up along the way can be more easily managed, and means that you can see exactly what’s happening at each stage. When you can see progressive results at the end of each step, it’s much less worrying than not seeing anything until a single big reveal at the end.
As you may have gathered if you’ve read the rest of this page, repeatability is the key to a smooth, successful data migration. Requirements or data structures *will* change as the new system gets deployed. If the data has been massaged manually, all the previous effort is wasted.
We’ll seldom touch your data by hand. Instead, we build a data pipeline, driven by rules and configuration parameters, that can be run automatically. When any changes are needed, you can just discard the current data, update the rules and run it again, at virtually zero cost - however many times you need to do it.
Typically, about 50% of a migration consists of simple mappings between source system and target. Another 30% is data with quality issues that need resolving. And it’s only the remaining 20% that tends to be tough problems requiring creativity, lots of manual labor or highly skilled coders.
We like to try and minimize the effort around first ‘simple’ part so we can concentrate on working with you to solve the harder problems. Building automated frameworks to handle most of the mapping can instantly eliminate a huge chunk of effort and time.
We’ve worked with many different data migrations, large or small, and in many different industries. But the kind of projects we love, and can bring the most benefit to, tend to fall into four main categories:
With CloverDX, IT can expose data with complex relationships to business users (often by transforming it into a human-readable, and actionable, form) for an easier data discovery phase.
Automated mapping and analysis makes project delivery faster, as well as keeping workflows flexible against incoming changes and variations in data.
With automated testing, you can save time by fixing issues as you develop, rather than waiting until a later test phase.
Automating validation rules, and the process by which data gets rejected, output for fixing and fed back into the process, ensures consistency across the project and a more streamlined process.
CloverDX can connect to, and ingest data from, any source system that your customers are using, and can also handle data in any format or any quality. So you can onboard faster, and neither you or your customers have to worry about your data formats.
Reusable rules speed up the import process by providing lists of errors that need fixing.
If any of these scenarios sound like a problem you're trying to overcome, get in touch and we can talk more about where we've helped others in similar situations.
A Workday implementation consultancy can now onboard significantly more clients, without the need for extra workforce.
Automating the mapping needed to move data from the clients’ legacy systems into Workday means that the consultants now only need to spend a few days configuring the framework rather than weeks (or even months) in Excel like they had to do before.