Data Migration with CloverDX

When it comes to data migrations, the actual moving of the data is often the easiest part. What’s difficult is the parts around the actual data movement - the data discovery, quality and cleansing parts - as well as managing the process at scale. Specialist data software can help enormously with the entire data migration process.

CloverDX, which at its foundation is an ETL (Extract, Transform and Load) tool, is ideally suited to data migrations. After all, what is a migration but Extracting data from one system and Loading it into another, usually with some Transformations in between.

Benefits of CloverDX for Data Migrations

  • Reduce time to delivery

CloverDX’s approach of creating workflows using a visual interface, with plenty of ready-made components, makes building the data processes you need fast and flexible.

Building a repeatable workflow that handles as many of the steps as possible automatically means you save time by not having to re-do work every time something in your data or requirements changes.

  • Reduce expensive surprises

A data migration is an iterative process. By creating a process that you can easily tweak and repeat you’re saving a huge amount of time and effort, as well as making your ‘go-live’ more predictable.

CloverDX workflows are built to be repeatable, so you can iterate and review changes quickly on your whole data set, not just an artificial test set. So when you get to your switch over point, you should already know exactly how your data will look.

The importance of repeatability for data migrations

Trying to manage a data migration manually can cause problems when there’s a small change in your data or your target system (which there inevitably will be). With a manual process you might need to start all over again and re-do all your work. With a system built for repeatability, you can easily go back and forth, make changes, and re-run the whole process automatically.

  • Automate complex processes

Automating as much of the data migration process as possible saves time, energy and frustration - and results in better delivery.

Not only does automation eliminate bottlenecks that can come from manual work, but it also reduces human error. Being able to perform as many of the migration steps as possible at the click of a button speeds up anything from a series of transformations to a full-scale automated mapping framework.

How 200,000 files with constantly changing schemas were migrated in just two months, using a tool that automatically creates mapping documents without human input

And automation enables you to test your workflows more effectively by making it easier to test the whole of your data set, rather than a sample.

  • Validate data before it gets to the target system

CloverDX has built in data quality features to clean your data before you move it to your new system. Perform basic checks such as validating emails or phone numbers, flagging missing values or checking data conforms to defined rules is simple with built-in components that are fully customizable.

Eliminating dirty data as part of your migration process not only improves your overall data quality, but also helps reduce storage costs (as you no longer have to pay to store redundant or duplicate data), and can speed up your processing times.

Case Study: Migrating Legacy Data
  • Build in data quality feedback loops

Automate your error handling by exporting any values that don’t conform to your rules, and define repeatable processes to fix errors and return cleaner data to your systems.

Managing Bad Data: 5 Things You Need To Know
  • Transform data at scale

You can’t usually get away with just exporting data from one place and importing it to another. There’s generally some transformations required to get the data into the target system in the way that you need it.

CloverDX is built for data transformations.

Out of the box components that you can define in a visual editor, or dive into and amend in code, make transformations such as these transparent and flexible:

  • Splitting or merging multiple fields
  • Validating fields
  • Converting timezones or currencies
  • Changing product codes
  • Updating naming conventions

And of course because you can work right in the code with everything in CloverDX, you can configure or build whatever you need to.

Fine-tuning your rules can take time, but with CloverDX it’s easy to adjust the rules and re-run the process, without touching the source data itself. So you can do it as many times as you need, so your business and technical teams agree and understand what’s happening and what the results will be. 

  • Keep everything transparent

Manual massaging (in Excel or some data wrangling tool like OpenRefine) may be clear to everyone at the time, but what’s been done will get lost over the coming months. Manual data migration processes mean there’s no way to keep track of changes other than writing (a lot of) documentation - and keeping it up to date.

Because CloverDX shows each step of the workflow, and automatically records what is happening, the whole process is easy to see and track back.

Deliver a successful data project - book a call now

In-House or External?

Whether to manage your data migration in-house or contract external resource to help - or to manage the entire project - depends on a number of factors.

  1. Your timescale

    Are you working to a hard deadline? The more time pressure you are under, the more you might be able to benefit from extra resource. Not only can an external party provide more manpower, but they might also be able to propose a faster or more scalable solution than you could implement.

  2. Your expertise

    Do you have enough people in-house with the skills to manage your migration? Do they know enough to cover the technical aspects (including up to date best practice and future-proofing) but also how to get the best out of the business users who will need to be closely involved in the process?

  3. Your other priorities

    Even if you can answer yes to #2 above, is handling your data migration the best use of your team’s time? For instance, if your business is built on providing a product or service to your clients, taking engineers’ time away from that core work to invest in an internal project could be a bad decision. 

If you do decide to outsource some or all of your data migration, make sure you ask the right questions of your potential partner:

  • Can they dedicate enough time to see the project through?
  • Do they have experience in both systems involved in the migration? If you can’t get experts in both systems, think about engaging someone with expertise in your target system, as that’s the one you’re likely to be less familiar with yourselves.
  • Do they understand best practice in your industry, and are they making effort to understand your business?
  • Can they provide more than just resource? You’re not just buying manpower, you should be getting added value and expertise too - your partner should be able to propose creative solutions that you wouldn’t have thought of.
  • Are you happy with their approach, ability to manage the project, and how they work?
  • Can they accommodate your changes and move quickly? Do they have technology or automation solutions in place to help deliver?
  • Remember that the project doesn’t end at migration. Systems need ongoing integration. Can your partners offer help, either in planning for this at the outset or advising on ongoing best practice?

The CloverDX Approach to Data Migrations

Iterative project management

Taking a data migration project as an iterative piece of work with several small steps means that any changes that come up along the way can be more easily managed, and means that you can see exactly what’s happening at each stage. When you can see progressive results at the end of each step, it’s much less worrying than not seeing anything until a single big reveal at the end.

Repeatable process

As you may have gathered if you’ve read the rest of this page, repeatability is the key to a smooth, successful data migration. Requirements or data structures *will* change as the new system gets deployed. If the data has been massaged manually, all the previous effort is wasted.  

We’ll seldom touch your data by hand. Instead, we build a data pipeline, driven by rules and configuration parameters, that can be run automatically. When any changes are needed, you can just discard the current data, update the rules and run it again, at virtually zero cost - however many times you need to do it.

Automated framework

Typically, about 50% of a migration consists of simple mappings between source system and target. Another 30% is data with quality issues that need resolving. And it’s only the remaining 20% that tends to be tough problems requiring creativity, lots of manual labor or highly skilled coders.

We like to try and minimize the effort around first ‘simple’ part so we can concentrate on working with you to solve the harder problems. Building automated frameworks to handle most of the mapping can instantly eliminate a huge chunk of effort and time.

Where CloverDX Can Help

We’ve worked with many different data migrations, large or small, and in many different industries. But the kind of projects we love, and can bring the most benefit to, tend to fall into four main categories:

Bespoke or complex scenarios

  • Where a system vendor is no longer available or unwilling to undertake the work
  • Where other consultancies maybe don’t understand the complexity of your project
  • Where you have heavily customized systems with lots of changes made over the years

Agile re-iteration

Complex migrations can’t be done in Excel. With Clover you build a ‘recipe’ that can be executed over and over, with new transformations and/or data.

Migration between environments

Moving data between multiple staging environments for dev/test/UAT and so on can easily be managed with CloverDX as part of the migration process.

Process in parallel

As CloverDX can run multiple processes in parallel, it can help with quicker turnaround (especially in cloud environments).

Collaborate between business and IT

With CloverDX, IT can expose data with complex relationships to business users (often by transforming it into a human-readable, and actionable, form) for an easier data discovery phase.

Extremely tight deadlines

  • Where there’s a project that’s going off the rails
  • Where your in-house resource isn’t enough to meet deadlines
  • Where your deadline calls for automation and some unorthodox solutions

Speed up with automation

Automated mapping and analysis makes project delivery faster, as well as keeping workflows flexible against incoming changes and variations in data.

Detect issues before the test phase

With automated testing, you can save time by fixing issues as you develop, rather than waiting until a later test phase.

Messy data not usable in your target system

  • Where you have heavily customized data sets with a lot of mess in them
  • Where a high volume of mess requires clever validation

Automate data validation

Automating validation rules, and the process by which data gets rejected, output for fixing and fed back into the process, ensures consistency across the project and a more streamlined process.

Data migration is a component of a bigger mission

  • Where you are providing a platform or service that relies on moving data efficiently
  • Where repeatedly onboarding new clients is taking too much time and effort
  • Where you’re entering a new market but aren’t sure how to handle the data part of what you need to do

Handle any source system easily

CloverDX can connect to, and ingest data from, any source system that your customers are using, and can also handle data in any format or any quality. So you can onboard faster, and neither you or your customers have to worry about your data formats.

Reusable validation rules

Reusable rules speed up the import process by providing lists of errors that need fixing.

If any of these scenarios sound like a problem you're trying to overcome, get in touch and we can talk more about where we've helped others in similar situations.

Data Migration Project Example with CloverDX

Automating data migration reduces onboarding time by 90%

Read the case study: Effectively Migrating Legacy Data into Workday

A Workday implementation consultancy can now onboard significantly more clients, without the need for extra workforce.

  • Before the project migrating each new customer took around 200 days.
  • After the automated data migration framework was deployed, migration now takes just 20 days.

Automating the mapping needed to move data from the clients’ legacy systems into Workday means that the consultants now only need to spend a few days configuring the framework rather than weeks (or even months) in Excel like they had to do before.

Project Highlights

  • Data loading: The migration framework can generically load data from Excel files based on configuration stored in the database
  • Data validation: The solution can validate each record for any given HCM workflow. The workflows are highly customized based on module, but additional modules are simple to add in
  • Reconciliation: The generic, database-driven reconciliation framework can be reused to reconcile data based upon a configurable key
  • Reporting: Report on any errors with input data, transformations, validations and output data

Ask us about your data migration project, or see CloverDX in action by booking a demo

Request a CloverDX demo