Data migration can be a daunting prospect, despite being a very common process for organizations to undertake. It’s not often something anyone looks forward to doing, but is usually a necessary part of a transition to something better.
This guide offers an overview of what a data migration is, some examples of when you might need to do one, and some tips for planning a successful project.
Data migration is the process of moving data from one place to another - typically between applications, storage systems or databases. There is one active data set before and after the migration. This is in contrast to data integration where data is active in both, or several, places.
Data migrations are one of the most common data processes and almost all companies will eventually undertake one. There can be many reasons for data migrations, including application replacements or upgrades, business process changes, data volume growth and performance requirements.
The data of course needs to be moved from one place to another, but it also needs to be made fit for purpose in the new system. This often requires data validation, correcting problems in the source or during transportation, converting data formats, and more complex data transformations such as merging values or calculating new ones.
There are three main approaches when it comes to data migration:
Whichever approach you choose, there are several different methodologies for undertaking a data migration.
A 'big bang' data migration is when you migrate all data in one operation. This may take a while, but for users there is a single point in time where they can no longer use the old data and the new system goes live. From their point of view, the change was made in a single 'big bang' event.
Big bang migrations typically have significant preparation periods and short down times, during which the system is unavailable. The ideal big bang migration has zero down time, but you can't always guarantee this.
The overall process can be visualized like this:
Only after data owners and all other stakeholders confirm that the migration was successful can the whole process be considered complete.
The fact that the migration itself only happens toward the end of the process is both a benefit and a drawback of a big bang migration. It can be beneficial, as users do not need to think about two different systems simultaneously and switch between two live systems.
On the other hand, the data migration being so late increases the burden on the planning, development and testing phases and insufficiencies in these areas can lead to expensive failures.
It's important to remember however that a data migration is never a one-time thing
A trickle migration can be likened to an agile approach to a data migration, breaking the migration down into many smaller sub-migrations, each with its own set of goals, data, deadlines and scope.
This approach allows stakeholders to verify the success of each individual phase, giving a stepwise indication of progress. Should any of the sub-processes fail, it is usually necessary to re-run only the failed process and lessons learned from that failure can be applied to subsequent runs.
Trickle migrations, however, may require more complex planning. They also place a higher burden on users of the data since they have to keep working with two systems while the ongoing overall migration takes place.
Each methodology has its own pros and cons. Regardless of the methodology, migrations can be expensive, especially if badly planned or executed.
Advantages of a big bang data migration:
Disadvantages of a big bang data migration:
Advantages of a trickle data migration:
Disadvantages of a trickle data migration:
The differences between a trickle and a big bang migration are considerable. Deciding which option to choose should be made very early on. The decision is often driven by several key questions:
There is no simple rule, however big bang migrations are normally selected where the scope is well defined from the outset and where deadlines or other project properties mandate it.
Conversely, trickle migrations are beneficial when the migration can be easily split into several different stages. They are also suitable when the scope is hard to define. In such cases, the trickle migration's phased migration approach allows you to migrate “easier” data first while dealing with the more complex processes later.
Another major consideration is the experience of your team. Each migration can pose different technical and project management challenges. Ensuring you have the most suitable people with the most relevant experience on your team can play a significant role in the final outcome. Teams that prefer an agile approach typically prefer trickle migrations and, conversely, teams who are more used to a waterfall methodologies prefer the big bang approach.
The importance of good planning for your data migration can’t be overestimated. A bad plan can result in project failure - and data migration failure can often jeopardize bigger business initiatives, not to mention leading to a loss of trust and a large bill.
However, a data migration is often a chance for businesses to dive deeply into and rethink their data. Investing time and effort into the planning process can not only mean a more successful migration project, but can pay off in more efficient systems and greater business value.
Finding the Right People
Contracting External Experts
Finding the Right Way
A data migration involves a number of stages:
Each stage has its own considerations and needs to be planned properly to deliver a successful project. Read more details in the post: 13 Stages of a Successful Data Migration
Typically, data migration occurs during an upgrade of existing hardware, transfer to a completely new system, or instances such as application replacements, business process changes, data volume growth, or the need for better performance.
Some examples of when a data migration might be needed:
Read more about how the CloverDX Data Integration Platform can help with data migration projects - reducing time to delivery, automating complex processes, handling errors and managing data at scale.Data Migration with CloverDX