Data migration can feel daunting, even though it’s a standard procedure that most organizations go through at some point. Careful planning and preparation will help to remove most of the potential pitfalls and, in most cases, allow a business to transition to a more effective and resilient way of working.
This guide offers an overview of what data migration is, examples of when it might be required, and some tips for ensuring a straightforward process.
Data migration is the act of moving data from one place to another - typically between applications, storage systems or databases.
There is one active data set before and after the migration. This is in contrast to data integration, where data is active in both or several places.
Almost all companies will eventually undertake data migration for various reasons, including application replacements or upgrades, business process changes, data volume growth and performance requirements.
Data needs to be moved to the new location and checked to ensure that it is fit for purpose in the new system. This often requires data validation, correcting problems in the source or during transportation, converting data formats and more complex data transformations such as merging values or calculating new ones.
There are three main approaches when it comes to data migration:
Whichever approach you choose, several different data migration methodologies are on offer.
A big bang data migration is when the entirety of your day is migrated all in a single operation. While the process may take a while, users experience a single point in time where they can no longer use old data before the new system goes live. As such, they experience the change as sudden and substantial – a 'big bang' event.
Big bang migrations typically require significant preparation periods and short downtimes, during which the system is unavailable. The ideal big bang migration has zero downtime, but guaranteeing this is usually impossible.
The overall process can be visualized like this:
The migration can only be considered complete when data owners and all other stakeholders have confirmed that migration has been successful.
The fact that the migration itself only happens at the end of the process is both a benefit and a drawback of big bang migration. Users do not need to handle two different systems simultaneously and manage the potential awkward ‘overlap’ between two live systems.
However, as data migration happens so late in the process, this increases the burden on the planning, development, and testing phases. Any problems in these areas can lead to expensive failures.
Remember: data migration is never a one-time thing
A trickle migration breaks the migration into smaller sub-migrations, each with its own set of goals, data, deadlines, and scope – effectively an agile approach to data migration.
Trickle migration allows stakeholders to verify the success of each individual phase, giving a continual indication of progress. Should any of the sub-processes fail, it is usually only necessary to re-run the failed process and prevent that failure from recurring on subsequent runs.
Trickle migration may require complex planning at the outset and also places a higher burden on users who need to work with two systems simultaneously while the ongoing overall migration takes place.
Each methodology has its pros and cons. Investing resources and attention in the planning and execution of your migration is what will ultimately determine its success.
Advantages of a big bang data migration
Disadvantages of a big bang data migration
Advantages of a trickle data migration
Disadvantages of a trickle data migration
Deciding whether to choose a big bang or trickle migration should be made as early as possible. Ask yourself the following questions:
There is no simple rule for choosing one over the other. However, big bang migrations are typically selected where the scope is well defined from the outset and where deadlines or other project properties mandate it.
Conversely, trickle migrations are beneficial when the migration can be easily split into several stages. They are also suitable when the scope is hard to define. In such cases, the trickle migration's phased migration approach allows you to migrate “easier” data first while dealing with more complex processes later.
Another major consideration is the experience of your team. Each migration can pose different technical and project management challenges. Teams that prefer an agile approach typically prefer trickle migrations, and teams that are more used to a waterfall methodology prefer the big bang approach.
The importance of good planning for your data migration can’t be overestimated. Poor planning can jeopardize business initiatives in one fell swoop and lead to a catastrophic loss of trust and a hefty bill to pay.
However, data migration also offers businesses the chance to dive deeply into and rethink their data. Investing time and effort into the planning process leads to a successful migration project and long-term rewards in the form of more efficient systems and greater business value.
Clearly define your goals. What would a successful migration look like? Remember that you don’t always have to move everything. In general, your main planning rule should be to find the smallest possible subset of the source data (smallest in terms of complexity rather than actual number of records) that will get you to your goal.
What will the process require from you? Considerations include (but are certainly not limited to):
Finding the Right People
Business users are the ones who understand the data. Make sure they’re involved in the process and invested in the chosen method. This is also a chance to seek out any potential unforeseen pitfalls ahead.
Contracting External Experts
If you want external help, getting an expert in both systems is ideal, but ensure they at least understand your target system. (After all, you probably already know a lot about your own source system). Take time evaluating potential partners, and make sure they combine technical expertise with an approach and manner that fits yours. Will they go the extra mile for you?
Finding the Right Way
Build a scalable, repeatable process from the outset. Believe us, it will save you time, effort and frustration later.Everything you should consider when planning a data migration: Read the Data Migration for Humans ebook
Data migration involves several stages:
Each stage has its considerations and needs to be considered to deliver a successful project.Read more details in the post: 13 Stages of a Successful Data Migration
Typically, data migration occurs when upgrading existing hardware, transferring to an entirely new system, or in instances such as application replacements, business process changes, data volume growth, or to unlock better performance.
Some examples of when a data migration might be needed:
The need to unify completely separate worlds when companies combine. IT stacks and data usually need to be joined to provide one unified system for the new entity.
When a current system struggles to keep up with performance requirements. For example, two legacy systems create performance problems, duplicate data, and compatibility. One company saw operating costs reduced by over 40 percent when they moved to a new ERP system. 27 million records needed to be migrated, but the project delivered substantial efficiency gains.
A potentially complex process, as it usually means migrating multiple applications. Even with cloud providers’ migration tools, custom development is often required.
Migrating data like this can be more complicated than it first appears because the move is rarely a 1:1 copy. The migration often involves data cleansing exercises or more complex changes if the target database behaves differently from the source (e.g. if the target is a columnar database while the source is a traditional relational database, the data must be transformed before it is loaded).
Learn more about moving to the cloud here: Migrating Data Workloads to the Cloud
A workday data migration implementation consultancy can now onboard significantly more clients, without the need for extra workforce.
Automating the mapping needed to move data from the clients’ legacy systems into Workday means that the consultants now only need to spend a few days configuring the framework rather than weeks (or even months) in Excel like they had to do before.
We’ve helped many businesses with their data migration projects, whether it’s helping a client move legacy data into a new system or creating a powerful and bespoke automation framework to process huge numbers of records against a strict deadline.
First things first, we do what we do with every project: sit down with you and find out what you need. It goes without saying that you want your data migration to proceed as quickly and reliably as possible. No two clients are the same, however, and we tailor the project to suit your needs.
Whatever your answers to the above – and any other questions you have – we then create a process that’s intuitive, effective, and, above all, bespoke to your business.
Our platform helps you plan and implement your ideal data migration process. Using a visual interface with a selection of ready-made components, it’s easy to build fast and flexible data processes.
Any workflows you build are repeatable, which means you save time by not needing to rework the entire process any time your data or requirements change.
At all times, your data stays in your control, with complete visibility, transparent documentation, and version control capabilities.
In short, data migration can be done however suits your business. And when it needs doing again… you’ll already have your ideal process in place.
Our demos are the best way to see how CloverDX works up close.
Your time is valuable, and we are serious about not wasting a moment of it. Here are three promises we make to everyone who signs up: