Nobody wants to go through a data migration. The sooner it’s done, the better. But a successful migration actually takes more thoughtful prep and planning than we often choose to invest. In this blog, we want to help lay some groundwork for better planning, sharing three common misconceptions about data migrations that hinder a project’s progress.
If any of these come as a surprise—or if you’re starting to plan your own migration and need more detailed insight—we encourage you to read our Data Migration for Humans e-book for a thorough look at choosing the right people, technology, and methods to involve in the undertaking.
A data migration project is part of a more complex, nuanced pursuit for innovation and improvement. Tossing old data into a new system without thinking it through is just not going to work out. The care required to properly migrate data goes far beyond IT just copying it from one location to another.
Imagine the process of relocating. When you plan your move, you always keep your new home in mind. You wouldn’t pack up and take every single thing with you just to unpack it all in the same setup as before. So you need to ask yourself: will the new dining room accommodate your current extendable table? How will your mattress fit through the bedroom door? Do you even plan on keeping these things? Even, dare I say it, that VCR from the basement?
A data migration likewise is an opportunity for the entire business to assess, judge, and discover what they have in their data beforehand. New systems require new data structures and different forms for how data gets stored, captured, and interpreted. They may also require completely new data that will have to be derived from your existing system. Like I said, way more involved than copying data.
No, it’s not fine. Trust me on this one. I mean, it’s okay as it is, in that you use it every day and it serves its purpose. But that’s not the only thing to consider. Let’s say you already know exactly what you want to migrate and what you want to leave out. The next step isn’t to jump straight to mapping (i.e. what data fields (attributes) goes where). First, you need to dig deep to see what’s really, truly in the data—not just what you think is in the data, or what the documentation suggests, or what other people say. Make sure you’re diligent about assessing all of it, not just the fraction that you work with every day.
Typically, you’ll find a lot of history and “bruises” hidden in the data that’s been sitting in the system, most likely for years. Because most of it remains largely untouched or unused daily, you really don’t notice. During a data migration, however, you’re charged with scrutinizing every single thing. And oh, the things you’ll find:
All of this must be dealt with so that you can prepare trustworthy and reliable data for the new system. You need the right data preparation tools to get this right.
Among all the data migration stages, this data quality stage is typically the hardest part of the migration, and it trickles through all phases of the project. Hopefully this comes as no surprise. The job for IT teams is to pull the data out and expose the irregularities to business users. There should also be continuous dialogue between the owner of the data and the migrator, assessing whether to fix issues in the source system or devise solutions that would correctly translate the data in transit.
Data Migration for Humans: What Makes a Data Migration Successful?A migration is a heavily iterative process. No matter how thoroughly you plan, you’ll still deal with new discoveries, omissions, additions, and surprises as you go through it. This iterative nature means you will most certainly have to go back and repeat steps that you’ve already finished. These characteristics usually apply just to creative processes, but a migration actually has an enormously important discovery component as well, given the years of history involved and numerous people’s imprinting changes and decisions into the data over time. Seldom can it be thoroughly analyzed and executed in a single go.
Moving your data to cloud? You need a strategy for getting your data there - on an ongoing basis where necessary. This clip is from our webinar on Strategy For Migrating Data To Cloud
For a successful data migration, taking actions in a repeatable fashion is key. What do I mean by this? Let’s frame repeatability by what it’s not. Say you just want to export all employee records from your HR system as an Excel in order to move them to a newer and better one. Naturally, you hope to simply import it into the new system, maybe define what goes where, and be done, right? In the real world, it’s just not going to happen like that. Instead, you’ll find the number of errors (bad formats, missing values, …) ranging from a few to thousands, depending on the system in question. So what do you do next? Either you go back to the old HR system and fix the problems, or, if that’s not possible, edit the Excel file.
After hours or days of massaging the Excel file, you finally can import it to the new system successfully. But what do you do with the new records have been added to the live system during those few days? Or even worse, what about those records that were modified in one way or another? Will you go in and manually find them and update the Excel? What if the new system’s requirements change? Perhaps a decision was made to add a few more attributes that previously weren’t part of the plan. In that case, you’ll have to export all the data again and then somehow match the newly requested attributes to records in the document that you’ve invested so much time in already. That just won’t cut it.
Repeatability means having the right method and tools that allow you to make adjustments and accommodate new changes quickly. In our example, it would mean having a program with a set of rules and transformations that can be applied repeatedly to the raw export data, instead of manually editing an Excel document. This program would then produce the desired output, without wasting so much time and energy. Automating operations like this not only saves time, but also proves to be vital when changes come and go during the project.
How a company perceives data migrations plays an important role in the outcome of a project in practice, so we wanted to put to rest some misconceptions we think inhibit positive outcomes. In reality, data migrations actually require detailed pre-planning, business and technical expertise and involvement, as well as a comprehensive data quality strategy in order to be successful. Remember, a migration is a heavily iterative process; repeatability is so important if you want to make it through to the other side.
If this blog offered some helpful hints or brought up some new questions, keep the momentum going and read on. Our Data Migration for Humans e-book explores the ins and outs of decision-making, including considerations to help you invest in the right people and technology as well as advice on how to avoid common pitfalls.