OMG: The Hidden Costs Of DIY ETL

We get it – there's something fun about rolling up your sleeves and getting a job done yourself.

It can make you feel more in control, more savvy.

But DIY ETL isn’t like painting your new backyard fence. That’s because there are plenty of hidden costs and downsides that you might not be aware of.

We’re not going to say ‘never do DIY ETL’, but it’d be irresponsible of us not to reveal the hidden costs involved when you do ETL yourself.

Let’s take a look at these hidden costs.

Growth becomes difficult when others depend on your DIY ETL project

When creating your own ETL, everything might seem fine at first. But when another person or process relies on your data project, things get tricky.

Why? Because now that it's part of your wider organization, you'll have to manage it differently. While you may have been able to stop the ETL for a day or two when only you were using it, your new dependencies will likely want it to run non-stop. In addition, you won't be able to deploy a change without first educating the other users on it.

And to support your data project, you'll need to manage the other aspects that come with it. This will involve supporting collaboration, versioning, documentation, scalability, stability, monitoring, and error handling.

When you find yourself implementing all of these supportive functions, you'll fast realize your DIY ETL has become way more complicated and time-consuming to support than you'd initially expected. You'll also realize that you could have bought all of these functions in an out-of-the-box solution like CloverDX.

Automation can be more difficult than it first seems

It makes sense to modernize your systems and automate your ETL processes so you can make use of the boost in productivity it brings.

Then rather than taking hours or days to process transactions, it can instead be done in minutes. As a result, you’ll accelerate many of your operations and bring more value to your customers.

But building automation yourself and creating your own ETL can lead to problems.

While automating data to go from point A to point B is straightforward, everything that comes with it is more difficult. Specifically, you'll need to automate moving data in a way that you can monitor, troubleshoot, and easily modify and adapt - doing this takes a lot of extra development effort which you might not be ready to invest in.

Reduced reliability and flexibility (very costly in the long-term)

When you go down the route of DIY ETL, things can progress quickly. Your teams can ‘scratch their own itch’ by building the solutions they want.

However, this excitement is often short-lived.

Why? Because when organizations build their own ETL processes, they often overlook two of the most important aspects: flexibility and reliability.

The truth is, your business and its data needs will change over time. So when you go down the DIY route, you’ll need to spend money on expensive developers (either in-house or outsourced) to continually update and evolve your ETL.

What’s more, you’ll continuously have to manage and solve errors in your ETL; when you DIY it, your teams won’t necessarily build a system that’s reliable. They may lack the experience and won’t build something that can last and can be easily adapted as needs change.

It's better to ditch the DIY and rely on a solution built by experienced experts who have been there and done it before. Then you’ll have an ETL that’s been designed from the ground up to be both reliable and flexible (and much less expensive over the long-term).

You don’t know what you don’t know (yet more hidden costs)

There are inevitably other things that crop up when you take the DIY route. These will only add to your issues and costs. These include:

• Over reliance on one person. DIY ETL is often led by one talented in-house IT specialist. Unfortunately, this ties you to them and their support. If they leave your business, things can get awfully expensive to understand, and adapting or updating your ETL processes can be difficult and time-consuming.
• You’ll get ‘bogged down’ with things. Yes, there can be ‘quick-wins’ with a DIY solution. But often there’s poor scoping due to lack of experience, meaning problems will quickly arise. As a result, timelines drag on and things never get finished.
• A ‘spaghetti-like’ infrastructure. A DIY attempt also gets - well - messy. This is particularly the case when one or more developers build a patchwork solution without looking at the ‘big picture’. This chaotic infrastructure can be hard to get visibility into and very costly to work with.

If we're being honest, these challenges are difficult to navigate and can creep up on even the most talented IT team. Fortunately, there’s a better way to go.

Why take on all these costs and risks if you don’t have to?

As you’ve seen, DIY ETL can feel appealing, but the reality can be hugely challenging.

From issues with reliability to a system that doesn’t embrace the full benefits of automation, many DIY ETL attempts are significantly more costly (especially in the long-run).

So, for those looking to cut costs, save time on maintenance, AND build a solution that delivers a solid foundation for the future, it’s worth exploring a tool like CloverDX.

If you're wondering whether you should ditch your homegrown ETL, take a look at our webinar where we discuss how to tell when it's time to move on to a more mature platform, and the benefits and challenges it brings: It's time to ditch your homegrown ETL. It doesn't have to hurt.