CloverDX is a new name for CloverETL Learn more
When it comes to your data migration process, conducting a ‘small’ move to the cloud isn’t always as easy as it sounds. When factoring in cost, maintenance and skills required, you may end up orchestrating an integration that’s far too difficult to sustain.
We faced a similar issue at CloverDX. What appeared to be a simple solution on paper transformed into a fascinating trial and error exercise (with a successful ending!).
Here’s our cloud migration journey, from start to finish, including our observations, predictions and tips.
Our idea was simple: create a basic financial reporting integration using data housed in our HubSpot account. The report tracks various events in HubSpot - such as the creation of an invoice or whether we’ve closed a new deal - and notifies the financial teams through an email.
However, HubSpot couldn’t create the reports we needed. So, we had to use other tools.
Initially, we opted to create the application on-premise.
In the following illustration, you’ll see that CloverDX is the main conductor of this integration. Through HubSpot’s “webhooks”, CloverDX is called directly whenever a new event notification occurs.
From here, CloverDX computes all the data necessary, packages it up and sends it out to the relevant team members.
How did we do it?
Version one’s tactic was to:
Resources required (we’ll talk about why this is important later on…)
Overall, the integration would have been easy to build. Since everything used was pretty standard, we had the skills and hardware required.
However, it turns out maintaining this solution would be time consuming as we would own and manage every single layer of the solution – from hardware, operating system, security to the CloverDX application itself. That’s a lot that can go wrong, especially if you don’t have a large IT team operating 24/7.
For your business, the case might be the same.
The on-premise cost predictions
Some further observations
Using this ‘old school’ architecture would mean:
2/10. It would be pretty painful to manage, to say the least.
This is where the cloud strategy came into play. In cloud, you’re free to pick from “bare metal” to fully managed serverless option like the AWS lambda, APIs, etc.
In this version, the fundamentals of the integration remained the same, only this time we ran it in the cloud rather than our own network. We kept a firewall and enabled an API gateway to filter out any unnecessary requests. HubSpot simply calls CloverDX, which then computes the data needed to produce the reports.
Overall, the skills needed in version two stayed largely the same as version one. The only (small) difference being that Amazon would take care of the database security and maintenance, and potentially would help us with the VM as well.
The cloud cost predictions
There were some slight differences to the cost outlook.
3/10. Version two was better, but only by a small percentage.
Finally, we come to our third version that turned out to be final: serverless.
That’s when we had to radically rethink the architecture. From a simple “naïve” solution in versions one & two we had to fully embrace the building blocks of the native cloud serverless world. For your typical sysadmin that meant learning completely new things.
Although we still kept the firewall and API gateway for security purposes, we completely changed the way HubSpot consumed and sent notifications.
Instead of HubSpot directly calling the CloverDX API, we used the AWS Lambda function to validate the HubSpot API call. You don’t want to allow just anyone to use your API endpoint and you also don’t want to spend too much processing time (read, money) on doing so. The HubSpot API call would then be submitted to a waiting queue (SQS). From here, CloverDX actively pulls data from the queue on a minute schedule and then sends out the necessary reports.
The use of an intermediate SQS queue allowed us to react to differing spikes in data volume, as well as prevent any lost messages, should CloverDX become unavailable.
To make the integration easier to manage, we installed the CloverDX into an AWS Fargate instance. As you’ll see from the diagram, everything is run as a serverless service.
For the final time, here are the requirements necessary for version three:
The serverless cost predictions
Our version three, in comparison to our previous cloud migration efforts, features an entirely new programming pattern for increased robustness and scalability.
Here are some final thoughts on the integration:
That said, this integration may lock you into an Amazon-only ecosystem. Which could make migrating the application into another cloud provider’s environment pretty difficult.
7/10. Overall, version three provided us with the streamlined and scalable reporting tool we needed. The only downside, of course, being the complexity of setting up the integration and some ongoing support requirements.
We hope you’ve enjoyed reading through our data migration ‘trial and error’ process.
As you can see, we initially prioritised simplicity over practicality. As much as an on-premise (or cloud) integration would be easier to build, it would most likely put extra strain on our IT and support teams. Thankfully, our experts created a solution that satisfied the whole team and fulfilled our financial reporting vision.
Before we let you get on with your day, here are some tips for those of you eager to migrate to the cloud:
If you’re interested in know more about how to move a data-intensive process into the cloud, please don’t hesitate to get in touch.