Automating complex processes in CloverDX" />
Are your data pipelines hobbled by manual processes?
It's time to look at automation.
Just imagine how your infrastructure (and wider business) would look if you enjoyed the benefits of automation. The perks include better:
- Scalability
- Consistency
- Data quality
- Productivity
- Governance
- Cost-effectiveness
Having been sold on the promise of automation, you're no doubt wondering what it looks like in the real world.
So, join us as we take a closer look at what automating data processes with CloverDX looks like in practice. But first, we'll unpack what you can automate with CloverDX.
What exactly can CloverDX automate?
The CloverDX platform resolves 100 percent of your data challenges and not just the easy 90 percent. What's more, automation is at the heart of the CloverDX philosophy - the goal is to automate all of your data solutions so they run seamlessly on autopilot.
Indeed, using CloverDX, you can automate:
- ETL (extract, transform, load – CloverDX enables you to do ELT as well)
- Error handling
- Data validation and profiling (explore Validator here)
- Orchestration
- Parallel processing
- File manipulation
- Calling scripts and APIs
- Logging and monitoring (here is a deep-dive on logging)
To automate processes with CloverDX, you’ll use CloverDX Designer to design, debug and iterate your data pipelines, and the CloverDX Server as the operations platform on the back end.
As a result, you can use CloverDX as your all-in-one tool to design, automate, operate and publish. This solves complexity and brings all those delectable benefits we listed earlier.
Exploring an automation use case
Having piqued your interest with the possibilities of CloverDX, let’s unpack a use case, and consider how automation could help a payroll company.
In this scenario, the business needs payroll data from several different clients to process their payslips for the month. This entails automating a data quality check on each of the employee data files received. Then, it's clear which is good data and which is bad, so the company can resolve any data quality flaws before it goes to the next step of the process.
To do this, the first step is to flesh out the data pipelines as a job flow in CloverDX designer.
What using CloverDX Designer looks like
With this tool, you can tie together operations into an end-to-end solution that's clearly visualized.
This is the process broken down in more detail:
- Defining the directory we want to look at to detect incoming files.
- Copying the files from the directory to the staging area.
- Entering files into the data quality firewall (the Profiler Probe will run analysis on data here).
- Rejecting data that fails.
- Accepting and processing files that pass the data quality process (and enriching if there is additional data).
Once processes are complete, you can then delete the staging area.
Crucial to the process is logging which payroll data files have been checked, and which have failed or passed. Naturally, this is all automated as well.
Watch the video below to see exactly how this process is built and runs in CloverDX
The CloverDX Designer is used for creating workflows, but to enact really powerful automation you turn to CloverDX Server.
What using CloverDX Server looks like
Using CloverDX Server, you can schedule processes from CloverDX Designer to run at set times (scheduling mode), or create a trigger (event listening) that starts a process.
With scheduling, there is the option for periodic scheduling (for example, something that runs every 60 minutes). We can also do micro-batching processes that run every few seconds for real-time situations.
Here's what scheduling looks like in CloverDX Server:
Or, you can use event listeners to kick off a process when something happens. For example, when a file lands in a particular directory or another workflow completes.
By using event triggers, it's straightforward to solve very complex scenarios by creating a chain reaction of triggers and automated processes.
It's worth noting that with CloverDX, you can also automate things outside of the platform. For example, processes in different cloud environments executing shell commands, or sending emails.
CloverDX Server allows you to monitor all your jobs and automations in one place. Now, you have a full view of all your business processes, and if anything fails, you can see it and take action.
Why walk when you can drive a Tesla?
From boosting productivity to reducing costs, the benefits of automation are tough to ignore. There's just no reason to continue using outdated manual processes when you can rely on a modern enterprise data platform.
We’ve shared a specific use case of automation, so you’ve got a taste of what’s possible with CloverDX. But, knowing that our platform can solve 100 percent of your data challenges, aren’t you curious to see how you could deploy it in your company?
If you’d like to chat with one of our specialist team to find out more, reach out today.
