The blueprint for scalable and efficient data ingestion

Jump to the section you need

Timestamps

• 2:46 – Defining data ingestion and onboarding
• 4:30 – The goals of a good ingestion solution
• 14:53 – Who CloverDX is
• 17:11 – The CloverDX blueprint
• 22:38 – Customer impact and case studies
• 26:51 – Walking through the blueprint
• 32:38 – Live Q&A: Monitoring and Coding Requirements
• 36:52 – Live Demo: From Trigger to Target
• 59:59 – Live Q&A: Scaling Data Volume
• 1:02:19 - Closing

   

Transcript

Introduction [1:06]

Kate Pidgeon: Hi everyone, thanks for joining us today. I’m Kate from CloverDX, and I’m here to help out on this session. We’re joined by CloverDX VP Marketing Pavel Najvar and VP Product Branislav Repček, who are going to walk us through a practical blueprint and some best practices for scalable data ingestion.

As always, if you’ve got any questions, just drop them into the Q&A box on your screen and we’ll answer them for you at the end. We’ll be sending out the recording and the slides, so those will be in your inbox by tomorrow. So with that, let’s get started — over to you, Pavel.

What data ingestion and onboarding really mean [2:46]

Pavel Najvar: Thank you, Kate. Good morning, good afternoon, good night — depending on where you’re joining us from. Today, Branislav and I wanted to share some experiences and ideas on how to deal with data ingestion. We’ve framed it as a blueprint — an amalgamation of the experiences we’ve had helping our customers solve this problem successfully.

I’ll talk for a bit and go through some slides at the beginning, then Branislav will take over and show you something much more tangible. So it’s half slides, half practical. We’re aiming for an informal session, so if you have any questions, feel free to drop them into the chat and we’ll either answer as we go or in the Q&A at the end.

Before we start, let’s agree on what we mean by data ingestion and onboarding. It’s a problem a lot of organisations face: getting outside data and new data sources into their organisation. Quite often, if you’re providing a service to your customers or working with partners, you’re consolidating data from multiple different sources. Typically it’s about homogenising or standardising that outside data into a master format that you define. So the problem is: how do we match the outside data into that format?

And it’s usually a repeated process — which sounds easy, but in practice “repeated” doesn’t mean you get the same data every time. Even from the same customer, you see variations: someone randomly adds or removes something, or cleverly improves things and forgets to tell you. So part of the challenge is really dealing with the variances in the data as time passes.

Three goals: accept data as-is, simplicity, resilience [4:30]

When we’re faced with a problem like this, let’s think about what we’re trying to achieve — and I have a few case studies that go along with these, distilled from what we’ve been doing with our customers.

The first objective is to be able to accept data as it is. That’s a competitive advantage: when you can tell a customer or partner, “we can accept your data as it is, you don’t have to do anything — we’re happy to take on the burden of adapting it to what we need,” that’s a huge boost in credibility. It’s also a technical simplification for them; they might not be able to reformat it, or they might, but with issues along the way that you don’t control. It’s a bit more effort on your end, but it gives you much more control over data quality — they’re good at producing their data, and you handle all the transformation.

I really like the soft side of this. We have a customer who quickly realised it wasn’t just about solving the technical problem — it gave them a big perception boost against their competitors. Being able to say “yes, we can do it, it’s not a problem, you don’t have to do anything,” and then actually delivering quickly during a POC, was a big win for them. So accepting data as-is is really important, including for the client–vendor relationship.

The second goal is simplicity — not necessarily that the overall implementation is simple (sometimes hard problems require hard solutions), but simplicity in who can do it and how. Organisations often rely on engineers to solve the problem, throwing every new client and data source at them, and that becomes a bottleneck. If the solution is designed so onboarding new data is simple, suddenly new teams and new people can take part, and the burden spreads across bigger teams — customer success teams, dedicated onboarding teams — who aren’t the technical resources normally used.

One example is FIA Tech, an important player in the fintech space. They serve more than 8,000 clients, consolidating data from the market. Their product, the Trade Data Network, is the hub that centralises transaction data from all sorts of sources — they solve the ingestion problem on behalf of their customers. It was a real success in enabling their connectivity teams to do that instead of the engineering team. They have a well-balanced interplay: the engineering team builds the building blocks, and the connectivity team — essentially customer success — uses them to onboard new sources very quickly.

The third goal is resilience: the ability to handle things as they change over time — and they will change. With hard-coded or purely manual solutions, any change takes time and effort. We’ve worked for quite some time with Availity [verify — formerly Dynamic Health], who work with the HL7 healthcare data standard. The keyword is “standard” — it is a standard, but a lot of providers introduce small variations: different date formats, or a date placed somewhere else in the file. So there are small, source-by-source changes. Using CloverDX, they set up a process with a set of rules that can be expanded and extended as new variations appear, so they can easily adapt.

The ultimate goals: automation and reusability [10:19]

All of this leads to two ultimate goals. The first is automation — moving the needle between what’s manual and what’s automated, and shifting as much as possible into the automated bucket. A fully automated solution isn’t always achievable, but anything that can be automated usually should be.

One of our clients in logistics, Sisk Fulfillment Services, is a great example. They’ve been around 40 years or more, processing four to four-and-a-half million packages per year — a really big load. To expedite shipping, they’d often run night shifts, with people waking at 3am to process the previous day’s data. When they introduced automation, that went away — people could wake at normal times, no more night shifts. It’s an interesting impact of automation on quality of life. And along the way, they increased the quality of the data too.

The second ultimate goal — the Holy Grail — is reusability. To scale, to improve the quality of ingested data, and to avoid “we have another source, so we have to do a lot of work and call John, who’ll be upset he has to do it again,” you need reusable pieces you can use multiple times. We’re trying to break the equation of “new client / new data source = new pipeline” — to turn it into an inequality, so a new source doesn’t mean building a new pipeline.

Again at FIA Tech, there’s great collaboration: the engineering team is called in when something genuinely new needs to change, and they do it once. The customer success or connectivity team, who are in touch with clients and do the actual onboarding, reuse those building blocks again and again, configuring and tweaking as needed — without calling engineering every single time. That’s a big achievement.

The reality: scattered systems and manual work [13:47]

So we’ve talked about the ultimate goals — reusability, automation, collaboration. But the reality we see when we come in is very different. It’s scattered systems, everyone with different requirements. We see “temporary-permanent” solutions, where someone writes a bunch of scripts to solve an immediate problem, and those scripts — meant to do one thing — are still there years later and become difficult to manage. Or customers still doing things purely manually, even copying and pasting cells from one Excel spreadsheet to another.

So there’s still a long way to go, and we’ve been helping customers along the way. That’s the preamble — stating the problem and some of the goals.

Who CloverDX is [14:53]

We have what we call the CloverDX blueprint for ingestion. But before we get there — for those who don’t know who CloverDX is: CloverDX, where Kate and I work, is a data integration platform. We’ve been around since 2007. What we bring to the market is a platform that’s ideal for solving data ingestion and other use cases.

The platform has three core pillars. First is the ability to build data pipelines, led by the engineering teams — your IT team, or a tech-savvy member of a specific team. We believe in collaboration between business users and engineering teams, so it’s aimed at developers and IT. On top of that, we bolt on a suite of tools that non-technical users can use to participate in those pipelines — collaboration is a big thing for us. Lately we’ve been adding AI into that process, both to enable use cases that weren’t possible before (like using AI to massage data in novel ways) and for productivity, to make our users more productive in the platform.

Branislav will show you some examples, so you’ll see the CloverDX platform in action. But all these principles apply to other tools as well — we’re describing a certain way of doing things, and showing what it looks like in CloverDX.

The blueprint: three principles [17:11]

Let’s go back to the blueprint. There are three things that form it — three things we believe you need in place to achieve the goals we set out.

First is modular architecture. How do we break the equation “new dataset = build a new pipeline”? With a modular architecture, a new input just means setting up a new reader — and even that isn’t always necessary, especially for data that shares the same or a similar format. The rest of the solution is reusable.

The second part is automation. There’s a rule I like — the rule of four: if you’re doing something more than four times, stop, and start thinking about how to automate it and how much you’d save. If several people are doing the same thing more than four times, it can add up to days wasted within a month. There’s also a challenge around abstraction — how you get to that modular place, and how you think about what’s repeated versus what needs to be done manually. It’s not always clear-cut, so the way we see it: if even half of it can be automated, it’s worth it.

The third part is division of roles and collaboration. We often see two roles — not always clear-cut, with some overlap. One is the engineering group; the other is the business-user group, typically the people dealing with customers and the actual onboarding of new sources. The engineers are responsible for building the framework and the orchestration, occasionally building connectors when necessary, and the actual operations and performance. Performance is often a key driver for organisations to move on from what they have — they may have built scripts ten years ago that worked fine, but now they need to onboard new clients to grow, and it takes too much time or hand-holding, so they need better performance.

The business group should be responsible for understanding the data — doing the data mapping, understanding the source data from the client or partner, and mapping it onto the target, homogenised structure the organisation needs. That might involve business logic, calculations, and further manipulation. They should also own data quality and stewardship: making sure the process is set up correctly and that everyday data quality is maintained, so when data needs attention, they can work through it quickly.

Mapping CloverDX tools to the blueprint [21:24]

The CloverDX platform is well set up for this, partly because we have tools for both groups, and they can work together tightly on a single pipeline. The engineering group has CloverDX Designer and Server — tools for developers to build orchestrations and data transformations.

For the business-user group, we have two tools that play different roles along the pipeline. One is CloverDX Wrangler, for data mapping, business logic, and self-service data manipulation. The other is Data Manager, a data stewardship and data quality tool that lets business users monitor and jump into the data — when the daily load or anything else needs attention, Data Manager is the right tool.

Customer impact and case studies [22:38]

Before the demo, I want to go through some of these cases. We’re seeing real impact from this blueprint, and I wanted to share it because it’s not just something we came up with — we’ve been using and refining this approach for years, and we’re happy to see customers who adopt it seeing real impact.

Sysco [verify], processing millions of packages a year, isn’t only seeing the shift away from waking up early — they’re also experiencing shorter processing times. When you think about how many packages they process a day, going from hours to minutes is a huge win.

FIA Tech, with the Trade Data Network, work with something like 110 different sources and manage to process everything in real time. Their previous tool was very business-user-friendly and used by the customer success team, but they realised they needed something more robust that followed better rules — more reliable, higher data quality. They started working with CloverDX and their engineering team, and now have a robust pipeline that lets them onboard new datasets quickly.

Another case study I like is Zywave, a SaaS company helping insurance companies process data. Before CloverDX, they were perceived as a bottleneck for onboarding new customers: the customer success team would work out what was needed with a new client, then hand it to engineering, who faced days or weeks of work to onboard them. We created an automated, modular framework that accelerated onboarding — they went from weeks to a couple of days. The impact was that they’re no longer seen as a bottleneck; in fact, they’re now seen as too fast, and the customer success teams have become the bottleneck — which, for the engineering team, is good news.

Ortec Finance, a Dutch company, prepare quarterly reports. There’s a number behind it — from five days to half a day to prepare a report — but there’s also an emotional side. They told us it went from a very stressful period every quarter to a rather laid-back, easy process. Eliminating stress is a big part of this too.

Walking through the blueprint [26:51]

Let’s talk about the blueprint itself — this is what you came to see — and then I’ll hand over to Branislav to show what it looks like in CloverDX.

We start with some incoming data. It could be a file — an Excel spreadsheet, a file appearing on SFTP or S3 — a message queue, or APIs. Essentially any data that needs to be onboarded. Usually it starts as a one-off file, but as the process goes on it becomes automated or semi-automated.

The first thing in the blueprint is the mode of operation — I call it the pipeline trigger. It can be semi-automated, with a human in the loop at the beginning: we see a lot of clients take this route, using something to trigger the processing — for example, a file they received by email that they don’t want to fully automate. We have tools (Branislav will show them) that let you upload data to the platform to be processed, or go somewhere and press a button. Or, better where possible, fully automated processes based on a schedule or a trigger that watches an SFTP folder and runs the pipeline when a file arrives.

Next, you need a connector that can read the data. CloverDX comes with a bunch of out-of-the-box connectors, or you can build your own components — they can be smart, doing early validations, and so on. This is where the engineers come in, if necessary — though often it’s not, since we can work with Excel files, databases, and standard formats out of the box.

Once you can get into the data, an important phase we call triage looks at the data and decides: is this something we can work with, or does it need someone’s attention? If the format’s fine and it’s a known piece of data, we go through the process — some processing, reformatting, calculations — and if that goes well, we write it to the target (a database, an API) and do any post-processing like logging or archiving the input files. All of that can be automated in CloverDX. That’s the happy path — the easy part.

So what happens if triage doesn’t pass? If triage sees a format we should be receiving but the file looks different — the format’s drifting, there are slight differences, or it’s a completely new format — we can hand it over to a user, typically someone who understands the data, who uses CloverDX Wrangler to provide the mapping and business logic for that new format. It’s easy, without engineers in the loop: they’re notified, the business user provides the logic and mapping, and once done, the data can flow through the happy path. So Wrangler lets business users provide the “secret sauce” to process unknown files.

Another part of the blueprint: once the process is running and we know the structure, there can still be problems. If processing finds issues, the data can stay in the platform, and Data Manager surfaces the records that need attention to users, who can work through them — and collaborate, if there are large volumes needing attention, spreading the work across a bigger team. Once their changes are approved, the data re-enters the process and is processed.

So that’s the full blueprint. Not everyone adopts all the tools or does all the parts, but this is the bigger picture of what can be done — and what’s often necessary to provide a high level of automation, high data quality, and high throughput, including cases where you need to step in and define a new mapping for an unknown or changing file.

We have time for the demo, but I see a bunch of questions, so maybe it makes sense to answer those first.

Live Q&A: monitoring and coding requirements [32:38]

Pavel Najvar: Kate, can we go through the questions together?

Kate Pidgeon: Yeah. I’ll read them out so everyone can see what you’re answering. The first question was: what’s the best way to monitor and troubleshoot automated ingestion pipelines?

Pavel Najvar: Branislav, this is one where you can show some of the monitoring facilities and the process operations dashboard.

Branislav Repček: Yeah — apologies, I was on mute. I wanted to start sharing my screen while we answer. There are all kinds of monitoring, and it’s a huge part of running any automatic process, whether it’s ingestion or anything else. What you should be seeing on my screen is CloverDX Server — the part of the platform where you see the monitoring side. We have monitors representing the different automated processes deployed on my instance — one processes customer data, one processes real estate, and so on.

For each process, you monitor things like APIs, scheduled jobs, or jobs triggered by other automation features. If something fails, you’ll see it — the square turns red or yellow depending on severity — and you can dig deeper, all the way to the logs. I don’t have any failures right now, but I can find something in history — you can see other things I tried to run before — and go quite deep to see what the problem was. In this case I was setting up the example and didn’t set it up correctly the first time. You can go all the way to the log files, which the technical team can understand — they’ll figure out whether it was just a connection issue that will fix itself once the connection is back, or whether they need to make changes to the process.

Kate Pidgeon: Hopefully that answers the question. We have two more — let’s go through one now and leave the last one to the end, and I guess we’ll be getting more questions. Someone’s asking: if I were to adopt a CloverDX blueprint, how much coding knowledge would my non-engineering or non-technical team need?

Pavel Najvar: Good question. We said CloverDX is an engineering-led platform, and there’s a lot of good stuff for engineers in it. But the tools meant and designed for business users — which Branislav will show — are very easy to use. And especially now: next week we’re releasing Clover AI Assistant, a productivity tool in the business-tools suite that makes them even easier, because you can essentially talk to Clover and it helps you do what you want. So we’re aiming at zero coding requirements on the business-user side. We can also support business users who are “dangerous” with a bit of coding or SQL — it’s a spectrum, not one or the other.

Anyway, Branislav, I think we can get to the demo. We had a bit of a detour, but let’s go back to it.

Live demo: from trigger to target [36:52]

Branislav Repček: I’ll show those parts of the platform too. Let’s follow the diagram Pavel showed a few slides back, going from the left to follow the process. We’ll start with the parts that help you at the beginning — automating the process — then look at how triage works, how Wrangler works, and so on, flowing through the diagram so you can see all the stages.

We talked about monitoring, which is often the first thing I show if everything’s running smoothly — it’s nice to look at. But to monitor something, you need something to monitor. In CloverDX there are all kinds of ways to automate your processes. Usually they run on a schedule — very common for ingestion — and I have a number of them loading data from various sources: files, APIs, and so on. An important part of a platform like this is being able to work with your automation simply. We have an interface for it, but also APIs so you can control it — some customers connect CloverDX Server to other tools for monitoring or control, so it fits into larger environments.

You can automate in different ways: scheduling, as I have here; APIs, of which I have a number supporting my use cases; or listeners that trigger when an event happens, for example when files arrive somewhere. If you do streaming processing with Kafka, you can create triggers for that too. The interesting part is the flexibility: you can pick any of these automation options around the same job. You might have a job that reads a file scheduled every hour — I have a bunch like that — then later decide the customer sending the data is too unpredictable for hourly runs, so you configure it as a listener that watches a location in S3, SFTP, or a local file system. You don’t need to change the job at all. When your engineers build the process — a job flow that orchestrates it — they don’t need to change it just because the configuration changes from time-triggered to event-triggered.

That’s the first part: fully automated triggers. But you can also have triggers that aren’t automated, where you want users to have some control over what gets triggered. This is one of the first interfaces usually presented to business users or domain experts, so they can run ad hoc processes. It can be manual loading of a file — they click, drop a file in, and something happens — or a load that doesn’t require a file at all and pulls data from an API or similar. This kind of process trigger lets users control their processes ad hoc. Often it’s a combination of both — manual triggers, through what we call data apps, and automated ones. I have a trigger for the customer load here, and another one over here; in the end, the same job runs, and I don’t have to modify the code to make it work.

The best part: someone has to build these applications, but the engineering team doesn’t have to design interfaces like this — CloverDX builds it for you. You just figure out what parameters you need. For something where you provide a file, you tell Clover you need a file, and it builds the interface automatically, where you can drag the file or browse your computer’s file system. So you get hassle-free creation of semi-automated or fully manual processes, even with a nice web interface people can trigger manually.

After that, when you run something, you get what we usually call triage — some customers call it connector matching, depending on what you do. This is where it gets a bit more technical, because business users usually aren’t involved here — they provide the requirements, but the IT team builds it in CloverDX Designer, the second part of the platform. Designer is meant for data engineers; they install it on their machines, and it’s a full development environment with tons of options for everything. You don’t want people who don’t want to code accessing this — but even so, it’s easy to follow the processes if you’re a little careful with planning and how you lay out the job flow. You can see the boxes and arrows — let me zoom in so you can follow the process.

This process figures out whether a file format is known — this one processes Excel files. If it knows the format, the file is processed automatically; if not, it reports on it, and you can look at it in the data app in the web application. This is a very common approach, though it works differently for every customer — some are much simpler than others. In principle they’re the same: grab data from the input, figure out which job to use to process it, and run it.

If you look inside, your data is loaded — what happens depends very much on what you’re doing. It can be as simple as reading data from somewhere, doing mappings (I’ll come back to this), and writing it — this one goes to Snowflake, for example. All kinds of things are possible. This is always done by the engineering team and hidden from business users; they don’t need to know anything about it, just that it works.

Now, the more complex job scans incoming files and tries to figure out which ones it knows. When you run it, it found two files it doesn’t know. Let me pick the first one. It created the mapping process for this — and this is where you involve your business users. We’ll go to Wrangler, one of the tools our users have. Let me look at the jobs; it created one for me. Let’s wait for the data to load. This is real data — not an example — so in production this would be an actual file. You can see a bunch of empty rows I have to get rid of, and the header wasn’t parsed properly — the first row is actually the header. So let’s fix that: I go in and edit it, and this doesn’t require any coding.

I fixed the header and it re-parsed the file — the empty rows are gone, I have proper names, and I can continue. I can add steps to make the transformation and figure out the mapping. This is the data format my IT team wants from me — they asked me to map 26 columns. Some can be mapped automatically; some I’ll just drop (I’m dropping a few randomly here because I don’t know them). You can create mappings, or even more complex processes, without any coding — that’s what we mean by a set of tools that let you create your jobs without learning to code.

Once done, you can do a test run. This one’s fast because it processes a small number of records. Now that I’m happy with it, I can tell my IT team I’m done — though in this case the whole thing is created by a framework. I can run this one; I have it as a manual trigger, but it’s also scheduled, so it runs on its own. When it runs, it processes the files it knows. This creates a smooth process for business users to follow, from when the source data arrives to where they can process it.

So we went from triage — the technical part — and followed the red arrow: when there was something Wrangler didn’t like, I was able to fix the transformation. Once I’m ready, the process will recognise the file next time and run it directly.

Now, another interesting part: data quality and processing often go hand in hand — in many cases they’re combined into one job, which is more complete than splitting them across two steps. This is typically something the data engineering team is involved in, and they can do anything, because in Designer they have about 200 components at their disposal — from simple transformations to all kinds of validations. You can even create your own libraries and share them. It’s easy to build more complex processes in Designer because you can work within a team, with version control.

Let me find one of the validation jobs — an interesting one. One thing to note: in Designer you can connect directly to the environment where your jobs are running, so you don’t have to work locally — your data these days is usually somewhere else, so you’re not uploading it to your machine or downloading files. I’m connecting to a server I’m running in AWS, but it could be running anywhere — on-premise or elsewhere in the cloud. When I do something here, it happens in the cloud — if I run the job, it runs in the CloverDX instance over there, and I don’t have to think about it. That’s a big advantage, because you can process large data volumes without needing everything on your machine or a large development machine.

Back to validation. Here I’m reading data from an Excel file — a common theme, especially with low-quality data, because anyone can type anything anywhere in Excel. I have multiple processes that validate the data. The first validates common properties — for example, whether required columns are empty. That’s a very simple validation. But you can also do something more complex: instead of just validating and saying “this data is incorrect, figure it out,” we try to help the domain expert by offering suggestions for how to fix the data — which you can do with Data Manager later. All the output from validation goes to Data Manager if any issues are found. This part isn’t very complicated — it does fuzzy matching to figure out the best match from a known set of values, in case a value in the data is incorrect.

When a job like this runs, the output is written to Data Manager, one of our other tools aimed at business users. Here, no coding is necessary for anything, because you’re just looking at your data. Let’s look at the customer dataset — names, addresses, phone numbers, emails. You can see the data isn’t very nice: lots of red triangles, which are errors. The validation — which you build in Designer — flags things like a missing last name. As a business-user tool, Data Manager lets me go in and fix it: here the first and last name are combined, and it’s probably supposed to be split like this. I’m editing the data, and when I’m happy I can approve it — my way of saying the record is done and should go to the target system. You can see it appeared over here. I can approve a bunch of them — let’s do just a few so we have some records to process.

This is designed for many users to work on your data at the same time, with different permissions, and you can do a two-level approval process where users edit and someone else reviews and approves. It’s aimed at organisations that, for example, have a large number of students helping onboard data — they hire a couple of hundred temporary workers for a month or two, onboard tons of data, fix it by going through the dataset, and a team lead approves it all. When they’re happy, it’s processed and goes wherever it needs to. In this case, the data goes into the warehouse for reporting, but it can be anything.

That’s the end of the process — and you need to trigger the final step that takes whatever was approved and runs it to completion. So we’re back at the automation features in Server. This is the “process approved customers” job, which triggers once per hour in my case — but I can also run it manually, or have a data app do the same thing. So I give business users control over whether to process some data out of band, or let it trigger on its own.

It runs, and if I refresh, my records will be gone from here — they’ll switch to a status we call “committed,” meaning they were successfully written to the target system. That completes the whole process. Looking at the slide again: I started here, went to triage, did something in Wrangler for files I don’t know or data I’m processing automatically; everything processed automatically went straight ahead; I validated the data; anything rejected went to Data Manager; anything that didn’t trigger validation errors went straight ahead — and now I’m at the target, wherever it is.

This final step is built again in Designer, usually a very simple job, because reading data from Data Manager once it’s been processed is typically simple. And often you want to archive your data — for compliance, or to investigate incidents later — so part of the process copies the data somewhere, archives the staging database, or zips the files and saves them.

So for every part of the process there’s a part of the CloverDX product. The green parts are built by the data engineers and require Designer — typically the difficult parts — but the business users, the domain experts, work with them in interfaces designed for them, without any coding at all. In Wrangler you might need to write a formula — if you need to multiply something by two, you write “2x” — that’s about as complex as it gets. Then the process goes to Data Manager and to the end.

A first look at Clover AI Assistant [56:49]

I have one last thing to show, which I haven’t plugged into the demo because we’re releasing it next week. Pavel mentioned Clover AI Assistant — it helps you work with your data in Wrangler without having to do anything resembling coding. Usually in Wrangler you create something by selecting steps: if I want to sort my data, I click “sort,” click the button, and it adds a step to sort by due date. That’s how you normally create jobs in Wrangler.

But with AI Assistant, you just type what you want. For example, I still have some errors here — I can look at what they are; this is because I have missing reference data, so it doesn’t join. So I type what I want, very simply — remove the rows with errors — and let’s see what the AI does. You can type something a lot more complex, of course. Actually, this whole job was created with AI, except for the sorting I did a minute ago — everything here was created by typing into the Ask Assistant box, getting a response, and either rejecting it (if it didn’t do what I expected, or I needed to modify my prompt) or accepting the solution and moving on.

Let’s give it a little time — it’s looking at what kinds of errors exist and trying to figure out transformation steps to remove them. It seems the internet is acting up a bit, and we’re getting dangerously close to the top of the hour — but I’m pretty much done. There’s one suggested step; I accepted it, it’s now part of my transformation, and the errors are gone — no more red marks anywhere. So this is the next generation of helping business users do something with their data, with even less coding than before.

Wrap-up [59:17]

Pavel Najvar: Thank you, Branislav — that was an interesting demonstration. Let’s get to the end of the session. We tried to show you a blueprint — a way of thinking and structuring things. You don’t necessarily have to use CloverDX to achieve good results, though obviously I like seeing CloverDX used for it, because it’s well suited to this collaborative way of working. We’re here to answer questions. If you have more, type them into the question box, or drop us an email at hello@cloverdx.com — either Branislav or myself is happy to get in touch.

Live Q&A: scaling data volume [59:59]

Kate Pidgeon: We have a question from someone who already has CloverDX: if I had to onboard twice the volume of data by next week, how much hands-on work would I need to do before I could rely on CloverDX’s automations to handle the onboarding?

Pavel Najvar: That’s a tough one — it really depends. It can be quite a lot: we’ve had customers spend months building complicated frameworks — not because it’s difficult, but because they’re building something reusable for hundreds or thousands of different data feeds. Or it can take a couple of weeks to set something up and you’re good to go. Branislav, further experiences?

Branislav Repček: Yes. If you already have the framework built in the style we mentioned — aligned with the blueprint — often there’s not much to do. When we’ve had customers expecting large data volumes — some have big seasonal changes, like a retailer before the holidays seeing five times their summer volume — usually they just make sure their infrastructure is up to speed. Sometimes they provision an extra instance of CloverDX, adding a node to a cluster to handle the load, because normally it’s not required. But if the framework is designed well and they’re not near the SLAs they need for their processes, usually nothing is needed — it just works. You check that the infrastructure can handle it, and that’s it. You don’t need to code anything or rebuild anything. If you build the framework well, it’ll just work on its own.

Closing [1:02:19]

Kate Pidgeon: I think that’s all the questions for now. If anyone has any last-minute ones, pop them in the box. If not, drop us an email — we’re always happy to talk about anything we’ve shown today, or to arrange a one-to-one demo. Just get in touch. So we’ll wrap up there. Thank you very much, Pavel and Branislav, for the presentation, and thank you to everyone who attended. It’s really nice to see you, and hopefully we’ll see you on another one of our webinars soon.

Pavel Najvar: Thank you, everyone.

Branislav Repček: Thanks, everyone — have a nice rest of your day.