Making data – trustworthy, accurate data – available to the people that need it can be a frustrating process. And it often falls to business systems analysts to be the bridge between business users (maybe in the Finance or HR teams for example) who need the data, and the IT or technical teams that need to supply the data.
In this post we'll show you how you can overcome some of the typical challenges you can face when trying to make reliable data available, and instead take control over data (even if you don't own it) and get it to users in a clean, usable format.
If you’re a business systems analyst (BSA), you probably want to make the process of getting data easier, but the realities of life can make this difficult. You likely have to deal with:
One of the unfortunate things about being in a liaison role between multiple teams is that you’re locked between forces that make your life a little difficult.
You need to provide access to, and governance over, data that you don’t own.
And you know the importance of getting that data quickly. Which means you don’t have time to wait for multiple other teams across the organization to implement some ‘perfect’ solution (which you know will likely never actually get across the finish line).
The solution? You need to work around these obstacles and create an ‘invisible’ solution, which also usually means something relatively low on budget and easy (and quick) to implement.
The goal of data democratization is to make data from across the business accessible, so people can be self-sufficient, without needing an excessive amount of manual work and without the downtimes that can come from dependencies on other teams.
Getting access to the data you own, or that’s in your own ‘silo’ is usually easy. But democratization means people getting access to data outside their silo. And that’s where things get tricky.
The problem with this option is that it’s too big of a project. It needs to be perfect to work as intended across an entire organization, and the realities of life mean it often isn’t perfect. There are a lot of dependencies on others, competing priorities and changing requirements.
The worst thing that happens with these kind of projects is that a big project gets started and abandoned halfway through, and years later another project gets started. So rather than one all-encompassing architecture you end up with lots of little pieces that serve different parts of the organization but are disconnected from each other.
So while it might be a slight improvement you’ve essentially ended up with just bigger silos.
Another option is that you give people access to the raw data, whether it’s the applications themselves or a data lake or lake house where you consolidate all the data and leave it to users to make sense of and be efficient with it.
The challenges with this option are firstly, and obviously, governance and security issues. But there’s also challenges around skillsets. New applications or storage such as data warehouses require skillets that business users might not have, and it can take a lot of time and effort to get usable data from that.
The final option is that you decide you’re not going to try and change too much of the architecture that’s already there, but you’re going to provide a layer on top to provide access to what’s there without too much hassle.
The potential pitfall with this option is that it can also easily balloon to a big complex project. But with the proper tooling and approach you can simplify it so this becomes the easiest one to get done and start seeing results from.
CloverDX is a data integration platform that can help technical teams (whether business systems analysts or IT teams) work together with users on the business side of the organization.
With CloverDX, it’s easier for business users to work with data on their own – but not in isolation. The data they’re working with is provided by the technical teams, which means it’s reliable, accurate, and pre-cleaned.
The checkmate that we talked about earlier comes down to making data that you don’t have control over available to end users.
You might not have control over the raw data or the source systems (the area in the grey box on the left in the diagram), but with CloverDX you can have control over the green section – transformation capabilities that can make data into something business users can work with.
You can build transformations that turn the raw data into high-quality, validated data that you can put in a Data Catalog, where business users can access, work with and export it via an easy-to-use interface.
The Data Catalog in CloverDX is the central bridge between you and the end users. You can publish data to it, and non-technical users can consume it.
You can make that data ‘trusted’ – you take data from the source and combine it, transform it or otherwise manage it to get it into a usable format. And as the connection is to the source system, the data users get is live whenever it’s requested.
You also have complete access control over what each user sees.
From the business user point of view, it’s a central place where they can find all the data from across the organization that they have permission to see. And they avoid data prep hell – it eliminates the pain of getting to the data in the first place, and having to spend hours (or more) making it into something usable.
Search results in the CloverDX Data Catalog
CloverDX is one platform with different interfaces for different users, meaning you or other technical users get access to all the power and configurability you need, and the complexity is hidden from the non-technical end users, who just get the data they need.
The Designer is where you build connectors and transformations. It allows you to connect with any source, and you can build even complex transformations for your data.
Server is where all the automation and deployment happens – from here you can deploy the connectors you’ve built to the Data Catalog, plus control which groups of users can see the data in their Data Catalog view.
CloverDX Wrangler is the business user interface where non-technical users can find and work with data that you’ve published to the Catalog.
With Wrangler, a self-service data transformation platform, business users don’t need to code, or understand the rest of the CloverDX platform, they can just work with ready-prepared data in a familiar interface.
Users can record each transformation step they do, so a whole series of steps can be repeated at the click of a button. At each point users can see what the data looks like, and see a validation summary that highlights any errors, such as an incorrect format.
How does CloverDX can help business users save time and improve data quality?
As well as creating data sources in CloverDX, you can also create data targets – so you can write data to anywhere. And like with data sources, you can add additional behavior to the target, so you’re writing not just raw data, but you can for example validate it, add to or enrich it.
A data target can define the format the target requires, so users can easily map data to that format with Wrangler’s visual, drag and drop interface.
Data mapping in CloverDX Wrangler
It’ll also give you immediate feedback if something is wrong (like you’re trying to map a number format to a field that requires text).
And once the end user (often the domain expert) has provided that mapping, they can export it from Wrangler, and your organization’s technical teams can integrate that mapping into a bigger process, within CloverDX Designer.
What’s happening in this example is that one part of a larger data workflow is using the mapping that the business user has created in Wrangler, and it's this mapping that's driving the overall processing of the files. The business user can contribute their expertise, though a simple, familiar interface, and not have to worry about the more technical parts of the process.
The whole end-to-end workflow can then be automated to run on a schedule or on a trigger.
And because everything runs on the same platform, you get complete visibility of all the jobs that have run, making it easy to investigate and troubleshoot any issues.
With CloverDX you get a data platform that enables both business users and technical users to work together, on the parts of the process they’re best at.
The domain experts get to work in an interface that’s simple to use, and what they produce can be integrated by technical users into larger pipelines that take care of the whole end to end process.