The transformation below is at the advanced end of the scale. However, even this could be just a smaller part of an even more complex process.
Higher levels of overall complexity can arise in various guises. For example, the above transformation could be integrated within a Jobflow as shown below, where one of the Jobflow components launches it.
Transformations can also be designated as shared, reusable “sub-transformations” as illustrated by the image below.
Your transformation may be simple but the complexity can come about due to the environment it operates in. For example, it might need to
Many companies deal address this using code but this brings with it many issues that get worse over time.
CloverDX Server provides a robust operational environment where transformations not only run, but can be automatically launched, scale and be monitored to deal with the most demanding situations.
Complexity often cannot be avoided. However, the more complex the scenario, the more important it is to simplify it to the fullest extent possible.
A lot of these simplifications follow the same principles that a programmer follows when writing complex code and CloverDX follows a lot of these.
Reusability is supported at many levels
Any transformation can be turned into a child transformation, allowing it to be fully componentized and shared between others
Data structure definitions can be shared across components and transformations.
These and other parameters can be shared across components and transformations.
Another programming inspired capability is to parameterize wherever possible. CloverDX allows you to create parameters that can be passed into all components.
This approach avoids hard-coding parameters in the component dialogs. There are many benefits to this, including allowing development, staging and production environments to be easily easily managed.
Even if your basic data transformations are simple, things can get complex when deployed into a real operational environment.
CloverDX provides many ways of launching data jobs, ensuring they run correctly and then taking actions if they don’t.
This significantly simplifies the level of work you would otherwise have to implement through code or cobbling together 3rd party solutions.
A jobflow is a way of orchestrating data transformations both independently of and in conjunction with external systems, applications and processes.
There are many ways to launch jobflows or data transformations.
API calls - CloverDX lets you easily define your own REST API endpoints through the CloverDX server UI that launch data jobs.
CloverDX provides many ways to address complexity not through simplification but through the ability to customize.
CloverDX encapsulates complex logic through the use of components in its visual design environment. However, you will inevitably encounter edge cases and this is when you will want to write code.
CloverDX allows you to do this in a number of ways.
Many components allow you to customize their behaviour by writing Java code. Your code can access and modify data as it flows through the transformation.
You have access to a fully fledged code debugger and can inspect data at any point in the transformation.
The source to a data transformation is XML, making it easy to access and modify through code. This allows you to create transformation templates and then make on-the-fly tweaks from code rather than having to use the visual designer.
As the transformation source is XML based, you can use any SCM tool such as Git, Mercurial or SVN. This allows you to manage and collaborate on transformations using repositories and to adopt a devops friendly approach to transformation management.