Quick Tip - Organizing Executions History

Reusability is very important topic when it comes to job design in CloverDX. We are strong advocates of the DRY principle, which can be a big help during development. There is small culprit when trying to make sense in multiple executions of same, highly configurable job.

Child job tagging

This is extremely helpful during development but may create all kinds of problems for DevOps when not managed properly. For example, the following image outlines multiple processes which read and process Payment files. If one or more of these processes fails, it is virtually impossible to see which file was not possible to load. It could be potentially even worse when the executed process could be a dummy wrapper for something more complex.

Execution tree without labels

As a developer, you can make browsing through an executions tree way more transparent. It is possible to change labels either via proper naming of any given Subgraph or through the Execution Label property configured directly from configuration dialogue or via Input mapping of ExecuteGraph/ExecuteJobflow. This results in better annotated execution structure and helps pinpoint issues more efficiently.

Execution tree with labels

Default job labels

In the case of the same top-level job running multiple times, we will find the picture very like the following one in Execution History. This is also not too helpful when dataLoad.jbf behaves different for each configuration, e.g. loading data for different regions.

It may not be straightforward to identify which region failed to refresh its data. But there is a way to fix it.

List of triggered jobs without labels defined

Execution Label property can be set also be set in a job file itself and as usual, it can be populated by any job parameter. In our example, dataLoad.jbf also includes one parameter, called REGION. This parameter is used as such a label.

Apply label to any CloverDX job

This allows for Executions History be annotated a little better than usual. Each individual trigger provides REGION as means of identification of the data target as well as the appropriate tag for the Executions History listing.

Making sense of execution history for labeled jobs is easier

Quite a different picture, right? I hope you will find this tip useful.

More from Tech Blog

  • Customizing metadata propagation

    Metadata propagation, i.e. the ability to push metadata out from connected components is in the product since CloverETL 4.0.0. A new addition in CloverDX... Java
  • CTL2 error handling - try/catch block

    Poor data quality, format changes and unreachable data sources are just a few examples of runtime problems that can wreak havoc on a seemingly robust data... Feature
  • Starting a new CloverDX project

    We often get questions such as 'What is a best practice for project structure?', 'How do you work on a single project in parallel?', 'What's the best... Best practice
  • Deployment templating for CloverDX Server

    As more and more companies move towards cloud or container deployments, CloverDX has introduced a number of features, supporting both an infrastructure as... CloverDX Server
  • Publishing data sites

    One of the frequently used features of CloverDX is Data Services. Data Services allows you to publish your CloverDX transformations as REST APIs. A less... API

Visit CloverDX Blog

Read On