Data transformation: Environment variables

What is an environment variable?

"An environment variable is a named value that can affect the way running process will behave on a computer."

In daily praxis we usually use environment variables with different syntax depending on the operation system. On UNIX-like systems we use them with the syntax: $variable_name, on DOS and Windows systems the syntax is: %variable_name%. To list the variables on UNIX-like system we can use env shell command, on DOS and Windows systems set cmd command. You can find more general information on environment variables here.


How we can use environment variables in CloverDX transformation?

It's very simple, you can use it in the same way as you routinely use graph parameters. If you want to add the username of the user under whom the data transformation process is running to your processed data, it's nothing more than adding a new field to metadata and write the following in CTL (Clover Transformation Language):

function transform(){
...
$0.username := '${USER}'; //UNIX-like systems
OR
$0.username := '${USERNAME}'; //DOS and Windows systems
...
}

But be careful, the value of environment variables can contain "bad characters" (,") that have to be replaced by '' in CloverDX. The safest way to use env variables in CloverDX is to enclose them in quotation marks  '''.

Overwriting environment variables & priority of parameter definitions

It's very helpful to use environment variables inside CloverDX transformation. But sometimes you want to define your own graph parameter with the same name as the existing environment variable has. And you may ask the question: "Is it possible?". I answer: "Yes, it is :-)." Because there is a hierarchy of graph parameter definitions:

  1. Parameter from external parameter file specified at the start of graph execution by -cfg option
  2. Parameter defined at the start of graph execution by -P option
  3. Parameter from external parameter file that is linked to the graph during the graph development
  4. Internal graph parameter
  5. Environment variable

Parameter definitions from the list are sorted by priority (highest to lowest). So if you have internal graph parameter with the same name as the environment variable, the value from internal parameter is always used in CloverDX.

More from Tech Blog

  • Server diagnostics: ChatGPT or Claude

    Administrating a healthy CloverDX Server usually means correlating information scattered across performance logs, job tracking tables, execution logs,... AI
  • Data Quality Examples on GitHub

    As our first set of examples, we published a project called DataQualityExamples which demonstrates how users can address their data quality issues with... CloverDX How-To
  • Introducing CloverDX Examples on GitHub

    We’re excited to introduce the CloverDX Examples repository on GitHub. The repository is a growing collection of helpful and practical examples designed to... CloverDX How-To
  • Send Email via Azure Communication SMTP

    There are many situations where it is helpful to get a message from your active pipelines. These may include logs, summaries of successful completion, or... Cloud
  • Connecting to REST APIs (OpenAPI)

    Connecting to REST APIs is a crucial aspect of modern data integration, and the REST Connector in CloverDX provides a streamlined approach. The video below... RESTConnector
  • Troubleshoot database-related performance

    Performance is undoubtedly one of the key factors when running data transformations. In this article, we will look at how to troubleshoot database-related... Performance

Visit CloverDX Blog

Read On