Data Transformation: Environment Variables in CloverDX Transformation

What is an environment variable?

"An invironment variable is a named value that can affect the way running process will behave on a computer."

In daily praxis we usually use environment variables with different syntax depending on the operation system. On UNIX-like systems we use them with the syntax: $variable_name, on DOS and Windows systems the syntax is: %variable_name%. To list the variables on UNIX-like system we can use env shell command, on DOS and Windows systems set cmd command. You can find more general information on environment variables here.

How we can use environment variables in CloverDX transformation?

It's very simple, you can use it in the same way as you routinely use graph parameters. If you want to add the username of the user under whom the data transformation process is running to your processed data, it's nothing more than adding a new field to metadata and write the following in CTL (Clover Transformation Language):

function transform(){
...
$0.username := '${USER}'; //UNIX-like systems
OR
$0.username := '${USERNAME}'; //DOS and Windows systems
...
}

But be careful, the value of environment variables can contain "bad characters" (,") that have to be replaced by '' in CloverDX. The safest way to use env variables in CloverDX is to enclose them in quotation marks  '''.

Overwriting environment variables & priority of parameter definitions

It's very helpful to use environment variables inside CloverDX transformation. But sometimes you want to define your own graph parameter with the same name as the existing environment variable has. And you may ask the question: "Is it possible?". I answer: "Yes, it is :-)." Because there is a hierarchy of graph parameter definitions:

1. Parameter from external parameter file specified at the start of graph execution by -cfg option
2. Parameter defined at the start of graph execution by -P option
3. Parameter from external parameter file that is linked to the graph during the graph development
4. Internal graph parameter
5. Environment variable

Parameter definitions from the list are sorted by priority (highest to lowest). So if you have internal graph parameter with the same name as the environment variable, the value from internal parameter is always used in CloverDX.