Hidden Features: Environment Variables in CloverDX Transformation

Using environment variables

"Environment variable is named value that can affect the way running process will behave on a computer."

In daily praxis we usually use environment variables with different syntax depending on operation system. On UNIX-like systems we use them with the syntax: $variable_name, on DOS and Windows systems the syntax is: %variable_name%. To list the variables on UNIX-like system we can use env shell command, on DOS and Windows systems set cmd command. You can find more general information on environment variables at en.wikipedie.org.


But enough of general information. Now how we can use environment variables in CloverDX transformation? It's very simple, you can use it in the same way as you routinely use graph parameters. So if you want to add username of the user under whom the transformation is running to your processed data, it's nothing more than adding a new field to metadata and write following in CTL (Clover Transformation Language):

function transform(){
...
$0.username := '${USER}'; //UNIX-like systems
OR
$0.username := '${USERNAME}'; //DOS and Windows systems
...
}

But be careful, value of environment variables can contain "bad characters" (,") that have to be escaped by '' in CloverDX. The safest way to use env variables in CloverDX is to enclose them in quotation marks  '''.

Overwriting environment variables & priority of parameter definitions

Often it's very helpful to use environment variables inside CloverDX transformation. But sometimes you want to define your own graph parameter with the same name as the existing environment variable has. And you may ask the question: "Is it possible?". I answer: "Yes, it is :-)." Because there is a hierarchy of graph parameter definitions:

  1. Parameter from external parameter file specified at the start of graph execution by -cfg option
  2. Parameter defined at the start of graph execution by -P option
  3. Parameter from external parameter file that is linked to the graph during the graph development
  4. Internal graph parameter
  5. Environment variable

Parameter definitions from the list are sorted by priority (highest to lowest). So if you have internal graph parameter with the same name as the environment variable, the value from internal parameter is always used in CloverDX.

Posted on August 11, 2009
DISCOVER  CloverDX Data Integration Platform  Design, automate and operate data jobs at scale. Learn more

Related Content

Subscribe to our blog


Where to go next