CloverDX Transformation Language: How to Extend CTL with Java Functions

In this article, I will walk you through the process of extending CTL (CloverDX Transformation Language) with your own custom functions implemented in Java. Think of this as adding your own Java functions to the standard CTL function library so that you can transparently use them wherever there’s a piece of CTL code—in Reformat transformations, Filter expressions, etc.

Why Java for Custom Functions (And Not CTL)?

Imagine you have some custom complex logic already implemented in Java packages, maybe some 3^rd party library with potentially dozens of outside dependencies. Let’s say you want to share that functionality across applications, CloverDX included. There might be thousands of lines of code that you don’t want to be re-implementing in CTL.

Other reasons for using Java over pure CTL are performance and the expressive power of Java. While performance begins to matter only with large volumes of data, CTL has never been designed to fully replace general purpose languages. CTL makes working with data records easy, however for very complex algorithms it lacks the expressiveness and eco-system of libraries that Java can give you.

//#CTL2
import "trans/filterFunctions.ctl";

function integer transform() {
    $out.0.field1 = filterChars($in.0.field1); // filterChars() is declared in filterFunctions.ctl
    return ALL;
}

Example Scenario

As an example, we’ll be implementing two functions:

• string getWords()
• string getWords(integer count)

These functions will produce “Lorem Ipsum”—Latin-like bogus text for testing. For this purpose, we’ll use an external loremipsum Java library by Sven Jacobs (see, we already have external dependencies!).

Architecture & Extensibility of CloverDX

CloverDX is an open platform that can be extended by users in many ways. The core of CloverDX lies in its modular “engine”. This is the execution layer that takes care of running your data transformations and that sits at the center of CloverDX Designer (Runtime) and Server. CTL and all user extensions are treated as “engine plugins”. For our purpose, we’ll need to extend the engine with our own plugin (Java package) containing our Lorem Ipsum functions.

CloverDX Designer presents another layer, working rather independently of the engine/Runtime. Designer is built on Eclipse, and it comes with its own set of plugins that form the GUI. For Designer to recognize our new functions we’ll need to make it aware of our engine plugin by wrapping it with a separate “Designer plugin”. This will make the new functions available for context assist, correct syntax highlighting in editors, etc.

Brief Overview

Here’s what we’ll do:

1. We’ll need our CloverDX Engine plugin first:
   • We’ll implement the desired CTL functions as a Java class that will extend org.jetel.ctl.extensions.TLFunctionLibrary.
   • Then, we’ll create a plugin.xml descriptor with ctlfunction extension definition.
   • And we’ll use org.jetel.ctl.extensions.TLFunctionAnnotation annotation to mark “entry point” methods that will serve as the implementation of our CTL functions (as you may have some additional supporting private methods that you don’t want to turn into CTL functions).
2. Secondly, we’ll add a Designer plugin:
   • For our functions to work in Designer, we’ll need another plugin.xml descriptor—this time with engineplugin extension definition.
   • Finally, we’ll import the engine plugin into this new Designer plugin.

Once finished, the new functions will be available in Designer dialogs (e.g. Source tab in Transform Editor, Reformat code, etc.).

So, let’s get to it!

Step-by-step Guide to Custom CTL Functions

Initial Setup

1. Download the external Lorem Ipsum Java library (http://loremipsum.sourceforge.net/).
2. Download, install and run CloverDX Designer (http://www.cloverdx.com/download).
3. Install Eclipse Plug-in Development Environment (PDE):
   • Help ⇒ Install New Software…
   • Select your Eclipse release update site named after the codename of your version
     (e.g. “Indigo – http://download.eclipse.org/releases/indigo”)
4. Search for General Purpose Tools / Eclipse Plug-in Development Environment.
5. Once installed, switch to Java perspective (Window→Perspective→Open Perspective→Java).

Creating the Engine Plugin

1. Start a new Java project (File ⇒ New ⇒ Other ⇒ Java/Java Project) and name it com.cloveretl.custom.loremipsum. Hit Next when done.
   
2. Add CloverDX Engine to the project build path:
   • Switch to the Libraries tab, click Add Library ⇒ CloverDX Engine.
   • Click Next, then Finish.
   
   Note: If you don't end-up with CloverDX Engine added to the project (see picture below) then try to right-click on the name of your project in PackageExplorer then Build Path→Add Libraries and pick CloverDX Engine.
3. Now that you’ve created the Java project, create a new folder named lib in the project root directory (right-click the project in Package Explorer or in Navigator to create that folder)
4. Put the “Lorem Ipsum” library into it (lib/loremipsum-1.0.jar)
   
5. Add the library to Java classpath (Project → Properties → Java Build Path section → Libraries tab → Add JARs… → Navigate to the lib directory → Choose loremipsum-1.0.jar) and click OK.
   
6. Create the main Java class. We’ll call it LoremIpsumCTLLibrary and it will extend org.jetel.ctl.extensions.TLFunctionLibrary.

   Let’s put the class into a package called com.cloveretl.customctl.loremipsum.

   TLFunctionLibrary comes from the CloverDX Library we added to the project earlier. However, you don’t need to worry about this for now, it is already a part of the code below).

   
   You don't need to set anything else here—you will paste the full source code of this class in the next step anyway (will work only if you kept the same name and package, otherwise you’ll have to do a bit of editing).
7. Copy & paste the following code. You can modify this later to suit your needs but for now this one class implements everything we’ll need for our two getWords() CTL functions.

   
   package com.cloveretl.custom.loremipsum;
   
   import org.jetel.ctl.Stack;
   import org.jetel.ctl.extensions.TLFunctionAnnotation;
   import org.jetel.ctl.extensions.TLFunctionCallContext;
   import org.jetel.ctl.extensions.TLFunctionLibrary;
   import org.jetel.ctl.extensions.TLFunctionPrototype;
   import de.svenjacobs.loremipsum.LoremIpsum;
    
   public class LoremIpsumCTLLibrary extends TLFunctionLibrary {
    
       LoremIpsum loremIpsum = new LoremIpsum();
    
       @Override
       public TLFunctionPrototype getExecutable(String functionName)
               throws IllegalArgumentException {
           TLFunctionPrototype ret =
                   "getWords".equals(functionName) ? new GetWordsFunction() : null;
           return ret;
       }
    
       @Override
       public String getName() {
           return "LoremIpsumCTLLibrary";
       } 
   
       @TLFunctionInitAnnotation
       public static void getWordsInit(TLFunctionCallContext context) {
           //for now, no initialization needed
           //this is called only once
           //can be used to set-up/initialize included 3rd party Java libraries
       }
   
       @TLFunctionAnnotation("Generates Lorem Ipsum text")
       public String getWords(TLFunctionCallContext context) {
           return this.loremIpsum.getWords();
       }
    
       @TLFunctionAnnotation("Generates Lorem Ipsum text")
       public String getWords(TLFunctionCallContext context, Integer count) {
           if(count==null) {
               return this.getWords(context);
           }
           return this.loremIpsum.getWords(count);
       }
    
       class GetWordsFunction implements TLFunctionPrototype {
           @Override
           public void execute(Stack stack, TLFunctionCallContext context) {
               if (context.getParams().length > 0) {
                   stack.push(getWords(context, stack.popInt()));
               }else{
                   stack.push(getWords(context));
               }
           }
           @Override
           public void init(TLFunctionCallContext context) { }
       }
   }
   

8. As the last step of this “engine plugin” part, let’s create the plugin.xml descriptor that the engine needs to properly load our code.
9. Create a new file called plugin.xml in the root of the engine plugin project:
   
10. Put the following code into plugin.xml:

    <plugin
       id="com.cloveretl.custom.loremipsum"
       version="0.0.0.devel"
       provider-name="CloverDX">
     
        <runtime>
            <library path="bin/"/>
            <library path="lib/loremipsum-1.0.jar"/>
        </runtime>
     
        <extension point-id="ctlfunction">
            <parameter id="libraryName" value="LoremIpsumCTLLibrary"/>
            <parameter id="className"
    value="com.cloveretl.customctl.loremipsum.LoremIpsumCTLLibrary"/>
        </extension>
    </plugin>

11. Your engine plugin is now done!

Alright, you’re half way through! Now let’s get on to setting up the CloverDX Designer plugin, so this library can be installed and used inside Designer.

Moving on To the Designer Plugin

1. Staying in the same workspace, create a new Plug-in project (not Java!) using File → New → Plug-in Development → Plug-in Project.
   
   1. Name the project com.cloveretl.designer.loremipsum.
2. Click Next
   
3. On the second page of the project wizard:
   • Uncheck Generate an activator, a Java class that controls the plug-in's lifecycle.
   • Uncheck This plug-in will make contributions to the UI (just ignore what it says for now).
   
4. After clicking Finish, choose No when Designer asks to switch to a Plugin perspective.
   
5. Now we have our Designer plugin project ready. Let’s put some classes in.
6. Create a “Class Loader Provider” Class
   • File ⇒ New ⇒ Class
   • Class name: CLProvider
   • Package is automatically prefilled to com.cloveretl.designer.loremipsum (you can leave that)
   • Superclass: com.cloveretl.gui.plugin.engine.AbstractClassLoaderProvider
   
7. Click Finish.
   Don’t worry that the AbstractClassLoaderProvider cannot be resolved in the code yet, we’ll fix it later.
8. Now double click META-INF/MANIFEST.MF (in the Explorer on the left).
   This will get you to Overview tab of the Plugin screen.
   
9. Switch to the Dependencies tab (bottom tabs)
10. Add com.cloveretl.gui plugin to the list of Required Plug-ins
    
11. Switch to the Extensions tab
12. Add com.cloveretl.gui.enginePlugin to Plugin extension
    
    
    
    
13. On the same screen, set the following:
    
    • Directory: plugins
    • classLoaderProvider: com.cloveretl.designer.loremipsum.CLProvider
14. Save all your changes (Ctrl-S) and review the previously created class (CLProvider.java). The missing dependencies should be resolved now.
    
15. Go back to the Designer plugin ("com.cloveretl.designer.loremipsum" in main tabs).
16. Switch to plugin.xml tab (bottom tabs).
17. Add the following lines to the enginePlugin section:

    <exportedPlugin
          pluginId="com.cloveretl.custom.loremipsum">
    </exportedPlugin>

    
    
18. Now we’ll import our engine plugin we created earlier into this Designer plugin.
19. Create a new directory called plugins under the com.cloveretl.designer.loremipsum project folder. As best practice, put your engine plugins into folders named after the plugin IDs.
20. Create a sub-directory called com.cloveretl.custom.loremipsum (see image below):
    
21. Right-click plugins/com.cloveretl.custom.loremipsum folder and select Import... → General → File system.
    • Set From directory to com.cloveretl.custom.loremipsum project location (the engine plugin project).
    • Check plugin.xml (in the right panel).
    • Check bin.
    • Check lib.
    
22. Click Finish and verify that files have been copied to the desired folder:
    
23. Open plugin.xml in the com.cloveretl.designer.loremipsum project—the one in the main directory, not the one that we have just imported to plugins folder!.
    • Switch to Build tab (bottom tabs).
    • Make sure the following items are checked in the Binary Build section:
      • META-INF
      • plugin.xml
      • plugins
    
24. Switch to Runtime tab and add the following folders to the classpath:
    • . (the Designer plugin folder)
    • plugins/com.cloveretl.custom.loremipsum/bin/
    • plugins/com.cloveretl.custom.loremipsum/lib/loremipsum-1.0.jar
    
    Note: you can use New button and just copy & paste the values from above.
25. Switch to build.properties tab and verify that the content is similar to this:
    
26. You are very close to victory now!
27. Right click com.cloveretl.designer.loremipsum (the Designer plugin) project and select “Export…” from the context menu.
    • Select Plug-in development → Deployable plug-ins and fragments.
    • On the Destination tab select Install into host. Repository: (no need to change anything else)
    
28. Click Finish.
29. In the dialog asking whether you trust your plugin, check the plugin and click Trust
    
30. Congratulations!  After restarting the Designer application, all your projects will be able to use your magnificent new CTL functions!

Now you can try your new functions anywhere in CloverDX, let's see it in DataGenerator's Source tab:

Troubleshooting

If you foolowed this how-to, you should end-up with the new functions present/available in your CloverDX Designer. However, if you experience issue during building or installing, you may look at the attached projects source code .zip archive. It should be enough to import into your Designer workspace and deploy/export as mentioned in step 27 above.

The archive also contains ready to use Eclipse/Designer Update Site: com.cloveretl.designer.loremipsum.updatesite - this may be used to install the plugin via Help → Install New Software and then adding local repository (note: copy the content of the updatesite to some local directory). For more details, consult Eclipse's manual).

Deploying to CloverDX Server

Well, you sucessfully deployed and tested your new plugin with CTL functions implementation in CloverDX Designer. Now is time to add this plugin to CloverDX Server, so transformations using your new functions also work in server environment.

1. First, locate file clover.properties. It should reside in cloverconf subdirectory of the main directory where your CloverDX Server is installed into.
2. Add configuration property engine.plugins.additional.src into the file (unless it is already there). This property should be set to a value which denotes a full path to the directory where CloverDX should look for your additional plugins. So make sure this is accessible (at least for reading) to whatever user CloverDX Server process runs under.
   Example:  engine.plugins.additional.src=/opt/cloverserver/myplugins
3. Go to CloverDX Designer, right click your com.cloveretl.custom.loremipsum project and select Export. Choose General → Archive File
   
4. In the Export Archive file dialog, unfold com.cloveretl.custom.loremipsum and make sure you have only these folders and files checked for export:
   
   • bin/
   • lib/
   • plugin.xml
   Also make sure, you choose Create only selected directories option.
   You may name your archive file whichever name you like. In our case, we use customloremipsum.zip
   
   
5. After clicking Finish, copy/move the newly created archive file into wherever your engine.plugins.additional.src points to. In our case, it is /opt/cloverserver/myplugins.
6. Extract the content of the archive into the directory. Make sure it resides in its own subdirectory, in which plugin.xml file is located.
   
7. Restart CloverDX Server so it picks up the newly added plugin.
8. Go to you CloverDX Server console, select Configuration → CloverDX Info → Plugins. You should see your plugin picked up & loaded by the server.
   
9. Congratulations ! Your plugin is now used by CloverDX Server and all transformations executed by it will be able to benefit from your newly created CTL function.