CloverDX Blog on Data Integration

Introducing: Integration of CloverDX with PHP

Written by Vaclav Matous | June 22, 2010

To witness the power of Clover, consider the following scenario. Our customer required an application whose main purpose was generating reports in xls format. What would seem like a simple task was presented with difficult challenges:

  • Reports had  many different formats specified by many parameters.
  • Reports were generated from heterogeneous sources – xls files, database tables and IBM iSeries files (former AS/400).
  • Application users modified data (mainly added records) in a report before its final confirmation.
  • The confirmation of a report invoked updates in several database tables and generated log records.
  • Short response times were needed for the most of operations.

We knew right away we would approach this situation with a web application written in PHP because web application architecture brought us plenty of needed advantages. But how can we quickly and reliably integrate heterogeneous sources? This is exactly the type of task that ETL tools were developed for. That’s why we built the whole solution using CloverDX Server with the Launch Services (equivalent of web services) feature. The following schema should illustrate the architecture of our solution.

Users of the application communicate only with the web application layer (1) that uses its own database for storing temporary data, setup information, and a history of reported records. This part also manages access permissions based on user roles.

When a user wants to view some report, he or she sets the proper parameters and other restrictions (e.g. type of records to be reported or dates scope) in the web application using his/her preferred web browser (IE, Firefox …). Confirming the filled parameters invokes a call to a Clover Launch Service (2) and a transformation connected with this service. The transformation integrates data from heterogeneous sources and stores it into a database dedicated to the web application. All stored records are identified by runID (transformation execution’s unique identifier within Clover Server) and the service returns the same runID as a response from its call.

The web application allows users to modify only certain data (according to partial runID). Users can insert, change or delete records (3). These changes have an effect only within the database of the web application and play only a temporary role until the user confirms the report. The confirmation again invokes  a call to the Clover Launch Service that then generates the needed reports, logs records about processed changes, and propagates changes into a history of reported records (4).

To summarize this solution, we moved the entire application logic into a CloverDX Server which allows us to solve such problems with ease. Furthermore we took advantage of Clover Launch Services – a combination of ETL integration power and online processing.

In my opinion, there are many similar problems where the concept described above cuts the time and effort to reach a solution.