The very typical usage of ETL tools is loading the data warehouse (DWH). So I decided to write a tutorial that will describe typical data warehouse tasks (slowly changing dimensions, date dimension, filling fact tables) and propose solutions with using of CloverDX. Let me guide you through the process of building data warehouse with CloverDX.
If you are a newbie in data warehousing I recommend you reading some of the books by Ralph Kimball or H. I. Imon.
Our sample data warehouse collects the information about sales for a small store chain that offers electronics like iPod, MP3, laptops etc.
The DB schema of my data warehouse is very simple. It consists of four dimensions (Customer, Product, Store and Date), one degenerate dimension (invoice number) and one fact table (Sales). Fact table stores two additive facts: units and total price. You can see complete DB scheme on figure below.
DB schema of sample DWH
One thing you will surely face when you are building a data warehouse is working with several types of slowly changing dimension (SCD). In this part of the tutorial I used the simplest SCD type.
When looking at data warehousing, a dimension table contains a surrogate key, natural key, and a set of attributes. On the contrary, a fact table contains a foreign key, measurements, and degenerated dimensions. Dimension tables provide descriptive or contextual information for the measurement of a fact table.
The simplest and surely the most popular SCD type among ETL developers is slowly changing dimension type 1. It doesn't store any history, so once the value in online transaction processing system (OLTP) has been changed, the value in DWH is immediately overwritten as well.
I decided to use SCD1 for store dimension which collects basic information about stores: address, store identifier, store manager etc. Each store is identified in OLTP by unique store number (natural key). But for DWH I have to generate an own surrogate identifier ID_D_STORE.
The basic idea of processing SCD1 is very simple: compare records in DWH and OLTP, insert missing records into DWH, update the DWH records according to OLTP. For all these tasks the attribute that helps us to find corresponding records is the natural key – STORE_NUMBER.
So let's go to develop a CloverDX graph. For better portability all inputs and output data are stored in csv files, thus you don't have to configure any database. The store dimension of DWH is stored in D_STORE.tbl file, the actual data from OLTP are stored in Store_25092009.csv. In both of these files we have to read, sort on natural key STORE_NUMBER and find the records that aren't in D_STORE (third output of DataIntersection). In this last step we will use DataIntersection component. Simultaneously (by second output of DataIntersection) we get the potential records that can be different in OLTP and DWH. These records are then filtered and only the records having any different value of any attribute are processed and new values are stored to D_STORE_update.tbl file. New records are written to D_STORE_insert.tbl file once ID_D_STORE attribute is added. ID_D_STORE attribute gains its value from sequence that we have already defined in CloverETL in advance. And that's all. You can see the resulting graph below.
CloverETL graph D_STORE_SCD1
If you want to read/write data from/to database easily replace UniversalDataReaders with DBInputTables and UniversalDataWriters with DBOutputTable components.
Part 2: Building a Data Warehouse with CloverDX - Slowly Changing Dimension Type 2
