• Blog
  • Contact
  • Sign in
CloverDX
Product
  • Overview
  • CloverDX Data Integration Platform
  • What's new in CloverDX 6
  • Pricing
  • CloverDX plans
  • Deployment
  • CloverDX on AWS
  • CloverDX on Azure
  • CloverDX on Google Cloud
  • CloverDX on-premise
  • Resources
  • Customer Portal
  • Documentation
  • Downloads & Licenses
  • Webinars
  • Academy & Training
  • Release Notes
  • CloverDX Forum
  • CloverDX Blog
  • Tech Blog
  • Other resources
isometric-illustration--product@2x 1

Get under the hood of CloverDX

See how CloverDX can benefit your business with a live demo. Simply get in touch with our team and we’ll handle the rest.

Book a demo
Solutions
  • By Industry
  • Banking
  • Capital Markets
  • Consultancy & Advisory
  • FinTech
  • Government Agencies
  • Healthcare
  • By Use Case
  • Data Quality
  • Data Ingest
  • Data Warehousing
  • Data Migration
  • Digital Transformation
  • Enterprise Data Management
  • Risk & Compliance
  • Anonymization
How F3 Group use CloverDX to ingest more client data - webinar
Customer interview

Formula 3: Staying Small And Agile While Working With Large Enterprise Ecosystems

Browse webinars
Services
  • Services
  • Onboarding & Training
  • Professional Services
  • Customer Support

More efficient, streamlined data feeds

Discover how Gain Theory automated their data ingestion and improved collaboration, productivity and time-to-delivery thanks to CloverDX.

 

Read case study
Customers
  • By Use Case
  • Analytics and BI
  • Data Ingest
  • Data Integration
  • Data Migration
  • Data Quality
  • Data Warehousing
  • Digital Transformation
  • By Industry
  • App & Platform Providers
  • Banking
  • Capital Markets
  • Consultancy & Advisory
  • E-Commerce
  • FinTech
  • Government
  • Healthcare
  • Logistics
  • Manufacturing
  • Retail
Migrating data to Workday - case study
Case study

Effectively Migrating Legacy Data Into Workday

Read customer story
Company
  • About CloverDX
  • Our story & leadership
  • Contact us
  • Partners
  • CloverDX Partners
  • Become a partner
Pricing
Demo
Trial

Metadata Propagation: It Makes Your Data Integration Jobs Much Easier

CloverDX How-To
Posted January 05, 2015
4 min read
Metadata Propagation: It Makes Your Data Integration Jobs Much Easier

In the CloverDX 4.0, along with subgraphs, we've introduced another very interesting feature: metadata propagation, that will make your data integration jobs much easier to prepare. Previously, you needed to assign metadata manually to every edge. In case you changed metadata in a graph, you would have to manually re-assign new metadata to each edge afterwards.

Data integration and ETL jobs are much easier thanks to new CloverETL feature - metadata propagation. Find out more about this feature.

However now, once you insert metadata on an edge, CloverDX will try to push it from left to right, and then from right to left, filling in each edge with the metadata where it expects it will fit, significantly reducing the need for metadata management. Also, in case you change metadata somewhere in the graph, the metadata everywhere else in the graph will adjust automatically to reflect these changes.

What does this mean for you?

A whole lot of time saved that you would otherwise spend placing metadata on edges. CloverDX 4.0 now will do this for you the majority of time. Also, this function reduces room for error when assigning metadata in a graph, plus makes changes in complex graphs much easier, as you really only have to change one important edge and the rest will be propagated automatically. And last but not least, it works as a fail-safe, making sure you will get the output with the expected metadata at the end of the graph.

Also in CloverDX 4.0, metadata can be embedded in components or in subgraphs, which means that many times the metadata will come pre-made for you, either from a component, or from a previously-built subgraph. And in these cases, CloverDX will take the metadata and propagate them through the rest of the graph.

When CloverDX propagates metadata, a yellow pop-up will blink on the screen to show you where the metadata will be assigned. Automatically assigned metadata are shown with a dashed grey line. Of course, there is always the option of assigning metadata for any particular edge yourself. These manually-assigned metadata are represented by solid line and will always have priority over automated metadata.

Metadata propagation makes your data integration jobs much easier

Different types of metadata propagation. Edge with manually-assigned metadata (solid line), edge with no metadata (red line) and edge with automatic metadata assigned (grey long dashes line).

Metadata propagation types

There are basically four types of metadata propagation. The first three are automatic; they assign metadata based on the components used, the type of graph, and the type of data. The last one – explicit metadata propagation - is semi-automated, as it is driven by user action when a user specifies metadata by selecting a reference edge. As a result, this edge will always have the same metadata as the referenced edge.

Metadata propagation and subgraphs

Where the metadata really shines is with subgraphs. When you create a subgraph, you can choose metadata that is required for the input or output (or even for both), so a subgraph will already have metadata embedded inside of it. This is especially handy when you are preparing customized connectors for CloverDX that will tap into your data and you are able to prescribe what kind of metadata will be produced by the connector. When you connect your subgraph to other components, this metadata will then automatically propagate throughout your graph. And when another user works with that subgraph, they won’t need to figure out and prepare the metadata for other components around it – the metadata will simply be there, automatically propagated throughout the graph.

Another very cool and clever way you can use subgraphs is as a template for your metadata (i.e. as a data target or a data source). You can create a customized component that contains a metadata template, which can be wrapped up for frequent reuse in any of your projects. And after adding this component into a graph, these metadata are propagated throughout the graph.

So, for instance, let’s say you have a specific format for Excel that needs to be written at the end of many of your graphs, with specific settings encoded in it. A savvy solution would be to wrap up that SpreadsheetDataWriter as a subgraph, and then configure it so that it insists on a specific format of metadata as its input. This SpreadsheetDataWriter subgraph would then create a unified end-product, across all projects, so that that all users and team members won’t have to worry about using the wrong format.

Metadata propagation makes your data integration jobs much easier

As you can see, subgraph is the source of metadata for the whole graph.

Using manually assigned metadata

Although in the majority of graphs, using automatically propagated metadata is a great time saver and preferable, there are times when we suggest using manually assigned metadata. This is true for longer data transformations that require a strictly defined output (for instance: you agreed on a specific format of an Excel sheet as the end product of a data transformation). In these situations, it is advisable to manually set the metadata for the graph’s last edge. Even though metadata propagation predicts and spreads your metadata throughout your transformation, you’ll still want to be completely sure that the metadata has propagated exactly as needed (in case someone made some change to a subgraph that alters the metadata, without your knowledge).

This is only a basic introduction to metadata. In the future, we will bring you other blogs and videos that will explain metadata propagation more deeply with examples and use cases.

If you'd like to see more about metadata propagation in action, check out our video here:

Share

Facebook icon Twitter icon LinkedIn icon Email icon
Try CloverDX for 45 days  Full access to Tech Support as if you were a customer

Newsletter

Subscribe

Join 54,000+ data-minded IT professionals. Get regular updates from the CloverDX blog. No spam. Unsubscribe anytime.

Related articles

Back to all articles
CloverDX 101 - some basic concepts explained
CloverDX How-To
7 min read

CloverDX 101: Some basic concepts explained

Continue reading
CloverDX Transformation Language How to Extend CTL with Java Functions
CloverDX How-To
8 min read

CloverDX Transformation Language: How to Extend CTL with Java Functions

Continue reading
Heres How to Connect to MemSQL with CloverDX (Plus a Few Tricks)
CloverDX How-To
4 min read

Here's How to Connect to MemSQL with CloverDX (Plus a Few Tricks)

Continue reading
CloverDX logo
Book a demo
Get the free trial
  • Company
  • Our story
  • Contact
  • Partners
  • Our partners
  • Become a partner
  • Product
  • Platform overview
  • Plans & Pricing
  • Customers
  • By Use Case
  • By Industry
  • Deployment
  • On-premise
  • AWS
  • Azure
  • Google Cloud
  • Services
  • Onboarding & Training
  • Professional Services
  • CloverCARE Support
  • Resources
  • Customer Portal
  • Documentation
  • Downloads & Licenses
  • Webinars
  • Academy & Training
  • Release Notes
  • CloverDX Forum
  • CloverDX Blog
  • Tech Blog
  • Other resources
Blog
4 steps to providing a data-driven customer experience
Data Integration
Implementing data democratization: 3 ways to make your data more accessible
Data Innovation
Data dictionary vs data catalog: what’s the difference?
Data Innovation
What is a ‘live’ data catalog and how can you use one in your organization?
Data Innovation
© 2023 CloverDX. All rights reserved.
  • info@cloverdx.com
  • sales@cloverdx.com
  • ●
  • Legal
  • Privacy Policy
  • Cookie Policy
  • EULA
  • Support Policy