• Blog
  • Contact
  • Sign in
CloverDX
Product
  • Overview
  • CloverDX Data Integration Platform
  • What's new in CloverDX 6
  • Pricing
  • CloverDX plans
  • Deployment
  • CloverDX on AWS
  • CloverDX on Azure
  • CloverDX on Google Cloud
  • CloverDX on-premise
  • Resources
  • Customer Portal
  • Documentation
  • Downloads & Licenses
  • Webinars
  • Academy & Training
  • Release Notes
  • CloverDX Forum
  • CloverDX Blog
  • Tech Blog
  • Other resources
isometric-illustration--product@2x 1

Get under the hood of CloverDX

See how CloverDX can benefit your business with a live demo. Simply get in touch with our team and we’ll handle the rest.

Book a demo
Solutions
  • By Industry
  • Banking
  • Capital Markets
  • Consultancy & Advisory
  • FinTech
  • Government Agencies
  • Healthcare
  • By Use Case
  • Data Quality
  • Data Ingest
  • Data Warehousing
  • Data Migration
  • Digital Transformation
  • Enterprise Data Management
  • Risk & Compliance
  • Anonymization
How F3 Group use CloverDX to ingest more client data - webinar
Customer interview

Formula 3: Staying Small And Agile While Working With Large Enterprise Ecosystems

Browse webinars
Services
  • Services
  • Onboarding & Training
  • Professional Services
  • Customer Support

More efficient, streamlined data feeds

Discover how Gain Theory automated their data ingestion and improved collaboration, productivity and time-to-delivery thanks to CloverDX.

 

Read case study
Customers
  • By Use Case
  • Analytics and BI
  • Data Ingest
  • Data Integration
  • Data Migration
  • Data Quality
  • Data Warehousing
  • Digital Transformation
  • By Industry
  • App & Platform Providers
  • Banking
  • Capital Markets
  • Consultancy & Advisory
  • E-Commerce
  • FinTech
  • Government
  • Healthcare
  • Logistics
  • Manufacturing
  • Retail
Migrating data to Workday - case study
Case study

Effectively Migrating Legacy Data Into Workday

Read customer story
Company
  • About CloverDX
  • Our story & leadership
  • Contact us
  • Partners
  • CloverDX Partners
  • Become a partner
Pricing
Demo
Trial

How to Design Versatile Subgraphs Using Optional Ports

CloverDX How-To
Posted June 30, 2015
4 min read
How to Design Versatile Subgraphs Using Optional Ports

Optional ports introduced in CloverDX 4.1 allow you to design generic and versatile subgraphs that replace potentially redundant variations of the same subgraph offering various combinations of inputs and outputs.

Designing Versatile Subgraphs Using Optional Ports

Example of a subgraph with optional ports

However, giving users such freedom of choice means you have to deal with numerous design challenges to handle the missing edge connections.

In this blog post, I will share three key concepts that will help you master the wizardry behind creating versatile subgraphs.

Demo Scenario

Designing Versatile Subgraphs Using Optional Ports

Sample subgraph with Optional ports

Let's assume we're building a subgraph called “Lean DataIntersection” - a „user friendly“ version of DataIntersection component that is forgiving in terms of what's connected to it (remember, standard DataIntersection needs all ports connected) and would take care of pre-sorting the inputs as a bonus (the two FastSort components).

Here are two sample scenarios showing how such a subgraph can be used:

Designing Versatile Subgraphs Using Optional Ports

Sample scenario 1: Notice only the middle output connected

Designing Versatile Subgraphs Using Optional Ports

Sample scenario 2: Using only one input connected

Setting Optional Ports

We want to set the second input and two output ports as optional so that users can freely to use the “forgiving” component.

Designing Versatile Subgraphs Using Optional Ports

Setting up optional ports

You can choose between two modes of optional ports (right-click the port in the vertical bar, or in outline):

Optional port (edge receives zero records)

If we select this mode the second FastSort will receive 0 input records (no edge connected). See the illustration below to view the results. Not exactly what we want, right?

Designing Versatile Subgraphs Using Optional Ports

Optional port set to receive zero records

Optional port (edge is removed)

Unfortunately, the second option is not much better at solving our dilemma either. In this case, instead of zero records, the edge would be removed completely at runtime and we would end up with a crippled subgraph illustrated below.

Designing Versatile Subgraphs Using Optional Ports

Removal of the edge caused error

Either way, merely setting ports to optional does not give you the results you'd expect. There's more to set than just the ports.

Key Concept I: Dynamically Enabling Components

One of the key factors to versatile subgraphs is to have Clover dynamically enable/disable parts of the subgraph that are affected by the missing input. In this case we want to disable the second FastSort and DataIntersection completely whenever the optional input is not connected.

Designing Versatile Subgraphs Using Optional Ports

Dynamically enable/disable portions of a subgraph

To set components to be enabled only if certain inputs/output are connected, go to „Enable“ menu of a component (right-click). In our case we're using “When Input Port 1 Is Connected”.

Designing Versatile Subgraphs Using Optional Ports

Dynamically enable/disable component based on a condition

IMPORTANT
The Enable condition is relative to the parent graph of our subgraph, not the component itself! Thus, you should read it as follows:
"When Input Port 1 (of this subgraph) Is Connected (in the calling parent graph)"

Designing Versatile Subgraphs Using Optional Ports

Instant indication of dynamically disabled component

The “?” icon indicates the component is set to enable/disable dynamically.

NOTE
Wondering why the top FastSort is also dynamically disabled? There's no optional port so that part will always work, right? Well, yes. Leaving it always enabled would work just fine but in the “one input port SimpleCopy” mode it would be sorting all the data without a real purpose and a good design avoids such costly operations whenever possible.

Key Concept II: Component Pass-through

Designing Versatile Subgraphs Using Optional Ports

User defined pass-through

When CloverDX disables a component, it needs to know how to bypass it. A disabled component is always replaced by a single edge and often it's simply obvious (e.g. disabling a sorter simply creates a “short-circuit”).

For complicated components like DataIntersection or subgraphs having multiple inputs and outputs you need to tell CloverDX which ports to connect.

Designing Versatile Subgraphs Using Optional Ports

Setting pass-through for components with multiple inputs and outputs

You can set pass-through for any component or subgraph by going to Edit and scrolling all the way down to Common properties and setting Pass Through Input Port and Pass Through Output Port.

Why do I need to set pass-through?

If we didn't set the DataIntersection pass-through properly, CloverDX would connect the first input port with the first output port (Port 0 > Port 0) by default which is not what we want.

However, you can typically ignore pass-through as the default behavior mostly works just fine.

Key Concept III: Metadata Propagation

Versatile generic graphs tend to depend heavily on metadata propagation rather than having everything predefined. For example, our Lean Data Intersection has no internal metadata whatsoever; everything is driven by what the parent graph provides by connecting edges with metadata to it.

This is where you can easily fall into a trap. Do not rely on metadata propagating from ports that are set as optional. Keep in mind that with some use-cases there will be no edge connected, thus no metadata!

The solution is to either use manually assigned metadata or use edge metadata (Select Metadata from Another Edge) so that your metadata is propagated from edges that are guaranteed to receive metadata at all times.

Dynamically disabling components connected to optional ports and setting correct pass-through will save you a lot of trouble with metadata propagation. In fact, if you do everything else right, you likely won't have problems with metadata propagation at all!

Conclusion

These three are basic concepts will help you change the way how you design versatile subgraphs. You can read more about more advanced use cases for optional ports in our follow-up blog. Remember to watch for the final version of CloverDX 4.1!

Share

Facebook icon Twitter icon LinkedIn icon Email icon
Try CloverDX for 45 days  Full access to Tech Support as if you were a customer

Newsletter

Subscribe

Join 54,000+ data-minded IT professionals. Get regular updates from the CloverDX blog. No spam. Unsubscribe anytime.

Related articles

Back to all articles
CloverDX 101 - some basic concepts explained
CloverDX How-To
7 min read

CloverDX 101: Some basic concepts explained

Continue reading
CloverDX Transformation Language How to Extend CTL with Java Functions
CloverDX How-To
8 min read

CloverDX Transformation Language: How to Extend CTL with Java Functions

Continue reading
Heres How to Connect to MemSQL with CloverDX (Plus a Few Tricks)
CloverDX How-To
4 min read

Here's How to Connect to MemSQL with CloverDX (Plus a Few Tricks)

Continue reading
CloverDX logo
Book a demo
Get the free trial
  • Company
  • Our story
  • Contact
  • Partners
  • Our partners
  • Become a partner
  • Product
  • Platform overview
  • Plans & Pricing
  • Customers
  • By Use Case
  • By Industry
  • Deployment
  • On-premise
  • AWS
  • Azure
  • Google Cloud
  • Services
  • Onboarding & Training
  • Professional Services
  • CloverCARE Support
  • Resources
  • Customer Portal
  • Documentation
  • Downloads & Licenses
  • Webinars
  • Academy & Training
  • Release Notes
  • CloverDX Forum
  • CloverDX Blog
  • Tech Blog
  • Other resources
Blog
Why data accessibility is essential for your digital transformation
Data Integration
4 barriers to making data-driven decisions
Data Strategy
4 steps to providing a data-driven customer experience
Data Integration
Implementing data democratization: 3 ways to make your data more accessible
Data Innovation
© 2023 CloverDX. All rights reserved.
  • info@cloverdx.com
  • sales@cloverdx.com
  • ●
  • Legal
  • Privacy Policy
  • Cookie Policy
  • EULA
  • Support Policy