Snowflake is a cloud-based data storage and analytics engine – or a “data warehouse as a service”. It allows you to store massive amounts of data in your warehouse and make it accessible to all your users while providing maximum query performance.
CloverDX enables you to easily connect to your Snowflake data warehouse, load or unload data and run various queries to manage your data.
As with many similar cloud-based databases, CloverDX offers two ways of connecting to Snowflake. The first way is via JDBC driver that allows the largest flexibility to run various queries on the target Snowflake instance. The only requirement in this case is a driver that can be downloaded directly from Snowflake’s website and then added to Designer or Server.
The second way – using a custom subgraph-based connector – allows you to connect to your Snowflake instance and perform high-performance bulk data loads that use all the power Snowflake offers.
The bulk writer component is implemented as a subgraph (a user-defined component) that performs several operations to load the data into Snowflake instance – it loads the data into staging area in Snowflake, then loads data from the staging area into a table and finally deletes any local temporary files.
This is the fastest way of getting your data to Snowflake, as the data is uploaded to the cloud in parallel in multiple threads. This means Snowflake can run its data loads in parallel as well. As such, this is the best way to load large volumes of data into your warehouse.
Sample code contains a project that shows how to:
Compare the performance of writes with the regular JDBC connection.
We'd be happy to walk you through how CloverDX's Snowflake connector simplifies bulk-loading.
