Reducing Salesforce Data Loading Time From 12 Hours to 41 Minutes
10 years ago, a script based approach to integration seemed like a good idea for a leading US wealth management firm.
9 years later, after very substantial growth in clients and data sources, the maintainability, error rates and lag times had become a serious threat to transparency and overall quality of service.
The company wanted to have complete control of their systems and steer clear of expensive ETL tools and inflexible platforms. This strategy served them well initially but as the business grew so did the complexity and the data volumes. The difficulties peaked as they developed a highly configurable Salesforce portal for their financial advisors.
With tens of millions of records of data being processed daily, things were out of hand. Maintaining ever-growing libraries of legacy scripts was now a serious problem and error rates had become very significant, requiring a full day of corrections. In the end, the published data was often 2 days out of date.
At this point the company realized that they had to squarely address the problem or the ability for the business to maintain a dominant position would be jeopardized. Choosing a platform partner was not just a question of product, it was equally about the team behind the product. CloverDX stood out from the competition in this respect with our experience in the integration field and the ability to co-engineer a solution.
Reducing Salesforce data loading time and freeing up resource
With this new solution, the company has been able to vastly reduce the Salesforce data loading time and free up significant resource for the team, so they can focus on adding value - not just fixing errors. The solution now sequences data appropriately to overcome the Salesforce data loading challenges, and data validation is now robust and reliable. It’s also enabled better information to be provided to their customers with data being pulled multiple times a day from different sources.
"The single best thing this company has done for data integration."
SVP of Technology
With improved data ingest, their platform can now also integrate with legacy systems; new integrations are significantly faster; customers can be onboarded more quickly; and the business can focus on their growth.
Needed to feed a complex Salesforce portal providing financial advisors a 360 degree view of the client
Tens of millions of records processed daily
Success of the new business model hinged on loading data, quickly and error free, into Salesforce
Needed to overcome the timeouts and batch restrictions imposed by the Salesforce API
Company needed not just product but expertise and a team to help accelerate the solution
Move away from hard to maintain scripts
Load times reduced from 12 hours down to 41 minutes
Error rate reduced to virtually nothing
Data sources due to double does not raise concern for the business
Dev team now focuses on adding value, not fighting fires
Can now integrate with any legacy accounting systems
Bolt on integrations that used to take a month now take less than a week
Data integration software and ETL tools provided by the CloverDX platform (formerly known as CloverETL) offer solutions for data management tasks such as data integration, data migration, or data quality. CloverDX is a vital part of enterprise solutions such as data warehousing, business intelligence (BI) or master data management (MDM). CloverDX Designer (formerly known as CloverETL Designer) is a visual data transformation designer that helps define data flows and transformations in a quick, visual, and intuitive way. CloverDX Server formerly known as CloverETL Server) is an enterprise ETL and data integration runtime environment. It offers a set of enterprise features such as automation, monitoring, user management, real-time ETL, data API services, clustering, or cloud data integration.