We are excited to announce the release of our newest integration with dbt Cloud, allowing you to orchestrate dbt Cloud jobs right from Fivetran’s platform. This integration means that all stages of data integration – extraction, loading and transformation – can now be managed through one platform with minimal engineering effort.

This isn’t our first rodeo

In 2022, Fivetran launched integrated scheduling for dbt Core™, our solution for orchestrating your open-source, self-hosted dbt data transformations. Integrated scheduling enables transformations to be triggered by the completion of Fivetran connector data syncs, ensuring that downstream tables – and the metrics and insights derived from them – remain up to date as new data becomes available. By tying transformations to the completion of data syncs, integrated scheduling also prevents redundant transformations from eating up compute costs.

Our team also began building our robust library of open-source, free dbt data models. These data models have helped data teams reduce engineering work by standardizing the base use cases for common SaaS data sources. This library continues to grow – including our new SAP data model – and the packages have been utilized by analytics engineers using both dbt Core and the fully managed dbt Cloud.

New capabilities provided by dbt Cloud orchestration

For data teams looking for a fully managed experience, similar to Fivetran, dbt Labs created dbt Cloud. dbt Cloud’s fully managed approach to data transformations means teams do not have to provision their own infrastructure, while also taking advantage of collaborative web-based development and navigation through a web UI. Many organizations have rapidly adopted dbt Cloud for these capabilities.

“dbt Cloud is designed to help teams manage the complexity of their modern data environments so they can ship trusted data faster,” says Luis Maldonado, VP Product of dbt Labs. “By automatically triggering dbt Cloud jobs to run when upstream data loads are complete, our over 1,000 shared customers can avoid tedious manual orchestration and have confidence that decisions are being made from the freshest data possible.”

The new Fivetran orchestration integration with dbt Cloud brings our integrated scheduling to dbt Cloud. Analysts and engineers can now schedule dbt Cloud jobs to run upon the successful load of new data from associated upstream connectors. You can set up, monitor and QA your pipelines directly from the Fivetran GUI with our robust run logs. Fully managed end-to-end data integration is now accessible to teams in an automated and accessible fashion.

“We know this will help data teams operate more efficiently and cost-effectively thanks to Fivetran’s data orchestration,” said Mac Noland, Co-Founder and Chief Data Officer at phData. “We even use this internally and find that our dbt Cloud and Fivetran instances are now fully synchronized and automated, reducing the time our team spends on pipeline build and maintenance to near null. This means our team - and our clients' teams - can now focus on more strategic projects while saving money since compute isn’t engaged until there’s new data available.”

The new dbt Cloud integration continues the Fivetran trend of simplifying data pipeline setup and maintenance, helping data teams reprioritize their attention while democratizing access to data and making it simpler and more reliable to use. Combined with Fivetran pre-built data models, the new integration offers a single environment from which it is possible to set up reports and dashboards in a matter of minutes.

“Our dbt Cloud integration delivers one platform to manage and automate end-to-end ELT pipelines with built-in choice. It allows for an organization’s entire pipeline to be fully managed and integrated, letting data teams focus more on data work versus pipeline management,” says Helge Scheil, EVP of Engineering at Fivetran. “This integration further underscores our commitment to providing our customers with a central platform that acts as a data foundation, regardless of the other tools they choose to build with.”

Getting started is dead simple

As with all other interactions involving Fivetran, getting started with dbt Cloud orchestration consists largely of navigating through a series of menus. The “Transformations” section of your Fivetran dashboard will present you with options for Quickstart data models, dbt Core and dbt Cloud.

Choose the dbt Cloud option, authenticate your account with a Service token, choose your project, and select “Save & Test” per usual.

From there you can use the “Add transformation” button to select jobs from your dbt Cloud project.

Set a schedule, either integrated - meaning dbt Cloud jobs run upon completion of upstream connector(s) - or custom if you want transformations to execute on fixed intervals or at fixed times. If you choose integrated, you then select which connectors’ successful data load you want to trigger the job run.