We’re excited to announce Alation’s Fivetran open connector framework (OCF) connector, designed to solve end-to-end cross-system lineage for business users. This add-on connector intelligently adds lineage metadata via Fivetran and end-to-end transparency through dbt Labs to cataloged data sources.

In the modern data stack, a key challenge data analysts face is the need for an end-to-end lineage view of the data pipeline, which is critical for data governance. Analysts want a consolidated view to understand the source data and see where and what changes have occurred. Alation’s Fivetran OCF connector provides that and more. 

In essence, Alation’s Fivetran OCF connector automatically extracts and incorporates metadata from Fivetran to reveal the upstream origins of the data in the destination database. 

Data analysts build reports and dashboards from data stored in central repositories such as data lakes and data warehouses. When analysts extract metadata from these central repositories, Alation is also triggered to extract metadata from Fivetran to provide details about upstream lineage. For transformations that take place within the central repository, the OCF connector by dbt provides all relevant lineage details, enabling full, end-to-end data lineage.

Solving the lineage challenge

A modern data stack is a well-oiled machine where the data seamlessly flows from source to insight. But with so much data coming from so many different sources and being transformed along the way, it’s challenging to properly govern the data without an end-to-end lineage view of the data pipeline.

But the insights are readily available, and Alation combines them. 

Fivetran pulls data from diverse sources to eliminate the need for manual extractions and clean-ups. It acts like a data pipeline concierge by performing tedious tasks and ensuring reliable data delivery. With pre-built connectors for hundreds of sources, Fivetran gets the data flowing quickly and securely, regardless of origin or destination.

Platforms like dbt enable data engineers to transform and refine the ingested data, ensuring consistency and quality for data analysis. As the data reaches the warehouse, dbt empowers the user to define data models and transformations using code, ensuring consistent and reproducible data quality. Thus, dbt offers a structured way to clean, organize and prepare the data for analysis without reinventing the wheel every time.

Alation centralizes metadata from disparate sources, delivering a unified view for organizations to discover, describe, govern and manage their data assets, including data, BI reports and AI models. It acts as your data catalog, centralizing all the information about the data assets by making them easily discoverable and understandable. Alation provides an end-to-end lineage for the data throughout its lifecycle. This empowers data analysts and engineers to collaborate effectively, avoid redundant work and ensure adherence to data governance best practices.

Despite different components providing internal lineage, Alation’s Fivetran OCF connector delivers insights on:

1. End-to-end cross-system understanding of the data sources.
2. Transformations the data has undergone during the course of its journey.
3. A consolidated view to help data engineers troubleshoot pipeline issues or assess change impact efficiently.

Alation’s Fivetran OCF connector in action 

Let us consider a company leveraging Fivetran-dbt as an ELT duo to prepare the data for analysis or consumption.

• Lisa is a data engineer responsible for collecting, managing and converting raw data into information, which analysts can then consume. Lisa’s goal is to ensure that the data is accessible and reliable.
• Jordan is an analyst and works with the finance team. He is an expert in finance, forecasting and estimation, and he needs access to accurate data.

In this scenario, the source of the data is Postgres. This data is extracted and loaded into the Snowflake ecosystem using Fivetran. In the Snowflake ecosystem, this data transforms twice and is eventually leveraged by Tableau.

Here, Alation’s Fivetran OCF connector along with the dbt OCF connector provide a seamless view of the lineage and transformations the data has undergone in the pipeline. Alation’s UI would show:

1. End-to-end lineage: Alation generates lineage between the current data source and other data sources cataloged in Alation that represent Fivetran connectors (upstream lineage).

a. Column-level lineage: The Fivetran OCF connector offers column-level lineage which gives data engineers a view of the origin, access and changes. The end-to-end lineage across the data pipeline can be viewed directly in Alation.

b. Consolidated view using compound layout: The compound layout view offers an unprecedented lineage view between metadata objects based on their logical alignment with data sources. 

c. Transformations: The dbt connector can be leveraged to view the SQL and Jinja code.

The end-to-end cross-system lineage across the data pipeline can be viewed directly in Alation to give data analysts a consolidated view of the data pipeline.

Your lineage challenges solved

Fivetran effectively handles the extract and load stages while dbt models data into analytics-ready tables. Combined, Fivetran and dbt act as a powerful duo that creates a complete and end-to-end pipeline.

The Alation Data Intelligence Platform adds several layers of value with improved data accessibility, quality and governance. Alation delivers trustworthy information that is easy to find, understand and use so data analysts are empowered to make better decisions and ensure a thriving data culture.      

Learn more about the Fivetran OCF connector in Alation’s documentation center.