SpotOn serves 5x faster reporting with Fivetran Transformations for dbt Core™

Key results:

• Fivetran helped SpotOn’s team centralize 30+ databases into Snowflake
• dbt Core helped them reduce thousands of lines of code into scalable data models
• Fivetran Transformations for dbt Core decreased reporting development time by weeks, while increasing automation

Background:

• Company Size: ~2000 employees
• Industry: SaaS
• ELT: dbt Core™, dbt Cloud™, Fivetran Transformations for dbt Core™‍
• Destination: Snowflake‍
• Cloud platform: AWS‍
• Use case: Powered by Fivetran‍
• Fivetran Connectors: MySQL, Postgres, MongoDB, DynamoDB
  

SpotOn is one of the fastest-growing software and payment companies with comprehensive, cloud-based technology for businesses of all types and sizes. With the acquisition of Appetize, SpotOn’s client base grew to include enterprise venues like Madison Square Garden,  Fenway Park, and MetLife Stadium, all with unique needs and complex data.

The SpotOn product team needed a way to efficiently turn their captured customer transaction data into fast, reliable, informative reporting for their clients. In parallel, their internal data team needed a way to harness data to optimize internal operations.

Before Fivetran

As the company scaled, turning data into reporting for customers and internal stakeholders grew more complex - collecting client data across 30, unconnected MySQL databases. Simultaneously, the engineering team used various databases to capture internal growth metrics but lacked a central repository where they could more efficiently generate reporting. 

When they did have the necessary data, it was transformed using stored procedures in Snowflake. But, as their data grew, so did their models, and eventually, there were over 2,000 lines of code behind a single table. To further complicate things, any changes made were not automatically monitored or logged without version control, making QA time intensive - upwards of 15 hours a week. And scaling required writing code from scratch for each new use case, as they didn’t have a modular approach to data modeling.

This ultimately cost SpotOn money and time: they were running heavy models in Snowflake, thus accruing costs, and spending resources on basic modeling processes. It was a resource-heavy process that delivered adequate but suboptimal results.

"With our previous solution, we had a trade-off between long load times or out-of-date data. Neither of these was satisfactory, so we needed a new solution. As a smaller dev team, we needed a solution that didn't require full-time data maintenance. We needed a solution that could be set up in weeks, not months." - Tom Gilbertson, Product Manager, Data & Analytics

The negative impacts on the business were notable:

• Inability to scale quickly to meet growing customer needs
• Complex logic that was slow and costly
• Resource-intensive processes to provide customers necessary reporting

Driving scalability and quality with Fivetran and dbt

These challenges inspired SpotOn to look for a new solution. And that’s when they turned to Fivetran, Snowflake, and dbt. This was ultimately an easy choice – they wanted the best-in-class products and knew the combination of these solutions would have an outsized impact without a significantly higher cost than their competitors.

Fivetran moves all of their clients’ customer data seamlessly from 30 disparate MySQL into Snowflake. This provides a secure foundation of data that can then be transformed at scale to meet their various analytical needs, creating two distinct use cases enabled by separate dbt technologies: client reporting and data-driven growth.

Fast, reliable client reporting:

With Fivetran Transformations for dbt Core, SpotOn automates faster, more reliable reporting for their clients, while saving time and cost. 

dbt Core modularizes the 2,000 lines of transformation code, making data models more readable, easier to debug and scalable with jinja templates. By integrating their dbt project into Fivetran, Fivetran Transformations for dbt Core can orchestrate those model runs automatically post-Fivetran connector load – no custom scripting, third-party tools, or DevOps required. 

SpotOn now manages and automates their entire ELT process at scale from one platform. Client-facing reporting is now generated faster, cutting down the dev time by several weeks. Clients now have always-on and reliable access to their data, ensuring no opportunities or optimizations are missed, expanding SpotOn’s wide portfolio of happy, data-driven customers.

"By integrating Fivetran and dbt (with Fivetran Transformations for dbt Core), Fivetran becomes not only the ELT tool but also the orchestration tool, which is great for reliability, scalability, and dev time." - Tom Gilbertson, Product Manager, Data & Analytics

Collaborative, data-driven growth:

It’s not just customers that reap the benefits of the modernized data stack but the internal team as well. SpotOn’s data engineering team of 2 supports their 10+ data analyst team by using dbt Cloud to turn large volumes of data from multiple databases - moved by Fivetran - into powerful internal analytics and BI. The web-based UI allows all team members to collaborate, regardless of their comfort-level with an IDE or command line. dbt Cloud also provides scheduling and built-in alerts, abstracting away the manual responsibility from the team.

This ensures that the team can continue to power data-driven growth without having to scale their headcount or sacrifice data integrity. By using cloud-based technologies, the entire team can participate in the analysis – expediting the time to insight while cutting down development time by 5x.

And they know the insights they provide are accurate, thanks to Metaplane. Metaplane monitors the pipeline and alerts if there are upstream issues in the application databases Fivetran extracts from. 

• With Fivetran, dbt, and Snowflake development time is cut down by weeks
• Fivetran moves data from 30+ databases into one centralized cloud data warehouse, Snowflake
• They write efficient, scaleable, governed data models with dbt Core
• Fivetran Transformations for dbt Core orchestrates and automates their data transformations, no third-party tools required
• dbt Cloud gives the lean engineering team of 2 a way to collaboratively produce internal analytics and BI 5x faster

"dbt allows us to version control our stored procedures, which previously didn't exist. We can take a more software engineering approach to data engineering," Gilbertson said. "Fivetran gives us a centralized, one-stop shop for our entire ELT pipeline – no third-party tools required. That made our processes more efficient and cut down on monitoring."

Outcomes

Ultimately, Fivetran Transformations for dbt Core decreased development - and development operations time - by a few weeks, while providing automation. SpotOn has also reaped additional benefits:

• Ability to scale, with a lean team, as their clients, and their data needs, grow
• Automated version control, data lineage graphs, and documentation ensure data integrity
• Synchronized connector load and model runs reduced computational costs 
• Reduced tech stack complexity_‍

Download this IDC report to learn how Fivetran drives millions in financial impact and enables new business initiatives for enterprises.

_{*dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.}