This is part 3 of a series on SAP ERP data integration. Read part 1 on top ERP data integration challenges enterprises face today and part 2 on the benefits of using a high-volume data replication solution.

To gain maximum value from your SAP ERP data, you need to be able to scale your data pipelines to support ever-increasing volumes of data, integrate that data with unstructured data systems and create real-time insights. Otherwise, you risk inaccurate and late business decisions and poor outcomes. 

The best and most cost-efficient way to achieve this goal is through high-volume data replication that uses log-based change data capture (CDC) and a data cloud platform.

Because the data volume generated by an SAP system is massive, any data replication solution must also be able to handle high volumes of data. In addition, to be able to realize near real-time access to your ERP data, the replication solution must also be able to capture changes from the SAP source and deliver that changed data efficiently, without much delay, to a target system, such as a cloud data (warehouse) platform.

Why use Fivetran’s high-volume data replication to convert SAP ERP data into business value 

Fivetran offers a best-in-class high-volume data replication solution with a log-based change data capture (CDC). Whether you’re on the latest S/4HANA implementation or using an ECC deployment based on non-HANA database technology, Fivetran can help. 

Our data replication technology first identifies record modifications, such as inserts, updates and deletes, from the database transaction log and then asynchronously processes those captured changes for downstream replication into the target source, such as a cloud-based platform. Data in cluster, pool and long-text tables can be made transparent for use in the target source during the pr. Not only does this allow real-time processing, but also has close to zero impact on the SAP database or application, because it doesn’t login on to both.

Fivetran provides out-of-the-box capabilities to facilitate downstream ELT:

“Soft delete” is a transformation to mark a row as deleted when it’s physically deleted on the source. The soft-deleted row can easily be identified on the target ODS (open data stream) as a delete that must be processed by ELT/ETL downstream. Without the soft delete, the alternative would be to use a resource-intensive query to identify what data is currently available in the target but no longer in the source.

Time series, also known as Fivetran’s history mode, appends all changed data as a new record in the target. This means that there is a full transaction log of all changes that happened on the source.  

It’s the ability to include special metadata from the source like the commit sequence number from the source, or the commit timestamp. This would allow any downstream ELT to make the best effect of source metadata.

Top benefits include:

Performance: Fivetran is highly efficient at capturing the data change on the source system to support large volumes of data replication. The agent technology allows the best use of available resources between source and target. 

Automation: The data validation feature ensures data is accurate and in sync. It also includes automated data extraction and load process from source to target.

Scale: It's easy to add additional data sources like systems and tables to various cloud destinations. 

Security: Fivetran follows industry standard practices to encrypt data in transit and at rest and provides robust security, privacy and governance from source to destination and in compliance with GDPR HIPAA, ISO, PC and SOC 2.

• AES256 encryption throughout the entire data replication pipeline, using industry best practices including a wallet and unique encryption certificates
  
• Support for OS-level and LDAP authentication, as well as custom plugin-based authentication
  
• 2-Step verification support with unique certificates, avoiding a so-called man-in-the-middle attack
  
• Pre-defined authorizations with full rights to make modifications, execute-only privileges, and read-only access
  
• Auditing/logging of all operations that may affect data replication
  

How Snowflake’s Data Cloud supports high-volume replication

Snowflake’s Data Cloud is a very good platform for managing, integrating and analyzing  SAP ERP data. Especially on performance and scalability does Snowflake offer flexibility with compute resources that scale linearly and the way snowflake storage can grow. Furthermore Snowflake has the ability for data sharing. A simple but powerful feature that provides live access of your data stored in Snowflake. It removes the need for copying data to other targets. Last Snowflake offers a secure and resilient environment by leveraging the most sophisticated cloud security technologies available.

Snowflake's platform, combined with Fivetran's high-volume data replication solution, allows you to spend minimal effort integrating and building and managing data pipelines while achieving near real-time access to your data.

Enterprise use cases for real-time access to SAP ERP data

The combined power of Snowflake and Fivetran can support a number of beneficial use cases across multiple industries. For example, when manufacturers can gain real-time access to ERP data and integrate it with other data sources like warehouse systems and supply chain data, they can optimize inventory management and improve reactions to disturbances to the supply chain.

Similarly, retailers can reduce product returns and increase customer retention by automating data ingestion and eliminating manual processes and errors. With accurate, detailed data they can best personalize the customer experience.

Here are some other common use cases for SAP ERP data that a high-volume data replication can support.

How Pitney Bowes achieved real-time SAP analytics with Fivetran and Snowflake

Fivetran and Snowflake have a track record of delivering successful consolidated real-time analytics solutions. Pitney Bowes, a global commerce services provider, chose Snowflake for its ability to deliver powerful analytics and Fivetran for its fast and continuous delivery of data from disparate sources.

Fivetran allowed Pitney Bowes to replicate its SAP ERP data continuously and provided low-latency log-based CDC, which enabled it to move large volumes of data without affecting the core application. In addition, most of the replication management is handled within the Fivetran interface itself, eliminating the manual oversight required for batch processing.

The Pitney Bowes Enterprise Information Management team has seen significantly faster processing times for both their Oracle and SAP systems. ETL jobs that previously took days now occur in less than an hour. And the technical footprint of the data acquisition process was also minimized.

Ready for real-time access to your SAP ERP data? Snowflake Partner Connect lets you quickly and easily create a Fivetran trial account that integrates with Snowflake and lets you deploy Fivetran connectors in under five minutes. Start your free Start your free trial today. 

Maximizing SAP ERP data with high-volume data replication

DOWNLOAD