“The State of Data Management Report,” a new global survey of 300 data and analytics leaders by Wakefield Research, sports a striking topline result: Enterprise data engineers spend nearly half their time building and maintaining data pipelines. This is a substantial investment, with respondents reporting a median of 12 data engineers who spend 44 percent of their time on ETL. The total average cost? $520,000 per company per year.

That allocation of resources might make sense if:

1. Data management and analytics outcomes were optimal
2. Data engineers could not add substantially more value elsewhere

As Wakefield Research found in its survey, though, neither condition holds.

[CTA_MODULE]

Problematic data, poor decisions

Many data and analytics leaders told Wakefield that their DIY data pipelines were unreliable and yielded error-prone data, and that these problems led to poor decision-making. Specifically:

• 71 percent of respondents said end users were making business decisions with old or error-prone data
• 85 percent said their enterprises had made bad decisions that cost them revenue
• 66 percent said their C-suite didn’t even know this was happening

“It would be one thing if the processes companies used for manually building and managing pipelines were optimized,” Fivetran CEO George Fraser notes, “but 80 percent of those surveyed admit they have to rebuild data pipelines after deployment — due to changing APIs, for example.”

For enterprise data teams, the issues don’t end with the questionable state of delivered data — survey respondents also struggled to get value from that data in a timely way:

• Only 13 percent reported being able to derive value from newly collected data in minutes or hours
• 76 percent said it took days or a week to prepare the data for revenue-impacting decisions, including 74 percent of companies with over $500 million in revenue

The high opportunity cost of DIY pipelines

Data and analytics leaders also clearly understand the high opportunity cost of DIY pipelines — building and maintaining them in-house means that data engineers have far less time to create advanced data models or enable sophisticated analysis.

That may mean, in turn, that business decisions aren’t based on the most powerful or relevant insights, and that business outcomes consequently suffer. “The State of Data Management Report” notes that:

• 69 percent of data and analytics leaders said business outcomes would improve if their data teams could contribute more to business decisions and spend less time on manual pipeline management
• 97 percent said business outcomes would improve if their data teams could spend more time on the analytics behind data-driven business decisions

Dig into the report ... and consider automated ELT

It’s well worth reading “The State of Data Management Report” in full — it includes insights into the impact of DIY data pipelines on business agility, the challenge of training new engineering talent, and the difficulty of affordably scaling up data pipeline production. You might also want to check out our in-depth post on why automated ELT delivers better results for the enterprise than DIY data pipelines.

[CTA_MODULE]