Embarking on the journey of data migration and modernization can often feel like setting sail into uncharted waters. As businesses race to harness the potential of cloud-based solutions, the urgency to upgrade legacy systems intensifies. Yet, this transformational voyage is anything but one-size-fits-all; it's a complex blend of technology, strategy and adaptability. 

Whether you're aiming to optimize existing systems, embark on full-scale modernization or strike a balance in between, you’ll need a deliberate strategy to help successfully navigate data migration and modernization. So, set sail with us as we explore this journey of transformation together.

Balancing business needs with the complexity of the future 

A lead architect at a large enterprise made the request: “We have to move our core data platform to the cloud now, and we need it done within the next quarter or two.” 

The challenge with this request is that the amount of work required to modernize existing ETL jobs by migrating to the cloud and the urgency of now don’t quite align. 

Embarking on a data transformation journey requires significant effort. Modernizing and migrating existing data and jobs to the cloud is not difficult, but it involves deliberate planning. From assessing your workloads, planning migration, deploying data and workloads, optimizing migrated workloads and more — there’s much to consider. 

These decisions involve considering system dependencies, compounding complexity, the business criticality of the workloads in question, and resource planning. To keep things simple and to complete the minimum necessary work to get a business up and running in the cloud, doing a straight lift-and-shift is typically the first priority of teams, then followed by data modernization.

But, when working with longstanding, legacy organizations, you grow accustomed to offering a multitude of approaches and options, especially when dealing with long-lived data platforms. While the move to modernize is accelerating, dealing with legacy systems and processes is a reality that needs optionality and flexibility. 

New workloads, cost optimization, flexibility and reliability are some of the driving factors behind the retirement of traditional data warehouse appliances, traditional RDBMS systems and Big Data / Hadoop platforms. New and expanding data workloads are taking off, leveraging the capabilities of cloud data platforms such as Google Cloud BigQuery. 

The seamless integration and scalability of BigQuery allows you to save valuable time and resources. Together, Fivetran and BigQuery are helping customers unlock insights to improve customer experiences, informed business decisions and strategies.

Ultimately, the goal is to deliver faster business outcomes and higher value through a sustainable, service-based approach that doesn’t require constant negative engineering oversight. 

When considering migration and modernization, several options are at your disposal. Here are three approaches that have proven effective, either individually or in parallel, depending on your specific needs:

1. The “Lift and Shift” approach 

This is the process of copying an existing application and its data to the cloud with minimal or no redesigning or modification. 

Why it works:

• Minimizes risk and disruption to existing jobs and processes.
• Accelerates on-prem escape velocity to the cloud.
• Preserves assets and investments.
• Minimizes the impact on people and reduces the need for extensive changes.

Before opting for lift and shift, consider these factors:

• The fixed-price model of on-premises may not align with a cloud consumption-based model, necessitating adjustments. 
• New data products will still be constrained by legacy architecture.
• Technical and business debt persists.
• Achieving modern data management principles, such as DataOps, DevOps, Security and Automation becomes more challenging. 

Lean towards this approach when time is the most critical dimension. You can leverage Google’s data warehouse migration framework to streamline your migration path.

2. The “Optimization” approach

The process of recognizing modernization is possible in some areas, but some traditional approaches are still required, resulting in a hybrid approach with lift-and-shift for some parts.

Why it works:

• Allows you to dip your toe into each “world” and optimize on both sides for time, risk tolerance or budget constraints.
• Capitalizes on low-hanging opportunities in the cloud.
• Focuses on the total cost of ownership (TCO) savings rather than a pure lift and shift migration.
• Initiates the process of data consolidation.

Before choosing optimization, consider these factors:

• Balancing speed, cost and skills between both “worlds” can become complex, potentially leading to extended TCO returns.
• Cloud options are more restricted compared to pure modernization.
• Technical debt tends to linger longer than anticipated, so be prepared for that.

Lean towards this approach when time is important, but you have some leeway to incrementally deliver value across your data estate.

3. The “Innovation” approach

The process of fully modernizing ETL to ELT, which ignores lift-and-shift and achieves true modernization.

Why it works:

• Embracing cloud-native solutions, such as Fivetran, that automatically move, replicate and centralize data to and from the cloud.
• Options range from SaaS to Serverless.
• Offers high development velocity and agility, leading to quicker time to market.
• Lowers overall TCO through effective cost controls.
• Eliminates technical debt.
• Fully leverages cloud advantages.
• Reduces infrastructure effort and costs.

Before going down a modernization path, consider these factors:

• The time to delivery is longer.
• Risk can be higher due to changes in existing processes (something that works today).
• Skill sets may need temporary augmentation and long-term retooling.

Lean towards this approach when modernization and removing legacy technical debt are top priorities. Moving quickly (time) is addressed by modernizing incrementally, getting quick, high-value wins and showing the value of your data program.

The opportunity in the innovation approach 

Boris Japes, co-founder and CEO at Census, used a poignant analogy on a recent podcast. He stressed the importance of avoiding building cathedrals and instead opting for “sheds,” or at most, “houses”. 

Build functional “sheds” that show value quickly. Tackling a 20-30 year legacy estate and constructing a new “cathedral” is never recommended.

To determine where you may fall in the continuum, ask yourself the following questions:

1. Which workloads and processes can be migrated with minimal effort?
2. Which processes have issues today and would benefit from a new approach?
3. What workloads are outdated, burdened with technical debt and require a complete overhaul?
4. Are there new workloads that could be easily deployed as cloud services using native cloud platforms?

5a) Do you have …

• Highly integrated data across your existing data platform?
• An independent, standalone data mart?
• Well-designed data and processes using standard ANSI SQL?
• A need to move off of legacy equipment quickly?

5b) Or do you have …

• A data platform with many independent data marts and other data applications that can be moved independently?
• Critical data and processes within your data platform that no longer perform well and need updating?
• New business requirements that can’t be met by reworking legacy processes?
• Changes to your data ecosystem, such as new data workloads, new data replication and transformation approaches, or new data consumption tools and visualization technologies?

Consider your legacy migration and modernization efforts through a lens of the following core dimensions:

• Stay outcome-focused. What do you hope to achieve with your migration and modernization efforts? Keep your goals in mind throughout the process, aiming for high-value, incremental wins with the shortest duration possible.
• Keep it simple. Focus on the most critical workloads and processes first; avoid trying to do too much too soon.
• No supertools. Recognize that there is no one-size-fits-all solution; choose tools and platforms tailored to your specific needs.
• Stay balanced (business and technical). Involve both business and technical stakeholders in the process; it’s not just a technical challenge, but a business challenge as well. 
• Automate … sooner. Identify opportunities for automation to save time and reduce errors sooner rather than later, particularly in data replication, transformation and testing.
• Be realistic. Understand that migration and modernization take time and effort. Avoid expecting overnight success or attempting to tackle everything at once — and don’t build cathedrals.
• Transform in a new way. Don't merely move your legacy systems to the cloud; use the opportunity to transform your data platform and processes in alignment with your business goals, minimizing technical debt.
• Modernize … if possible. If feasible, prioritize modernization as it offers the most advantages, providing flexibility and scalability for future needs.

Lastly, we found it helpful to visually map out desired outcomes, use cases and data workloads to identify the areas of opportunity to focus on first. Below is an example that we used with customers in the past.

The voyage of data migration and modernization is a complex yet vital journey for organizations in today's dynamic landscape. As we've navigated through the strategies, challenges and insights in this blog post, one thing remains clear: there is no one-size-fits-all approach. 

Success lies in careful planning, adaptability and a clear understanding of your unique objectives. 

Remember, this journey is not just about upgrading technology; it's about unlocking the power of your data to drive better decisions and outcomes. So, as you set sail on your own data migration and modernization adventure, trust in your strategy, embrace the changes and navigate towards a future where your data becomes your greatest strategic asset.

To learn more about how Fivetran can help you modernize your application, visit this page. 

Google Cloud also offers a wide range of services and features that can help you improve performance, reliability, security and cost-effectiveness of your applications and workloads.

If you’d like to learn more, visit the Migration Center, or if you’re ready, get started. Request an assessment of your current IT landscape.