Meta-Orchestration of Data Pipeline Tools Across Multi-Cloud and Hybrid IT Environments

Meta-orchestration has become a significant data pipeline management strategy over the past few years.  According to a recent SAPinsider survey on data management in the cloud, IT leaders are under pressure to centralize data from multiple sources for greater intelligence and real-time insight while minimizing costs. That’s no small task.

The cloud offers these leaders a flexible, agile path toward data consolidation across SAP and non-SAP platforms. However, moving to a multi-cloud or hybrid IT approach introduces complexity and requires an update to the way they manage and optimize complex data pipelines.

In this article, we’ll take a closer look at the findings from the survey. In addition, we’ll deep dive into how companies are maturing their enterprise data management strategies to meta-orchestrate the data management tools used along their data pipelines.

Benchmarks for Data Management

In November 2021, SAPinsider released the results of its benchmarking survey, Data Management and Data Warehousing in the Cloud. The report examines the challenges, strategies, and trends in these areas. Common themes include:

• A thoughtful, selective move to the cloud
• The proliferation of numerous disparate third-party data management tools and solutions
• Meta-orchestration to manage cloud and data complexity of the automated processes required to manage complex data pipelines that span both hybrid and cloud-only environments

A Thoughtful, Selective Move to the Cloud

According to SAPinsider, 61% of respondents are architecting enterprise-wide data strategies that include on-premises, private cloud, and public cloud systems.

However, few are making a complete move to the cloud. Companies are selective about their cloud deployments, only making a move when it makes the most sense. Customer relationship management (CRM) systems are most likely to be in the cloud now. Within the next two years, respondents indicate that analytics/BI, ERP, and CRM systems are most likely to migrate to the cloud, if not there already.

While companies are selective about their cloud deployments, the resulting hybrid IT model poses a challenge from an integration standpoint. Consolidating data into a centralized cloud data lake or data warehouse helps but doesn’t address the next trend: an abundance of specialized data tools and solutions that reside in a myriad of internal or external environments.

The Proliferation of Data Management Tools

Tools for data storage, management, and ETL abound — enterprises have an average of 23 tools for data storage and warehousing, as well as another 20 for ETL. Individually, these are all powerful data engineering tools — names like Azure Data Factory, Google BigQuery, Pentaho Data Integration, Informatica, SAP Data Services, and Snowflake are recognizable even beyond the world of data. However, when orchestrated to work in concert with one another, they can do so much more.

Regarding SAP implementations, 90% of the survey respondents need to integrate non-SAP and SAP data. On average, they have 11 other tools that are directly integrated with their SAP solutions. Despite this, only 66% have an enterprise data strategy that spans both SAP and non-SAP data.

Meta-Orchestration to Manage Cloud and Data Complexity

A mature enterprise data strategy almost always includes a data pipeline orchestration approach. After all, an enormous amount of complexity comes with reliably moving data across an enterprise data pipeline. In fact, according to the SAPinsider report, over 40% of enterprises are currently using or actively implementing data orchestration platforms to support the management of their data pipelines.

A service orchestration and automation platform (SOAP) is a type of data orchestration tool. SOAP is a newer category that Gartner introduced in 2020. As an evolution from traditional workload automation, SOAPs are used to centrally schedule and control all automated processes within the applications and databases that are part of a data pipeline.

SOAPs do not replace the tools within your data pipeline. Instead, they act as meta-orchestrators by connecting to each tool and allowing end-users to manage their data tools from a single platform centrally.

SOAPs are ideal for managing data flows across hybrid data landscapes because they:

• Make it possible to create visual workflows of your data pipeline
• Enable agile data management methodologies like DataOps
• Encourage cross-functional collaboration between the teams building and using the data
• Offer real-time event triggers to move data in real-time (versus a batch approach)
• Keep information flowing as it should throughout on-premises and cloud systems
• Allow centralized visibility into errors, and the ability to quickly root-cause problems

The most mature enterprise data strategies recognize that big data isn’t a shiny new thing anymore. Data is no longer a cost center — it’s a key driver of business performance.

Orchestration Specific to SAP Customers

A meta-orchestration strategy helps streamline data flows across SAP and non-SAP environments alike, successfully integrating multiple non-SAP data sources without creating data silos in the process. Data pipeline orchestration also helps data flow smoothly between SAP ECC on-prem, SAP S/4HANA, and RISE with SAP S/4HANA Cloud environments. The numbers in the survey prove it — among those who are most satisfied that they’re meeting business requirements, 78% have an enterprise-wide data strategy for SAP and non-SAP data.

Summary

Need to centralize data from multiple sources, enable real-time data flows, centrally control data tools from a single platform, and minimize data management costs? IT leaders are increasingly looking to SOAPs to help achieve these goals. Most important, SOAPs are designed to meta-orchestrate the automated processes within each of the tools they use across their hybrid IT environment.

For all the significant investments enterprises are making in data and the cloud, it’s no wonder that managing this complexity requires an orchestration solution. If you’d like to learn more about data orchestration, check out the SAPinsider data management benchmark report, the What is a DataOps Orchestration Solution? blog post, and the How to Use UAC to Orchestrate Your Big Data Pipelines webinar.