How to Supervise Your Integration Platform: Challenges and Best Practices

In this fifth and penultimate article in a series devoted to the benefits of having an integration platform within an Information System (IS), we’re going to take a look at a very important component, but one that is often overlooked or even relegated to the background when building a platform. And yet, this is an essential component that should be taken into account right from the design stage to ensure both the platform’s resilience and a solid level of information on its health and activity. This is the Observability of the platform.

In this article, we’ll look at the issues and challenges surrounding Observability and then describe the different types of Key Performance Indicators (KPIs) and their characteristics. We’ll go on to explore the Observability tools that Microsoft makes available in Azure and conclude with ways of rendering these indicators via dashboards.

Observability: difficulties and challenges

In the past, with On-Premise or SaaS integration platforms, like BizTalk or Tibco, all tool supervision and message tracking was carried out within the tool itself. However, in Azure, the paradigm shifts, as this platform is made up of a suite of heterogeneous components, which offers new possibilities, but also presents different challenges.

The difficulties with an integration platform in Azure

The three main difficulties with an Integration platform in Azure are:

Resilience and performance

An integration platform is an important part of the information system, playing a central role in interapplication exchanges. This is a critical aspect, one that must be a priority for it to work properly, which will greatly facilitate its adoption by other teams. For this reason, the platform needs to be resilient and high-performance, able to withstand load peaks while maintaining acceptable processing times for business teams.

Traceability of exchanges

One of the main difficulties is the ability to monitor exchanges to ensure that they are processed nominally and to detect any anomalies or failures as quickly as possible, whether in processing or in the partners involved, so as to have as little impact as possible on the activities of the rest of the company.

Cost tracking

Unlike legacy platforms, where the cost was fixed by the licenses required to use the tool, in Azure, costs depend on each component and can vary greatly depending on the choice of components used, the level of service chosen, and the way they are used. This makes it essential to closely monitor the consumption of each component in order to implement scaling mechanisms and optimally adapt the cost of the platform to its actual use.

Integration platform challenges in Azure

Having discussed the main difficulties, we can identify three main challenges that arise from them. Of course, there are also other challenges of varying degrees of importance, depending on the context in which the platform is evolving. The main challenges are:

Monitoring in the various Azure services

In Azure, as in legacy solutions, we no longer have a single tool to monitor, but a suite of heterogeneous Azure services and it’s important to be sure they are working properly and to react as quickly as possible to any breakdowns or failures, whether in PaaS or related services, be they linked to networks, security, or storage.

Message tracking

In the same vein, it’s also important to be able to track each message as it passes through each service on the platform in order to gain an exhaustive view of processing and, especially, to be able to supervise each “point of failure.” One way of ensuring this monitoring is to have a unique correlation ID for each message (even if the message is duplicated or split during processing), which will enable the message to be supervised at each stage of processing and thus facilitate rerunning in the event of failure.

Visibility for consuming teams using the platform

Finally, the last challenge has been inherent to integration platforms for a very long time: for this platform’s consuming teams, it’s often seen as a “black box” where, once a message has been ingested, there is no longer any visibility on the ongoing processing and any anomalies encountered. To remedy this, it’s essential to have supervision solutions designed for both the teams in charge of operating the platform and the consuming (or business) teams, so that they can autonomously monitor the processing of their messages.

To respond to these various difficulties and challenges, we can take a look at the indicators and tools available in Azure for ensuring optimal observability.

Technical vs. functional observability

For platform monitoring, two main types of indicators are used: technical indicators and functional indicators. Let’s dive into their specific features and benefits.

Technical indicators

The technical indicators of an integration platform fall into two main categories: logs and metrics. These indicators provide crucial information on platform performance, availability, incidents, and security. Logs capture the events and incidents that occur within the various services, while metrics provide data on resource utilization, response times, and error rates.

These indicators are intended and useful for technical profiles, including run and operations teams, who use the data to monitor the health of the platform, quickly diagnose and resolve problems, and ensure optimum security against potential threats. By monitoring these indicators, these teams can ensure that the platform remains high-performing and available, capable of responding effectively to the company’s needs.

Functional indicators

The functional indicators of an integration platform, mainly based on logs, are essential for monitoring transactions, detecting errors and exceptions, as well as ensuring data integrity. These indicators are used to monitor the progress of interapplication transactions in real time, identify any anomalies or exceptions occurring during exchanges, and check that transferred data remains consistent and complete. This information is intended for a wide range of profiles, both technical and functional, including run teams, platform consumers, operational staff, and managers. By using these indicators, these teams can guarantee rigorous monitoring of processes, act rapidly in the event of a problem, and maintain optimum exchange quality within the information system. These indicators also provide an overview of how the platform is used by the various IS teams, as well as a complete mapping between the different applications.

Now that we’ve looked at the different types of indicators, we’ll examine the Azure services involved in collecting and storing them.

Observability services for Azure AIS

From a more concrete point of view, we’re going to look at the different departments involved in collecting the indicators needed to make the platform observable. In Azure, there are two main complementary services for collecting indicators (logs, metrics, etc.) from PaaS services:

• Application Insights for Data Factory, Function App, API Management, and Logic App Standard
• Log Analytics for Event Grid, Service Bus, and Logic App Consumption

Diagram of the different KPI collection services based on AIS services

Application Insights is a service performance and availability monitoring service, with the following main features:

• Service performance and availability monitoring.
• Performance and health monitoring: response times, success rates, and query failures to identify pain points and anomalies.
• Collection and analysis of logs, metrics, and custom events: diagnose errors and exceptions and monitor external calls and third-party services used by the service.
• Viewing of interactions within services.

Azure Log Analytics is a log management service used to collect, analyze, and view service telemetry data. Its key features include:

• Data collection: aggregation of logs and metrics from various PaaS services, as well as related platform-critical services such as Application Gateway and Key Vault.
• Log analysis: use of the Kusto Query Language (KQL) to search for and analyze collected data.

Additionally, Log Analytics acts as a reservoir for logs and metrics rendered in Application Insights.

The two components also share common features, including:

• Custom Dashboards: creation of interactive dashboards to view indicators (logs or metrics) in real time.
• Alerts and notifications: configuration of alerts based on specific conditions to alert teams in the event of a problem or to take remediation action.

Next, we’ll focus on one of the major challenges in setting up a platform: viewing supervision indicators. This enables business teams to both see their exchanges and monitor the platform’s consumption and health.

Supervision visualization solution

Based on our experience in guiding our clients through setting up integration platforms, we’ve become convinced that separating technical and functional supervision is essential, as these indicators are not intended for the same user personas and don’t have the same aims and timeframes. Consequently, we’ll dive into two solutions for different supervision needs.

Supervision of technical indicators

The set of dashboards for technical supervision is designed to monitor the health of the platform, a responsibility that falls to one or more teams with technical skills and rights over the Azure ecosystem to remedy any incidents. For these reasons, Microsoft offers a directly integrated dashboard service: Azure Workbooks. These have the advantage of being directly usable from Application Insights or Log Analytics at no extra cost. These are very appealing, since it’s possible to use ready-made templates supplied by Microsoft or to build custom templates to meet specific needs for tracking particular indicators directly integrated into the platform.

Example of an Azure Workbook for technical supervision

To learn more about the Azure Workbook service, take a look at our previous articles:

• Azure Workbooks: Azure’s Integrated Dashboarding Tool
• How to Create Your Own Dashboards with Azure Workbooks

Supervision of functional indicators

Unlike technical indicators, functional indicators are intended for a broader range of profiles. Of course, the teams in charge of monitoring the platform are still very much involved, but let’s not forget the platform’s consumer teams either, who also need a view of their interapplication exchanges for several reasons:

• Knowing the health of their exchanges.
• Being able to autonomously search for potential anomalies during processing.
• Having quantified data on their consumption.

So it’s beneficial to take this supervision out of Azure in order to give access to these different profiles, who are not necessarily familiar with Azure and its portal or who don’t need to have access to it for security reasons. For this reason, it’s beneficial to use external tools such as Power BI, custom tools, or market solutions like DataDog, Splunk, and others.

At Cellenza, drawing on our expertise and experience, we designed “Integration Cockpit,” a functional supervision tool specially adapted to the needs of integration, enabling supervision throughout the processing of exchanges.

Example of the Integration Cockpit dashboard

Observability: Key Takeaways

In this article, we examined the challenges involved in implementing observability for an integration platform in Azure, given the heterogeneity of the services that comprise it. So, it’s essential to include the design of this supervision strategy right from the start of the project, just like other essential components, such as security. Technical supervision will make it easier to monitor the health of the platform and keep a close eye on costs. Functional supervision will encourage business teams to adopt the platform, while making it possible to map interapplication exchanges and their volume.

In the next and final article in our series on the reasons for setting up an integration platform and the reasons for doing so, we’ll look at the governance and organization required to bring this project to fruition.

Interested in learning more about developing a modern integration platform? Contact us or read the other posts in this series:

• Why Develop a Modern Integration Platform?
• Why Use Azure Integration Services?

• How Can You Use AIS to Adapt Your Integration Platform to Azure Services?