On Wednesday June 14, the European Parliament adopted its position on the AI regulation by a large majority (499 in...
10 January 2023
Lire cet article en Français
Data Mesh Adoption Framework: Introduction to Data Mesh
Data Mesh is a new data management architecture that aims to improve the governance and management of data within an organization. In this series of articles, we will discover an evolutionary framework for adopting Data Mesh.
The problems we are solving
Most companies have experienced building analytics platforms around data warehouse and data lake architectures. The usual pain points of these architectures are:
Main issues with centralized data architectures
- The central analytics team(s) often lack the knowledge of the business domain possessed by the operational systems’ teams => This results in blurry ownership of topics/data and bottlenecks when implementing use cases.
- Changes in the operational model can have unforeseen consequences in the analytical model => This often results in friction between data and business teams.
- The complexity of governance of central repositories increases with applied regulations, volume, velocity, and variety => This usually leads to issues related to data quality in the analytical models.
- Weak contracts between producers and consumers often lead to strong dependencies, coupling, and possible chaos related to data sharing and reuse.
The pain experienced in multiple data warehouses and data lake implementations leads to a new data management architecture called Data Mesh.
The founding theory
The term Data Mesh was introduced in an article by Zhamak Dehghani called Data Mesh Principles and Logical Architecture in 2019 and is based on four principles that bundle well-known concepts (Domain-Driven Design DDD, Product Thinking):
The principles of Data Mesh and their interplay
The domain ownership principle mandates the domain teams to take responsibility for their data:
- According to this principle, analytical data should be composed around domains similar to the team boundaries aligning with the system’s bounded context.
- Following the domain-driven distributed architecture, analytical and operational data ownership is moved to the domain teams, away from the central data team.
The data as a product principle projects a product thinking philosophy onto analytical data:
- This principle means that there are consumers for the data beyond the domain.
- The domain team is responsible for satisfying the needs of other domains by providing high-quality data => Domain data should be treated as a public API.
The idea behind the self-serve data infrastructure platform is to adopt platform thinking to data infrastructure:
- A dedicated data platform team provides domain-agnostic functionality, tools, and systems to build, execute, and maintain interoperable data products for all domains.
- With its platform, the data platform team enables domain teams to consume and create data products.
The federated governance principle achieves interoperability of all data products through standardization:
- It is promoted through the whole Data Mesh by the governance guild. The main goal of federated governance is to create a data ecosystem with adherence to organizational rules and industry regulations.
Criteria of readiness
While Data Mesh has many potential benefits, it is not always the best solution for every organization. This is because Data Mesh is a complex and potentially expensive undertaking, and it may not be the right fit for all organizations depending on their specific needs and resources.
When considering whether to adopt a Data Mesh approach, it is important to carefully evaluate the readiness of your organization. The following are some key criteria that can help in assessing whether your company is ready for Data Mesh adoption:
Criteria of readiness for Data Mesh adoption
- Data technology at core: Data Mesh is a data-centric approach, so it is important that your organization’s technology infrastructure is able to support it. This means having robust data management systems in place, as well as the technical expertise to implement and maintain a Data Mesh.
- Domain-oriented organization: Data Mesh relies on the concept of data domains, which are defined areas of expertise within an organization. In order for Data Mesh to be successful, your organization should be structured in a way that supports the creation and management of these domains.
- Executive support: Data Mesh requires buy-in and support from the highest levels of an organization. Without executive support, it can be difficult to secure the resources and funding necessary to implement and maintain a Data Mesh.
- Data-oriented strategy: Data Mesh should be viewed as a strategic initiative, rather than just a technical implementation. In order for it to be successful, your organization should have a clear data-oriented strategy in place that aligns with your overall business goals.
- Organization complexity: Data Mesh can be a complex undertaking and may not be the best fit for organizations with high levels of complexity. Before deciding to adopt a Data Mesh approach, it is important to carefully assess the complexity of your organization and determine if it is manageable within the context of Data Mesh.
- Long-term commitment: Data Mesh is not a short-term project, but rather a long-term commitment to improving data management and governance within an organization. In order to be successful, your organization must be willing to commit the necessary resources and effort over the long term.
Overall, these criteria can help you determine whether your organization is ready to adopt a Data Mesh approach. It is important to carefully consider each of these factors and assess your organization’s readiness before making a decision.
Conclusion
Data Mesh is a promising new approach to data management that can provide many benefits. However, it is not the right fit for every organization, and careful consideration should be given to the readiness of your organization before deciding to adopt a Data Mesh approach.
If you still think Data Mesh is the right solution for you, the next part of this series will tackle the iterative approach to implement Data Mesh and its projection on the 4 founding principles.
You want to know more abaout Data Mesh? Read our other posts:
