Leveraging Pyramid Analytics to Migrate to Data Mesh Architecture

This Blog has been written by Mastaneh Nikroo  from our Partner Reply.

In today’s world, data plays a crucial role in organisations and is one of the determinants of their success. All businesses and enterprises are focusing on their data management approach and creating added value through their data products. Data is growing at a faster rate than ever due to new technologies such as IoT and Cloud. Indeed, 90% of the world’s data was created in the last two years and every two years, the volume of data across the world doubles in size.

By growing the size of organisations and also the amount of data they produce or consume, the existence of efficient data management and data architecture is inevitable. An inefficient data architecture creates bottlenecks in accessing the necessary data for different teams and increases the environmental footprint of the data storage. By optimising the data flows and accesses, Data Mesh creates value for the enterprise, all stakeholders, and the environment.

In this article, we will explore Data Mesh Architecture and how Pyramid Analytics can be used to migrate to this data architecture. The first section introduces the concept of data mesh and its importance and necessity in data governance. The next section explores the breadth of tools in Pyramid Analytics’ platform that can help facilitate with the transition to a Data Mesh Architecture.

What is a Data Mesh

Originally, organisations used central data teams to manage their data, however this process proved to be inefficient in large organisations with different domain teams, as a central data team would often become a bottleneck by being unable to grant access to necessary data sources for different domain teams on time.
As a result, Data Mesh Architecture was developed as a much more streamlined way of managing data. Data Mesh decentralises the ownership of data from the central team to domain teams and allows the domain teams to conduct cross-domain data analysis on their own. This results in domain-orientated decentralisation which creates better and more efficient access to data sources in each domain team, which in turn creates business value.  The Data Mesh Architecture generates a platform for creating, monitoring, discovering and accessing data products.

Main Characteristics of Data Mesh

There are four main features that distinguishes a Data Mesh when compared to other types of data architecture:

• Domain Ownership: This shifts the responsibility and accountability of the data to the domain teams.
• Self-Serve Data Platform: This feature provides access to high-level infrastructure abstractions for users.
• Data as a Product: It solves the problems related to data quality and data silos. According to this principle, data is not a siloed product but a business product that benefits different stakeholders.
• Federated Computational Governance: creates shared responsibility between the domains and the central Data team and allows adequate autonomy for each domain.

Thanks to these principles, each domain team will be in charge of its own operational and analytical data, data security, and compliance. Data Mesh enables the domain teams to be able to ingest operational data, create their own analytical data, and grant access for other teams to use their published data.

Using the Pyramid Tenant System to Migrate to Data Mesh Architecture

The Content Management System (CMS) in the Pyramid platform facilitates the migration to Data Mesh Architecture. CMS provides the possibility to define different groups of users and refer to each group as a tenant. All users in a tenancy share the same privileges in terms of accessing the software. Besides, the Multi Tenancy feature enables different tenants to co-exist on a single instance of the platform and the central data team has visibility and control across all tenants. 

In addition, CMS provides the chance for the central data team to assign domain admins who can freely manage all the users, roles, data sources, profiles, logs and licencing within their respective tenant domains. Content Management System (CMS) in the Pyramid platform facilitates the migration to Data Mesh Architecture. CMS provides the possibility to define different groups of users and refer to each group as a tenant. This allows different tenants to coexist on a single instance of the platform and the central data team has visibility and control across all tenants. In addition, the CMS admin (enterprise admin) provides the chance for the central data team to assign domain admins who can freely manage all the users, roles, data sources, profiles, logs and licensing within their respective tenant domains. A tenant admin can set up tenant-specific data source connections which cannot be viewed or accessed by users in a different tenant. Users within a tenant can be added to specific roles which the tenant or enterprise admin can use to control row-level security to the connected data sources.

How would this look like in the CMS:

Overview of total available Data Products in the System

Available Data Sources in the Data Landscape

Selection of Curated Data sources with Access Management and Metadata Information

Cross-Tenant Content and Roles

Cross-tenant content refers to the content seen and accessible by different tenants without having to create duplicates and it can only be set and managed by the Central Data Team. Cross-tenant roles allow users to join roles outside of their own tenancy. This feature allows domain admins to add and remove users, from outside their domain/tenancy, to existing  roles without Enterprise admins.

Conclusion:

In summary, Data Mesh Architecture offers a much more efficient way of managing data and through Pyramid’s CMS and other features such as Multi-Tenancy, the transition to data mesh can be achieved with relative ease.

by Mastaneh Nikroo from our Partner Reply, read the full whitepaper on DataMesh and ESG here: https://www.reply.com/data-reply/en/Data-Mesh-ESG-and-Leveraging-Pyramid-Anaytics