For a few years now, I’ve been asked the following question about the management and activation of digital and marketing data on a regular basis:
Should the management of marketing data be decentralized or centralized (BI/IT team vs. marketing team)?
Before starting to discuss the issue, you should understand that this debate between centralization and decentralization has always existed. The giants of digital (of which Netflix was the precursor) have noted over time that adopting a decentralized (microservices) architecture, which is by nature more flexible, enables a faster development time for their products and digital applications, in addition to fostering innovation internally.
Microservices architectures are characterized by independent services that can function autonomously, often developed by small teams. On the other hand, monolithic architectures are more rigid.
The following charts illustrate the structural difference between these two types of architecture.
Representation of monolithic architecture
Representation of microservices architecture
Domain-driven design, a software development methodology, has had a strong influence on microservices architecture. Basically, this architecture type is modeled on a business domain which in turn defines the reasonable scope of the service being designed.
To provide an example, if order management is the targeted domain, a microservice devoted to automated management (creation, modification or cancellation of an order) can be developed. However, the service will not offer payment functionalities, since managing these transactions would belong to another domain that is independent of ordering.
At the organizational level, this can result in the use of small decentralized technical teams whose primary goal is to produce high-quality services within a specific business domain.
It’s important to understand that different domains still intercommunicate—otherwise it would be impossible for a company to produce a consistent product. Basically, no matter how many domains this type of architecture is based on, its complexity is invisible to the end user.
The data mesh
Data mesh is a new model inspired both by microservices architectures and domain-driven design and it has enabled many technology firms to grow at a rapid pace. It encourages decentralized data management organized by business domain.
The main goal of a data mesh is to reduce the amount of time between the collection and exploitation of data (the lag that exists between insight and activation). In other words, it aims to reduce the distance separating operational and analytical data.
Data producer, data owner and data consumer are very important terms in this model. The data producer is the person generating the data; the data owner is the person to whom the emitted or produced data belong; the data consumer is the user. The idea of a data mesh is to decentralize data management for the data owner, who is sometimes also the data producer and one of the data consumers. In other words, it requires the development of a data ecosystem that can support these different roles.
A data mesh is often a component of a data strategy, since its implementation requires a culture, processes, talent, technology and, lastly, investment.
Data mesh basics
A data mesh is an architecture based on the following four main principles:
1. Data must belong to a specific business domain and be managed within this domain
In this model, data must reside within the business domain to which it belongs. The domain (operations, customer, marketing, financial, human resources, product) is therefore the main owner of this data and, as a result, is responsible for its valuation process.
2. Data are treated like a product
Each domain has the responsibility of developing high-quality data “products” that enable other domains to easily discover, understand and use the data, or in other words to make the data consumable.
3. Infrastructure must be created in “self-serve” mode
To be able to build and deploy this type of model, self-serve tools must be made available. These will allow you to mask the complexity of the design process for data products.
4. Governance must be federated with automated policy management
Using APIs, “products” from different domains can exchange their data. Given that the interoperability and discoverability of data is crucial, it’s imperative to establish global, standardized rules that will support seamless exchanges. Product managers are required to define these overall rules and oversee their application. Adopting a data mesh entails one difficulty: determining whether identified policies and standards should be centralized or decentralized.
Why is this being talked about now?
In 2022, with the acceleration of the shift to digital, many organizations want to orient themselves towards intelligence and data, meaning companies are looking to quickly enhance their data to support their growth and achieve their business ambitions.
The reality is that traditional data architectures can’t respond fast enough to such a large volume of demands. What’s more, organizations sometimes house domains that use huge volumes of complex, diverse and changeable data (such as marketing, for example). Traditional architectures are often incapable of supporting the achievement of business objectives as quickly as companies would like, and the inability of an organization to meet these new challenges could lead to a negative impact on their strategic advantage.
This desire and need to accelerate the data valuation process is exactly what the data mesh architecture was created for.
Most companies want to use their data to improve customer experience, discover new business opportunities, reduce their operating costs, help their employees make better decisions and share their data with partners in order to ultimately enhance their service offering.
These ambitions are rarely all attainable without a data strategy. A data mesh must be a serious consideration for IT and business intelligence teams, because the true value of analytics is in the activation of the information collected. Organizations that are able to reduce the time required to exploit data will be able to benefit from a real return on their investment in data initiatives.
On the other hand, if your company is already reaching its targets in terms of data, using a data mesh isn’t necessary.
According to data mesh evangelist Zhamak Dehghani, the main reasons driving companies to adopt this model are:
- expectations that remain unsatisfied given the data;
- the volume, diversity and complexity of data sources;
- evolving use cases for data;
- a shifting business context requiring flexibility and agility;
- disillusion with traditional models after unrewarding investments in data infrastructure that didn’t have a positive impact on company results.
If after several years your company has still not managed to exploit its data, you might be interested in considering this new data management philosophy, especially if your company is already using microservices architecture as part of its operations.
Marketing data are good data mesh candidates
Marketing seems to be a good candidate for application of the data mesh idea. Marketing data are usually voluminous and diverse—for example, they include advertising data (Google Ads, Facebook Ads), web and mobile behavioural data (Google Analytics, Adobe Analytics), and attitudinal (surveys) and financial data (revenue and profit by client or product).
These data are changing and evolving (especially in digital) because a lot of marketing data come from the platforms that produced them. Google generates data from its advertising platforms, as does Facebook and Salesforce. Comprehension of these data is linked to the originating platforms. The metrics presented in the platforms are not standardized (in terms of the way in which they’re calculated) and are therefore often not intercomparable.
In digital marketing, transitioning from holding information to taking action is crucial. The end goal of a digital initiative is often the optimization of customer experience or of marketing and advertising campaigns. IT often finds it hard to understand these types of data since it is neither the owner nor the consumer. This can lead to delays in establishing strategies to enhance marketing data.
Integrating a data mesh into marketing involves clarifying responsibilities. As a result, marketing should be the sole manager and owner of the data generated by its domain. In order to be able to consume these data, marketing must be treated as a product, which means creating a solution that enables the discovery of the data but also the ability to use them easily. The product idea involves a Service Level Agreement or SLA, so the department must guarantee a minimum level of service and quality that is satisfactory for the final consumers.
The creation of data products cannot happen without a reliable technological infrastructure. To this end, a few vendors such as Google and Snowflake have already established technology to support the data mesh model.
The marketing department must be part of the federated governance of company data. It must participate in the establishment of governance rules and policies that enable the sound management of data in every sector of the company while preserving the special features that belong to marketing. Confidentiality, protection, quality and accessibility of data are the pillars of sound management.
I hope this article has piqued your curiosity about data mesh, a recent data architecture paradigm that aims to tackle a persistent, central challenge within analytical science: acceleration of the cycle of data development.
