Data Mesh
Data Mesh: A Comprehensive Guide
Inmon’s "Local vs. Global" concept was the prophecy; Data Mesh is the modern fulfillment.
Data Mesh was introduced by Zhamak Dehghani (around 2019) to solve a specific problem: The Monolithic Bottleneck. As companies grew, the central Data Engineering team became a bottleneck because they didn't understand the domain data (Marketing, Finance, Logistics) as well as the experts in those departments did.
The 4 Pillars of Data Mesh
1. Domain-Oriented Decentralized Data Ownership
The Concept: Instead of one massive "Central Data Team" owning all the tables, ownership is pushed out to the business teams (Domains).
How it works:
The Checkout Team (Software Engineers) doesn't just write the app; they also own the Checkout Data Products.
The Logistics Team owns the Shipment Data Products.
Inmon Translation: This is exactly Inmon’s "Local Data Warehouse" concept. The people who create the data are responsible for cleaning and structuring it because they know what it actually means.
Why? It removes the bottleneck. You don't have to file a ticket with the "Data Team" to fix a column; the domain team fixes it themselves.
2. Data as a Product (DaaP)
The Concept: Domain teams shouldn't just dump raw logs into S3 and say "Good luck!" They must treat their data like a consumer product (like an iPhone).
The Product Features: A Data Product must be:
Discoverable: Registered in a catalog.
Addressable: Has a permanent unique ID/URL.
Trustworthy: Has SLAs (e.g., "Updated every hour, 99.9% uptime").
Self-Describing: Includes documentation and schema.
Inmon Translation: This is the "Summarized/Integrated Layer" but built by the domain. Instead of a messy table, they publish a clean, versioned "Product" for others to consume.
3. Self-Serve Data Infrastructure as a Platform
The Concept: If every domain team (Marketing, Sales) has to build their own Spark cluster and Airflow server, it will be chaos.
The Solution: A central "Platform Team" builds the infrastructure (the roads), but the domains drive the cars (the data).
What they provide: A "vending machine" for infrastructure.
"I need a Snowflake Schema." -> Click a button -> You get one with permissions set up.
"I need a Data Catalog entry." -> API call -> Registered.
Key Benefit: It hides the complexity of the underlying tech stack from the domain experts.
4. Federated Computational Governance
The Concept: If everyone does what they want, you get anarchy (Inmon’s "Spider Web"). You need rules that are enforced automatically by code, not by a committee meeting.
The "Federated" Part: The rules are decided by a group of representatives from each domain (like the UN), not a dictator.
The "Computational" Part: The rules are automated.
Rule: "All Personal Identifiable Information (PII) must be encrypted."
Implementation: The Platform automatically rejects any Data Product that exposes unencrypted PII.
Inmon Translation: This is the "Corporate Data Model" and "Metadata Repository" from Chapter 6. It ensures that even though data is distributed, "Customer ID" means the same thing in London and New York.
Key Terminology Cheat Sheet
Data Product
A clean, managed dataset + metadata + code + access policies. It is the unit of value.
A "Gold" Table with an API and an SLA.
Domain
A business boundary (e.g., "Orders", "Users", "Inventory").
Inmon’s "Subject Area".
Data Quantum
The smallest unit of a Data Mesh (Code + Data + Metadata).
A Microservice, but for data.
Polyglot Storage
Storing data in the best format for the job (Graph for relationships, Relational for finance).
Inmon’s "Multiple Platforms for Detail Data".
The Comparison: Monolith vs. Mesh
Ownership
Central Data Team (Bottleneck)
Domain Teams (Decentralized)
Architecture
Ingest -> Clean -> Serve (ETL pipeline)
Domains publish Products -> Consumers subscribe
Governance
Top-down (Central Control)
Federated (Global standards, Local autonomy)
Thinking
"Data as an Asset" to be collected.
"Data as a Product" to be sold/served.
When to Use Data Mesh?
Do NOT use Data Mesh if:
You are a small startup.
You have < 3 domains.
Your data team is < 10 people.
Why? The overhead of building the "Self-Serve Platform" is huge. It solves a scale problem you don't have yet.
Use Data Mesh if:
You are a large enterprise (e.g., Netflix, Uber, JPMorgan).
The central data team is a massive bottleneck.
Domain teams are technically capable of managing their own data.
Connection to "Inmon" Knowledge
You can think of Data Mesh as Inmon's Distributed Data Warehouse (Chapter 6 of the book "Building the Data Warehouse") combined with Product Thinking.
Inmon said: "Distribute the warehouse to local teams to remove bottlenecks."
Zhamak added: "And make sure those local teams treat their data like a product, using a self-serve platform."
Last updated