Why R&D Data Belongs in the Lakehouse - and Why Agents Need It There

Why R&D Data Belongs in the Lakehouse - and Why Agents Need It There | Databricks Blog Skip to main content

Summary

Cellcentric’s Data Hub is a governed context layer for data and AI, built on Databricks using Unity Catalog and Lakehouse Federation, which provides a single user interface for employees and an MCP server for agents.

It solves the core data engineering problem of integrating scattered R&D data from various source systems (like IoT telemetry, SAP, and MES) into a unified, AI-ready product, which is the hard requirement for industrial AI.

By making documentation a first-class quality metric, the platform dramatically accelerates complex R&D investigations from weeks to days, delivering cumulative value as each new data product adds reviewed business context to the platform.

The setup At cellcentric, a joint venture of Daimler Truck and Volvo Group, we develop and build hydrogen fuel cell systems for heavy-duty applications. Our work is R&D-heavy, engineering-heavy, and data-heavy. The questions our teams ask rarely fit inside one source system. They cross product hierarchy, manufacturing history, rework, laboratory evidence, test telemetry, and domain knowledge. That is the central challenge for industrial AI in R&D. An agent is only useful if it can reason over the same governed context an engineer needs to trust an answer: where the data came from, what it means, how complete it is, which caveats matter, and whether the user is allowed to see it. Turning that into AI-ready context starts with data engineering before model selection. That is why we spent the last four years building our data foundation on Azure and Databricks. Unity Catalog was part of the architecture from the beginning. Lakehouse Federation brought on-premises SQL sources into the lakehouse pattern. Delta Sharing helped us exchange data across boundaries. Databricks Asset Bundles gave us a production path for pipelines and data products. The result is the Data Hub : our governed context layer for data and AI, with a user interface for employees and an MCP server for agents. The Data Hub began as a data products platform. In hindsight, that foundation is exactly why it works as an AI platform. The Fuel Cell Passport The foundation is easiest to explain through one data product: the Fuel Cell Passport . It brings together five enterprise source systems, including SAP S/4HANA, two MES systems for manufacturing and rework, a laboratory database, and an IoT telemetry platform. It models seven hierarchy levels from system down to raw-material batch, refreshes daily, and uses a state-based temporal model so teams can answer both point-in-time configuration questions and full rework-history questions. Daily data quality checks monitor whether the product is complete enough to support engineering, quality, and manufacturing investigations. The name reflects the traceability and lifecycle concepts of a product passport, but the Fuel Cell Passport is an internal engineering data product - not a regulatory compliance artifact like the EU Digital Product Passport (DPP), though the same traceability foundation would support one.

Figure 1: Five enterprise source systems converge through Unity Catalog into the Fuel Cell Passport data product.

That sounds like a conventional lakehouse success story: integrate the sources, model the domain, govern access, make the data reusable. For AI, the structural point is the attached context: a governed product around the modeled data. Context as a quality metric Unity Catalog gives us the governed structure: tables, columns, ownership, lineage, classifications, and permissions. The Data Hub adds the product layer around it. A data product combines ownership, lifecycle state, domain assignment, linked governed assets, and context entries that explain what the product is for and how it should be used. That distinction matters for AI. Agents need more than schema metadata, and they need more than a document. They need to understand which business questions a data product supports, how the important tables relate, what caveats matter, and which surrounding assets should be used with it. We persist that richer context in the data product layer, including extensive markdown catalog entries that are written during the development process while the project context is still fresh. This changed how we think about data quality. Completeness and freshness still matter, but they are no longer enough. For AI-ready data, context coverage has become a first-class quality metric. In our marketplace, every data product carries a context-coverage badge. It shows whether table descriptions are present and how much column-level documentation exists across the product's attached tables. The badge made the metric visible and actionable. Coverage went up because engineers could see the gap directly in the product surface where the data would be consumed. The metric is not a claim that every semantic ambiguity has been solved. It is a practical proxy that makes missing context visible early enough to fix. Historically, context coverage improved because the badge turned documentation from an after-the-fact cleanup task into something engineers could see, measure, and improve as part of delivery. Today, we have 27 published data products, all with rich markdown documentation. Published products average 90% column-comment coverage, with most column comments AI-assisted during data engineering work and reviewed by the engineer before merge. The catalog entry adds another layer of context: a long-form markdown summary of the development workflow, the domain decisions, the table relationships, the caveats, and the intended ways to consume the product. The important shift is that documentation becomes part of the engineering workflow, then becomes available as structured product context for the next human or agent that needs to reason over the data. One UI for humans, one MCP for agents The Data Hub is the layer that makes that substrate consumable. For human users, it is a marketplace and workbench. Employees can discover data products, see owners and lifecycle status, open linked dashboards and apps, query Unity Catalog-governed data, and use a chat interface for natural-language exploration. For AI clients, the same context is exposed through MCP. Any MCP-aware coding agent or assistant can access the same Unity Catalog metadata and data product context that the Data Hub uses.

Figure 2:...