From experiment to insight: how Dotmatics Luma and Databricks make AI-ready science a reality

From experiment to insight: how Dotmatics Luma and Databricks make AI-ready science a reality | Databricks Blog Skip to main content

Summary

Getting the data foundation right (continuous capture, harmonization, and governance) is what actually unlocks trustworthy AI in science; AI trained on fragmented data can't be trusted.

Luma and Databricks are complementary by design: Luma brings scientific context and instrument connectivity, Databricks brings enterprise-scale storage, governance, and AI tooling, together greater than the sum of their parts.

The winning model is "start where the data pain is worst, prove value fast, then expand," making this a repeatable foundation rather than a one-time integration project.

The gap between scientific data and scientific insight Modern scientific workflows generate data on an extraordinary scale.  A single organization might run hundreds of instruments across wet labs and partner networks. Each produces data, and most of the time that data lives in silos, disconnected from the very decisions it’s meant to inform. The problem isn’t volume, but rather context.  Maintaining the integrity and context of scientific data as it moves across instruments, analyses, and decisions is crucial. When context is lost, scientists spend time reconstructing or repeating results instead of advancing research. When AI models are trained on fragmented, unharmonized data, the outputs can’t always be trusted ( Figure 1 ). Figure 1 . Dotmatics Luma and Databricks transform fragmented instrument outputs into a continuous, connected pipeline of structured, AI-ready scientific data. Closing that gap requires two things working in concert.  A platform purpose-built for scientific data, and the enterprise-grade infrastructure to support it at scale. That's what Luma, Dotmatics' scientific intelligence platform, and Databricks were each respectively built to do. Together, they deliver something neither can provide alone. What Dotmatics Luma and Databricks deliver together Luma is the scientific operating layer for modern R&D.  Luma captures instrument outputs continuously and automatically, without disrupting existing workflows, bringing data into a harmonized, structured scientific record in real time. It can also handle billions of scientific data points daily. That harmonization step is what makes everything downstream possible.  Unstructured raw outputs become structured, FAIR-compliant (Findable, Accessible, Interoperable, and Reusable) data that is ready for analysis, modeling, and AI applications the moment it arrives. Because the scientific record is continuous and structured, AI can be applied to the entire record, identifying patterns across experiments, suggesting what to run next, and even generating plain-language standard operating procedures (SOPs) that scientists can follow immediately. Databricks is the foundation upon which Luma is built.  This provides the scalable, governed infrastructure needed to store, manage, and activate that data across the enterprise. It allows scientific data to sit alongside finance, procurement, and business intelligence systems, connecting research outcomes to the broader organizational context. Delta Sharing enables seamless data exchange with third-party collaborators including contract research organizations (CROs) and academic partners, without compromising governance or data integrity. How Dotmatics Luma and Databricks work better together Luma is purpose-built for science, and Databricks is purpose-built for scalable data and AI.   Luma runs natively on Databricks, so organizations get deep scientific capability and enterprise-grade data infrastructure as a unified stack, not a patchwork of integrations. That unified stack works because each platform contributes something the other doesn’t. Complementary by design.  Luma provides the   instrument connectivity, harmonization logic, scientific context, and a FAIR-compliant data foundation, all built specifically for R&D. The use of open, extensible ecosystems for both biology and chemistry ensures that users are leveraging workflows designed by scientists, for scientists. Databricks brings the data and AI infrastructure, with scalable storage, governance, and the tools to activate that data across the enterprise. Together, the stack is greater than the sum of its parts ( Figure 2 ). Figure 2. Luma and Databricks form a unified stack, with scientific capability on top, enterprise data infrastructure beneath, and AI-ready insight as the output. The result is a faster path to AI-ready science, without sacrificing the rigor that science demands. Luma is built for workflows where data must be auditable, decisions must be traceable, and AI outputs must hold up under scrutiny, spanning everything from early discovery through regulatory submission. That's the standard this partnership is built to meet. An example application area: When chromatography data fragments The typical chromatography workflow is full of operational drag. SOPs can vary across teams and sites, instruments often come from different vendors with their own proprietary data systems and file types, and results are manually exported, reformatted, and loaded into an electronic lab notebook (ELN). This approach can strip out metadata, lineage, and experimental context resulting in difficult cross-site comparisons and underlying data that often ends up buried or inaccessible. This type of siloed data is precisely what we want to avoid when working in an AI environment. Scientific continuity is crucial, and Luma enables this by acting as the orchestration layer that enables faster scientific decisions with continuity across the entire research lifecycle. This includes: experiment design automated job delivery and data acquisition automated chromatogram analysis through connectivity with powerful tools one-click reporting and sharing easy data comparison

Importantly, the metadata, lineage, and experimental context are preserved throughout the digital thread. This is where Virscidian’s Analytical Studio comes in. In 2024, Dotmatics purchased Virscidian, which owns the powerful Analytical Studio chromatography processing software. On its own, this software provides tremendous potential for accelerating drug discovery, due to its capabilities in automating complex liquid chromatography–mass spectrometry (LC/MS) data processing, high-throughput experimentation (HTE), and purification...