Governance beyond security: knowledge, context & ontology on the lakehouse

Governance beyond security: knowledge, context & ontology on the lakehouse | Databricks Blog Skip to main content

Summary

The governance artifacts most teams treat as compliance overhead — classification tags, de-identification policies, data contracts, model cards, lineage — are the raw material of enterprise data semantics; the audit work you already do becomes your AI foundation.

A catalog-centered agentic lifecycle lets AI agents build, test, de-identify, and deploy from curated Unity Catalog metadata, certifying both the data product and the agent through shared human gates. Production PHI never leaves the governed boundary.

Because meaning and context live in the catalog instead of the more expensive LLM tokens, cheaper models can serve most needs with more trust.

Ask most organizations what data governance for AI means, and you’ll hear a security answer: lock it down, restrict access, pass the audit. In healthcare, security is non-negotiable — but it’s incomplete. Security tells you who can touch data. It says nothing about what the data means , whether it can be trusted , or whether an AI model should ever learn from it . Our Data Empowerment Program (DEP) starts from a different premise: governance is knowledge, context, and ontology ; not just controls. Artifacts most teams treat as compliance overhead, such as classification tags, de-identification policies, model cards, and data contracts are raw material for enterprise data semantics. When seen this way, you are not choosing between governance and AI, but instead, governance helps build AI. New approaches to governance need to be implemented in the AI era. The only question is whether you do the work later just to pass the audit, or now, to lay the foundation your AI runs on. Our goal is to show that the security and governance work you already do is the foundation your AI runs on. Govern the data well enough, and AI can run on cheaper models with more trust. Governance must think broader across five pillars through one lens Start with the lens that every governance artifact contributes to semantics. Every classification tag is a concept. Every model card is context. Every data contract is a shared definition. Every lineage link is a relationship. Read that way, the security stack you already run is the first draft of your ontology, and the catalog is where it lives. Governance then stops being one thing and becomes five facets of a single discipline: the data itself and how it’s controlled, the AI built on top of it, the people who need to understand it, the products that carry it into the business, and the shared context that ties all four together. It’s the same lens, but from five fronts. With DEP, we envision semantics through five pillars: Data Governance — Catalog, quality, curation, lineage, and with security & compliance built in such as PII classification, access control, HIPAA/GDPR, and AI-specific privacy risks. Knowledge (AI/ML) Governance — Model documentation, governance, and responsible-AI standards such as bias & fairness, explainability, human oversight, and EU AI Act readiness. Data Literacy — Training, self-service enablement, practitioner certification, and KPIs such as adoption rates, usage metrics, and program ROI. Data Management — Architecture, data engineering, and data product contracts should include schema agreements, SLAs & quality thresholds, and producer/consumer obligations. Ontology — Glossary, taxonomy, knowledge graph - culminates in an AI semantic layer. This includes context for LLMs, RAG grounding, and chat-query readiness.

Operationalizing the vision through agents Our five-pillar vision ends as only slideware unless the platform can carry it out into something operational. Once your governance artifacts live as structured, machine-readable metadata, they stop being just documentation and start being instruction sets for agents. When we refer to "agent", we are thinking about it in two ways: “build agents” that assemble and deliver data products, and “analytic agents” that answer business questions on top of them; each bound to a single data product. Let’s start with build agents. Build agents automate the delivery lifecycle of data products from source mapping through ETL, testing, and de-identification to a production release. Everything they need lives in Unity Catalog as governed metadata: source-to-target mappings, business definitions, classification tiers, deidentification policies, data contracts, and model cards. The platform derives from tags, comments, certified flags, lineage, and glossary-linked terms. The catalog isn’t where you just document governance; it’s the runtime the agents execute against. Each agent works in a loop. It reads instructions from the catalog; does one concrete task such as generating pipeline code, running a test suite, producing de-identified data, or deploying a certified dataset; and then writes the evidence back as test outcomes, quality scores, lineage, or change capture data. This repeats.

FIG 1 — CATALOG-CENTERED AGENTIC ARCHITECTURE. Unity Catalog curates the metadata. Five AI agents consume that metadata to do lifecycle work — ETL generation, testing, curation validation, de-identification, deployment — and write their results back to the catalog.

In practice, we sequence the De-ID and Testing agents first. They eliminate the highest risks and the heaviest manual work up front. Starting where the payback is fastest helps build momentum early. As we continue through the loop, no agent acts on data the catalog doesn’t describe. Modern catalogs make this approach scalable because it can auto-generate column and table descriptions for a steward to approve, classify sensitive fields automatically, and capture column-level lineage without anyone maintaining it by hand. The human role shifts from authoring the metadata to approving it , which is exactly the kind of judgment work humans should be doing. The data & AI build lifecycle: proof & continuous context Build agents operate within an end-to-end lifecycle designed to release two assets simultaneously: the governed data product (mapping, curation, pipeline) and the analytic agent running on top of it (semantic layer, prompt configs, eval suites). This approach marks a fundamental shift from pipeline-centric engineering (moving data from point A to B) to context-centric engineering (making data understandable and actionable for LLMs). Rather than certifying code quality...