How Unity Catalog managed tables bring interoperability, performance, and unified governance to the Lakehouse

How Unity Catalog managed tables bring interoperability, performance, and unified governance to the Lakehouse | Databricks Blog Skip to main content

Summary

External engines, including Apache Spark, Apache Flink, and DuckDB, can now create and write to UC managed Delta tables, with governance enforced centrally in Unity Catalog

UC managed tables use Predictive Optimization to improve query performance and lower storage costs while remaining fully interoperable across external engines. Existing external tables can be upgraded in place with ALTER TABLE SET MANAGED , with no data rewrite while preserving interoperability

External access is built on the open UC Delta APIs from the open-source Unity Catalog (UC OSS) project, so supported engines work with both Databricks UC and UC OSS.

Unity Catalog was designed for interoperability at scale. Enterprises have the flexibility to run any engine on a single copy of data with governance enforced centrally. Today, we are taking interoperability a step further. We're excited to announce that external access to Unity Catalog (UC) managed Delta tables is now in Public Preview. Over the past year, a growing list of engines, from Apache Spark and Flink to now Starburst and DuckDB, have integrated with UC managed Delta tables, and the ecosystem continues to grow. Enterprises can get the best of both worlds: the price-performance benefits of UC managed tables and the flexibility to use the engines that fit each workload. Unity Catalog remains the central governance layer, ensuring that access policies are enforced consistently across engines. Until now, teams that needed multi-engine access used external tables, which lack performance optimizations and governance guarantees. With external access now in Public Preview, that trade-off is gone. UC managed tables are the clear choice for the lakehouse across price, performance, and interoperability. Why teams choose Unity Catalog managed tables UC managed tables combine the interoperability of Delta Lake and Apache Iceberg with automatic, built-in capabilities that improve query performance, lower storage costs, and keep tables healthy over time. They are also the foundation for platform features, such as disaster recovery and Zerobus ingestion . Databricks automatically tunes managed tables through Predictive Optimization. Predictive Optimization cleans up storage, collects query statistics, and automatically selects Liquid clustering columns to evolve table layouts as query patterns change. Together, these benefits can deliver up to 50% storage cost savings and 20x faster queries while eliminating the need for data teams to tune tables manually. We recently announced the General Availability of catalog commits , which make Unity Catalog the commit coordinator for UC managed tables. Rather than having external clients commit directly against cloud storage, writes to UC managed Delta tables are coordinated through the catalog. This makes Unity Catalog the source of truth for table state, which unlocks key benefits like safe external writes, multi-statement transactions, and auditing of external operations. Create, read, and write from external engines Now in Public Preview, external engines can: Create, read, and write to UC managed Delta tables with catalog commits from Apache Spark (batch and streaming), Apache Flink, Starburst, DuckDB, and Streamnative. Keep governance centralized in Unity Catalog across all engines. Benefit from Predictive Optimization, which lowers costs and improves performance across all engines.

External access is built on Open APIs for metadata, credential vending, and commit coordination. Credential vending issues short-lived, scoped credentials based on UC privileges, while the Unity Catalog Delta API gives Delta engines a versioned contract for create, read, and write operations. Because these APIs are open and implemented in the Unity Catalog OSS project , any engine can integrate with create, read, and write support. The result is a growing ecosystem of engines that operate on the same governed copy of data in Databricks UC and in UC OSS Partners are already building around these APIs in different ways. Starburst, for example, integrates with Unity Catalog and Delta Lake to support creating, reading, and writing to UC managed Delta tables. "Starburst shares Databricks’ vision of openness and interoperability across the data ecosystem. By integrating with Unity Catalog managed tables, we are enabling customers to create a single source of truth for all their data, with centralized governance and the flexibility to leverage the tools of their choice." —Justin Borgman, CEO, Starburst Data Expanding the ecosystem through Delta Kernel External access to UC managed tables is part of a broader investment in the Delta Lake and Unity Catalog OSS ecosystem. The goal is to align Delta with the catalog-managed model pioneered by Iceberg so enterprises can benefit from the performance and consistency guarantees of a catalog while maintaining interoperability with any engine. We’re expanding the Delta ecosystem with Delta Kernel , which provides libraries for reading, writing, and committing to Delta tables without reimplementing the protocol from scratch. Paired with the UC Delta APIs in UC OSS, Delta Kernel allows an engine to integrate seamlessly with a Databricks-managed catalog or a self-hosted UC OSS deployment. For example, DuckDB built its Delta and Unity Catalog OSS extensions on Delta Kernel. As of DuckDB v1.5.1, it can read and write UC managed Delta tables directly. This reflects our commitment to an open, interoperable ecosystem: as the Delta protocol evolves, engines built on the Kernel inherit the latest features automatically, so customers always have access to the latest and greatest capabilities no matter which engine they choose. "DuckDB users value being able to work directly with open lakehouse data from a lightweight analytical engine. Delta Lake and Unity Catalog integration help bring that experience to governed enterprise tables, including workflows where data written through one engine can be read consistently by another." —Hannes Mühleisen, CEO, DuckDB Labs We’re continuing to work with the community to develop more Kernel-based integrations with UC managed Delta tables. For example, we’re taking major strides to bring Flink streaming workloads into UC managed tables with a new Kernel-based Flink connector . Centralized governance Centralized...