Apache Iceberg Lakehouse: Snowflake & Google Cloud

skip\_to\_content

Blog/From Iceberg to Intelligence: The AI-Ready Borderless Lakehouse with Snowflake and Google Cloud

Jul 29, 2026/11 min read

Copy post link

Open in ClaudeOpen in ChatGPT

From Iceberg to Intelligence: The AI-Ready Borderless Lakehouse with Snowflake and Google Cloud



Iceberg: From silo to interoperability

Every database used to own its data, which worked when organizations had one analytics engine. Today's data teams often combine Apache Spark™, BigQuery, Gemini Enterprise Agent Platform, Snowflake (including Snowflake's CoCo and CoWork) and other services depending on the job — often within the same pipeline.

This created a dichotomy — choose flexibility or consistency:

• Let every team use its preferred engine and replicate data across systems, introducing redundancy, inconsistency and risk.
• Force everyone through one engine and sacrifice the specialization and autonomy that multi-engine architectures provide.

The industry needed a third option, and it emerged from a fundamental insight known as the Data Locality principle: It is faster, cheaper and more secure to move a small piece of executable code to where data already lives than to move massive volumes of data across a network.

These ideas converge on a single pattern: Instead of copying data to each engine, bring each engine's compute to the data. All data sits in the customer's own storage bucket, and all engines agree on how it is physically laid out so each can read and write directly to the same table.

This requires an agreed-upon open table format — a shared language all engines understand.

The industry converged on Apache Iceberg ™ as that format. Developed at Netflix for petabyte-scale table management and later donated to the Apache Software Foundation, Iceberg is supported by Spark, Trino, Flink, BigQuery, Snowflake and dozens of other engines. With Iceberg as the shared language, every engine participates without proprietary adapters.

Catalog: From Iceberg to lakehouse

An open format largely solves interoperability, but it introduces a new question: Who is in charge? When multiple engines can read and write the same files, someone must manage table metadata, enforce access policies, coordinate concurrent writers and ensure no engine sees stale or inconsistent state. That role belongs to the catalog, which serves as the lakehouse's governance layer. It is the single authority that knows which tables exist, what their schemas look like, who can access them and where data files physically reside. Without a catalog, open data is ungoverned data.

The catalog also controls storage access through vended credentials. When an engine requests table data, the catalog returns short-lived, narrowly scoped storage tokens rather than standing bucket credentials, so engines do not gain persistent access to the underlying storage layer.

For this model to work across engines built by different vendors, catalogs need a standardized integration protocol. Without one, every engine would require a custom integration with every catalog. The Iceberg REST Catalog (IRC) is the open API specification that solves this: It defines how clients discover namespaces, load table metadata and commit updates. Every catalog exposes an IRC endpoint, and every engine connects as a client.

This combination of vended credentials and a standard protocol makes it practical for engines to cross catalog boundaries — what the industry calls federation. When a Snowflake query reaches into Google Cloud's Lakehouse catalog, or a BigQuery job reads from Snowflake Horizon, each catalog issues only the narrow, time-limited tokens the requesting engine actually needs, so governance is enforced regardless of which engine initiates the request. True interoperability requires this in both directions: inbound federation, where engines external to your catalog can read and write your Iceberg tables, and outbound federation, where your compute can read and write Iceberg tables managed by a different catalog.

Together, the catalog and vended credentials provide a single-authority governance layer enforcing security, controlling access and centralizing audits. Customers can now get bidirectional access to Iceberg tables across catalogs, allowing for zero-copy architecture across Iceberg-compatible engines.

The key architectural question becomes: Who manages the catalog? Customers can deploy a DIY self-managed catalog or more likely use a managed Iceberg REST Catalog.

For enterprises who want to DIY, they can deploy Apache Polaris™, a popular open source IRC implementation, originally co-created by Snowflake and Dremio and donated to the Apache Software Foundation. This gives these organizations full control, but it also shifts full operational responsibility — including infrastructure, scaling, patching and availability — to the customer's team. Unlike managed solutions, DIY catalogs are not serverless and do not scale to zero when not in use, creating an ongoing operational cost.

In practice, most organizations choose a managed IRC so they can focus on data rather than catalog operations. When Snowflake and Google Cloud are both in the picture, each provides a natural fit:

• **Lakehouse runtime catalog** (Google Cloud's managed Iceberg catalog, part of Lakehouse for Apache Iceberg) is a serverless and scalable metastore that serves as a single source of truth for your data lakehouse. It allows multiple engines (BigQuery, Google-managed Spark, Apache Spark, Trino, Snowflake) to access the same copy of data across open formats like Apache Iceberg. Federation allows agents and engines in Google Cloud to access data across borders from catalogs in Snowflake, Databricks and AWS Glue.
• [**Snowflake Horizon...