Streaming Data into Apache Iceberg with Snowflake

Streaming Data into Apache Iceberg with Snowflake

Skip to content

Blog / Data Engineering / Stream to Apache Iceberg Easily with Snowpipe Streaming

Aug 24, 2026 / 7 min read Data Engineering Copy post link Open in Claude Open in ChatGPT

Stream to Apache Iceberg Easily with Snowpipe Streaming

Dave Matthews

Full-scale streaming evaluation without the infrastructure lift

For a team trying to find out whether Snowpipe Streaming holds up at 1M+ TPS, you want the shortest path to a real answer. Streaming at scale has a reputation. It's the workload teams put off, because standing it up entails S3 buckets, an EKS cluster, Kafka topics, IAM policies, a security review and sign-off from three platform teams who each have their own backlogs. That setup can require additional infrastructure provisioning and coordination before testing begins. Proving these pipelines work at the scale and handle production volume is usually its own project. In our demo environment, we were able to set up and run this evaluation in an afternoon. You can use our Snowpipe Streaming High-Performance Architecture to stream data into Apache Iceberg™ format at over 1M TPS, entirely inside Snowflake infrastructure. A container running in Snowpark Container Services (SPCS) generates the load, streams it through the SDK and lands it in a Snowflake-managed Iceberg table. That makes it ready to query in Snowflake right away, with role-based access control (RBAC), lineage and masking all applied as it arrives. In our test environment, we stood up the demo, tested it end to end, measured throughput and tore it down in an afternoon, as we did in our test. This post will walk you through how to actually build an end to end streaming to Iceberg demo, using this GitHub Repo . This example is in python but you can test the SDK options (Java, Python, Node, REST), so that you can credibly recommend this to your business with real results to show.

Architecture diagram: The pattern

We wanted to build a demo where everything runs inside Snowflake infrastructure. Multiple Docker containers running in Snowpark Container Services ( SPCS ) generate synthetic test data at your target transactions per second and stream it using the Snowpipe Streaming SDK . It lands in a Snowflake-managed Iceberg table : open format, queryable within seconds of landing, governed in Snowflake. Authentication is easy: SPCS injects a short-lived OAuth token into the container telling the Streaming SDK to use it. You don't need to manage any secrets.

Figure 1: Snowpark Container Service to Iceberg

Why SPCS to generate the load?

To be clear, Snowpipe Streaming doesn't need SPCS. The SDK runs anywhere: your laptop, an EC2 box, a Kubernetes pod. Putting it in SPCS is what gets you to the test faster since the service runs entirely on infrastructure Snowflake already manages, with nothing new to provision or additional sign-offs to manage for the test itself.

Landing directly into Iceberg

The target is a Snowflake-managed Iceberg table. Rows stream in through the SDK and land as Parquet with Iceberg metadata, managed by Snowflake. Point it at your own S3 external volume later if you want, or leave it managed after the evaluation. The code stays the same either way. For the evaluation it means less setup. A table on Snowflake managed storage skips the external volume, storage integration and cloud storage permissions you'd otherwise have to configure first. When you're ready to productionize, you've got three clear paths: Snowflake-managed Iceberg with an external volume: Parquet files are stored in your S3 bucket. An external engine reads the Parquet files directly from S3, using the Horizon REST Catalog for metadata. Snowflake-managed Iceberg on Snowflake storage: The files are in Snowflake's internal storage. This option avoids configuring an external volume and associated cloud storage. Leverages Iceberg features for compatibility going forward. Native Snowflake table: No separately managed external storage infrastructure is required for this configuration.

Demo: See it running

Figure 2: Demo of end to end streaming

When you run the demo script you can see the SPCS service starts, the producer begins generating the load, and the consumer streams it into the Iceberg table. You can switch to Snowsight, and see the row count climb here as well — millions of rows, queryable as they land.

The code: Building it yourself

To help understand how this all works, here is some pseudocode to help visualize how this works. The full code is in the repo, and there are several alternative demos linked below. Start with a Snowflake-managed Iceberg table and a pipe:

-- 1. The target table CREATE OR REPLACE ICEBERG TABLE events ( event_id STRING, event_ts TIMESTAMP_NTZ, payload VARIANT ) CATALOG = 'SNOWFLAKE' BASE_LOCATION = 'events/' ICEBERG_VERSION = 3;

-- 2. The streaming pipe (extracts typed fields from the SDK's VARIANT payload) CREATE OR REPLACE PIPE events_pipe AS COPY INTO events (event_id, event_ts, payload) FROM ( SELECT $1:event_id::STRING, $1:event_ts::TIMESTAMP_NTZ, $1:payload::VARIANT FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING')) );

Snowpipe Streaming’s high-performance architecture supports both v2 and v3, but omitting the parameter defaults the table to v2.

Next, the consumer:

from snowflake.ingest.streaming import StreamingIngestClient

SPCS injects credentials automatically - no keys, no secrets

props = { "account": "YOUR_ACCOUNT", "user": "YOUR_USER", "role": "STREAMING_SERVICE_ROLE", "url": "https://YOUR_ACCOUNT.snowflakecomputing.com", "authorization_type": "SPCS", "spcs_token_path": "/snowflake/session/token", }

Connect to the pipe

client = StreamingIngestClient( client_name="my_consumer", db_name="STREAMING_DEMO", schema_name="PUBLIC", pipe_name="events_pipe", properties=props, )

Open a channel and stream rows

channel, status = client.open_channel("ch_1")

for i, row in enumerate(generate_load()): channel.append_row(row, offset_token=str(i))

Close cleanly

channel.close() client.close()

Then you deploy the consumer to SPCS with a short service spec. And you watch it land:

SELECT COUNT(*) AS rows_landed, MAX(event_ts) AS latest_event FROM streaming_demo.public.events;

Java, Node and the REST interface follow the same shape: open a channel, insert rows, flush. There's a working example for each in the repo, so you can test whichever matches your stack. To try this yourself you can check...