RepoSnowflake (Arctic)Snowflake (Arctic)published Jul 17, 2026seen 4w

Snowflake-Labs/snowflake-event-table-to-otlp-exporter

Python

Open original ↗

Captured source

source ↗
GH
GitHub/github.com/Snowflake-Labs/snowflake-event-table-to-otlp-exporter
Snowflake-Labs/snowflake-event-table-to-otlp-exporter repository metadata
Source ↗
published Jul 17, 2026seen 4wcaptured 4whttp 200method plain

Snowflake-Labs/snowflake-event-table-to-otlp-exporter

Description: A Python application to convert Snowflake Event Table rows into OpenTelemetry OTLP/JSON, and push them to a supported backend. Currently only Kafka is supported.

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-07-17T15:22:41Z

Pushed: 2026-07-22T17:32:03Z

Default branch: main

Fork: no

Archived: no

README:

Snowflake Event Table to OTLP Exporter

A utility to translate Snowflake Event Table rows into OTLP_JSON, and push to a compatible backend for consumption. As of now, only the OpenTelemetry Collector `kafkareceiver` is supported.

> [!WARNING] > This code is very early in development. Please conduct thorough testing to ensure no telemetry records are lost, and that transforms are properly handled. If you find any problems, please open either a GitHub Issue or PR with a fix.

Architecture

> [!IMPORTANT] > Custom event table recommended. It's recommended to use a custom event table with this application to avoid granting the application runtime role privileged access to your event table (e.g. TRUNCATE). See the [deploy guide](docs/deploy-snowflake.md) for details.

A Snowflake Stream is created and attached to the source Event Table to track new rows. A Python application checks SYSTEM$STREAM_HAS_DATA on a schedule. When true, transforms the Event Table rows into OTLP_JSON, and pushes to a Kafka topic.

This is designed to be run either as a Snowflake Snowpark Task (gated by a WHEN clause on SYSTEM$STREAM_HAS_DATA to makes empty ticks near-free), or as a long-running external service on your own infrastructure (e.g. in a Python venv or as a container).

The shared core (et2otlp.pipeline.run_once) runs byte-identical under a local venv/container poll loop and as a Snowpark stored procedure — only the entrypoints and config sourcing differ. Event Table rows are routed by RECORD_TYPE to per-signal Kafka topics (otlp_logs, otlp_spans, otlp_metrics), and offset advancement is tied to Kafka delivery inside one transaction for at-least-once delivery.

For more details, see:

  • [docs/local-configuration.md](docs/local-configuration.md) for all dev/.env / ET2OTLP_* environment variables
  • [docs/architecture.md](docs/architecture.md) for the full pipeline diagram, transaction sequence, and signal-routing details
  • [docs/transform.md](docs/transform.md) for the Event Table → OTLP field mapping, value-encoding rules, and schema-drift handling.

Quick start (local)

uv sync
cp dev/.env.example dev/.env # fill in Snowflake + Kafka (see docs/local-configuration.md)
cp dev/connections.toml.example ~/.snowflake/connections.toml
set -a && source dev/.env && set +a
uv run et2otlp-local

Run the tests:

uv run pytest -q

The local end-to-end harness ([dev/docker-compose.yaml](dev/docker-compose.yaml) + [dev/otel-collector-config.kafka.yaml](dev/otel-collector-config.kafka.yaml)) is in dev/ for deployment external to Snowflake (Python venv, VM, container, etc). The bundled dev/.env.example sets ET2OTLP_KAFKA_SECURITY_PROTOCOL=PLAINTEXT for the embedded broker; the collector config ships with auth disabled — uncomment SASL_SSL or mTLS blocks in that YAML locally for secured Kafka (see [docs/local-configuration.md](docs/local-configuration.md)). Kafka security defaults to SASL_SSL in the Python runner and Snowflake path when unset.

For the full Snowflake deployment walkthrough (SQL scripts, RBAC, procedure deployment, task creation), see [docs/deploy-snowflake.md](docs/deploy-snowflake.md).

---

Security

Kafka delivery defaults to SASL_SSL (encrypted + authenticated); PLAINTEXT is for a local/dev broker only. Store credentials (key-pair passphrase, Kafka SASL creds) in a secrets manager, and prefer Snowflake Workload Identity Federation when running in a supported environment (CSP workload identity / Kubernetes OIDC) to avoid managing a private key entirely. See [docs/deploy-snowflake.md#production-hardening](docs/deploy-snowflake.md#production-hardening) for the full checklist (secrets, WIF, PAT rotation, mTLS/ACLs, and pip-audit).

---

Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](.github/CONTRIBUTING.md) for more details.

License and Legal

Copyright (c) Snowflake Inc. All rights reserved. Licensed under the [Apache License 2.0](LICENSE). See [LEGAL.md](LEGAL.md) for additional terms. The code in this repository is not part of the Snowflake Service and is provided as-is.