databricks/zerobus-sdk-py

databricks/zerobus-sdk-py

Description: Zerobus SDK for Python

Language: Python

License: NOASSERTION

Stars: 33

Forks: 3

Open issues: 0

Created: 2025-09-03T13:11:45Z

Pushed: 2026-03-06T14:27:51Z

Default branch: main

Fork: no

Archived: yes

README: > This repository has been moved to [databricks/zerobus-sdk](https://github.com/databricks/zerobus-sdk/tree/main/python). This repo is archived and read-only. All new development, issues, and pull requests should go to the monorepo. > > | SDK | Link | > |-----|------| > | Rust | databricks/zerobus-sdk/rust | > | TypeScript | databricks/zerobus-sdk/typescript | > | Java | databricks/zerobus-sdk/java | > | Python | databricks/zerobus-sdk/python | > | Go | databricks/zerobus-sdk/go |

Databricks Zerobus Ingest SDK for Python

GA: This SDK is generally available and supported for production use cases. Minor and patch version updates will not contain breaking changes. Major version updates may include breaking changes.

We are keen to hear feedback from you on this SDK. Please file issues, and we will address them.

The Databricks Zerobus Ingest SDK for Python provides a high-performance client for ingesting data directly into Databricks Delta tables using the Zerobus streaming protocol.

Table of Contents

• [Disclaimer](#disclaimer)
• [Features](#features)
• [Requirements](#requirements)
• [Quick Start User Guide](#quick-start-user-guide)
• [Prerequisites](#prerequisites)
• [Installation](#installation)
• [Choose Your Serialization Format](#choose-your-serialization-format)
• [Option 1: Using JSON (Simplest)](#option-1-using-json-simplest)
• [Option 2: Using Protocol Buffers](#option-2-using-protocol-buffers)
• [Usage Examples](#usage-examples)
• [JSON Examples](#json-examples)
• [Protocol Buffer Examples](#protocol-buffer-examples)
• [Authentication](#authentication)
• [Configuration](#configuration)
• [Error Handling](#error-handling)
• [API Reference](#api-reference)
• [Best Practices](#best-practices)
• [Handling Stream Failures](#handling-stream-failures)
• [Performance Tips](#performance-tips)
• [Debugging](#debugging)

Features

• Rust-backed performance: Native Rust implementation with Python bindings for maximum throughput and minimal latency
• High-throughput ingestion: Optimized for high-volume data ingestion with native async/await support
• Automatic recovery: Built-in retry and recovery mechanisms from the Rust SDK
• Flexible configuration: Customizable stream behavior and timeouts
• Multiple serialization formats: Support for JSON and Protocol Buffers
• OAuth 2.0 authentication: Secure authentication with client credentials
• Type safety: Rust's type system ensures reliability and correctness
• Sync and Async support: Both synchronous and asynchronous Python APIs
• Zero-copy operations: Efficient data handling with minimal overhead

Architecture

The Python SDK is a thin wrapper around the Databricks Zerobus Rust SDK, built using PyO3 bindings:

┌─────────────────────────────────────────┐
│ Python Application Code │
│ (Your code using the Python SDK API) │
└─────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Python SDK (Thin Wrapper) │
│ • API compatibility layer │
│ • Python types & error handling │
└─────────────────────────────────────────┘
│
▼ (PyO3 bindings)
┌─────────────────────────────────────────┐
│ Rust Core Implementation │
│ • gRPC communication │
│ • OAuth 2.0 authentication │
│ • Stream management & recovery │
│ • Protocol encoding/decoding │
└─────────────────────────────────────────┘

This architecture provides:

• Native performance through Rust's zero-cost abstractions
• Memory safety without garbage collection overhead
• Single source of truth for all SDK implementations
• Python-friendly API with full type hints and IDE support

Requirements

Runtime Requirements

• Python: 3.9 or higher
• Databricks workspace with Zerobus access enabled

Dependencies

• protobuf >= 4.25.0, = 2.28.1, .cloud.databricks.com/o=

- **Workspace URL**: The part before `/o=` → `https://.cloud.databricks.com`
- **Workspace ID**: The part after `/o=` → ``

> **Note:** The examples above show AWS endpoints (`.cloud.databricks.com`). For Azure deployments, the workspace URL will be `https://.azuredatabricks.net`.

Example:
- Full URL: `https://dbc-a1b2c3d4-e5f6.cloud.databricks.com/o=1234567890123456`
- Workspace URL: `https://dbc-a1b2c3d4-e5f6.cloud.databricks.com`
- Workspace ID: `1234567890123456`

#### 2. Create a Delta Table

Create a table using Databricks SQL:

CREATE TABLE .default.air_quality ( device_name STRING, temp INT, humidity BIGINT ) USING DELTA;

Replace `` with your catalog name (e.g., `main`).

#### 3. Create a Service Principal

1. Navigate to **Settings > Identity and Access** in your Databricks workspace
2. Click **Service principals** and create a new service principal
3. Generate a new secret for the service principal and save it securely
4. Grant the following permissions:
- `USE_CATALOG` on the catalog (e.g., `main`)
- `USE_SCHEMA` on the schema (e.g., `default`)
- `MODIFY` and `SELECT` on the table (e.g., `air_quality`)

Grant permissions using SQL:

-- Grant catalog permission GRANT USE CATALOG ON CATALOG TO ``;

-- Grant schema permission GRANT USE SCHEMA ON SCHEMA .default TO ``;

-- Grant table permissions GRANT SELECT, MODIFY ON TABLE .default.air_quality TO ``;

### Installation

#### From PyPI (Recommended)

Install the latest stable version using pip:

pip install databricks-zerobus-ingest-sdk

Pre-built wheels are available for:
- **Linux**: x86_64, aarch64 (manylinux)
- **macOS**: x86_64, arm64 (universal2)
- **Windows**: x86_64

#### From Source

Building from source requires the **Rust toolchain** (install from [rustup.rs](https://rustup.rs/)).

Install Rust (if not already installed)

curl...