databricks/zerobus-sdk-go

databricks/zerobus-sdk-go

Description: Zerobus SDK for Go

Language: Go

License: NOASSERTION

Stars: 6

Forks: 1

Open issues: 0

Created: 2025-11-19T13:44:36Z

Pushed: 2026-03-06T14:18:59Z

Default branch: main

Fork: no

Archived: yes

README: > This repository has been moved to [databricks/zerobus-sdk](https://github.com/databricks/zerobus-sdk/tree/main/go). This repo is archived and read-only. All new development, issues, and pull requests should go to the monorepo. > > | SDK | Link | > |-----|------| > | Rust | databricks/zerobus-sdk/rust | > | TypeScript | databricks/zerobus-sdk/typescript | > | Java | databricks/zerobus-sdk/java | > | Python | databricks/zerobus-sdk/python | > | Go | databricks/zerobus-sdk/go |

Databricks Zerobus Ingest SDK for Go

GA: This SDK is generally available and supported for production use cases. Minor and patch version updates will not contain breaking changes. Major version updates may include breaking changes.

We are keen to hear feedback from you on this SDK. Please file issues, and we will address them.

The Databricks Zerobus Ingest SDK for Go provides a high-performance client for ingesting data directly into Databricks Delta tables using the Zerobus streaming protocol.

Table of Contents

• [Overview](#overview)
• [Features](#features)
• [Getting Started](#getting-started)
• [For SDK Users (Install from pkg.go.dev)](#for-sdk-users-install-from-pkggodev)
• [For Contributors (Build from Source)](#for-contributors-build-from-source)
• [Quick Start](#quick-start)
• [Repository Structure](#repository-structure)
• [How It Works](#how-it-works)
• [Architecture Overview](#architecture-overview)
• [Data Flow](#data-flow)
• [Authentication Flow](#authentication-flow)
• [Custom Authentication](#custom-authentication)
• [Usage Guide](#usage-guide)
• [1. Initialize the SDK](#1-initialize-the-sdk)
• [2. Configure Authentication](#2-configure-authentication)
• [3. Create a Stream](#3-create-a-stream)
• [4. Ingest Data](#4-ingest-data)
• [5. Handle Acknowledgments](#5-handle-acknowledgments)
• [6. Error Handling](#6-error-handling)
• [7. Close the Stream](#7-close-the-stream)
• [Configuration Options](#configuration-options)
• [Error Handling](#error-handling)
• [Examples](#examples)
• [Tests](#tests)
• [Performance Benchmarks](#performance-benchmarks)
• [Best Practices](#best-practices)
• [Migration Guide](#migration-guide)
• [API Reference](#api-reference)
• [Building from Source](#building-from-source)
• [Community and Contributing](#community-and-contributing)
• [License](#license)
• [Requirements](#requirements)

Overview

The Zerobus Go SDK provides a robust, CGO-based wrapper around the high-performance Rust implementation for ingesting large volumes of data into Databricks Delta tables. It abstracts the complexity of the Zerobus service and handles authentication, retries, stream recovery, and acknowledgment tracking automatically.

What is Zerobus? Zerobus is a high-throughput streaming service for direct data ingestion into Databricks Delta tables, optimized for real-time data pipelines and high-volume workloads.

This SDK wraps the Rust zerobus-sdk-rs using CGO and FFI (Foreign Function Interface), providing an idiomatic Go API while leveraging the performance and reliability of the underlying Rust implementation.

Features

• Static Linking - Self-contained binaries with no runtime dependencies or LD_LIBRARY_PATH configuration
• High-throughput streaming ingestion into Databricks Delta tables
• Automatic OAuth 2.0 authentication with Unity Catalog
• Simple JSON ingestion - No code generation required for basic use cases
• Protocol Buffers support for type-safe, efficient data encoding
• Batch ingestion - Ingest multiple records at once for maximum throughput
• Backpressure control to manage memory usage
• Automatic retry and recovery for transient failures
• Configurable timeouts and retry policies
• Immediate offset returns for ingested records
• Graceful stream management - Proper flushing and resource cleanup

Getting Started

Choose your installation path:

| Path | When to Use | |------|-------------| | [Standard Installation](#installation) | You want to use the SDK in your project (via go get) | | [Development Setup](#development-setup) | You want to contribute or build from source (via git clone) |

Installation

The Zerobus Go SDK is a CGO-based wrapper around a high-performance Rust core. For the best experience, use a tagged release which includes pre-built binaries.

Prerequisites:

• Go 1.21+
• CGO enabled (enabled by default)
• C compiler (gcc or clang)

Installation Steps:

# Add the SDK to your project (use a tagged version for pre-built binaries)
go get github.com/databricks/zerobus-sdk-go@latest

> Note: Tagged releases (e.g., v1.0.0) come with pre-built Rust libraries for Linux, macOS, and Windows. If you use @main or a commit hash, you will need to have Rust installed and run go generate to build the library yourself.

In your code:

import zerobus "github.com/databricks/zerobus-sdk-go"

func main() {
sdk, err := zerobus.NewZerobusSdk(
"https://your-shard.zerobus.region.cloud.databricks.com",
"https://your-workspace.cloud.databricks.com",
)
if err != nil {
log.Fatal(err)
}
defer sdk.Free()
// ... create streams and ingest data
}

> Note: After the initial go generate step, regular go build works normally. The Rust library is statically linked into your binary.

Development Setup

Prerequisites:

• Same as above (Go, CGO, Rust, C compiler, Databricks workspace)
• Git

Setup:

# Clone the repository
git clone https://github.com/databricks/zerobus-sdk-go.git
cd zerobus-sdk-go
go generate # Builds Rust FFI
make build # Builds everything

See [Building from Source](#building-from-source) for more build options and [Community and...