Object Storage + WAL: Lakebase Postgres for the agentic era

Object Storage + WAL: Lakebase Postgres for the agentic era | Databricks Blog Skip to main content

Agents that interact with a traditional OLTP database often create bottlenecks at the storage layer. New deployments, copies, restores, and replicas all mean moving around large volumes of data which is time-consuming and expensive. The polar opposite is true for object storage. Amazon S3, for example, is cheap, performant, almost invisible to operate. It creates a scalable, cost-effective storage layer for agent memory. Which brings us to the question: Can object storage sit underneath a transactional database and make it easier for agents to work with? This question is what started Lakebase Postgres. The answer does not just depend on how fast your object store is, but rather where you place the source of truth. Two OLTP models The usual mental model for OLTP is data-centric. Data is organized into tables with rows and columns, each representing an entity. Storage is the place where the current state lives, and the database's job is to store and retrieve it. But there is a second model: transaction-centric. Here the database is a journal of transactions. Each entry is an operation, and storage is a timeline of those operations rather than a snapshot of the present. The current state is one thing you can derive from the timeline. For years the data-centric model was the only one that mattered in practice, because what the operations team asked of a database were reads and writes against the present. Over the past few years, that has dramatically changed. The operations that agent workloads ask for are almost all operations on transaction history: Give me an isolated copy of production to work in Put it back the way it was before my last three statements Show me what this table looked like before the migration Run twenty of these at once, and delete nineteen of them in an hour

These are all queries about the timeline. A database that only stores the present delivers copies and backups, which are slow and expensive. However, Postgres already contains this timeline: it is called the write-ahead log (WAL). The writing in the WAL Postgres’ WAL records every modification before it reaches the data files. It originally existed so Postgres could recover: if the server died between the log write and the data file write, a WAL replay closed the gap. But WAL contents are interesting far beyond recovery. Take a table and an insert:

Before that change reaches the users table on disk, Postgres appends it to the WAL. The log is binary, but pg_waldump will render it. The records for this insert look roughly like this:

These are four records, and one transaction. Note how each has a log sequence number (LSN), a monotonically increasing identifier. The heap and btree lines also name the exact 8 KB page that changed. The log does not say "a row was added." It says which page, in which relation, at which point in the timeline. Read that as a recovery mechanism and it is a list of work to redo after a crash. But if you read it as a transaction journal , it is something else: A complete, ordered, byte-level account of every page the database has ever changed, with a unique name on every entry. That name, the LSN, is the part that matters most. It means the timeline is already addressable. Nothing needs to be added to Postgres to make "the database as of a point in time" a well-defined thing. It only needs a storage layer that keeps the log around and can answer questions against it. The log becomes the source of truth In a conventional Postgres deployment, the WAL is a means to an end. The data files are the database , the log protects them, and the log is trimmed once its records are safely applied. Storage is simply a disk attached to the machine running Postgres, and everything about the database's identity is tied to that machine. Now, let’s invert it. Make the log the database , and the data files a derived, cached representation of it. Then you can keep the full timeline, and you no longer have to move data to copy or rewind the database. History becomes addressable, so a database “copy” becomes a pointer instead of a second set of files. This makes deployments, restores, and replicas cheap enough to treat like code. That’s what we did in Lakebase Postgres. Concretely, we split the system into two layers: The compute layer The compute layer runs standard Postgres. It parses SQL, plans and executes queries, enforces MVCC, manages locks and indexes. Nothing in the query engine is rewritten. What changes is what the compute node is responsible for: it exists to execute work, not to preserve data. It has RAM for shared buffers and local NVMe as a page cache, and it can start, stop, scale, or die at any moment without putting durability at risk. The storage layer The storage layer owns correctness, durability, and history. It outlives any individual compute node, and it is built from three components with distinct jobs: Safekeepers replicate the WAL . When the compute node generates WAL records, it streams them to several safekeepers, and a transaction is committed once a quorum acknowledges the record through a Paxos-based protocol . Durability is a property of replication and consensus rather than of one machine's fsync . The pageserver turns WAL into pages . It combines base pages with committed WAL records to materialize the version of a page that a given query needs, and it persists those materialized versions into object storage asynchronously. Object storage holds long-term, immutable history. Materialized page versions and historical states arekept as an append-only record rather than a mutable filesystem.

The write path What does the write path look like? A commit in this system follows these steps: Postgres applies changes in memory. Buffers are updated, indexes are modified, WAL records are generated exactly as usual. Instead of flushing WAL to a local filesystem, the compute node streams it over the network to the safekeepers. The transaction is committed once a quorum of safekeepers has acknowledged the record. That is the point where the client hears success. Page materialization happens afterward, in the storage layer, off the transaction's critical path. A commit never waits for pages to be written or uploaded.

This design might get an obvious objection: that step 2 adds a network hop to the commit path. But any Postgres deployment that takes durability...