How celld Actually Achieves RPO Zero

The first post in this series touched on this briefly: celld doesn’t always wait for a bucket round trip on every write — it waits for a durability proof, and a peer’s disk can satisfy that proof just as validly as object storage can. This post is the full version of that mechanism, including the parts the docs leave open.

LTX didn’t start with celld

The transaction format celld uses — LTX, for Lite Transaction file — was built by Fly.io for LiteFS, its FUSE-based system for replicating SQLite across a cluster. It has nothing to do with celld originally, and predates it by years.

The idea is simple to state and fiddly to get right: SQLite tracks each transaction through its own rollback journal — created when a transaction begins, deleted the instant it commits. That deletion is a reliable, atomic signal that a transaction just completed, and it’s the exact moment LiteFS intercepts: rather than letting the journal simply vanish, it captures the pages that changed and repackages them into an LTX file instead — a sorted, checksummed list of pages, tagged with a monotonically increasing transaction ID so a replica always knows exactly where it stands relative to the source.

SQLite transaction commits
        │
        ▼
journal deletion intercepted
        │
        ▼
changed pages sorted + checksummed
        │
        ▼
     LTX file
   (tagged with TXID)

Litestream — a separate, older project for streaming SQLite backups to object storage — adopted the same format in its 0.5 rewrite, replacing its previous approach of shipping raw WAL segments.

What the rewrite actually fixed

The old model shipped WAL segments more or less as SQLite produced them. Restoring a database meant replaying however many segments had accumulated since the last snapshot — potentially thousands of small files, applied one at a time.

LTX’s answer is hierarchical compaction. Litestream compacts recent transaction files upward through three tiers:

per-transaction LTX files
        │
        ▼
  Level 1: 30-second windows
        │
        ▼
  Level 2: 5-minute windows
        │
        ▼
   Level 3: hourly windows

Smaller, more frequent files get merged into larger, less frequent ones, with duplicate page writes collapsed — “newer wins” when the same page changed more than once in a window. A restore reads down through this hierarchy rather than replaying every transaction since the beginning of time. Fly.io’s own numbers put the result at “a dozen or so files on average” for a restore that used to require thousands.

The rewrite also dropped Litestream’s old “generations” concept — a mechanism for handling multiple overlapping replication streams — in favour of the monotonically increasing TXIDs LTX already carries. One less piece of state to reconcile.

celld includes its own Rust implementation of LTX, separate from Litestream’s Go original — consistent with celld itself being a Rust project built around embedded V8 and Tokio. Two independent implementations of the same wire format is a normal thing to happen and also a normal place for subtle divergence to hide; it’s not a criticism, just a reason not to assume the two are interchangeable in every edge case.

The write path celld’s docs actually describe

Here’s the mechanism in full — the brief mention in the first post doesn’t cover the two-mode split.

celld runs in one of two modes, and they acknowledge writes differently.

Fleet mode — two or more nodes, the default and recommended configuration:

write arrives at owning node
        │
        ▼
  sent to a peer node
        │
        ▼
peer confirms it's on disk
        │
        ▼
      ACK
        │
        ▼
(bucket upload happens
 asynchronously, after)

Bucket mode — a single node, no peers available:

write arrives at the only node
        │
        ▼
   sent to the bucket
        │
        ▼
  bucket confirms the PUT
        │
        ▼
      ACK

Both configurations claim the same thing — RPO=0, no acknowledged write is ever lost. But the mechanism achieving that guarantee is different in kind, not just in latency. Fleet mode’s hot path is peer-to-peer replication: a write is durable once a second machine has it on its own disk, which is structurally the same idea as Postgres’s synchronous_commit waiting on a standby, not “wait for object storage.” The bucket write still happens — it’s how a cell’s state gets rebuilt from scratch on a cold start, and how a fully-dead fleet recovers — but it isn’t on the critical path of every acknowledged write.

That guarantee has a boundary worth stating rather than leaving implicit. In fleet mode, a write is durable the moment a peer has it on disk — but it isn’t durable against every failure until the asynchronous bucket upload lands too. If the owning node and every peer holding that write’s only copy die before that upload completes, the write is gone despite having been acknowledged. RPO=0 here means no single-node failure loses an acknowledged write, not that a correlated loss of every replica holding a given write is impossible.

Bucket mode is the fallback for the case where no peers exist: nothing else to lean on, so the object store round trip has to be in the critical path. celld’s own documentation is candid that this is the slower path — “much slower than a follower fsync,” in its own words.

This is worth being precise about because “no Raft cluster inside celld” — a claim worth making about the ownership model — doesn’t mean there’s no peer replication anywhere in the system. There is. It’s just a narrower, purpose-built protocol (send to a peer, wait for its disk) rather than a general consensus algorithm, and it lives on the write-durability path rather than the ownership-arbitration path covered in the previous post.

This is a feature you can dial, not an implicit default

Two settings make this concrete rather than a matter of inference:

CELLD_OUTPUT_GATE, default 1, is what enforces the durability wait at all — whichever mode is active, the gate is what makes celld hold the response until a durability proof exists. Setting it to 0 removes that wait entirely. celld’s own docs describe this plainly as accepting possible write loss. It’s there, presumably, for workloads that would rather take the latency win and accept the risk — but it’s not a setting you’d want on by accident.

CELLD_LTX_DURABILITY_TIMEOUT_SECS, default 10, bounds how long a write will wait for its durability proof before giving up. Ten seconds is a long time to hold an HTTP response open; it’s a generous budget that presumably exists to tolerate a slow bucket PUT in bucket mode without failing writes that would otherwise have succeeded a moment later.

Both are the kind of detail that only shows up in documentation, not in a diagram — and both confirm the durability model is a deliberately engineered, tunable mechanism, not an emergent property of “SQLite plus object storage.”

Why this isn’t just “backing SQLite up to S3”

It’s worth being explicit about the difference, because from a distance the two can look similar.

A periodic snapshot-to-S3 backup has a recovery point measured in the snapshot interval — minutes or hours of potential loss — and a slow restore, since you’re pulling back a full file and possibly replaying whatever changed since.

LTX-based replication is continuous and transaction-scoped, and critically, the replicated copy — whichever one satisfied the durability gate, peer disk or bucket — is the actual mechanism a new owner uses to resume a cell after a takeover, not a disaster-recovery artifact that only matters in a true catastrophe. The “Follow one request” section of the first post in this series — a node claiming ownership, restoring SQLite, and resuming — depends on this replicated state being current to the last acknowledged write, not to the last nightly snapshot.

What’s still unclear

Two things aren’t settled by the documentation as it stands, and are worth naming rather than guessing at:

Whether multiple SQLite commits get batched into a single LTX segment and a single peer-send, or whether every commit triggers its own round trip. This matters for write throughput under load, and the docs don’t say either way.

What “one or two other nodes” actually means operationally in fleet mode — is the replication factor configurable, and what happens to the durability guarantee during a rolling restart when the follower count temporarily drops to zero or one.

The full picture

Put together with the previous post, the full shape looks like this: object storage arbitrates who owns a cell, through compare-and-swap and epoch-prefixed isolation. Durability of what that owner writes is a separate concern, satisfied preferentially by peer replication and only by the bucket itself when no peer exists. The bucket is the thing everyone eventually agrees on — but on the hot path, in the common configuration, it’s not the thing you wait for.

That’s a more careful design than “put SQLite on S3,” and a more interesting one. It’s also a reminder that any high-level architecture sketch — including the opening post in this series — is a starting point, not a substitute for reading what the project’s own documentation says its code actually does.