How celld Turns Object Storage Into a Coordination Service

In the first post about celld, I described object storage as more than backup — it participates in coordination. That claim is worth taking apart properly, because the mechanism is more specific, and more interesting, than “objects have version numbers.”

The bar an object store has to clear

celld’s docs are unusually precise about this. A bucket only qualifies as a coordination substrate if it provides:

• Conditional create — a write fails if the object already exists
• Conditional overwrite — a write fails if the object changed since it was last read
• Read-after-write consistency — a read immediately after a write returns that write, not a stale one
• Ranged reads — the store returns exactly the byte range requested

S3, Cloudflare R2, Google Cloud Storage, Azure Blob Storage, and Tigris meet all four. Backblaze B2, Hetzner, and DigitalOcean Spaces — despite being marketed as “S3-compatible” — do not.

That distinction matters more than it looks. “S3-compatible” is a claim about the request/response shape of an API. “Safe to build a lock service on” is a claim about consistency semantics under concurrent access. The two are not the same claim, and conflating them is exactly the kind of mistake that would surface only under a network partition, at the worst possible time.

It’s also worth noting how recent this is. S3 didn’t support conditional writes as a general feature until August 2024, extended to If-Match checks in November 2024. Cell ownership arbitration via bucket CAS is only a viable pattern because object storage vendors have been quietly retrofitting exactly the primitive this design depends on.

Ownership is a race, and the bucket is the referee

Acquiring a cell reduces to one of two operations:

no record exists
      │
      ▼
  conditional create
      │
      ▼
  bucket accepts exactly one

record exists, owner presumed gone
      │
      ▼
  compare-and-swap on the previous record
      │
      ▼
  bucket accepts exactly one

If two nodes race to claim the same cell, both send a write, and the bucket’s own conditional-write semantics guarantee only one of them lands. The loser doesn’t get an ambiguous answer — it gets a rejected write, and it backs off.

There is no election, no quorum, no leader to ask. The bucket’s compare-and-swap operation is the entire arbitration mechanism.

Fencing, but not the way you’d expect

If you know the classic distributed-locking literature — Kleppmann’s 2016 argument against Redlock — you’d expect fencing to work like this: the lock service hands out a monotonically increasing token, the client attaches it to every write, and the protected resource checks the token and rejects anything from a superseded holder.

celld does something related but structurally different.

Every activation of a cell — a takeover, or a node waking a hibernated cell — bumps an epoch number. That epoch isn’t just a field in a metadata record. It’s baked directly into the storage path:

cells/agent-123/ltx/e94/   ← old owner's writes
cells/agent-123/ltx/e95/   ← current owner's writes

A node that has lost ownership but doesn’t know it yet — say, a partition-survivor that thinks it’s still authoritative — can keep writing. Nothing stops it. But its writes land in e94/, and nothing reads from e94/ anymore. Recovery selects the current epoch’s lineage. The stale writer isn’t rejected; it’s quarantined into irrelevance.

That’s a meaningfully different failure mode than the classic model. Kleppmann’s fencing requires the resource itself to be token-aware. celld’s version requires the reader to know which prefix is current — the isolation is structural, not enforced by a check on every write.

The design adds a second, complementary layer on top: self-fencing leases. Each node holds a lease in the bucket with a published expiry. If that expiry passes without renewal — because the node is partitioned, overloaded, or simply hasn’t heard back — the node doesn’t wait to be told it’s been superseded. It fences itself: stops the cell, fails every in-flight request it hasn’t completed.

lease expiry passes
        │
        ▼
node stops the cell itself
        │
        ▼
fails in-flight requests
        │
        ▼
(does not wait to be told)

This is a more paranoid design than “the lock service will eventually tell you you’re wrong.” It assumes a partitioned node might never hear that message, and builds in a self-destruct instead.

Consensus didn’t disappear — it moved into someone else’s black box

It’s tempting to read all this as “celld doesn’t need consensus.” That’s not quite right.

S3’s read-after-write consistency guarantee — the one celld depends on for correctness — is itself the output of an internal, replicated metadata system that Amazon operates and you never see. Before December 2020, S3 didn’t even provide strong consistency by default; getting there was itself a distributed-systems achievement AWS shipped as an announcement, not a given property of “object storage” in the abstract.

This is the same trick Google’s Chubby lock service played twenty years earlier: hide a Paxos-based consensus system behind a simple lock API, so application developers get to think about locks instead of quorums. celld does the same thing, except the consensus system it’s hiding behind isn’t one you run — it’s whichever provider’s bucket you pointed it at.

your application
        │
        ▼
     celld
        │
        ▼
 object store API
        │
        ▼
provider's internal
replicated metadata store
        │
        ▼
   (actual consensus,
    happens here,
    you never see it)

The coordination problem is real and it’s being solved. It’s just being solved several layers further down than an etcd or ZooKeeper deployment would put it — and by a system you don’t operate, can’t inspect, and have historically been surprised by (S3’s own consistency model is the case in point).

What this trades away

Three costs are worth naming plainly, because none of them are hypothetical:

Latency. celld’s own documentation concedes an object-store round trip is “much slower than a follower fsync.” A co-located etcd or Raft cluster typically resolves a normal request — a read, or a conditional write against an already-elected leader — in low single-digit milliseconds. Failover is a different story: when the leader actually dies, a new one isn’t chosen until an election timeout expires, which for etcd defaults to around a second. An S3 or R2 conditional write sits in between those two numbers and is far more consistent about it — no leader to lose, but also no path to single-digit-millisecond latency on the happy path either. You’re now subject to someone else’s network path and someone else’s load, not just your own.

Opacity. You can’t debug the provider’s internal consensus system the way you can debug a self-hosted etcd cluster. When something goes wrong at that layer, you get an HTTP status code, not a Raft log to inspect.

Primitive poverty. A single conditionally-writable object gives you leader election and nothing else. No watches, no multi-key transactions, no sub-second lease renewal across many keys at once. This is the same limitation that makes cross-cell transactions hard, discussed in the first post — the coordination primitive here is deliberately minimal, and minimal primitives don’t compose into richer ones for free.

None of this makes the design wrong. It makes it a specific bet: that the latency and opacity costs of leaning on a hyperscale object store’s internal consistency guarantees are worth not having to run, patch, and reason about your own consensus cluster.

The reframe

The interesting question this raises generalizes past celld:

If your durability layer already offers a strong-enough write primitive, do you need a separate coordination service at all?

For a long time the answer was structurally yes — object stores were eventually consistent, so coordination required purpose-built systems with their own quorums. That’s no longer strictly true. S3, R2, and a handful of others now expose exactly the primitive a lock service needs.

Whether that’s the right trade for a given system still depends entirely on your latency budget and your tolerance for depending on infrastructure you can’t see inside. But it’s no longer a question you can dismiss by assuming object storage can’t do this. It can. The question is whether you want it to.

The next post in this series takes apart the other half — how celld actually guarantees durability, and the LTX replication format underneath all of this.