Skip to content

3. The task-theft incident

The symptom made no sense: cloud agents launching fine, booting fine — then their tasks completing as failures within seconds, with claimed_by: null in the wreckage, before the box had even finished booting.

The hunt

We suspected the box (a boot bug?), then the queue (a lease bug?), then the proof gate (over-strict?). All innocent. The actual thief was the warm worker on my own Mac — the helpful little process from Tier 2 whose job is: "if any tasks are pending, claim and run them."

The fleet had been built on the same task table as everything else (by design — one boring queue, post 1). A neutrino's work order sat in agent:<id> as a pending task. The Mac's drain loop asked the Worker "anything pending?" — and the unscoped pending count said yes. The drain claimed the fleet's task, had no idea what to do with a box work-order, and fast-failed it. The box then booted into an empty queue and idled to TTL. Every component functioned exactly as written. The composition was the bug.

Why it took hours

Because nothing was broken. No errors, no crashes — just work evaporating between two correct subsystems. Distributed-systems debugging truism, paid for again: when two loops share a resource, the bug lives in neither loop. The tell, in hindsight, was claimed_by: null — the channel drain claimed anonymously, while boxes always claim as themselves. The forensics were in the metadata all along.

The fix: ownership, stated three times

  1. The unscoped pending count now excludes agent:% and broker:% queues — the drain's wake-up signal can't even see fleet work (countPendingTasks, lib/task-store.ts).
  2. task_claim refuses fleet queues outright with a typed error, fleet_queue_off_limits — defense against any other well-meaning drainer, present or future.
  3. isFleetQueue() is a named, exported predicate — the ownership rule is code, not comment.

One incident, three fences, because the second occurrence of a composition bug is unforgivable.

The lesson

Namespaces are not organization, they're authorization. A queue name is a claim about who may consume it; if that claim isn't enforced, every consumer you add is a future incident. We now treat "who drains this?" as a schema-level question — answered in the store, not in the politeness of callers.

Next: the credential problem that scale forces — Cookies never leave home.

MIT licensed. Built for people who want to own their agent.