Minh Tri Le · i’m building group conversations for my agent team platform: one chat where a human and several ai agents work together. the first hard question wasn’t about models or prompts. it was this:
what happens when a process dies halfway through an agent’s turn?
on the beam, processes die all the time. that’s the point. supervisors restart them, and life goes on. but “restart” is only useful if the thing you restart knows where it was. so in this design, every agent turn is a row in a durable ledger. not a message on a bus. a row.
this post walks through how that works. the code is still in review (it’s the first real slice of the feature), so treat this as a design i’m shipping, not a victory lap.
one locked transaction
when a message lands in the group, two things have to agree: the message’s sequence number, and the list of agents who should respond to it (the dispatches).
if those are written separately, you get fun bugs. a message with a number but no dispatch. a dispatch pointing at a message that never committed. two concurrent posts that both think they’re seq 41.
so numbering and dispatch happen in one locked transaction. the database also enforces it: there is at most one dispatch per (message, target participant). if the same turn gets admitted twice, by a retry or by two nodes racing, the second one collapses into the first. it lands once.
commands from clients carry an id plus a fingerprint of their content, so a retried command is recognised as the same command, and a different command reusing the same id is rejected instead of silently merged.
pubsub fires after commit, and it’s only a hint
after the transaction commits, a pubsub message goes out. the important part is what that message means:
“go look.” not “here’s the work.”
the work lives in the database. the signal just says something changed. if a node misses the signal (it was restarting, the network blipped), nothing is lost. the dispatch is still sitting there, waiting to be claimed.
this is the one rule i’d tattoo on every event-driven system: if losing a message loses work, your bus is your database, and it’s a bad one.
leases, and the turns nobody finished
an agent that picks up a dispatch claims it with a lease: an id and an expiry time. while the run is alive, it holds the lease. if the process dies, the lease just… expires.
expired claims that never got linked to a result get reconciled. that’s how orphans get found, instead of a turn sitting in “running” forever while the ui spins.
the sink is idempotent, the ids are stable
results go through a terminal sink that is idempotent, and every event id is deterministic: derived from the message and the participant, not from a random uuid generated at write time.
roughly:
group-message:<message_id>:participant:<participant_id>so if a run crashes and replays, it rewrites the same history instead of appending a duplicate. the agent’s context only advances once the events behind it are durable.
put together: retries are safe, and results land once. that’s effectively-once behaviour built out of machinery that can crash and retry. it is not magic exactly-once delivery, and i’m not claiming it is. it’s at-least-once plus idempotency, done carefully.
things i refused to do
- write token streams to the ledger. partial output is a live delta keyed by the dispatch id. if you reload the page mid-turn, the ui rebuilds a “running” placeholder from the dispatch state. the ledger only gets the final result.
- fake it on failure. a failed turn shows up as a failed turn, redacted. nobody writes text on the agent’s behalf to make the chat look tidy.
routing never trusts the model’s @mentions
this one matters for security more than reliability.
when agent a wants agent b to take over, it doesn’t write “@b can you do this” and hope a regex catches it. routing only uses stable participant ids. handoff is a typed tool call, checked against the group roster. prose never dispatches.
and when one agent’s text shows up in another agent’s context, it’s wrapped as external, untrusted content, not as instructions.
why so strict? because if model-written prose can decide who runs next, then so can anything that got injected into that prose. a web page, a tool result, a file. the moment text can route work, prompt injection can route work.
the honest numbers
the pull request for this slice: +1988 / −54, 98 tests passing, 8 commits.
one of those commits was the feature. the other seven were making the tests deterministic.
that ratio is the real lesson. concurrency bugs don’t fail loudly. they fail sometimes. a test that passes 49 times out of 50 is worse than no test, because it teaches you to hit “re-run”.
the whole thing ships behind a feature flag, off by default, until the next slices land.
if you’ve built something similar, i’d love to hear how you make agent turns survive a crash. what did i miss?