Durable agent execution

You ask a friend to pay the electricity bill online. They text "on it", and then their phone dies. Did the payment go through? Pay again and you may pay twice; don't, and the power may get cut. Nobody did anything wrong; you just can't tell what happened.

A coding agentCoding agent An agent that works inside a code repository, reading files, editing code and running commands toward a task. lives in that situation. One run can last twenty minutes of copying a repository, editing files, running tests and saving commits, and meanwhile the laptop can sleep, the network can drop or the process can die. The hard question is not "how do I try again?" but "did it already happen?". A blind retry can push a change twice; no retry can lose work silently.

Making work survive crashes like these is called durable executionDurable execution Running a workflow so its progress is saved step by step, and after a crash it resumes where it stopped without redoing finished steps.. This lesson covers four ideas: why agentAgent A system in which a model, in a loop, chooses the next action until it decides it is done or hits a stop condition. runs need it, how to keep a single owner per run, what to do when the outcome is unknown, and how to prove the design works. The case study is Tack's runnerRunner A worker program that picks up work, runs it close to the code and credentials it needs, and reports back. fleet, the machines that execute its agents.

Why agent runs are different

A short request to a web server can usually be retried without much thought. An agent run can't. It lasts minutes or hours, so failing halfway is not rare. It changes things outside the system, and restarting does not undo a commit or a deploy. And the story of what happened is spread across several machines.

Retries and timeouts in general are covered in Observability, audit and reliability. This lesson is about who owns the work.

Workers that ask for work

The central server can call the machines that do the work, or each worker can ask "anything for me?" and report back. The second style is a pull model, the same one GitHub Actions runners use. Workers only make outgoing connections, so they can sit behind a firewall; the server never needs keys to get into any machine; and a dead worker simply stops asking.

One owner at a time: leases and fencing tokens

Think of a "reserved" sign on a restaurant table that is valid only until 8 pm. If the guest never arrives, the table is not blocked all night. In software that sign is a leaseLease A lock that expires unless its owner keeps renewing it with regular check-ins, so a worker that vanishes cannot hold work forever.: a claim on a piece of work that expires on its own. The worker holding it keeps it alive by checking in regularly with small messages called heartbeats. If they stop, the lease runs out and another worker can take over.

Expiry creates a new problem. Worker A freezes for a while, its lease expires, and worker B takes the job. Then A wakes up, unaware, and tries to save its results. Now two workers think they own the same run.

The fix works like numbered tickets at a deli counter. Each time the work is handed out, it gets a number one higher than before: A got 33, B got 34. Every write must show its number, and the record only accepts the current one, so A's 33 is turned away. That number is a fencing tokenFencing token A number that goes up each time work changes hands, so the system can reject writes from a previous owner whose lock has expired.. A frozen worker may still be alive, but it can no longer change the record.

When nobody knows what happened: stop, don't guess

Back to the friend with the dead phone: the sensible move is to check before paying again, and to ask if you can't check. When a runner crashes after starting an agent, the server can't prove whether the agent is still running or what it changed. The tempting rule is "the lease expired, start it again". Tack does the opposite.

Two habits make that safe. The runner writes down what it is about to do before doing it, in a local log called a journal, so after a restart it can report what it knows. And every message is idempotent: sending it twice has the same effect as sending it once, so a repeated report does no harm.

The cost is that a person sometimes has to step in. For agents whose changes are hard to reverse, that cost is usually worth paying.

Proving it: kill the worker halfway

A design like this is only as good as its tests. The honest one is blunt: start a real run, kill the worker mid-task, and check that the run is not lost, no duplicate starts, and the retry finishes once someone decides.

Putting it into practice

• List the effects that can't be repeated safely, such as pushes, deploys, payments and emails. That is where you need idempotencyIdempotency The property that performing an operation twice has the same effect as performing it once. or a human decision.
• Weigh build against adopt. An engine like Temporal gives strong guarantees for extra infrastructure and a stricter coding style. A narrower design of your own is cheaper to run but needs serious crash testing.
• Keep keys where the work runs. A pull model lets credentials stay on the executing machine.

Go deeper

Tack's own documentation covers its runners in more depth than this lesson:

• Introduction explains why Tack splits the board from the runner and how each survives the other's failure.
• Agent Runners & Fleet Execution is the operator's guide: enrolling runners, where keys and workspaces live, and which parts are proven end to end today.
• Recovery Runbook walks through a crash step by step: what the runner reports, how the server decides, and how a person resolves an unclear case.

Key takeaways

• No outside effect is guaranteed to happen exactly once; aim for one valid owner at a time plus actions that are safe to repeat.
• A lease keeps work from being blocked forever; a fencing token keeps an old owner from overwriting the new one.
• Fencing protects your record, not the world, so unknown outcomes need an explicit rule.
• When an effect can't be reversed, stopping for a person beats retrying blindly.
• A durable design is only credible if a test kills the worker halfway through.

Check yourself