Observability, audit and reliability

In a web service, a log line like "order created, 34 ms" is usually enough, because the code explains the rest. An agentAgent A system in which a model, in a loop, chooses the next action until it decides it is done or hits a stop condition.'s loop code is the same on every run; what changes is what the model decided. "Turn finished, failed, 212 s" explains nothing. To learn why an agent failed, deleted a file or burned through tokensToken A small chunk of text, often part of a word, that a model reads and writes and that providers charge by., you have to rebuild its trajectory: the sequence of decisions and actions it took.

This lesson covers three disciplines that are often blurred together: observabilityObservability Being able to answer new questions about system behavior from the events, traces and metrics it emits. (understanding what happened), audit (proving who authorized what) and reliability (making failures, retries and stops leave the system consistent).

Tracing what an agent did

Observability means answering new questions about a system from the data it already records; for an agent, being able to replay any run. The vocabulary comes from distributed systems: a trace groups everything that happened in one operation, and a span is one step inside it, with a start, an end and a parent. For an agent, that means task, then turn, then each model call and tool call.

Here is one run as a trace. Look at the fourth line: what made this run slow was a person approving a command, not the model.

run 123  "fix the login test"                 tokens   time
├─ model call #1 → search the code             3,100   2.1 s
├─ tool search   allowed  ok                           0.1 s
├─ model call #2 → run the tests               4,900   2.8 s
├─ tool shell    needs approval → approved after 38 s
├─ tool shell    ran  failed                           9.4 s
├─ model call #3 → edit the auth file          7,400   3.5 s
└─ result ok  finished with a final message   total 81 s

A trace answers what a flat log cannot: why the run was slow, why it cost what it did, and which files it touched.

What to record

Record the model that actually answered, not just the one requested. It is the cheapest defense against a provider or gateway quietly serving a different model under the same name.

Audit is not observability

Four terms sound alike but ask different questions:

• Observability: why did it behave this way?
• Audit trail: who did or authorized what, under which rule?
• Tamper-evidentTamper-evident A record whose modification is detectable, which is not the same as impossible to modify or delete.: if someone altered the record, would we notice? It detects changes.
• Immutable: can the record be altered at all? It prevents changes.

A trace can keep only a sample of runs and still be useful. An audit trail cannot, because the missing action is exactly the one someone will ask about. With agents, "who did this?" is a chain: a model, the harnessHarness The software around the model that runs its loop, carries out the tools it requests, prepares what it sees and enforces limits. that ran it, a queued task, a policy and perhaps a person's approval. Lose one link and nobody is accountable.

Hash chains

A hash is a short fingerprint of some data; change one character and the fingerprint changes completely. In a hash chain, each log entry also stores the fingerprint of the one before it. Edit or delete an entry in the middle and the link to the next one breaks.

Be precise about what that buys. With no secret key and nothing stored elsewhere, a chain catches clumsy or accidental edits, but not someone with write access who knows how it is built: they can recompute every fingerprint, or cut off the last entries, and nothing will show. Stronger steps are a fingerprint that needs a key kept on another machine, regularly copying the latest fingerprint to another system, write-once storage in a separate account and, finally, separation of duties: whoever runs the agents does not control the audit store.

Know where each piece of evidence lives. docket audits every decision that is not a plain "allow": denials, approval requests, approvals, cancellations. Allowed actions land in the trace, which is not chained and expires after 30 days (README).

Retries and doing things twice

Not every error deserves a retry:

The standard recipe: wait longer after each failure, add a little randomness so clients don't retry in lockstep, and respect the provider's "try again after" hint. A circuit breaker stops calling a provider that keeps failing.

Retries mean the same action may run twice, and the model repeats itself too. So any tool that changes something outside, such as charging a card or sending an email, should be idempotent: running it twice has the same effect as once. A common way is an idempotencyIdempotency The property that performing an operation twice has the same effect as performing it once. key, a unique label on each request so a repeat is recognized.

Stopping an agent safely

Stopping an agent is not killing its process. A forced kill can leave an unanswered tool request in the history, child processes running, half-finished effects, and no record of who stopped what.

Cooperative cancellation separates asking to stop from stopping. The request is saved durably at once; the running work checks for it at safe points and stops with its state consistent. The price is that it is not instant.

Tack reaches the same conclusion from the other side: the coding tools it runs start their commands in their own process sessions, so Tack refuses any adapterAdapter A small set of extra weights trained and attached to a model that otherwise stays unchanged, so its behavior changes without retraining the whole model. that claims it can cancel them (Agent Runners & Fleet Execution).

Putting it into practice

• Name the threat before picking an audit mechanism. A bug, a careless operator and a hostile administrator need different levels of protection.
• Decide where promptsPrompt Everything sent to the model in one call: instructions, conversation messages and any tool definitions. will be stored before adopting a tracing platform.
• Measure how long "cancel" really takes before trusting the button.
• Require every tool with outside effects to say whether it is safe to run twice.

Go deeper

The docket and Tack documentation show traces, audit and cancellation in a working system:

• Security model describes what docket's audit log records, how rotation keeps the chain checkable one file back, and what the log cannot see.
• Workflow guide shows how to read a task's trace, follow a run live and cancel one, including what "cancel requested" means while work is still stopping.
• Architecture explains, in its dispatch section, the states a task moves through, which failures are retried and how cancellation reaches running work.
• Agent runners and fleet execution covers Tack's idempotent execution requestsExecution request A saved record asking for a piece of work to run under given rules, built so that submitting it twice still creates only one., numbered attempts, why cancelling is only ever a request, and recovery that waits for a person.

Key takeaways

• An agent is debugged by its trajectory, not its log lines: record decisions, measured tokens, the model that answered and why the run stopped.
• Observability, audit trail, tamper-evident and immutable are different requirements; use the precise word.
• A local hash chain catches clumsy edits, not someone with write access who knows how it works.
• Retry only temporary failures, with growing delays, and make actions with outside effects safe to repeat.
• Stopping is cooperative: a saved request, safe checkpoints and a record of when work really stopped.

Check yourself