Context, routing and cost

In ordinary software, running a function one more time costs almost nothing. In a system built on a language model, every step an agentAgent A system in which a model, in a loop, chooses the next action until it decides it is done or hits a stop condition. takes costs tokensToken A small chunk of text, often part of a word, that a model reads and writes and that providers charge by., time and money, and that cost follows from design: how much text you send, which model answers and how many rounds you allow. The model's working memory, its context windowContext window The maximum number of tokens a model can read in one call; the model keeps nothing between calls., is also limited, and a crowded one hurts the model's attentionAttention The mechanism that lets a model decide, for each token, how much to draw on every other token in the text. well before anything visibly breaks.

This lesson covers five ideas: why an agent's cost grows faster than expected, keeping the context small on purpose, measured versus estimated cost, budgetsBudget A hard limit on tokens, calls, time or money, enforced before the resource is spent. as stop signs, and choosing a model per role. Tokens and context windows are introduced in LLMs as software components.

Why agent cost grows fast

A model call is priced by the text going in and the text coming out, counted in tokens. The model also remembers nothing between calls, so every round of an agent's loop resends everything so far: instructions, files read, output of every command.

Picture a meeting where, before each new point, someone reads all the minutes so far out loud. The tenth point costs far more than the first, and a 20-round agent loopAgent loop The repeating cycle behind an agent: ask the model, run the tools it asked for, feed the results back and repeat, until it is done or hits a set limit. is not 20 equal calls for the same reason. Unless the history is trimmed, input cost grows roughly with the square of the number of rounds.

Keeping the context small on purpose

Deciding what goes into each call, in what form and order, is called context engineeringContext engineering Deciding what goes into each model call, in what order and within what token budget.: treating the context as a budget, not a bucket. Typical moves: cut long tool output, summarize old history, fetch only relevant documents (RAG), and hand subtasks to helper agents with their own context.

Three failures come up again and again:

• Overflow. The request no longer fits, the provider rejects it, and the round's work is lost unless the system planned for it.
• One giant output. A single command that prints a huge log fills the context in one step.
• Summaries that break the conversation. A request to use a tool and the tool's answer belong together. A summary that keeps one and drops the other leaves a conversation the provider refuses.

The product callout below follows these moves in the order docket applies them before each model call.

Measured versus estimated

There are two ways to know how many tokens something contains. An estimate is a rule of thumb, such as "about four characters per token"; it exists before the call, so it serves to decide what fits, with a margin. A measurement is the count the provider reports after the call; it is exact, so it is the one for budgets and accounting. Reporting an estimate as a measurement misleads whoever decides from that number.

Even exact token counts do not give the real bill. That depends on current prices, discounts, negotiated plans and middlemen's fees. Hence the rule: tokens are measured; dollars are estimated.

That gap is the general lesson: a cost control is only as good as its source of truth. In production, that source is the provider's or gateway's invoice, and internal numbers should be checked against it.

Budgets as stop signs

A runaway loop is an agent that keeps going without progress, each round costing more than the last. The defense is several hard limits at once, since each catches a different kind of loop:

Choosing a model per role

Not every step needs the most capable, most expensive model. Model routingModel routing Assigning each request or role to a specific model, usually to balance cost and quality. is choosing which model answers each request: fixed per role, by how hard the request looks, as a cascade (try a cheap model, move up if a check fails), as a fallbackFallback Sending a failed request to another provider or model automatically. when a provider is down, or by cost and speed. The trade-offs: instructions tuned for one model can fail on another, failures get harder to reproduce, and a cascade sometimes pays twice.

Putting it into practice

• Label every cost figure as measured or estimated, and check it against the real invoice.
• Test with a small context window before trusting a system with a large one.
• Set limits before the first deployment, and allow no spending without an explicit trigger.
• Prove a cheaper model with evidence: compare it on the same tasks with evals before assigning it a role.

Go deeper

The docket and Tack documentation show these controls in working systems:

• Workflow guide walks a task through a team of agents, including a budget cap stopping work between steps and reading measured tokens next to a labeled estimate.
• Troubleshooting covers growing context and runaway turns: the per-turn limits, how to check what is resent every round, and trimming tool output for a small window.
• Architecture explains the cheap and strong model classes, and how the hand-off between steps drops less important fields before it ever cuts the summary.
• Agent runners and fleet execution describes, per coding tool, which usage Tack can really measure, and why an unknown cost reads "Not measured".

Key takeaways

• The whole history is resent every round, so cost grows much faster than the number of steps.
• Keep context small on purpose, and never split a tool request from its answer when summarizing.
• Estimate to decide, measure to account, and never present an estimate as money spent.
• Several limits at once stop runaway loops; each catches a different failure.
• Routing by role saves money, but evalsEval A measurement of how well the whole system behaves on realistic cases, including the steps it took along the way, scored with imperfect checks and read with statistics., not assumptions, show whether quality holds.

Check yourself