LLMs as software components

Most of software engineering rests on one quiet assumption: give a function the same input and it returns the same result. Types, tests and API contracts all depend on it. A large language model (LLMLLM A large language model: a neural network trained to predict the next token, then tuned to follow instructions.) breaks that assumption, and it fails in an awkward way: its wrong answers look right.

This lesson treats the LLM as a component with an unusual contract. It covers how that contract differs, what tokensToken A small chunk of text, often part of a word, that a model reads and writes and that providers charge by. are and why they cost, why the model remembers nothing between calls, what "temperatureTemperature A setting that controls how random the model's next-token choice is: low gives predictable output, high gives more varied output." does and does not do, and how to turn the model's text into data you can trust. The case studies are docket, which calls models, and Adapta, which serves them.

A different contract

A program calls a model by sending messages, such as an instruction ("extract the contact details") and the user's text, and gets text back. The web request itself is exact; the content is not. A successful response means "the model answered", not "the answer is right".

Tokens: what the model reads and what you pay for

Models don't read letters or words. They read tokens: fragments of text, often a short word or part of a longer one. In English a token averages about four characters; Spanish and many other languages need more tokens for the same content. Tokens matter because:

• they are billed, usually at different prices for input and output;
• they are limited: each model has a maximum context windowContext window The maximum number of tokens a model can read in one call; the model keeps nothing between calls., the total text, input plus output, it can handle in one call;
• they take time: the model writes one token at a time.

No memory between calls

The model remembers nothing from one call to the next. A chat app that seems to remember is resending the whole history every time. So each turn costs more than the last, someone must decide what to drop when the history no longer fits, and anything left out does not exist for the model.

The history is sent as messages with roles: system for the developer's instructions, user, assistant for the model's earlier replies and, where tools are supported, tool. But before the model sees them, all messages are joined into one piece of text, with markers where each role starts and ends (Adapta does this in adapta/core/chat_templates.py).

That matters for security: there is no wall between instructions and data. The system role carries more weight only because the model was trained to follow it, so text hidden in a document can compete with your instructions (see Sandboxing and agent security).

Temperature is not determinism

For each next token, the model chooses among likely options. Temperature sets how adventurous that choice is. Near 0, it almost always takes the most likely token; higher values explore alternatives.

Who sets the default is a design decision too. docket leaves it to the provider unless the caller chooses, so it never guesses a model's defaults. Adapta, being the server, applies its own configured default and caps the answer length.

Turning text into data you can trust

For a person, Name: Ana. Email: ana@example.com is perfect. For a program that must store that email, it is free text whose shape can change between replies. There are three layers of defense:

1. Ask for it in the instructions ("reply with JSON only"). This works almost always.
2. Use structured outputStructured output Making the model reply in a fixed data format defined by a schema, usually JSON Schema, so programs can read it reliably.: the provider forces the reply into valid JSON or a schema you define.
3. Always validate in your own code. A schema can check that the email is text; it cannot tell a real address from an invented one.

Then decide what happens when validation fails: retry, ask the model to fix it, hand it to a person, or reject it. That decision is AI engineering in miniature.

Failures that look correct

• Cut-off replies. If the model hits its length limit, JSON or a tool request may be cut in half. docket detects this and ends the turn as invalid without running any tool.
• Valid shape, invented content. It passes every type check.
• Instructions quietly ignored, such as replying in another language.
• Empty output not counted as an error. In Adapta's data generation, an unparseable reply yields nothing, but only crashes count toward the failure rate, so that passage silently contributes no data.

Putting it into practice

• Validate in code, not only in the instructions.
• Write down a response for each expected failure: retry, ask for a correction, escalate to a person, or reject.
• Track usefulness, not just responses: "the model answered" and "the answer was useful" are different numbers. See Evaluating models.

If the model will request actions rather than just produce text, the next step is From LLM app to agent.

Go deeper

The documentation of Adapta and docket shows these ideas from both sides, the server and the caller:

• Learning the system explains what a language model computes, what tokens are and why they drive cost and limits.
• Architecture traces one chat request through Adapta, including the step that checks the promptPrompt Everything sent to the model in one call: instructions, conversation messages and any tool definitions. fits the context window.
• Consuming the API shows how a program calls Adapta and what comes back: the answer, token usage and a consistent error format.
• Models, gateways, and coding harnesses explains how docket connects to model providers, and what "compatible" does and does not guarantee.

Key takeaways

• A successful call only means the model answered; correctness has to be checked separately.
• Tokens cost money, space and time; keep real counts and estimates apart.
• The model has no memory: history is resent every time, and all roles end up in one piece of text.
• Temperature 0 does not guarantee repeatable answers, and repeatable does not mean correct.
• Always validate output in your own code, and decide in advance what happens when validation fails.

Check yourself