Serving models

A trained model is a large file of numbers, its weights. To use it, an application needs an inferenceInference Running a trained model to produce an output for a new input. server: a program that loads the weights into memory, takes requests and generates answers, fast and cheaply enough to be useful. It is mostly a classic infrastructure problem (capacity, many users at once, isolation, authentication) with a few quirks of its own.

This lesson covers five decisions: whether to run models yourself or use a hosted service, how to make a model fit in memory, what the engine that runs it does, what the common API offers when many clients share a server, and what drives cost and speed. Adapta is the server-side example; docket, the client-side one.

Run it yourself or use a hosted API

It is not a binary choice: many teams use a hosted model for hard tasks and a local one for high-volume or sensitive work.

Making the model fit

A model's size is counted in parameters, the individual numbers it learned. At 16 bits each, a 7-billion-parameter model needs about 14 GB of memory for the weights alone. QuantizationQuantization Storing a model's weights as less precise numbers, using fewer bits, to cut memory and cost at some loss of accuracy. stores each number with less precision, say 4 bits instead of 16, much like a compressed photo keeps most of the picture in a smaller file. The model uses less memory and runs faster, at a quality cost that depends on the model, the method and the task.

Two file formats come up often. GGUFGGUF A single-file model format used by llama.cpp and similar tools to run models locally, holding the weights and their metadata, usually quantized., used by the popular llama.cpp engine, is a single file with the weights (often quantized) and everything needed to use them. safetensorsSafetensors A file format for model weights that holds only the numbers and a description of them, so opening it cannot run hidden code. is the usual format in the Hugging Face ecosystem, typically at full or half precision.

What the engine does

Generating text has two phases. First the engine reads the whole promptPrompt Everything sent to the model in one call: instructions, conversation messages and any tool definitions. at once (prefill). Then it writes the answer one tokenToken A small chunk of text, often part of a word, that a model reads and writes and that providers charge by., a word or piece of a word, at a time (decode). To avoid reprocessing the prompt at every step, it keeps notes on what it has read: the KV cache. Those notes grow with the length of the context and with the number of requests in progress, which is why long contexts and many simultaneous users compete for the same memory.

One base model, many behaviors

A LoRALoRA Low-rank adaptation: a cheaper way to fine-tune that trains small add-on matrices instead of all of the model's weights. adapter is a small file applied on top of a base model to change its behavior, a bit like a lens on a camera. One base can then serve many behaviors, choosing the adapter per request. What this costs depends on the engine. In Adapta's llama.cpp backend, every combination of base and adapter is a full copy in memory. In vLLM, several adapters share one loaded base and can be served in the same batch, up to a configured limit.

The OpenAI-compatible API

OpenAI's chat completions request format has become the common language between applications and models; most servers and providers accept it. You can switch providers by changing an address and a model name. But "compatible" does not mean "equivalent". Support for tools, structured outputStructured output Making the model reply in a fixed data format defined by a schema, usually JSON Schema, so programs can read it reliably., streaming the answer as it is written, or cached-token reporting varies between servers.

Sharing a server

When several teams or customers share a server, a setup called multi-tenancyMulti-tenancy Serving several isolated customers or projects from one deployment., every call has to answer two questions: who is calling, and what may they reach. (docket, by contrast, is beta software for a single operator, with no tenants or quotas, per its README.)

Put together, a chat request to Adapta goes through these steps:

What drives cost and speed

• Output length. The prompt is read in parallel, but the answer is written one token at a time. Long answers dominate the wait, and providers usually charge more for output tokens.
• Context size. More context means more work during prefill and a larger KV cache.
• Batching. Grouping requests uses the GPU better, but a single request may wait longer.
• Prompt caching. Many services reuse a stable, identical start of the prompt, such as system instructions and tool definitions, at a lower price.
• Model size. The biggest factor. Hence model routingModel routing Assigning each request or role to a specific model, usually to balance cost and quality.: sending each task to the smallest model that handles it well.

Routing and budgetsBudget A hard limit on tokens, calls, time or money, enforced before the resource is spent. are covered in depth in Context, routing and cost.

Putting it into practice

The central decision is which traffic justifies running your own inference.

• Data that can't leave your infrastructure makes self-hosting a requirement rather than an option.
• Steady, high volume pays for fixed capacity; bursty traffic is usually cheaper per token on a hosted API.
• Someone has to operate it: the GPUs, the model upgrades and the endpoint's security. An unaudited server exposed to the network is a bigger risk than a hosted API.
• Measure latency and quality with your own traffic before committing. Generic numbers rarely transfer.

Go deeper

Adapta's documentation covers the server side in more depth, and docket's covers the client side:

• Learning the system explains GGUF, quantization and why a loaded model takes requests one at a time, with everyday analogies.
• Architecture maps Adapta's containers and follows a chat request from the key check to the answer.
• Operations runbook gives memory sizes per model and sets up the optional vLLM backend for many adapters.
• Models, gateways, and coding harnesses shows how docket registers an OpenAI-compatible endpointOpenAI-compatible endpoint A model API that follows the request format OpenAI popularized for chat, so existing client code works without changes. and lists what that compatibility does not cover.

Key takeaways

• Serving a model is infrastructure work: memory, concurrency, authentication and cost, with the KV cache as the central resource.
• Quantization trades some quality for memory and speed; evaluate under conditions close to serving.
• The OpenAI-compatible API makes switching providers easy, but does not guarantee the same features.
• Adapters let one base model carry many behaviors; the cost depends on whether the engine shares the base across adapters.
• Output length and model size dominate latency; routing and caching are the main levers.

Check yourself