When an organization goes from one prototype to several teams using language models, the same problems appear: provider keys copied into ten repositories, spending nobody can attribute, one provider outage taking the application down, and no central record of what was sent where.
A fixes this with one shared entrance in front of every model. This lesson covers what a gateway is, what to check before trusting each of its features, how it differs from a router or a client library, and where it stops helping. The case study is docket, which is not a gateway but a program that can talk to one.
What a gateway is
Think of a building's front desk: visitors don't get keys to every office; they sign in once, get a badge, and the desk decides where they may go and keeps a log.
In the diagram, many callers share one gateway, which holds the rules and reaches many models.
- Callersapps and agents
- App A
- App B
- An agent
- Gatewayone entrance that holds the rules
- Keys per team
- Budgets
- Fallback and limits
- Cache and logs
- Modelsmany providers behind one address
- Closed providers
- Open-weight host
- Your own model server
Applications know one address and one gateway-issued key; the real provider keys live only inside the gateway.
What it usually offers, and what to check
| Feature | What it solves | What to check |
|---|---|---|
| Keys per team or app | each one can be revoked on its own | who administers the gateway |
| Budgets | a spending cap per team or project | whether the cap is strict or approximate |
| Fallback and retries | another provider or model when one fails | the model can change without you noticing |
| Load balancing | spreading traffic across accounts or regions | results are harder to reproduce |
| Rate limits | one app can't use up everyone's quota | counted per request or per token |
| Caching | not paying twice for the same question | stale answers, or someone else's |
| Logs | tokens, cost and speed per call | prompts end up stored |
| Guardrails | filtering content or hiding personal data | false alarms and extra delay |
A reuses an earlier answer when a new question is close enough in meaning, judged with embeddings. It saves money, but similar questions can need different answers, so use it only when the answer does not depend on who asks or when.
Gateway, router or SDK
- An SDK, such as Vercel's AI SDK or the LiteLLM library, lives inside your application and gives one way to call many providers. It unifies code, but each application keeps its own keys and rules.
- A router decides which model or provider serves each request, in your code or inside a gateway. OpenRouter, for example, picks a provider by availability and price, with fallback on by default.
- A gateway is a separate service all traffic passes through, so it centralizes keys, limits, caching and logs.
They can be combined. Routing by role is a design choice you can make with or without a gateway.
docket as a gateway client
In docket
- Coding toolworks on the repository
- its own provider settings
- Model policywhich model each role uses
- set with docket's own commands
- Gatewaythe endpoint docket's loop calls
- OpenRouter
- Vercel AI Gateway
- any registered compatible endpoint
It is deliberately a plain client:
- Keys: one per provider, from settings, environment or a local secrets file. No per-team keys or budgets: it is beta software for a single operator (README).
- Retries: it retries a failed step a couple of times, on top of whatever the gateway does, and does not yet respect a provider's "try again after" hint.
- No : a failed request is never moved to a cheaper model.
- No gateway extras: no gateway-specific options, no discovery of catalogs or prices.
- Cost: it records the token counts the endpoint reports, but leaves OpenRouter and Vercel models unpriced on purpose, because a marketplace changes prices per model and per account.
Where a gateway stops helping
A gateway controls what is sent to which model. It does not control what your system does with the answer. If the model asks to delete a folder or move money, the gateway only sees a message requesting a tool. Whether that action may happen is for the agent's runtime to decide, through policies, approvals and audit; see Governance and human-in-the-loop.
- Agent runtimedecides what happens with each answer
- Policies
- Approvals
- Audit
- Gatewaydecides what goes to which model
- Keys
- Budgets
- Cache and logs
- Model providersproduce the answers
Other limits to plan for:
- Delay: one more network hop, and more if or a cache make their own calls.
- Single point of failure: if the gateway is down, every model is down at once.
- Stacked retries: if both the application and the gateway retry, one error can multiply calls and cost.
- Reproducibility: with fallback or load balancing, the same call may reach a different model. Record who answered.
- Where data goes: the gateway sees every and answer and may store them. Check where it processes and keeps data. OpenRouter, for instance, can restrict routing to providers that keep no data and offers in-region processing only to enterprise customers.
Putting it into practice
- Start from what you want to centralize and who will run it, not from which product to pick.
- Size it to the organization. One team with one provider may need only a client library and careful key handling; several teams need per-team keys, cost attribution and a central record, which is where a gateway pays off.
- With regulated data, decide first where it may be processed; that rules out options before any feature comparison.
- Keep model access and agent governance apart. You need both, in different layers.
Go deeper
The docket and Tack documentation show what calling models through a gateway looks like in practice:
- Models, gateways and harnesses sets up OpenRouter and Vercel AI Gateway, mixes gateways by role, and lists what "compatible" does and does not cover.
- Architecture explains why docket runs its own and reaches every model through a single plain client, with no vendor SDK.
- Troubleshooting walks through the usual endpoint errors: no endpoint configured, an unreachable server, and requests the provider rejects.
- Agent runners and fleet execution records, per coding tool, whether the model that answered is confirmed, and why a gateway can leave it unconfirmed.
Sources
- LiteLLM Proxy
- OpenRouter: provider routing
- Vercel AI Gateway
- Cloudflare AI Gateway: overview, compatible endpoint, guardrails
- Portkey AI Gateway
- Kong AI Gateway
- Envoy AI Gateway is now Agent Router
- Azure API Management: AI gateway capabilities
- Hugging Face Inference Providers
Key takeaways
- A gateway is one shared entrance to many models: the place for keys, limits, caching and logs.
- SDKs, routers and gateways solve different problems and can be combined.
- Check what each budget, cache and fallback really covers before relying on it.
- docket is a plain gateway client: one format, its own retries, no fallback, no prices for marketplace models.
- A gateway does not govern what agents do, adds delay and is a single point of failure.
Check yourself
What separates a gateway from an SDK that supports many providers?
The SDK lives inside each application, which keeps its own keys and rules. The gateway is a separate service all traffic passes through, so keys, limits, caching and logs are managed in one place.
Your gateway has automatic fallback and your application also retries. What can go wrong?
Retries multiply, so one error can become many calls and more cost. Fallback can also change which model answered without the application knowing, which makes results hard to reproduce.
Why doesn't a gateway with guardrails replace agent governance?
It filters the text going in and out, but it does not decide whether a tool may run. Approving, denying and auditing actions belongs to the agent's runtime.