GateLLM: LLM gateway with failover

An LLM gateway is a routing layer between your applications and your model providers. LLM stands for large language model. Your app calls the gateway instead of the provider, and the gateway handles credentials, failover, usage and cost in one place.

Without a gateway, every app holds its own provider key and its own retry code. A revoked key breaks each app separately, and the provider invoice shows one total with no split by product. A gateway moves those concerns out of application code. Your teams change a routing rule in one file, and no app needs a redeploy.

GateLLM is the gateway La Boétie runs in production for its own products. Each app receives one virtual key, and each key has an ordered chain of providers.

Your app sends a normal request to the gateway, with its virtual key in place of a provider credential. GateLLM picks a provider from the chain of that key, forwards the request, streams the answer back and records usage. The app keeps using its normal SDK and changes two environment values.

A virtual key is a gateway credential that stands in for a provider key. Your apps never see the real credentials, and you can rotate those without a restart.

Failover is the move from one provider to the next in a chain. A good gateway separates failures by cause, because each cause needs a different response. GateLLM treats four cases.

The last row is a deliberate default. Moving a request to another model can change its answer, so an app must opt in to that behaviour.

Prompt caching lets a provider reuse the start of a long prompt at a lower price. The cache lives with the provider, so a conversation that jumps between providers loses it on every jump.

GateLLM keeps an active conversation on the provider that last served it, as long as that provider can still take requests. Clients mark a conversation with the x-gatellm-session header. Long agent sessions gain the most from this, because each turn resends a large context.

GateLLM speaks the Anthropic Messages API and the OpenAI Chat Completions API, and it covers the main model providers, the large cloud platforms and compatible endpoints.

Cloud endpoints matter for data residency. A team that must keep inference in Europe can point a chain at a European region and leave the public endpoints out of it.

A gateway server is one more service to run. Some apps prefer the same routing and failover inside their own process, with no server and no virtual key. GateLLM ships that mode too.

Provider invoices show spend by account. A gateway sees every call, so it can tell you which product, route and model produced the cost. GateLLM stores usage and cost per key, provider and model. Request and response bodies stay out of that store, so usage reporting creates no second copy of your prompts.

Your team can read service health, the state of each provider, usage and Prometheus metrics. An alert fires each time a provider goes down or runs out of credit.

An API gateway manages ordinary web traffic: routing, authentication and rate limits per request. An LLM gateway adds what model traffic needs. Cost depends on tokens. Answers stream for seconds or minutes, so the gateway must pass a stream through while it counts. Each retry costs money.

Many teams run both. The API gateway faces the public, and the LLM gateway sits between your backend and the model providers.

Three failures look alike from an app.

A rate limit means you sent too many requests in a short window. Waiting usually fixes it. Another provider can also take the request, and the answer then comes from a different model.

Spent credit means the account has hit its limit and will refuse every request until a reset time. The right reaction is to skip that provider until the reset.

An outage means the provider is failing for everyone. A resend with backoff covers a short failure. A longer failure needs another provider from the chain.

A naive retry loop treats all three the same way, which turns a spent account into a stream of failed calls. GateLLM reads the response of the provider and marks its state, so later requests skip a spent or revoked provider without trying it first.

Migration takes four steps, and the app code barely changes.

1. Create a virtual key for the app and give it a chain with one provider, the one it uses today.
2. Change the base URL and the key in the environment of the app.
3. Watch the usage route for a day and confirm the totals match your expectations.
4. Add a second provider to the chain, then decide whether the key should fail over on outages.

Roll back by restoring the two environment variables. Moving one app at a time keeps each change small.

A gateway becomes the place that holds every model credential, so judge it as a sensitive service.

• Clients hold a revocable virtual key.
• Provider tokens stay outside the configuration.
• The usage store keeps no request or response body.
• Admin access needs a separate key.
• Running in process keeps credentials inside your app.

A gateway adds a component, so it needs a reason. Use this table.

These questions apply to any gateway.

1. Does failover separate outages from spent credit and from rate limits?
2. Can a key refuse failover, for apps where a model change is unacceptable?
3. Does a long conversation stay on one provider, to protect prompt caching?
4. Does the gateway store request bodies, and can you turn that off?
5. Does it add a server to your stack, and is there a mode without one?
6. Can you read cost per key, model and route without a separate tool?
7. Do the cloud endpoints you need exist in your region?

Treat the gateway as shared infrastructure with one owner. That team holds the provider tokens, reviews changes to chains and reads the alerts. Product teams receive a virtual key and a chain that fits their needs. A change of model or provider then goes through a configuration review, and the record shows who approved it.

A good chain follows from what the app needs.

Test each link before you rely on it. A backup that nobody has called in months can fail on the day you need it.

Read cost per key each week. A sudden change usually points to one feature, and per key data shows which one.

Long answers stream for minutes, so a proxy with a short timeout will cut them off. Check every layer between your app and the provider: load balancer, reverse proxy and client. A dropped stream partway through an answer costs tokens and returns nothing useful, so your app needs a plan for partial answers.

1. Which providers receive our prompts?
2. In which region does inference run?
3. Do those providers train on API data?
4. Does the gateway keep prompts?
5. How long is usage data kept?
6. Who can read it?

Write the answers down per app. Your legal team will ask, and a gateway gives you one place to enforce them.

• Provider keys sit in several repositories.
• Finance cannot split the model bill by product.
• A provider outage has taken a feature down.
• Each team wrote its own retry code.
• Nobody can list which apps use which model.
• Moving to a new model means redeploying several apps.

Two or more of these usually justify one.

A thin proxy is simple to write. The expensive parts arrive later: failure handling, usage accuracy and every new provider format.