---
title: "LLM gateway with failover: one virtual key per app"
description: "LLM gateway with failover on empty balances, revoked tokens and outages. Each app has one virtual key, tracked cost per model and 0 request bodies stored."
canonical: "https://laboetie.io/products/gatellm"
language: en
---
# GateLLM: LLM gateway with failover

GateLLM is an LLM gateway with failover that gives each app one virtual key and an ordered chain of providers. When a provider balance runs out or a token is revoked, the same request goes to the next provider in the chain. GateLLM tracks usage and cost per key, model and route and stores no request body.

Status: Live. Access: By request. One key per app, automatic failover and cost tracked per app.

## Capabilities

- Every virtual key belongs to one app and carries an ordered provider chain.
- Requests fail over when a provider balance runs out, a token is revoked or a provider goes down.
- Conversation stickiness preserves the prompt cache.
- You see usage and cost per key, model and route. Request bodies are not stored.
- In process use needs no server. Prometheus metrics and alerts are included.

## LLM gateway definition

An LLM gateway is a service between your apps and your model providers that holds the real credentials and routes each call. A client of GateLLM receives the gateway URL and a virtual key in place of a provider credential. Because GateLLM injects the real token on outbound calls, the app configuration holds no provider secret.

## Failover triggers in GateLLM

GateLLM fails over at once on empty balances and revoked tokens: a 400 balance error marks an API key as spent, and a 401 marks a token as revoked for ten minutes. A 502, 503, 504, 529 or dropped connection is first resent with backoff. By default the client receives the error for a plain 429 rate limit and for an outage that outlasts those resends. A key opts in to failover on either case through its failoverOn setting. The provider then stays held back for 30 seconds, or for its retry-after value up to five minutes.

## Virtual keys per app

Each app gets one virtual key. A key lists its chain, the providers that serve it in order. A provider is one credential shared by every key that chains it, so all those keys skip it once one of them has spent it. Tokens stay out of the configuration file. Each provider names a token file or an environment variable that GateLLM reads on every request, so a replaced token takes effect on the next call.

## Prompt cache and provider stickiness

GateLLM keeps a conversation on the provider that last served it while that provider is available and the conversation stays active within a window you set. The example configuration uses 3,600 seconds. Because prompt caches sit in one provider account, a conversation moved to another account pays for the whole prompt again.

## LLM cost tracking per app

GateLLM reads token counts from each answer stream and computes cost at API list rates. SQLite stores each usage record with its key, route, tags, provider, model and session. Prometheus metrics expose the same counters. Alerts go to the log, Telegram or a Slack webhook when a provider is spent or a key has no provider left.

## In process use with no server

In process mode runs the same gateway inside a Node app and needs no server or virtual key. The app passes its providers and chain to GateLLM and gives the GateLLM fetch function to its model client. An app with several tasks gives one chain per route, and every route shares failover state, stickiness, usage and alerts. The gateway has no npm dependency.


## What an LLM gateway is

An **LLM gateway** is a routing layer between your applications and your model providers. LLM stands for large language model. Your app calls the gateway instead of the provider, and the gateway handles credentials, failover, usage and cost in one place.

Without a gateway, every app holds its own provider key and its own retry code. A revoked key breaks each app separately, and the provider invoice shows one total with no split by product. A gateway moves those concerns out of application code. Your teams change a routing rule in one file, and no app needs a redeploy.

GateLLM is the gateway La Boétie runs in production for its own products. Each app receives one virtual key, and each key has an ordered chain of providers.

## How a request moves through GateLLM

Your app sends a normal request to the gateway, with its virtual key in place of a provider credential. GateLLM picks a provider from the chain of that key, forwards the request, streams the answer back and records usage. The app keeps using its normal SDK and changes two environment values.

## Virtual keys

A **virtual key** is a gateway credential that stands in for a provider key. Your apps never see the real credentials, and you can rotate those without a restart.

| Problem | Without a gateway | With a virtual key |
|---|---|---|
| A credential leaks | Replace it in every app that copied it | Revoke one key |
| Finance asks which product spent what | The invoice shows one account | Usage is stored per key |
| A provider goes down | Each app needs its own fallback | The chain of the key handles it |

## Failover: what triggers it

Failover is the move from one provider to the next in a chain. A good gateway separates failures by cause, because each cause needs a different response. GateLLM treats four cases.

| Event | What GateLLM does |
|---|---|
| Credit spent on a provider | Marks it spent and moves to the next link until the reset time |
| A revoked credential (HTTP 401) | Marks the provider revoked and moves on |
| An outage (HTTP 502, 503, 504, 529 or a dropped connection) | Resends with backoff |
| Outage outlasts the resends, or a plain rate limit | Returns the error to the client, unless the key opts in to failover |

The last row is a deliberate default. Moving a request to another model can change its answer, so an app must opt in to that behaviour.

## Conversation stickiness and prompt caching

**Prompt caching** lets a provider reuse the start of a long prompt at a lower price. The cache lives with the provider, so a conversation that jumps between providers loses it on every jump.

GateLLM keeps an active conversation on the provider that last served it, as long as that provider can still take requests. Clients mark a conversation with the `x-gatellm-session` header. Long agent sessions gain the most from this, because each turn resends a large context.

## Providers and cloud endpoints

GateLLM speaks the Anthropic Messages API and the OpenAI Chat Completions API, and it covers the main model providers, the large cloud platforms and compatible endpoints.

Cloud endpoints matter for data residency. A team that must keep inference in Europe can point a chain at a European region and leave the public endpoints out of it.

## Running GateLLM in process

A gateway server is one more service to run. Some apps prefer the same routing and failover inside their own process, with no server and no virtual key. GateLLM ships that mode too.

| Mode | Best for | You run |
|---|---|---|
| Gateway server | Several apps, or clients you do not control | One service |
| In process | One Node app that needs routing and failover | No extra service |

## Usage and cost tracking

Provider invoices show spend by account. A gateway sees every call, so it can tell you which product, route and model produced the cost. GateLLM stores usage and cost per key, provider and model. Request and response bodies stay out of that store, so usage reporting creates no second copy of your prompts.

## Monitoring and alerts

Your team can read service health, the state of each provider, usage and Prometheus metrics. An alert fires each time a provider goes down or runs out of credit.

## LLM gateway and API gateway: the difference

An **API gateway** manages ordinary web traffic: routing, authentication and rate limits per request. An LLM gateway adds what model traffic needs. Cost depends on tokens. Answers stream for seconds or minutes, so the gateway must pass a stream through while it counts. Each retry costs money.

| Concern | API gateway | LLM gateway |
|---|---|---|
| Unit of cost | Request | Input and output tokens per model |
| Response | Usually complete | Streamed |
| Failure handling | Retry the same upstream | Move along a chain of model providers |
| Key management | API consumers | Virtual keys in front of provider credentials |
| Session behaviour | Usually stateless | Conversations stay on one provider for prompt caching |

Many teams run both. The API gateway faces the public, and the LLM gateway sits between your backend and the model providers.

## Rate limits, spent credit and outages

Three failures look alike from an app.

**A rate limit** means you sent too many requests in a short window. Waiting usually fixes it. Another provider can also take the request, and the answer then comes from a different model.

**Spent credit** means the account has hit its limit and will refuse every request until a reset time. The right reaction is to skip that provider until the reset.

**An outage** means the provider is failing for everyone. A resend with backoff covers a short failure. A longer failure needs another provider from the chain.

A naive retry loop treats all three the same way, which turns a spent account into a stream of failed calls. GateLLM reads the response of the provider and marks its state, so later requests skip a spent or revoked provider without trying it first.

## How to migrate an app to a gateway

Migration takes four steps, and the app code barely changes.

1. Create a virtual key for the app and give it a chain with one provider, the one it uses today.
2. Change the base URL and the key in the environment of the app.
3. Watch the usage route for a day and confirm the totals match your expectations.
4. Add a second provider to the chain, then decide whether the key should fail over on outages.

Roll back by restoring the two environment variables. Moving one app at a time keeps each change small.

## Security

A gateway becomes the place that holds every model credential, so judge it as a sensitive service.

- Clients hold a revocable virtual key.
- Provider tokens stay outside the configuration.
- The usage store keeps no request or response body.
- Admin access needs a separate key.
- Running in process keeps credentials inside your app.

## LLM gateway or direct calls

A gateway adds a component, so it needs a reason. Use this table.

| Your situation | Recommendation |
|---|---|
| One app, one provider, low volume | Call the provider directly |
| One app that must survive a provider outage | Use the in process mode |
| Several apps sharing providers | Run a gateway server |
| Finance needs cost per product | Use virtual keys |
| Data must stay in one region | Use a chain limited to regional endpoints |

## What to check when you choose an LLM gateway

These questions apply to any gateway.

1. Does failover separate outages from spent credit and from rate limits?
2. Can a key refuse failover, for apps where a model change is unacceptable?
3. Does a long conversation stay on one provider, to protect prompt caching?
4. Does the gateway store request bodies, and can you turn that off?
5. Does it add a server to your stack, and is there a mode without one?
6. Can you read cost per key, model and route without a separate tool?
7. Do the cloud endpoints you need exist in your region?

## Who should run the gateway

Treat the gateway as shared infrastructure with one owner. That team holds the provider tokens, reviews changes to chains and reads the alerts. Product teams receive a virtual key and a chain that fits their needs. A change of model or provider then goes through a configuration review, and the record shows who approved it.

## Terms worth knowing

| Term | Meaning |
|---|---|
| Chain | The ordered list of providers for a key |
| Virtual key | A gateway credential that stands in for a provider key |
| Stickiness | Keeping a conversation on one provider |
| Route | A named task with its own chain, in process |
| Spent | A provider out of credit until its reset time |

## Choosing a provider chain

A good chain follows from what the app needs.

| App need | Chain design |
|---|---|
| Consistent answers | Two links to the same model on different clouds |
| Lowest cost | A cheap model first, a stronger one behind it |
| Data in one region | Regional endpoints only |
| High availability | Different providers on different clouds |

Test each link before you rely on it. A backup that nobody has called in months can fail on the day you need it.

## Common cost leaks in LLM apps

| Leak | Example | Fix |
|---|---|---|
| Oversized context | Sending a whole document per question | Send the relevant passages |
| Wrong model | A top model for simple classification | Route simple tasks to a smaller model |
| Lost prompt cache | A conversation moving between providers | Keep it on one provider |
| Retry storms | A loop calling a spent account | Stop calling until the reset |
| Unbounded output | No limit on response length | Set a maximum |
| Forgotten test traffic | A staging job left running | Give it its own key |

Read cost per key each week. A sudden change usually points to one feature, and per key data shows which one.

## Streaming and timeouts

Long answers stream for minutes, so a proxy with a short timeout will cut them off. Check every layer between your app and the provider: load balancer, reverse proxy and client. A dropped stream partway through an answer costs tokens and returns nothing useful, so your app needs a plan for partial answers.

## Data and compliance questions

1. Which providers receive our prompts?
2. In which region does inference run?
3. Do those providers train on API data?
4. Does the gateway keep prompts?
5. How long is usage data kept?
6. Who can read it?

Write the answers down per app. Your legal team will ask, and a gateway gives you one place to enforce them.

## Signs you need a gateway

- Provider keys sit in several repositories.
- Finance cannot split the model bill by product.
- A provider outage has taken a feature down.
- Each team wrote its own retry code.
- Nobody can list which apps use which model.
- Moving to a new model means redeploying several apps.

Two or more of these usually justify one.

## Build or adopt a gateway

| Question | Write your own | Adopt one |
|---|---|---|
| First version | A small proxy looks easy | Ready to run |
| Streaming and token counts | Each dialect needs care | Handled |
| Provider changes | Your team tracks them | Handled upstream |
| Control | Total | Within its options |

A thin proxy is simple to write. The expensive parts arrive later: failure handling, usage accuracy and every new provider format.

## Testing failover before you need it

Schedule a drill. Revoke a test key, point one app at a chain with a dead first link, and confirm that traffic moves and an alert fires. Then read the usage report to confirm the requests landed where you expected. Repeat it after each provider change.

## Next step

GateLLM runs in production for La Boétie's own products, and access is on request. [Send a brief](https://laboetie.io/brief) with your providers, your apps and your failure history, and the studio will reply with a setup. [DocRead](https://laboetie.io/products/docread) routes its model calls through GateLLM, and [PRbot](https://laboetie.io/products/prbot) shares its gateway core.


## Frequently asked questions

### What is an LLM gateway?

An LLM gateway is middleware between applications and model providers. It gives apps one endpoint, holds the provider credentials, routes calls and records usage. GateLLM is one such gateway: each app receives a virtual key and the gateway URL, and the real token stays on the host. A provider change then means editing a chain in the configuration while the app code stays as it was.

### What is the difference between an LLM gateway and an LLM proxy?

An LLM proxy forwards a call to a provider. An LLM gateway adds decisions around that call: which key is asking, which provider comes next and what the call cost. GateLLM forwards answers unbuffered, so server sent events stay live. Around that forwarding it adds virtual keys, an ordered provider chain and metering per key, model and route.

### Can an LLM gateway fail over in the middle of a stream?

GateLLM fails over before an answer starts. Once a stream has begun, an upstream that dies ends the answer to the client with an error, and the app handles the partial output. GateLLM buffers the request body up to a cap for a possible resend. A body past that cap streams to the provider once, so that request has no failover.

### Which providers can GateLLM route to?

GateLLM has provider kinds for Anthropic, OpenAI and DeepSeek, for Anthropic compatible and OpenAI compatible endpoints, and for Azure OpenAI, Google Vertex AI and Amazon Bedrock. Every provider in one chain must use the same dialect, which keeps Anthropic chains and OpenAI chains separate. A chain link can change the model name, because one model has a different name on each cloud.

### Does GateLLM store prompts or responses?

No. GateLLM stores token counts, cost, key, route, tags, provider, model and session in SQLite. It writes no request or answer body to storage or logs. Metering reads the tokens from the stream and forwards each chunk untouched. A team that needs prompt level debugging keeps that logging inside its own app.

### What happens when every provider in a chain is unavailable?

The client receives one error that explains why GateLLM skipped each link, and an alert reports a key with no provider left. A second alert goes out when the key is served again. A revoked token comes back into the chain after ten minutes. A provider held back after an outage returns after 30 seconds, or after its retry-after value up to five minutes.

### Do apps need code changes to use an LLM gateway?

An app using GateLLM changes two values: the base URL and the key. The gateway URL replaces the provider URL. The virtual key replaces the provider credential. The request format stays the one the provider already speaks. An app can add tags to its calls with the x-gatellm-tags header and name a conversation with the x-gatellm-session header.