AI gateway
A proxy between your app and model providers that adds logging, caching, limits, billing and fallbacks.
Instead of calling each provider directly, the app calls the gateway, which forwards the request. You get one place to see traffic and cost, switch providers and enforce limits. Some gateways also bill several providers from one balance.
Related terms
- Model routing and fallback: Sending each request to the model that fits it, and to another one when the first fails.
- Cost cap: A hard spending limit enforced by your code, per user, per day or per month.
- LLM observability: Tracing every model call with inputs, outputs, latency, tokens, cost and the user outcome.
