Model routing and fallback
Sending each request to the model that fits it, and to another one when the first fails.
Cheap, fast models handle simple steps; strong models handle hard ones. A fallback chain covers outages, rate limits and refusals. Models change often, so the model name should be configuration, not code scattered across the app.
Related terms
- Evals: A repeatable test set that scores an AI feature, so changes can be measured instead of guessed.
- AI gateway: A proxy between your app and model providers that adds logging, caching, limits, billing and fallbacks.
- Tokens: The pieces of text models read and write, and the unit you pay for.
