Batch API (LLM)
Sending many model requests as one job that finishes within hours instead of seconds, usually at a large discount.
Major providers offer batch endpoints for work nobody waits on: classifying a backlog, enriching a catalog, running evals overnight, generating descriptions. You upload a file of requests and collect results later. The trade is latency for price and separate rate limits; it does not suit anything a user is waiting for.
Related terms
- Tokens: The pieces of text models read and write, and the unit you pay for.
- Cost cap: A hard spending limit enforced by your code, per user, per day or per month.
- Prompt caching: Reusing an identical prompt prefix across requests so it is cheaper and faster to process.
