Semantic cache
Reusing a stored model answer when a new question means the same as an earlier one, matched by embedding similarity rather than exact text.
“What is your refund policy” and “how do refunds work” can share one answer, which saves the model call and its latency. The risk is a threshold set too loose: “cancel my order” and “cancel my subscription” look similar and need different answers. It is a different thing from prompt caching, where the provider reuses the processed beginning of a prompt and the answer is still generated fresh.
Related terms
- Prompt caching: Reusing an identical prompt prefix across requests so it is cheaper and faster to process.
- AI gateway: A proxy between your app and model providers that adds logging, caching, limits, billing and fallbacks.
- Embeddings: Vectors of numbers that place texts with similar meaning close together.
