Quantization
Storing a model’s weights with fewer bits, so it needs less memory and runs faster, at a small cost in quality.
A model stored in 16 bits per weight needs about two bytes per parameter. Common 4-bit formats need roughly a third of that, which is what lets mid-sized open models run on a single workstation. The memory needed is about parameters times bits per weight divided by eight, plus room for the context.
Related terms
- Local LLM: A language model running on hardware you control instead of a provider’s API, so documents stay where you decide.
- Tokens: The pieces of text models read and write, and the unit you pay for.
- Model routing and fallback: Sending each request to the model that fits it, and to another one when the first fails.
