Local LLM
A language model running on hardware you control instead of a provider’s API, so documents stay where you decide.
Open models can run on your own server, a workstation or a host in your region. You trade the convenience and quality of the largest hosted models for control over where data goes and a fixed cost. Whether a model fits depends on its size and quantization; how fast it answers depends mostly on memory bandwidth.
Related terms
- Quantization: Storing a model’s weights with fewer bits, so it needs less memory and runs faster, at a small cost in quality.
- RAG (retrieval-augmented generation): Finding relevant documents first and putting them into the prompt, so the model answers from them.
- Tokens: The pieces of text models read and write, and the unit you pay for.
