Free tool
A language model on your own hardware, or an API?
Law firms, accountants, clinics and plants ask the same thing: can we use AI on our documents without sending them outside? Pick a model, the hardware and how much your team asks, and see if it fits, how fast it answers and what it costs against an API.
Where your data lives
| Option | Where the documents go | Who runs it |
|---|---|---|
| Your own hardware | Stays in your office or server room | You (or me, under contract) |
| A model at an EU host | An EU data centre under a contract you sign | The host and you |
| A public API | The provider’s servers, under its terms and data settings | The provider |
How the numbers are calculated
Memory: parameters × bits per weight ÷ 8, using the bits measured in llama.cpp, plus 10% and 2 GB for context and the runtime, and 16 GB left for the operating system. Speed: generating text is limited by memory bandwidth (Databricks), so tokens per second ≈ bandwidth ÷ model size × 60%. The overheads and the 60% are my assumptions. Reading a long input before the first word adds seconds that this simple model does not count.
Hardware numbers
Memory and bandwidth come from the makers: AMD lists 256-bit LPDDR5x-8000 for the Ryzen AI Max+ 395, which is 256 GB/s, and up to 128 GB; Apple lists 614 GB/s for the M5 Max with the 40-core GPU and 1.2 TB/s for the M5 Ultra, up to 512 GB. Prices, power draw and API prices are examples. Type in your own quote.
When local makes sense
Rarely on cost alone: a busy team pays an API less than the electricity and hardware of its own box. It makes sense when documents must not leave the company, when you need it to work offline, or when you want a Polish model like Bielik that you control.
Questions
Can we run AI on our documents without sending them out?
Yes. A model on your own hardware or at an EU host, with search over your documents (RAG), answers questions without the documents leaving the infrastructure you choose.
Is Bielik good enough?
For Polish text it is a strong open model, and the 11B version fits comfortably on a 128 GB machine. Whether it is good enough depends on your questions: test it on 30 to 50 real ones before deciding.
What is quantization?
Storing model weights with fewer bits. A 4-bit model needs about a third of the memory of a 16-bit one and runs faster, with a small loss of quality.
Can you set it up for us?
Yes: hardware or an EU host, the model, search over your documents, a simple panel and a test set that measures how often answers are right. The configurator shows the price and the start date.
Sources
- llama.cpp, quantize README (bits per weight: Q4_K_M 4.89, Q5_K_M 5.70, Q8_0 8.50, F16 16.0)
- Databricks, “LLM Inference Performance Engineering: Best Practices” (decoding is memory-bound)
- AMD, Ryzen AI Max+ 395 specifications (256-bit LPDDR5x-8000, up to 128 GB)
- Apple, Mac Studio technical specifications (M5 Max up to 614 GB/s, M5 Ultra 1.2 TB/s, up to 512 GB)
- SpeakLeash on Hugging Face (Bielik 11B v3 and Minitron 7B v3)
Want AI on your documents, with the data staying where you decide?
I set up the model, the search over your documents and a panel for your team, measure how often it answers right, and look after it afterwards.
