Which models run on a local LLM server in 2026?
Last updated
Which models run on a local LLM server in 2026 depends on its memory. A 24 GB GPU serves 8B to 32B models and mixture-of-experts models like Qwen3-30B-A3B. A 70B model at 4-bit needs roughly 40 GB (approximate), so 64 GB or more.
Olares One → Local LLM server → vLLM · Ollama · OpenAI-compatible API → Your models, served on your network
Which models run on a local LLM server in 2026?
The models that run on a local LLM server in 2026 are set by its memory first and its speed second. At 4-bit, a model needs roughly 0.5-0.6 GB per billion parameters plus a few GB for context, by our approximate rule. A 24 GB server therefore holds models up to about 32B.
| Model size | Approx. memory at 4-bit | Server class that fits it |
|---|---|---|
| 8B | ~5 GB | Any 24 GB GPU, with room for many users |
| 14B | ~9 GB | Any 24 GB GPU |
| 32B | ~19-20 GB | 24 GB GPU with little room left; 32 GB GPU more comfortably |
| 70B | ~40 GB | 64 GB or larger unified memory, or several GPUs |
| 120B | ~65 GB | 128 GB unified memory, or a 24 GB GPU spilling into system RAM |
Why do mixture-of-experts models suit a server?
Mixture-of-experts models suit a server because they take the memory of a large model and run at nearly the speed of a small one. Each token passes through a few experts, so only a small part of the model works per token, while the whole model stays loaded. On 24 GB, we'd serve one as the main model.
Qwen3-30B-A3B is the example Olares tested. On Olares One it ran at 157 tokens per second with one request and about 81 per request at 8 concurrent, on vLLM, in Olares' own test. We'd pick a dense 32B instead only if it clearly answers your team's tasks better.
What can't a 24 GB server serve well?
A 24 GB server can't serve models above about 32B at team speed. gpt-oss-120b runs on Olares One by spilling into its 96 GB of system memory, at about 36 tokens per second in Olares' own test. That's usable for one person and slow for a group. Dense 70B models spill worse, because every token touches every weight.
The largest open models are out of reach entirely. According to Olares' FAQ, Kimi K2, at about 1 trillion parameters, needs about 1,024 GB of VRAM and is not in the Olares Market.
How should a team mix its models?
A team does well with one fast general model served on vLLM, a small model for quick jobs like tagging, and an embedding model for document search. Larger or specialist models can sit in Ollama and load on demand. They share VRAM with the main model, so leave headroom.
For the hardware behind this mix, read a server for a 32B model and local LLM server hardware.
Questions & answers
Can a local LLM server run image models too?
Yes, if it has a capable GPU. In Olares' own test on Olares One, FLUX.1 dev with Nunchaku FP4 made the first image in 15.51 s and later images in 8.32 s each.
Can I run several models at once on 24 GB?
Small ones, yes. Larger models compete for VRAM, so most servers keep one main model loaded and swap others in on demand.
Sources for this entry
- Olares One FAQ (Olares docs) (Company-published)
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Local LLM Server. “Which models run on a local LLM server in 2026?.” Content reviewed 13 November 2026.
https://localllmserver.com/models-for-a-local-llm-server/