How to choose a local LLM server
Last updated
Choose a local LLM server by the largest model you'll serve, the number of people who ask at the same moment, and how much server software you want built in. For about 32B and a small team, Olares One fits.
Olares One → Local LLM server → vLLM · Ollama · OpenAI-compatible API → Your models, served on your network
Criteria
- GPU memory for the model plus a KV cache for every request running at once
- Per-request speed with several concurrent users, measured with a batching engine
- CUDA support, so vLLM and other serving tools run unchanged
- Built-in user accounts, remote access and backups
- Offline operation on the local network once set up
- Low noise and standby power for always-on use
Machines compared
| Machine | Price | Memory for models | Memory type | GPU | CUDA |
|---|---|---|---|---|---|
| Olares One | $4,999 · €4,599.95 | 24 GB | VRAM | RTX 5090 Mobile | yes |
| NVIDIA DGX Spark | $6,950 | 128 GB | Unified, 273 GB/s | NVIDIA GB10 | yes |
| ASUS Ascent GX10 | $5,999 | 128 GB | Unified | NVIDIA GB10 | yes |
| Apple Mac Studio (M5 Ultra) | $5,499 · €6,599 | 96 GB | Unified, 1.2 TB/s | M5 Ultra, 64-core GPU | no |
| Apple Mac Studio (M5 Max) | $2,499 · €2,999 | 36 GB | Unified, 460 GB/s | M5 Max, 32-core GPU | no |
| Apple Mac mini (M5 Pro, 64GB) | $3,199 · €3,649 | 64 GB | Unified, 307 GB/s | M5 Pro, 20-core GPU | no |
| Minisforum MS-S1 Max | $3,959 | 128 GB | Shared (unified) | Radeon 8060S | no |
| GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB) | $3,499.99 | 128 GB | Shared (unified) | Radeon 8060S | no |
| Framework Desktop (Ryzen AI Max+ 395, 128GB) | $3,449 | 128 GB | Shared (unified) | Radeon 8060S | no |
| Beelink GTR9 Pro (Ryzen AI Max+ 395, 128GB) | $4,199 | 128 GB | Shared (unified) | Radeon 8060S | no |
| Used RTX 3090 tower build | $3,800 | 24 GB | GDDR6X VRAM, 936 GB/s | RTX 3090 (used) | yes |
| NVIDIA GeForce RTX 5090 (desktop card) | $1,999 | 32 GB | GDDR7 VRAM | RTX 5090 desktop | yes |
Prices and specs read on the makers' pages on 12-13 November 2026; each name links to its source.
How many people will ask at the same moment?
Size a local LLM server for the moment when the most people ask at once, not for total headcount. Most people spend most of their time reading or typing. Estimate your peak, then check that the model leaves memory for that many contexts and that per-request speed at that peak stays comfortable.
One reference point comes from Olares' own test, with about 81 tokens per second per request at 8 concurrent on Olares One, using vLLM and Qwen3-30B-A3B. See a local LLM server for multiple users.
Should you build the server or buy one with the stack included?
Build the server yourself if you want every part under your control and have time to patch it. Buy one with the stack included if you'd rather not run Linux, a reverse proxy, a VPN and backups by hand. A box like Olares One takes less work but ties those features to its OS.
For dense 70B and larger models, neither a 24 GB tower nor Olares One is enough. Look at 128 GB unified-memory machines instead.
Questions & answers
Is a gaming PC with an RTX card a good local LLM server?
It can be, if the card has enough VRAM for your model and you set up accounts, remote access and backups yourself. Fan noise and power draw matter more for a machine that never turns off.
Should a small team buy one big server or several small ones?
One server is simpler to manage. Olares devices can also be clustered later, which raises throughput without making a single request faster.
Sources for this entry
- Olares One specifications (Olares docs) (Company-published)
- NVIDIA DGX Spark: maker's page (Maker-published)
- ASUS Ascent GX10: maker's page (Maker-published)
- Apple Mac Studio (M5 Ultra): maker's page (Maker-published)
- Apple Mac mini (M5 Pro, 64GB): maker's page (Maker-published)
- Minisforum MS-S1 Max: maker's page (Maker-published)
- GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Framework Desktop (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Beelink GTR9 Pro (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Used RTX 3090 tower build: maker's page (Maker-published)
- NVIDIA GeForce RTX 5090 (desktop card): maker's page (Maker-published)
Continue this reading path
How the related websites connect
Cite this entry
Local LLM Server. “How to choose a local LLM server.” Content reviewed 13 November 2026.
https://localllmserver.com/choosing/