How to run a local LLM server
Last updated
To run a local LLM server, pick a model that fits your GPU memory, install a serving engine (Ollama or vLLM), add a chat interface such as Open WebUI, expose an OpenAI-compatible API, then set up accounts, remote access and backups.
Olares One → Local LLM server → vLLM · Ollama · OpenAI-compatible API → Your models, served on your network
How do you run a local LLM server?
How to run a local LLM server comes down to a few steps. Choose a model that fits your GPU memory, serve it with Ollama or vLLM, and put Open WebUI in front for people. Apps connect through the engine's OpenAI-compatible API. Then add accounts, a VPN for remote use and backups.
The first two choices decide whether the rest goes smoothly. Model size sets the hardware. At 4-bit, a 14B model needs roughly 9 GB and a 32B model roughly 19-20 GB, plus a few GB for context, by our approximate rule. Each extra user needs memory for their own context on top of that, which local LLM server hardware covers.
Which serving engine should you start with?
Start with Ollama if one person will use the server at a time, and move to vLLM once several people ask at the same moment. That's our recommendation. Ollama installs in minutes and swaps models easily. vLLM batches requests, so each user keeps a fast stream under load. Both expose an OpenAI-compatible API, so the switch doesn't break your apps.
In Olares' own test on Olares One with Qwen3-30B-A3B, vLLM gave about 81 tokens per second per request at 8 concurrent, and Ollama gave 32. The full comparison is in vLLM or Ollama.
What does the do-it-yourself path look like?
The do-it-yourself path runs on a Linux machine with an NVIDIA GPU. Install the NVIDIA driver, CUDA and Docker. Run Ollama or vLLM in one container and Open WebUI in another. Put a reverse proxy with TLS in front, and give remote users a VPN instead of opening ports to the internet.
Schedule backups of the chat database and any documents you index. Every piece is standard, and every update is yours to apply.
How does the same setup work on Olares One?
On Olares One, the operating system already provides accounts, a built-in VPN and backups, so the setup shrinks to activation and two installs. Activation needs an internet connection, an Olares ID and the LarePass app, according to Olares' FAQ. Ollama and Open WebUI then install from the Olares Market with one click each.
After activation the box can work entirely offline on the local network, though the Market needs internet. Read securing a local LLM server before inviting users, and load-test with as many parallel requests as your busiest hour brings.
Questions & answers
What is the easiest way to run a local LLM server?
Ollama with Open WebUI on a machine that has enough VRAM for your model. On Olares One both are one-click installs from the Olares Market.
Do I need Docker?
On a plain Linux server, containers make Ollama, vLLM and Open WebUI easy to update and remove. Olares OS runs apps on its own Kubernetes-based system, so you install from its Market instead.
Sources for this entry
- Olares One FAQ (Olares docs) (Company-published)
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
- Olares One product page (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Local LLM Server. “How to run a local LLM server.” Content reviewed 13 November 2026.
https://localllmserver.com/how-to-run-a-local-llm-server/