Run Open-Source LLMs Locally: The 2026 Guide
From zero to a private, offline LLM stack on your own hardware: Ollama, Open WebUI, quantization, and the models worth running in 2026.
- 1
Install Ollama
Install Ollama from ollama.com or brew install ollama. It ships with a model runtime and CLI.
- 2
Pull a model
Run `ollama pull qwen3:8b` — a strong 8B parameter model for daily questions. For bigger GPUs, `ollama pull llama3.3:70b`.
- 3
Serve the API
Run `ollama serve` to expose OpenAI-compatible endpoint at localhost:11434/v1.
- 4
Add Open WebUI
Spin up Open WebUI with `docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui` for a ChatGPT-grade chat UI.
- 5
Verify & tune
Open http://localhost:3000, create an account, and check model swap followed by GPU utilization with `ollama ps`.
Running state-of-the-art open models on your own laptop is a 25-minute project in 2026 — and a growing privacy default. This guide runs the complete stack locally with zero cloud accounts.
Why go local
Local inference removes per-token cost, keeps prompts and data off third-party servers, and gives you freedom to swap any open-weight model. For teams handling sensitive code or research notes, it is the only sane baseline.
Hardware budget
An M-series Mac or any 16GB-VRAM GPU runs an 8B model comfortably at 40-60 tok/s; 70B-class models want 32GB+ or quantization to 4-bit.
Next steps
Point your agent tools at http://localhost:11434/v1 (OpenAI-compatible), and pin favorites in Open WebUI so the whole team shares one local endpoint.