How to Run DeepSeek V4-Flash Locally

You’re going to serve DeepSeek V4-Flash-0731 - the MIT-licensed, 284B/13B-MoE agentic model that tops the official Terminal-Bench 2.1 charts at 82.7 - on your own GPUs, sized honestly to your hardware. This guide is the practical companion to our VRAM reality check: same official figures, but with the exact commands to make it run.

What you need to know up front: this is a 284B-parameter MoE (13B active per token), and the smallest community build is 82.5GB. This is not a laptop project. The good news is that compute is cheap - the active params are small - so memory is the entire game.

Step 1: Pick your quantization tier

Community GGUFs (unsloth’s Dynamic 2.0 builds) come in a ladder from 82.5GB to 162GB, and the official repo ships at 166.9GB. Rule of thumb from the VRAM guide: VRAM needed ≈ weights + 10–15% headroom for runtime and KV cache. With the 1M-token context, cache grows into the tens of GB for long prompts.

Step 2: Install a runner

Option A - Ollama (recommended to start). One install, and it spreads big models across all your GPUs automatically:

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows (PowerShell)
irm https://ollama.com/install.ps1 | iex

Option B - plain llama.cpp (if you want the raw server and its flags):

# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh

# Windows
winget install llama.cpp

Both verify with a version check - you should see a version number, not an error:

ollama --version

Step 3: Pull the GGUF for your hardware

The tag is hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:<quant>. Replace <quant> with the tier from Step 1 - for a 2x80GB box, that’s UD-Q3_K_M:

# Ollama (substitute your tier in the tag)
ollama run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q3_K_M

The first ollama run pulls the weights, so expect a long progress bar for a 128GB file. Later runs are instant.

With plain llama.cpp the same model works directly from the HF tree:

# terminal chat (same :UD- tag syntax)
llama cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q3_K_M

# OpenAI-compatible server on port 8080
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q3_K_M

Step 4: Verify it loaded onto the GPU

This is the step most people skip, and the one that catches real problems:

ollama ps

Expected output shape (a row for your model, Processor column):

NAME                                              ID            SIZE    PROCESSOR   UNTIL
hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q3_K_M  bcfb190ca3a7  128 GB  100% GPU    4 minutes from now

The Processor column is the whole story:

• 100% GPU - fully in VRAM, best speed. On a 2x80GB box, Ollama loads the model spread across both cards (that’s the documented behavior when a model doesn’t fit a single GPU).
• 48%/52% CPU/GPU - partially offloaded; works, slower.
• 100% CPU - your GPU was not discovered. Skip to Troubleshooting, error #3.

Step 5: Chat and call it from code

In the terminal runner, just type. For the API, DeepSeek’s official card recommends temperature = 1.0, top_p = 0.95 for agentic scenarios (top_p = 1.0 otherwise), and reasoning effort levels low / high / max - with a recommended 384K maximum output at high/max effort. OpenAI-compatible call:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q3_K_M",
    "messages": [
      {"role": "user", "content": "Write a bash script that finds the 10 largest files in a directory tree."}
    ],
    "temperature": 1.0,
    "top_p": 0.95
  }'

Any tool that takes a custom OpenAI base URL now points at http://localhost:11434/v1 - your agent stack is running the 82.7 Terminal-Bench model locally.

How to verify it worked

Run this exact sequence and confirm each check:

ollama ps                      # 1. your model listed, GPU in Processor
curl http://localhost:11434/api/tags   # 2. JSON lists your tag
ollama run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q3_K_M "Say hello in three words."
# 3. responds promptly (vs seconds of CPU crawl)

A slow CPU-level crawl on a model that shows 100% GPU in ollama ps is expected first-token warm-up latency for 128GB of weights; sustained slowness means the model is not actually on the GPU.

Troubleshooting

1. ollama pull dies halfway, “transfer interrupted”. The download is huge; the connection dropped. Network resume is flaky for 100GB+ files. Best fix: derive the artifact locally - download the official repo once (166.9GB) and let the runner pick the quantization, or retry ollama run overnight on a stable connection.

2. The model loads, then errors out of memory. Your budget math was off. Recheck Step 1 with the 10-15% headroom rule: a 128GB build needs ~140GB+ of total memory across your cards, and context (KV cache) eats more at 1M tokens. Steps: drop a quant tier, reduce context, or add memory. A single 80GB card cannot run any V4-Flash build - do not spend a day trying.

3. ollama ps shows 100% CPU - GPU not discovered. Follow the official GPU-discovery checklist, in order: reboot; update drivers; on Linux confirm the container runtime works with docker run --gpus all ubuntu nvidia-smi; for NVIDIA load the uvm driver with sudo nvidia-modprobe -u; for AMD, Ollama ships ROCm 7 libraries and requires a ROCm 7 kernel driver - an older driver makes discovery hang and fall back to CPU. Logs on Linux: journalctl -u ollama -f. On macOS: ~/.ollama/logs/server.log.

4. “model not found” when running the tag. Tag typos are the classic cause - the tree uses hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF (that exact repo) with a suffix like :UD-Q3_K_M. Run ollama ls to list exactly what you have pulled, and compare the tag character by character.

5. Responses are slow even though ollama ps says GPU. Two usual suspects: context is set very high (1M context on 128GB of weights means the KV cache can dominate), and single-user single-request throughput expectations. Lower the context for interactive use, and confirm you are not running CPU-bound layers by checking the CPU/GPU split in ollama ps.

6. You want the DSpark speedup but don’t see it. DSpark is optional speculative decoding - the draft weights are in the GGUF tree (DSPARK Q8_0 ~10.9GB / BF16 ~11.3GB). Not every runner enables it by default. unsloth’s DeepSeek-V4 docs cover enabling it per runner; with it on, decoding reports up to 2x faster (community claim, not official DeepSeek figures).

When local beats the API

Before you buy GPUs, the honest math matters: DeepSeek’s official price for the 0731 build is $0.14 in / $0.28 out per million tokens (cache hits $0.0028), and our V4-Flash vs Qwen3.8-Max reconciliation shows why below a 192GB-class node, self-hosting mostly loses on cost. Local wins on three fronts only: privacy and air-gapped data, infinite agent-loop volume where per-call charges add up, and having the weights in-house under MIT terms. If none of those apply to you, use the API and skip the GPU budget - our model chooser has the full comparison.