How to Run Coding Agents with Ollama: The Complete Local Setup Guide

Also read: Coding Agent Pricing in 2026 · Local-First Wins: Ollama’s $65M Round · Best Coding Agents Decision Guide

Ollama crossed 9 million builders and closed $65M because developers want local-first AI. But most coding agents ship with cloud-first defaults. The gap between “Ollama is installed” and “my coding agent actually uses it” is where most people quit.

This guide closes that gap. You will have a working local coding agent by the end — no cloud API key required, no subscription, no per-token bill.

Why local models for coding agents?

Three reasons that matter more than ideology:

1. Cost. A single Claude Code session on Opus 4 can burn $2-5 in tokens. A local model on a single GPU costs $0 per session after hardware. For teams running 10+ sessions a day, local saves hundreds monthly.

2. Latency. Cloud APIs add 200-800ms of network round-trip on every call. Local inference on a decent GPU returns tokens in 30-80ms. Over a 200-token response, that is 30-40 seconds of network wait you eliminate.

3. Privacy. Your codebase never leaves your machine. For regulated industries, defense, or proprietary codebases, this is not optional — it is policy.

The tradeoff is real: local models trail frontier cloud models on complex multi-file refactors and nuanced architectural decisions. But for the 80% of coding tasks that are straightforward — bug fixes, renames, test generation, boilerplate, documentation — local models are good enough, and improving fast.

Step 1: Install Ollama

Ollama runs on macOS, Linux, and Windows (via WSL2 or native). The install is one command on each platform.

macOS / Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows (PowerShell as admin):

winget install Ollama.Ollama

Verify it works:

ollama --version
ollama serve   # starts the local server on port 11434

Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1. This is the key detail — every coding agent that supports custom OpenAI-compatible endpoints can talk to Ollama without any special integration.

Step 2: Pick the right model for coding

Not all local models write good code. Here is what actually works for coding agents, based on real usage across terminalblog’s test suite.

Start here: If you have 8GB VRAM, grab Qwen3-Coder-7B. If you have 16GB, grab Mistral-Small-24B or Qwen3-30B-A3B. If you have 48GB+, DeepSeek-Coder-V3 gives you near-frontier performance.

# Quick start — 7B model, fits on most GPUs
ollama pull qwen3-coder:7b

# 16GB VRAM sweet spot
ollama pull mistral-small:24b

# Maximum local quality (48GB+ VRAM)
ollama pull deepseek-coder-v3-0324

Pro tip: Use the :q4_K_M quantization tag for the best speed-to-quality balance. Example: ollama pull qwen3-coder:7b-q4_K_M.

Step 3: Wire Ollama into your coding agent

Each agent connects differently. Here is the actual config for the most popular ones.

Claude Code (via OpenAI-compatible proxy)

Claude Code does not directly support Ollama, but it supports custom API endpoints through provider configuration. Set the ANTHROPIC_BASE_URL or use a proxy like LiteLLM to bridge Ollama’s OpenAI-compatible API.

# Option A: LiteLLM proxy (recommended)
pip install litellm
litellm --model ollama/qwen3-coder:7b --port 4000

# Then point Claude Code to it
export ANTHROPIC_BASE_URL=http://localhost:4000

OpenCode

OpenCode has native Ollama support. Add to your config:

{
  "providers": {
    "ollama": {
      "baseURL": "http://localhost:11434/v1",
      "apiKey": "ollama",
      "models": {
        "default": "qwen3-coder:7b"
      }
    }
  }
}

Hermes Agent

Hermes routes to any OpenAI-compatible endpoint. Add Ollama as a custom provider:

hermes config set provider ollama
hermes config set ollama.base_url http://localhost:11434/v1
hermes config set ollama.model qwen3-coder:7b
hermes config set ollama.api_key ollama

Kilo Code CLI

Kilo supports custom providers in its config. Set:

kilocode config set provider openai-compatible
kilocode config set base_url http://localhost:11434/v1
kilocode config set model qwen3-coder:7b
kilocode config set api_key ollama

Goose

Goose reads from config.yaml. Add the Ollama provider:

provider:
  type: openai-compatible
  base_url: http://localhost:11434/v1
  model: qwen3-coder:7b
  api_key: ollama

OpenClaw

OpenClaw supports Ollama natively through its provider picker:

openclaw --provider ollama --model qwen3-coder:7b

Step 4: Validate it works

Before you rely on local models for real work, run a quick smoke test.

# 1. Confirm Ollama is serving
curl http://localhost:11434/v1/models | head -20

# 2. Quick completion test
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder:7b",
    "messages": [{"role": "user", "content": "Write a Python function that checks if a string is a palindrome"}],
    "max_tokens": 500
  }' | python -m json.tool

# 3. Test in your actual coding agent
# Open a new session and ask: "What files are in the current directory?"
# The agent should list your files — this confirms the model + tool use loop works.

What to watch for:

• If the model responds but tool calls fail, the model may not support function calling. Switch to Qwen3-Coder or Mistral-Small — both have strong tool-use training.
• If responses are extremely slow, check nvidia-smi or htop to confirm GPU utilization. If VRAM is full, the model is CPU-offloading, which is 5-10x slower.
• If you get empty responses, check that ollama serve is running and the model name matches exactly.

Step 5: Tune for coding workloads

Default Ollama settings are optimized for chat, not coding. These tweaks help.

Increase context window

# Set a larger context window for code-heavy sessions
OLLAMA_NUM_CTX=32768 ollama serve

# Or per-model in Modelfile
cat > Modelfile << 'EOF'
FROM qwen3-coder:7b
PARAMETER num_ctx 32768
PARAMETER temperature 0.1
EOF
ollama create qwen3-coder-coding -f Modelfile

Lower temperature for deterministic code

# In your Modelfile or API call
PARAMETER temperature 0.1

Lower temperature means less creative variation — exactly what you want for code generation where correctness matters more than diversity.

Adjust repeat penalty

PARAMETER repeat_penalty 1.1

This prevents the model from getting stuck in code-generation loops, a common failure mode with smaller local models.

What local models cannot do (yet)

Honest limitations so you do not waste time:

1. Complex multi-file refactors. Local models under 30B parameters struggle with changes that span 5+ files and require understanding architectural relationships. For these tasks, switch to a cloud model.

2. Large codebase comprehension. Context windows on local models cap at 32-128K tokens. If your codebase context exceeds that, the model operates blind. Cloud models with 200K+ context handle this better.

3. Advanced reasoning chains. Tasks like “trace the execution path through 15 functions across 3 modules” still favor frontier models. Local models can handle simpler traces.

4. Model upgrades. Cloud providers ship new models monthly. Local models only improve when you manually pull updates. You are your own upgrade path.

The pragmatic approach: Use local for routine work (80% of tasks) and switch to cloud for complex reasoning (20% of tasks). Most coding agents let you change models mid-session — use that.

Cost comparison: local vs cloud

*GPU amortized over 24 months: RTX 4090 ($1,600) = $67/mo; A6000 ($4,500) = $188/mo.

For solo developers, the math is simple. For teams, local becomes cost-effective within 6-12 months of a single GPU purchase.

Troubleshooting common issues

Model not loading / “out of memory”:

ollama ps                    # check what is loaded
ollama stop                  # unload all models
ollama pull qwen3-coder:7b-q4_K_M  # lighter quantization

Slow generation (< 5 tokens/sec): Check GPU utilization. If the model is partially on CPU, you need a smaller model or more VRAM. Run nvidia-smi to verify the model fits entirely in GPU memory.

Agent disconnects after first response: Some coding agents expect cloud-style streaming. Ollama supports streaming, but if your agent has timeout issues, increase the timeout setting or use a smaller model that responds faster.

Tool calls / function calling not working: Not all Ollama models support function calling well. Stick to models with explicit tool-use training: Qwen3-Coder, Mistral-Small, and DeepSeek-Coder are the safest choices. Check with ollama show <model> for supported features.

Related articles

• Coding Agent Pricing in 2026 — full cost breakdown across all 15 agents
• The Case for Lightweight Coding Agents — why less overhead wins
• Open Source vs Commercial Coding Agents — operator fit, not ideology
• Local-First Wins: Ollama’s $65M Round — why local-first is mainstream