DeepSeek-Coder-V2 scored higher than GPT-4 Turbo on HumanEval when it was released in mid-2024. That is not a marketing claim—it is a benchmark result that surprised a lot of people who assumed local models were a compromise. The assumption was reasonable a year ago. It is not reasonable now.
< We will cover which models are genuinely worth your time, how to run them without a PhD in MLOps, what hardware you actually need, and where local models still fall short—because they do fall short in specific, predictable ways.Why Local Models Have Caught Up—and Where They Haven't
The short version: parameter-efficient architectures, better training data curation, and the distillation of frontier model outputs have compressed what used to require 175 billion parameters into models that fit on a consumer GPU. Mistral's mixture-of-experts approach, Meta's work on Llama 3, and Chinese labs like DeepSeek and Alibaba have all pushed the frontier down in size while keeping benchmark scores up.
On coding tasks specifically, the gap closed first. HumanEval, MBPP, and LiveCodeBench all show that models in the 7B–34B range now compete with early GPT-4 on Python, JavaScript, and SQL generation. The reason is straightforward: code has a ground truth. You can run the output and check it. That makes training signal much cleaner than, say, creative writing, so smaller models learn faster.
Where local models still genuinely lag: very long context reasoning (anything requiring coherent synthesis across 100,000+ tokens), multi-step tool use where the model must chain API calls reliably, and tasks that require real-time web knowledge. If your workflow involves summarizing a 300-page codebase in one shot or autonomously browsing the web to research a bug, you will still hit the ceiling faster locally. For most day-to-day developer tasks—autocomplete, test generation, refactoring, explaining code, writing documentation—the ceiling is high enough that you won't reach it.
The Four Models Worth Running Right Now
This is not an exhaustive catalog. These are the four that a working developer should actually try, in rough order of how often they will be the right answer.
Qwen2.5-Coder-32B-Instruct
Released by Alibaba's Qwen team in October 2024, this is currently the best open-weight model for code on most benchmarks. It outperforms DeepSeek-Coder-V2-Lite and matches or beats GPT-4o on HumanEval and MultiPL-E. The 32B parameter count is the sweet spot: you get genuinely strong reasoning without needing a data-center GPU. On a Mac Studio with 96 GB unified memory, it runs comfortably at around 15–20 tokens per second. On a single RTX 4090 (24 GB VRAM), you need 4-bit quantization via GGUF, which drops quality slightly but keeps it competitive. This is the model to try first if your primary use case is writing and reviewing code.
DeepSeek-Coder-V2
DeepSeek's second-generation coding model, released June 2024, uses a mixture-of-experts architecture with 236B total parameters but only 21B active at inference time. That architecture is what lets it punch above its weight on a memory budget. The full model requires more VRAM than most developers have—realistically you are running the Lite variant (16B) or a quantized version. Even the Lite variant beats GPT-3.5 comfortably on coding tasks and is competitive with early GPT-4. DeepSeek's training methodology is unusually well documented in their technical report, which is worth reading if you care about why a model behaves the way it does.
Meta Llama 3.1 70B
This is the general-purpose choice when you need strong reasoning across domains, not just code. Llama 3.1 70B scores well on MMLU, MATH, and reading comprehension, and its instruction-following is reliable enough that it handles complex multi-step prompts without falling apart. It requires either a multi-GPU setup on consumer hardware (two 3090s or two 4090s) or Apple Silicon with at least 64 GB unified memory. For a developer who wants one model for code, documentation, architecture discussions, and general Q&A, 70B is the answer. The 8B variant is much easier to run and still respectable, but the quality drop on complex reasoning is real and noticeable.
Mistral 7B / Mistral Nemo 12B
The original Mistral 7B remains the best argument for small models. It fits on any GPU with 8 GB VRAM, runs at 40+ tokens per second on modern hardware, and is genuinely useful for autocomplete, simple refactoring, and scripting tasks. Mistral Nemo (12B, released mid-2024 in partnership with NVIDIA) extends this with a 128K context window and noticeably better instruction following. If your machine is modest—say, a laptop with an RTX 3060—Nemo 12B is where you should start. You will outgrow it for complex tasks, but it will cover 70% of what most developers need from an AI assistant.
Getting These Models Running in Under 30 Minutes
Ollama is the right answer for most developers. It is an open-source tool that handles model downloading, quantization management, and serving a local API endpoint—all with a single CLI. You install it, type ollama run qwen2.5-coder:32b, and it downloads the quantized GGUF model and starts a server at localhost:11434 that speaks the OpenAI API format. Any tool built for OpenAI—Continue.dev, Cursor via local model config, Aider—can point at that endpoint with a one-line config change.
The alternative is LM Studio, which provides a GUI and is easier if you are not comfortable with the command line. It supports the same GGUF format, has a built-in chat interface, and also exposes an OpenAI-compatible server. The tradeoff is that Ollama is faster to script and integrate into automated workflows, while LM Studio is easier to experiment with when you are still deciding which model you want.
For the Llama models specifically, you need to accept Meta's license on Hugging Face before the weights download. This takes about 90 seconds and is automatic through Ollama's model library. DeepSeek and Qwen models have more permissive licenses and download without a gate.
- Install Ollama from ollama.ai (macOS, Linux, Windows via WSL2 are all supported natively as of late 2024).
- Pull your chosen model:
ollama pull qwen2.5-coder:32b-instruct-q4_K_M. Theq4_K_Msuffix specifies 4-bit quantization with a K-quant method—this is the best quality-to-size tradeoff for most use cases. The model will be roughly 20 GB on disk. - Test the API:
curl http://localhost:11434/v1/chat/completions -d '{"model":"qwen2.5-coder:32b-instruct-q4_K_M","messages":[{"role":"user","content":"Write a Python function to flatten a nested list"}]}' - Connect your editor: In Continue.dev (VS Code extension), set the model provider to Ollama and select your pulled model. In Aider, pass
--openai-api-base http://localhost:11434/v1 --model ollama/qwen2.5-coder:32b-instruct-q4_K_M.
The whole process, including model download time on a fast connection, is under 30 minutes. The main failure mode is running out of VRAM and getting a cryptic error. Check your available VRAM first with nvidia-smi and cross-reference against the model card's memory requirements before pulling.
Hardware Reality: What You Actually Need
There is a lot of vague advice about this online. Here is the specific breakdown.
Apple Silicon (M2 Pro / M3 Pro and above): This is genuinely the best consumer platform for local LLMs right now. Unified memory means the GPU and CPU share the same pool, so a Mac Studio with 96 GB can load Llama 3.1 70B in full precision. The M3 Max with 128 GB is overkill for most developers but future-proofs you as models grow. An M2 Pro MacBook Pro with 32 GB handles Qwen2.5-Coder-32B at 4-bit quantization with acceptable speed—roughly 10–12 tokens per second, which is fast enough for interactive use. The Metal backend in llama.cpp (which Ollama uses under the hood) is well-optimized and keeps improving.
NVIDIA GPUs: An RTX 4090 with 24 GB VRAM is the best single-card consumer option. At 4-bit quantization, it fits Qwen2.5-Coder-32B. An RTX 3090 or 4080 (16 GB) limits you to 13B–14B models at 4-bit. Two 3090s connected via NVLink give you 48 GB effective VRAM and can run 70B models. Older cards with 8–12 GB VRAM limit you to Mistral-class models, which are still useful but not competitive with GPT-4.
AMD GPUs: ROCm support has improved considerably in 2024. An RX 7900 XTX (24 GB VRAM) now runs most models that the RTX 4090 can handle, using the ROCm backend for llama.cpp. The toolchain is still rougher than CUDA, and you will spend more time on setup. Worth it if you have the hardware; not worth buying AMD specifically for this purpose.
CPU-only: A modern CPU with enough RAM can run quantized models, just slowly. An M2 MacBook Air with 16 GB unified memory can run Mistral 7B at 3–4 tokens per second via Metal—technically viable for batch tasks but too slow for interactive coding assistance. On an x86 CPU, the AVX-512 instruction set matters; newer Intel and AMD desktop processors handle it, most laptops do not.
The practical recommendation: if you are buying hardware specifically for local LLMs, a Mac Studio with 64–96 GB unified memory is the most cost-effective option per dollar per token per second as of late 2024. If you are using existing hardware, run what you have and size up only if you hit a specific wall.
Integrating Local Models into a Real Developer Workflow
The model running locally is worth nothing if it is not actually in your loop. Here is what integration looks like for the three workflows where developers get the most value.
In-editor autocomplete and chat
Continue.dev is the open-source VS Code and JetBrains extension built specifically for this. It supports Ollama natively, lets you configure different models for autocomplete (where speed matters, so use a 7B–14B model) versus chat (where quality matters, so use the 32B). The tab-completion works like GitHub Copilot. The chat sidebar handles multi-file context by letting you @-mention files from your workspace. This is the integration most developers should set up first.
Agentic coding with Aider
Aider is a terminal-based AI coding assistant that can edit multiple files simultaneously, run tests, and iterate until tests pass. It was designed for GPT-4 but works with local models via the OpenAI-compatible Ollama endpoint. The caveat: agentic workflows that require many sequential reasoning steps expose weaknesses in smaller models faster. For Aider-style use, Qwen2.5-Coder-32B and Llama 3.1 70B are the minimums that produce reliable multi-step results. Below that, the model starts making edits that break other parts of the codebase in ways it fails to notice.
CI/CD and automated code review
This is where local models shine in ways cloud models simply cannot: you can run them inside a private network with no data leaving your infrastructure. A common pattern is a pre-commit hook or CI job that sends a diff to the local model and asks it to flag potential bugs, security issues, or style violations. You are not blocked by rate limits, you pay no per-token costs, and proprietary code never reaches a third-party server. The latency is higher than a cloud API if your hardware is modest, but for asynchronous CI use that rarely matters.
For all three workflows, the prompt engineering that works with GPT-4 largely transfers. Local instruction-tuned models respond to the same patterns: be explicit about the output format, provide context rather than assuming the model knows your codebase, and break complex tasks into smaller prompts rather than asking for everything at once.
Where Local Models Will Frustrate You
Being honest about this matters. There are specific scenarios where switching back to a cloud model is the right call, and knowing them in advance saves frustration.
Very large context windows used heavily: Qwen2.5-Coder supports up to 128K context tokens in principle, but inference speed drops steeply as you fill that window, and quality degrades on tasks requiring coherent reasoning across the full context. If you regularly work with contexts above 32K tokens—large codebases loaded wholesale, long conversation histories—you will notice quality and speed problems that cloud models with more compute handle better.
Frontier reasoning tasks: OpenAI's o1 and o3 models, and Google's Gemini 2.0 series, use inference-time compute scaling that is not available in any open-weight model as of early 2025. Tasks requiring hard mathematical reasoning or multi-step logical deduction across many variables still go to the frontier models for a reason. This affects developers working on algorithmic problems, optimization, or formal verification more than it affects typical application developers.
Unstable or misconfigured hardware: A model that occasionally produces garbled output because your GPU is thermally throttling is worse than no model at all—it introduces subtle bugs that are hard to catch. If you are running on a laptop that heats up under load, or a desktop with questionable cooling, check your temperatures under sustained inference load before trusting the output for anything important. nvidia-smi dmon and sensors (Linux) are your friends here.
Initial setup time: The first time you do this, expect to spend half a day on model selection, download, configuration, and editor integration. That is a one-time cost, but it is real. If you need AI assistance working this afternoon, use a cloud API today and set up local tomorrow.
Quantization, Model Formats, and What the Acronyms Mean
You will see terms like GGUF, Q4_K_M, GPTQ, and AWQ constantly when downloading models. Understanding them takes five minutes and saves significant confusion.
GGUF is the file format used by llama.cpp and therefore Ollama and LM Studio. It bundles model weights and metadata into a single file and supports loading layers across CPU and GPU memory, which is what makes it possible to run large models on consumer hardware. If you are using Ollama, you are always using GGUF.
Quantization compresses the model's weights from 16-bit or 32-bit floats to lower precision (4-bit, 5-bit, 8-bit) to reduce memory requirements. The quality tradeoff is real but often smaller than you expect. On coding tasks, a Q4_K_M quantized model typically scores within 2–5% of the full-precision version on benchmarks—often less than the noise between different prompt phrasings.
The most common quantization levels you will encounter:
- Q8_0: 8-bit, minimal quality loss, roughly half the memory of full precision. Use this if you have the VRAM.
- Q4_K_M: 4-bit with K-quant method (mixed precision for sensitive layers). Best general-purpose choice when memory is constrained. This is what you want for the 32B models on a 24 GB GPU.
- Q3_K_M / Q2_K: Very aggressive quantization. Quality degrades noticeably. Only worth it to squeeze a larger model into very limited memory—and usually it is better to just run a smaller model at Q4.
GPTQ and AWQ are alternative quantization methods designed for CUDA inference via the transformers library rather than llama.cpp. They can offer better speed on NVIDIA hardware for some model architectures, but the toolchain is more complex. Most developers should ignore these until they have outgrown GGUF-based workflows.
The practical rule: start with Q4_K_M in GGUF format via Ollama. If you have excess VRAM, upgrade to Q5_K_M or Q8_0. Do not go below Q4 unless you have no other option.
Frequently Asked Questions
Can a local LLM really beat GPT-4 on coding tasks?
On specific benchmarks like HumanEval and MultiPL-E, yes—Qwen2.5-Coder-32B and DeepSeek-Coder-V2 both match or exceed GPT-4's scores on code generation. In practice, the comparison is more nuanced: GPT-4o has stronger reasoning on complex multi-step problems and handles ambiguous instructions better, but for well-defined coding tasks like function generation, test writing, and refactoring, the best local models are genuinely competitive.
What is the minimum GPU VRAM needed to run a useful local LLM?
8 GB VRAM is enough to run Mistral 7B at 4-bit quantization, which handles basic coding assistance, scripting, and simple refactoring. For serious coding work with a model that competes with GPT-4, you want 24 GB VRAM (RTX 4090 or equivalent) to run 32B models at 4-bit quantization. Apple Silicon with 32+ GB unified memory is also viable and in some ways preferable.
Is Ollama safe to use with proprietary source code?
Yes, assuming your machine is not compromised. Ollama runs entirely locally; it makes no network calls with your prompts or completions. The model weights are downloaded once from Ollama's servers, after which inference is fully. If you are working in an air-gapped environment, you can pre-download the GGUF files and load them directly without any internet connectivity at inference time.
How does Qwen2.5-Coder compare to GitHub Copilot?
Qwen2.5-Coder-32B competes with Copilot's underlying model on benchmark coding tasks and often produces more complete, context-aware completions when given sufficient context. The practical difference is in integration quality: Copilot's VS Code extension has years of polish and telemetry-driven UX improvements. Continue.dev with Qwen locally is close but not identical. If data privacy is a concern, local Qwen is the clear choice. If you want zero setup and your code is not sensitive, Copilot's convenience is real.
What is the difference between Llama 3.1 8B and 70B for developers?
For simple tasks—autocomplete, docstring generation, explaining a function—8B is adequate and fast. For complex tasks—refactoring across multiple files, debugging non-obvious logic errors, explaining an architectural tradeoff—70B produces noticeably better results. The quality gap is largest on tasks that require holding multiple constraints in mind simultaneously. If you run 8B for a week on real work and hit a ceiling, that is your signal to move to 70B.
Can local models run in a CI/CD pipeline without internet access?
Yes, and this is one of the strongest use cases for local models. You run Ollama (or a llama.cpp server) on a build agent inside your private network. The agent receives code diffs via your CI system, sends them to the local model for review, and posts results back to your pull request tool. No proprietary code leaves your network. The setup requires a machine with adequate GPU resources on your CI infrastructure, but the model itself runs as a standard HTTP service.
Do local LLMs work on Windows, or do you need Linux?
Ollama has had native Windows support since late 2024 and works without WSL2 on Windows 11. CUDA inference on NVIDIA GPUs works on Windows. LM Studio has always had a polished Windows installer. Linux gives you more flexibility and slightly better performance in some configurations, but Windows is now a fully supported platform for local model inference.
How often do local model options change, and how do I keep up?
The local LLM space moves fast—major new models appear roughly every one to three months. The most reliable way to track what is currently best for coding is the HuggingFace Open LLM Leaderboard filtered to coding benchmarks, and the BigCode Evaluation Harness results published alongside new model releases. EvalPlus.ai publishes HumanEval+ results that are harder to game than the original benchmark. Following the Qwen, DeepSeek, and Meta research blogs directly is also worthwhile.