How to Pick a Local Coding AI for Any GPU

A local coding AI is useful only when its weights, KV cache, and runtime scratch space all fit in your GPU memory with headroom to spare. Fitting the weights alone proves nothing: a model that barely loads can hold a conversation, but it cannot refactor five files. This guide walks every memory tier from 4GB to 512GB and shows where each one runs out of runway.

The three-part memory budget: weights, KV cache, and runtime scratch

Three things fight for your GPU memory the moment a model loads, and fitting only the first one proves nothing. The weights are the file you download. The KV cache is the running set of notes the model keeps on every token of your prompt and its own replies, which is what saves it from rereading the whole conversation per word. The runtime, whether that is llama.cpp or another inference engine, then takes its own scratch space just to do the math.

The KV cache is the part every sizing chart omits. A few thousand tokens of code and chat can consume hundreds of megabytes, and it keeps climbing as the conversation grows. On an 8GB card running a 9B model at roughly 5.75GB of weights, that leaves a little over 2GB of room, and one long file plus a back-and-forth session can eat it. The video's worked example, from the Claude Code hardware guide published September 20, 2026, makes the point plainly: on a tight card, context tips you over long before the weights do.

When memory runs short you have two dials, and most people only know one. Drop to a smaller quant so the weights take less, or trim the context so the cache takes less. Context is usually the cheaper thing to give up, which is why every size in the tiers below is a weights-only number and why you should always leave room on top.

4GB to 12GB: small models that handle single-file work

At 4GB to 12GB you get a real assistant for focused, single-file work, and the right spend is precision rather than size. The video's picks in this tier are 4B models such as SparkX 2.5 at about 2.6GB quantized, and Neo Horse at roughly 3GB at a higher-quality setting, which it reports beating its own base model by about six points across a ten-round agentic benchmark run. Treat both figures as the speaker's measurements, not independent results.

4GB cards: sharp scalpel, short reach

A 4B model explains code you did not write, fixes one contained bug, and makes small edits. It loses the thread when a task spans five files and needs a plan. One catch the charts leave out: SparkX 2.5 uses its own attention design, so your runtime must ship support for it or the file will not load at all.

8GB cards: the first junior pair-programmer

At 8GB a 9B model finally fits. Ornith 1.5 9B, released in August 2026 per the video and openly licensed, runs at about 5.75GB at 4-bit and scores in the mid-80s on the hard agent benchmark the speaker cites, which is strong for the size class. Keep prompts lean and it holds one file and a real conversation about it.

12GB cards: more bits, not a bigger brain

The common mistake at 12GB is jumping to a larger model. Running the same 9B at 8-bit, about 9.75GB of weights, gives you a steadier version of the model you already trust. Quantization, storing each weight in fewer bits, takes that 9B from roughly 18GB at 16-bit to under 6GB at 4-bit. The safe zone for most coding models is 4 to 5 bits; below about 3 bits quality drops sharply, which is why 1-bit builds are a party trick rather than a daily driver.

16GB to 32GB: the developer sweet spot for multi-file edits

Sixteen to 32GB is where most gaming cards live and where multi-file changes and local test runs start working. The trick at 16GB is non-uniform quantization: instead of squeezing every layer equally, the build measures how far each tensor can be compressed before outputs suffer, then spends more bits on sensitive layers and fewer on the rest under a fixed size target. The video reports a 27B model, Qwen 3.8 27B from the Qwen team, squeezed under 12GB this way while matching full-precision scores on one math benchmark and one coding benchmark exactly.

At 24GB, the classic sweet spot card, that same model runs at about 16.5GB of weights with the rest of the card free for a long file and a long conversation. This is the tier where a real change works end to end: rename a function in one file, update its callers in another, and fix the test that breaks in a third, all held in the model's view at once.

At 32GB you face a choice that teaches a rule. Max out the 27B at 6-bit, about 22GB, or step up to a 35B mixture-of-experts model at roughly 25GB. A mixture-of-experts model fires only a few of its parts per token, so the video's example runs about 3 billion active parameters per token at the speed of a small model while carrying 35B of knowledge. The part people miss: you still must fit all 35B in memory. Sparsity buys speed, not space, and the hardest reasoning can feel thinner than the headline size suggests.

48GB to 128GB: agentic coding and the PCIe offload bottleneck

This is the first tier where a model can run a coding agent that works a task across many steps: read the error, edit the file, run the tests, read the new error, and go again while holding the whole chain in context. The workhorse in the video is Qwen3-Coder-Next, an 80B model with native context of about a quarter million tokens. Reported weights footprints: roughly 38GB at a compact 4-bit on 48GB, about 48.5GB at full 4-bit on 64GB, a figure the speaker attributes to the official Qwen repository, and about 66GB at 6-bit on 80GB.

One single-person report in the video, flagged as anecdote rather than a controlled test, got useful output from this 80B model on a 24GB GPU plus 64GB of system RAM by offloading layers, with mixed reliability. That trick exposes the real bottleneck at this tier: the bus between GPU and system memory. When the model keeps reaching across PCIe for offloaded parts, the fast chip sits idle waiting on data, and the pain peaks during the first long prompt read. A runtime that prefetches during compute hides some of it; a careless setup wastes half the chip's time.

94GB versus 96GB: why bandwidth beats raw capacity

One gigabyte of capacity tells you nothing about speed, and the 94GB versus 96GB comparison proves it. The 94GB card is an H100 NVL, a data-center part with HBM memory and a high-speed interconnect between chips. The 96GB is an RTX PRO 6000 workstation card with GDDR7 and a slower link. Per the video, the workstation card has edged ahead on single-card raw throughput in published benchmarks, but the data-center link moves hundreds of gigabytes per second against the workstation's low tens, so across four cards the interconnect wins outright. Same chart, one gigabyte apart, opposite answers depending on whether your workload is one card or many.

141GB to 512GB: frontier models, runtime branches, and SSD streaming

At the top of the ladder, capacity is no longer the hard part; component placement and runtime support are. GLM 5.3 Flash from GLM's developers reportedly runs from about 120GB at a low quant to 240GB at a high one across the 141GB to 288GB range, with the full GLM 5.3 filling roughly 280GB to 467GB at 512GB. The catch: the architecture was not yet merged into the standard open-source runtime as of the video, so the working path is a branch maintained by the Unsloth team or their desktop app. Download weights without it and nothing runs.

When sparser beats bigger at 256GB

At 256GB you could load GLM 5.3 Flash at a high quant, about 200GB, or run MiniMax N3 from MiniMax at around 208GB. MiniMax is a 426B mixture-of-experts model that fires only 23B parameters per token, so it runs faster and leaves headroom for context. On the same box, the video argues, the sparser model is the one you would rather use all day.

SSD streaming: the newest idea on the chart

DeepSeek v4.1 Flash, released September 10, 2026 per the video and openly licensed on DeepSeek's repositories, is a 552B model. The Mac engine DwarfStar keeps a lookup structure permanently on the SSD and streams it in on demand, because that table is read infrequently and predictably enough to fetch from disk in time, while the hot core weights stay in fast memory. That is how a compressed version runs on a single 128GB Mac, and it is why raising the quality setting adds about 142GB while the on-disk table stays put. The video calls this experimental and finicky, and it is right to: do not bet a deadline on it yet. It also breaks the weights-only chart entirely, because part of the model never enters your memory at all.

Tier-by-tier: what each VRAM class can actually do

The table below compresses the whole climb into one view. Model names and figures are the video's recommendations and claims from September 2026; verify current weights and quant availability on Hugging Face before downloading, because this space moves weekly.

GPU memory	Model size that works	What it is good for	Where it stops
4GB	4B, ~2.6GB	Single-file fixes, code explanation	Any multi-file plan
8GB	9B, ~5.75GB	Junior pair-programming, one file plus chat	Long context on a tight card
12GB	9B at 8-bit, ~9.75GB	Same brain, cleaner outputs	Multi-file refactoring
16GB	27B non-uniform quant, <12GB	Real 27B quality on a gaming card	Long conversations at low quants
24GB	27B, ~16.5GB	Multi-file edits, running your test suite	Autonomous agent loops
48-128GB	80B, 38-66GB	Agentic multi-step coding with recovery	PCIe offload bottlenecks
141-512GB	426B-552B MoE and dense	Frontier local inference	Runtime support, bandwidth, cost

The pattern repeats at every rung: the right pick is the setup that leaves headroom, not the biggest model that squeezes in. If a model almost fits, drop one quant level or trim the context before buying a new GPU.

Frequently asked questions

• Why does my local model crash after a long conversation? The KV cache grows with every token of prompt and reply. On a tight card, a few thousand tokens of code and chat can consume hundreds of megabytes, and once your headroom is gone the runtime either spills to slower memory or fails. Trim the context or drop one quant level.

• Can I run a 70B-class model on a 24GB GPU? Only with offloading, and the video's one reported attempt was mixed in reliability. Layers pushed into system RAM must cross PCIe, the GPU idles while waiting, and the first long prompt read is the worst phase. It can work, but it is not a dependable daily setup.

• Does a mixture-of-experts model save GPU memory? No. Sparsity saves compute, because only a few billion parameters fire per token, but you must still fit the full parameter count in memory. A 426B MoE at 208GB runs fast precisely because it fires a fraction of its weights, not because it uses less memory.

• Is 512GB of unified memory the same as 512GB of GPU VRAM? No. Unified memory on a Mac is one pool shared with the whole system at lower bandwidth, while two data-center cards delivering 512GB total are fast but require splitting the model across both. The same number on the spec sheet describes completely different machines.

• What quantization level is safe for coding models? The video puts the safe zone at 4 to 5 bits, where benchmark scores barely move, and warns that quality drops off sharply below about 3 bits. When memory is tight, prefer a 9B at 8-bit over a larger model at an extreme quant.

Where this guide travels next

This walkthrough first appeared as a video, and the same is true of a lot of hard-won hardware knowledge: it lives in a 19-minute recording that nobody can search, quote, or skim. Communities such as Crazystack typescript and creators like Dev doido publish in the same space, and written versions of their material travel further than the original stream.

If you have explanations, benchmarks, or opinions sitting inside your own YouTube videos, Skala blog turns that footage into a structured, searchable article: paste the video URL, it transcribes the talk, and you get a draft you can edit and publish. The three-part memory budget in this piece started as a spoken aside; your best explanation deserves the same treatment. Visit Skala Blog to convert your next video into an article.

Source video