What a Local Model Is Actually Good For

Every number here comes from a SQLite database on my desk, queried the morning I wrote this. The failure numbers are in the same tables as the good ones.

There are two Macs on my desk, wired together with a single Ethernet cable. The Mac mini has an M4 and 16 GB of memory. The MacBook Pro has an M5 and 24 GB. On the day I set this up, the mini was sitting at 83% memory free with 1.2 GB of swap, and the MacBook, the bigger machine and the one I actually work on, was at 34% free with 18.3 GB of its 19.5 GB swap in use.

So the smaller machine became the inference host. That is the whole origin of the project: not a thesis about local models, just a measurement about which computer had room to think.

The cable runs at 1000baseT, about 100 MB/s in practice. That sounds fast until you compare it to unified memory, which moves at 100+ GB/s, roughly a thousand times quicker. You cannot pool RAM across two machines over a wire that slow. What you can do is send text one way and get text back, which is exactly the shape of an inference request. The model lives where the memory is. The work lives where the code is. Only tokens cross.

The thing I built

A code reviewer that never stops. It watches git-tracked files in two repositories, hands each changed version to a Qwen3-8B model running under mlx_lm.server on the mini, and writes what comes back into a local database. A second agent re-checks findings with a different model family. A third agent, Claude Opus, is the only part that costs money; it grades the whole operation periodically and writes calibration rules back into the system.

In its first 60.4 hours of operation, measured from the first review at 23:46 on July 31st to the last one at 12:13 on August 3rd, it completed 1,767 review runs across 705 distinct versions of 457 files. Average time per review: 12.23 seconds. Total model time: about six hours of the sixty.

Through the local model, that came to 5,041,271 prompt tokens and 72,124 completion tokens.

The marginal cost of those five million tokens was zero.

Cost, stated carefully

I want to be precise here, because this is where writing about local models usually goes wrong.

I have a measured figure for what the same work costs through a hosted coding agent: $0.29 to $0.43 per headless invocation, in cache creation alone. It is tempting to multiply that by 1,767 and print a savings number with a dollar sign in front of it. I am not going to, because that would be modeling a system nobody ran. I never sent 1,767 reviews to a paid endpoint. The comparison is real; the counterfactual spend is fiction.

What I can say is what actually happened. The local half cost nothing per token. The paid half, 20 audits by Opus, cost $32.46, and I know that because the system records cost_usd on every audit row.

The interesting property is not the size of the number. It is the shape of the cost curve. A hosted model charges per unit of work, so every additional review is a decision. A local model charges for the hardware once and then charges nothing, so additional reviews are free and the only real constraint is time. Measured capacity on this setup is roughly 14,000 reviews a day. Actual need, based on how fast I change code, is about 250. There is no economic reason to ever throttle it.

That changes what you build. When each review costs a third of a dollar, you review pull requests. When reviews are free, you review every version of every file, including the ones nobody asked about, including the same file eleven times in an afternoon. You stop rationing.

Privacy, which is the part that needs no argument

The code never leaves the two machines. Not the diffs, not the file paths, not the surrounding context the model needs to judge anything. There is a configuration constant in the project that lists every host the system is allowed to send inference to:

ALLOWED_ENDPOINT_HOSTS = ("127.0.0.1", "localhost", "10.42.42.1", "10.42.42.2")

Two loopback addresses and the two ends of the cable. That is not a policy document or a data-processing addendum. It is four strings, and a request to any other host does not happen.

For my own repositories this is a preference. For anyone working under a compliance regime (client code under NDA, anything regulated, anything a security review would have to reason about), it is the difference between a tool being usable and being a non-starter. The audit story for "the model runs on that computer over there, on a cable, with a four-host allowlist" is short enough to fit in a sentence.

Speed, in the sense that matters

The local model is not fast. It runs at about 16 tokens per second, and a real 14.6 KB diff takes 6.2 seconds to review. A frontier model on someone else's hardware would be quicker and better.

But latency per request is the wrong measure for work that happens while I sleep. What matters is that there is no queue, no rate limit, no quota to spend, and no service to be degraded on a Tuesday. The reviewer runs at background priority and takes the machine's leftovers. It has been running continuously since I turned it on.

There is a related lesson I paid for. Qwen3 is a hybrid reasoning model, and if you leave thinking enabled it will spend its entire budget in a reasoning field and return an empty content field. Downstream, that reads as "no findings": a silent wrong answer rather than an error. enable_thinking: false is mandatory, and I only found that because the results looked plausible and empty at the same time.

The part where I tell you it did not work

Of the 452 findings the local model produced, 182 survived far enough to reach the paid audit.

Opus upheld 3 of them. It rejected 179.

That is a 1.6% precision rate at the boundary where I start spending money, and I am not going to characterize it as anything other than what it is. The local model is noisy. It reports control-flow bugs that are not bugs, security vulnerabilities in code that does not do the thing it claims, null-handling defects in paths that cannot be null. It is confident about all of it.

So: local inference solves cost. It solves privacy. It solves availability. It does not solve quality, and no amount of running it longer will change that, because the volume is not the constraint. The model's judgment is.

Here is the thing I did not expect. That result does not make the system worthless. It makes the architecture the whole product.

Letting a model be wrong cheaply

Findings pass through five filters before anything bills. The order is deliberate: free and deterministic first, expensive and stochastic last.

The first filter uses no model at all. It takes each finding's quoted evidence and searches for that exact string in the actual file. If the model quoted code that is not there (invented a line, cited a line number that does not exist, quoted something the diff removed), the finding dies immediately. That filter discarded 85 findings at a cost of zero. Hallucinations die for no tokens and no dollars.

The second filter asks whether the finding is material, which turns out to be a separate question from whether it is true. This one I learned by watching it fail: a finding that said "the variable name total is too short" was upheld three votes to zero at maximum confidence. Every voter was correct. The name is short. Majority voting cannot save you from a true and worthless claim, so materiality has to be a gate rather than a vote.

Then three lenses vote, then a different model family re-checks, and only then does Opus see anything.

The design principle underneath is one I would keep even if I swapped every component: spend compute in inverse proportion to how likely it is to be wasted. A free string search should run before a local model call, which should run before a model of a different family, which should run before the one that bills. If you invert that order you are paying frontier prices to discover that a finding quoted code that was never in the file.

And the honest reading of my own numbers is that this ordering is not yet aggressive enough. 182 findings reached the layer that costs money and 179 of them should have been stopped earlier. The free filter is doing its job, with 85 caught for nothing. The middle filters are not. That is the measured defect, it is the next thing to fix, and I would rather publish it than publish the token count on its own.

What I did about it

I went looking for where the loss happened, and it was sitting in the vote table. The three lenses meant to challenge every finding had cast 90 votes and refuted none of them. Not a few. None. A verification stage with a 0% refute rate is not verifying anything; it is a rubber stamp with a latency cost, and it is how 179 findings walked into the layer that bills.

I replaced them on August 1st with three lenses that have to cite an artifact before they are allowed to agree. Across 1,010 votes the new set refutes 341, and the useful part is that they disagree with each other: one refutes at 52%, the next at 45%, the third at 5%. Three lenses that disagree are asking three different questions. Three that never refuse anything were asking one question and hearing one answer.

Here is the part I have to say plainly: it has not worked yet. The 17 audits since the rebuild reviewed 159 findings and upheld zero of them. All three upheld findings on record came from the very first audit I ever ran. The intermediate metric moved and the outcome did not, and if I reported that refute rate as a win I would be making precisely the mistake this post is about.

I still think it is the tractable kind of problem, and the reason is structural rather than optimistic. The failure lives in prompts, thresholds, and ordering rather than in model weights, so an iteration costs an afternoon instead of a training run. Every rejection is stored with the auditor's reasoning, so there are now 179 labelled negatives sitting in the same database as the findings they describe. The audit layer already writes rules back into the reviewer, 82 of them so far. The loop is built and it is turning. It has not produced a measured gain, and until it does, that is what I will say about it.

What else falls out of running your own

Two things I did not anticipate.

The first is that the system needed a way for its agents to stay out of each other's way, and the obvious mechanism was wrong. I started with a heartbeat table: one row per agent, overwritten. That tells you who is alive. It cannot tell you what anyone is holding, and the dangerous state is not that a process holds a lock but that it held one and then died. So the heartbeats became a small bus with four primitives: what is true, what is held, an append-only log, and what the human has permitted. Everything expires. The mechanism that recovers from a dead agent is the same one that grants a live one permission.

The second is that giving software your own machine forces the safety question immediately. There is an agent whose job is to give the computer back to me when I need it, and it found 6.5 GB reclaimable across 40 idle processes. The brief was to close background apps and restore them later, and closing them cannot do that: an ended process loses its windows and unsaved work. Suspending one keeps everything. Same memory back, no data loss, and restoring becomes a property of the mechanism rather than a promise.

Four capabilities cannot be granted through any interface, in any combination:

NEVER_GRANTABLE = ("delete_files", "uninstall_apps", "system_settings", "kill_processes")

The API returns 403 to anything asking for them. They are irreversible, or they need privileges this account does not have, so they are only ever proposed for me to run myself. The local model writes the human-readable explanation of what it wants and never selects the process, for the same reason the protected-name list is compiled in rather than configured.

None of that is specific to local inference. It is just that when the model lives on your hardware and can touch your machine, you write the boundary down on day one instead of day ninety.

The reframe

I set out to find whether a small model on cheap hardware could replace a large model on someone else's. It cannot, and the number that proves it is 3 out of 182.

But that framing was the mistake. The local model is not a cheaper version of the expensive one. It is a different instrument: one that generates candidate volume at no marginal cost, on hardware I control, without a quota. Volume is worthless on its own. 452 findings, mostly wrong, is noise. Volume becomes useful the moment you build something that can discard it for free.

The five million tokens were never the achievement. The 85 findings that died in a string search, before anything was spent, are closer to it. What local inference actually bought me is the freedom to be wrong at scale and pay for it in electricity, plus the obligation, which I had not expected, to build the thing that decides which wrongness is worth a dollar.