GoodEnough

The Question

"Just run it locally" is the reflexive answer to LLM cost. It is rarely tested honestly. GoodEnough asks the specific version of that question: can a quantized 1.7B model running on a commodity CPU hold answer quality within a fixed margin of a hosted 70B model, and what does it actually cost you in latency to find out?

The design is preregistered on purpose. The non-inferiority margin (10 points) was fixed before any data was collected, so the study cannot be talked into a flattering conclusion after the fact. The local model is Qwen3-1.7B (Q4_K_M) on an ordinary CPU; the hosted baseline is Llama-3.3-70B via Groq. Quality is measured across ~1,700 items spanning 8 MMLU domains plus GSM8K, at 95% confidence.

Why non-inferiority, and why paired

The wrong test here is "is local better than hosted." Nobody expects a 1.7B model to beat a 70B one. The right test is "is local not meaningfully worse" for a given task, which is exactly what a non-inferiority design measures: local minus hosted has to clear a predetermined lower bound, not zero.

Every item is run through both models, so the comparison is paired: the same question, scored the same way, differing only in the model. That lets the analysis use exact paired-item statistics (McNemar for the paired accuracy comparison, Clopper-Pearson for exact binomial intervals) instead of large-sample approximations that get shaky on per-domain slices. Splits and seeds are frozen, so a re-run reproduces the same items in the same order.

The apparatus

The eval harness is dependency-free Python by design: paired-item scoring, frozen splits and seeds, and a resumable, budget-aware runner that can stop and pick back up without double-spending, instrumenting every request for token count, cost, and latency. The output is a per-domain map of where the small model stays inside the margin and where it falls out, rather than a single headline percentage.

The tradeoff the numbers actually show

The interesting result is not "local is fine" or "local is bad." It is the shape of the tradeoff, and it runs in opposite directions on the two axes people conflate:

• Cost: local inference costs roughly 90% less per query. On a commodity CPU the marginal cost of a query is essentially electricity, against a metered hosted API.
• Latency: local is roughly 8x higher p50 latency (on the order of ~2.4s versus ~0.27s per item). Groq's hosted 70B is dramatically faster than a 1.7B model on a CPU. The cheap option is the slow option here, which is the part the "just run it locally" reflex usually skips.

Naming the latency cost honestly is the point. A cost win that quietly ships an 8x latency regression is not a free lunch, and a study that only reported the cost number would be misleading.

Status and what I will not claim yet

The study is mid-data-collection. The design, apparatus, cost, and the latency multiple are real and measured; the headline quality verdict is not final. I am deliberately not stating "the small model is non-inferior on N of 8 domains" until the full hosted run and the per-slice map are complete, because that is the one number the whole study exists to produce and it has to come from the data, not from an expectation. The latency figure is reported as a rough multiple pending the exact p50 from the finished run.

The value here is the discipline: a margin fixed before data, exact paired statistics, a reproducible apparatus, and a tradeoff reported in both directions instead of the flattering one.

Tech

Python (dependency-free) · Qwen3-1.7B GGUF (Q4_K_M) local · Llama-3.3-70B via Groq · MMLU + GSM8K · exact paired-item statistics (McNemar / Clopper-Pearson) · resumable budget-aware runner with per-request cost and latency instrumentation.

github.com/sahilmehta17/GoodEnough →