LLM Benchmarks · Measured November 2026

Best Local LLM 2026:What We Measured on Each Machine, at Real Context Lengths

Short answerThe best local LLM 2026 turned out to be several answers, because the right model depends on the box you own and the work you give it. On one NVIDIA DGX Spark, Nemotron 3.5 Bolt on the TensorFold engine was the strongest all-rounder we measured: 72 tokens per second for one user, 258 tokens per second across 16 users, and it found a planted fact at 250,000 tokens of context. Qwen 3.8 27B with the DFlash2 drafter is the best dense model on the same box at 49 tokens per second. On a Mac Studio M5 Max, Nemotron 3.5 Lightning answers one user at 195 to 241 tokens per second. Every number here comes from 128k-token context windows, not toy prompts.

Petronella Technology Group, Inc. runs a private AI fleet: four DGX Spark units, four MSI GB10 units, two Mac Studio M5 Max machines and a 192 GB Ryzen AI Max+ PRO 495 box. Every machine in it answers a different buyer's question, so instead of one leaderboard we report which model wins on each box, at the context lengths real work arrives in, measured with our published benchmark methodology.

128k-token windows|Prefill to 120k tokens|Steady-state ladders|Paired quality gate
250k
tokens of context recalled by Nemotron 3.5 Bolt on one Spark
3.2x
single-user speedup from the DFlash2 drafter on Qwen 3.8 27B
241 tok/s
Nemotron 3.5 Lightning, one user, Mac Studio M5 Max 64 GB
0.96
CMMC RAG accuracy, GLM 5.3 Flash on the 192 GB box
Best local LLM test bench: DGX Spark units, Mac Studio M5 Max and a 192 GB Ryzen AI box at Petronella Technology Group, Inc.
The verdict

The Best Local LLM 2026 Ranks, by Machine

One row per machine. Speed for one user is the steady decode rate a person feels while a reply streams. Speed for many agents is the aggregate throughput at the busiest concurrency level we measured, with the per-user rate beside it. Reading time is how long the model takes to start answering a 32,768-token prompt, the size of a modest codebase or a long contract.

MachineBest model we measuredEngineOne userMany agents (per user)Reads 32k tokens inLong-context recall
1x DGX Spark class (GB10, 128 GB)Nemotron 3.5 Bolt (Mia recipe)TensorFold72.4 tok/s258 tok/s at 16 users (16.9)13.4 sPass at 16k, 120k and 250k
1x DGX Spark classQwen 3.8 27B + DFlash2 drafterTensorFold 1.2.748.7 tok/s182 tok/s at 8 users (23.2)26.4 sPass at 16k and 120k
1x DGX Spark classgpt-oss-20bvLLM45.8 tok/s381 tok/s at 16 users (24.5)5.4 sRe-test pending
Mac Studio M5 Max, 40-core GPU, 64 GBNemotron 3.5 Lightning, MLX 4-bitTensorFold 1.2.7241 tok/s375 tok/s at 4 users (95.7)6.7 sPass at 16k and 120k
Mac Studio M5 Max, 32-core GPU, 36 GBNemotron 3.5 Lightning, MLX 4-bitTensorFold 1.2.7195 tok/s238 tok/s at 2 users (122)9.0 sPass at 16k and 120k
Ryzen AI Max+ PRO 495, 192 GBGLM 5.3 Flash (Maya-S IQ2_XXS, about 2 bits per weight, 98 GB)Project Maya llama.cpp build22.9 to 28.7 tok/s24.5 tok/s at 4 users (6.1)about 3 minutes (173 tok/s reading)CMMC RAG 0.96 agentic

The Mac rows and the Spark rows above use different models, so they answer "what is the best thing to run on this machine," not "which machine is faster." The same-model comparison is in the Mac Studio section below. The 192 GB row was measured with our earlier single-stream harness, before the full methodology suite; it is labeled that way wherever it appears.

Planning a private AI deployment? See how we build on-premises AI
Why most local LLM benchmarks mislead

A Local LLM Is Only Useful at the Context Length You Actually Work With

Most published local LLM numbers come from a 40-token prompt, a 2,000-token window and one request at a time. That measures how fast a model types, not whether it can read a 200-page contract or your compliance evidence and answer correctly. A model that streams 200 tokens per second but takes four minutes to read a 32k prompt, or that quietly truncates anything past 8k, is of little use in real work.

So we tested the way the work arrives. Every server ran with a 128k-token window (smaller only where the hardware could not hold it, and we say so). Each model went through the same suite, documented in full on our LLM benchmark methodology page:

  • Quality gate first. Facts, arithmetic and a planted "needle" sentence at 16k and 120k tokens. A speed number only counts next to a model that passed.
  • Prefill curve. Time to first token at 1k, 4k, 16k, 32k, 64k and 120k tokens (and 250k where the window allows), three repetitions each, with a unique nonce on every prompt so no cache can help.
  • Agent capacity at 32k. One to sixteen agents each send a unique 32k-token request at the same instant: how long until the last one starts answering, and how fast each one then streams.
  • Steady-state ladders. 120-second measurement windows after a 20-second ramp, four repetitions in randomized order, from one user up to twice the server's slot count. We report the median and flag any level whose run-to-run variation is high.
  • Engine restart repeat. The prefill curve and one ladder run again after a cold restart, to catch numbers that only exist on a warm cache.
  • Metadata from the running process. Engine version, checkpoint, drafter, container image and flags are read from the live server, not from our notes.

Speculative decoding (a small "drafter" model that proposes tokens the big model verifies) is recorded on or off for every row; it changes speed dramatically, and some published numbers leave it out.

The machines

Five Local LLM Machines, Three Memory Architectures

All of these are unified-memory machines: the CPU and GPU share one pool, which is what lets a desktop-sized box hold a 30 to 120 billion parameter model. Memory bandwidth, not raw compute, sets most of the single-user decode speed.

MachineMemoryBandwidth (vendor spec)Units in our fleet
NVIDIA DGX Spark / MSI GB10 (Grace Blackwell GB10)128 GB LPDDR5x unified273 GB/s4 DGX Spark + 4 MSI GB10
Apple Mac Studio, M5 Max 32-core GPU36 GB unified460 GB/s1
Apple Mac Studio, M5 Max 40-core GPU64 GB unified614 GB/s1
AMD Ryzen AI Max+ PRO 495 (Minisforum MS-S1 MAX)192 GB LPDDR5x unified, up to 160 GB as GPU memorynot published by AMD1

Specifications are the vendors' own: NVIDIA's DGX Spark hardware guide, Apple's Mac Studio technical specifications and AMD's Ryzen AI Halo announcement (links in Sources). AMD does not publish a bandwidth figure for the 192 GB part, so we do not quote one.

One DGX Spark

Best Local LLM on One DGX Spark: Bolt for Breadth, Qwen 27B for Dense Quality

A single GB10 box has 128 GB of memory and modest bandwidth, so the winners are models that either activate few parameters per token (mixture-of-experts designs such as Nemotron 3.5 Bolt and gpt-oss) or that pair a dense model with a fast drafter.

Nemotron 3.5 Bolt on the Mia TensorFold recipe

Bolt was the most complete result on one box. It passed every gate question, recalled the planted fact at 16k, 120k and 250k tokens, and its prefill held above 2,000 tokens per second from 4k to 120k (2,436 tok/s at 32k, 2,017 at 120k) and was still 1,602 at 250k. One user saw 72.4 tokens per second; sixteen users together pulled 258 tokens per second while each still got about 17. When sixteen agents each dropped a fresh 32k-token request at once, all sixteen completed, with the last one starting to answer after about 219 seconds. That is the real cost of heavy parallel reading on one box, and it is why agent fleets want more than one unit.

Qwen 3.8 27B with and without the DFlash2 drafter

Qwen 3.8 27B is a dense model, so it is slow on 273 GB/s of bandwidth by itself: 15.1 tokens per second for one user. Adding the DFlash2 drafter on the same box, with the same checkpoint inside the same 128k window, took it to 48.7 tokens per second, a 3.2x speedup. The drafter kept paying off under load: 182 tokens per second at eight users against 89 without it. Prefill does not change with a drafter (about 1,250 tokens per second at 32k either way), so a 32k prompt waits about 26 seconds in both cases. Qwen answered one of the eight gate questions wrong (a five-digit multiplication) and passed both needles.

gpt-oss-120b and gpt-oss-20b on vLLM

OpenAI's open-weight models read fastest of everything on the Spark: gpt-oss-20b started answering a 32k prompt in 5.4 seconds and gpt-oss-120b in 9.4. gpt-oss-20b also carried the most users, 381 tokens per second at sixteen. One user saw 45.8 tokens per second on the 20b and 34.1 on the 120b. Their quality gate is being re-run: both are reasoning models and our first gate allowed only 64 output tokens, which cut their thinking off before the answer. We will not publish a speed claim for them without the gate beside it.

Gemma 4 31B and Nemotron 3.5 Lightning on the Spark

Results pending (re-run 2026-11-13). Our first run of these two was invalid: a server left over from an earlier test answered on the benchmark port, so the harness measured the wrong model. We discarded those rows rather than publish them, fixed the harness so it refuses a busy port, and re-ran both. Gemma 4 31B with its DFlash drafter fits six concurrent 128k streams on one Spark, not eight, so both Gemma runs use six for a fair with-and-without comparison.

Want these numbers for the model you plan to run? Ask for a benchmark on your workload
Two and four Sparks

When a DGX Spark Cluster Pays Off

Larger mixture-of-experts models such as DeepSeek V4.1 Flash and GLM 5.3 Flash do not fit one 128 GB box at a useful quality, so they run split across two or four units. On our four-unit Spark cluster, DeepSeek V4.1 Flash 1017 on the Pico engine measured 74 tokens per second for one user and 214 at eight users, with prefill around 6,000 to 7,000 tokens per second; the Mia recipe on the same cluster measured 65 and 163. These cluster numbers come from our trial runs on November 11, before this methodology suite existed, and the four-unit lanes serve production work, so they have not yet been re-run under the full suite.

The practical answer to "should I buy a second or fourth Spark" depends on what you need. If you need a bigger model, a cluster is the only way to run it locally. If you need more agents on a model that already fits one box, separate single-box replicas usually give more total throughput than one split model, because splitting adds network synchronization to every token. Our earlier GB10 versus H200 comparison shows the same pattern from the data-center side, and the GB10 cluster scaling deep dive covers every topology from one unit to eight.

Mac Studio M5 Max

Mac Studio M5 Max for Local LLMs: Fast Decode, Tight Memory

The Mac Studio's advantage is memory bandwidth: 460 GB/s on the 32-core GPU and 614 GB/s on the 40-core GPU, against 273 GB/s on a GB10. That shows up directly in single-user decode. On the same TensorFold 1.2.7 engine, Nemotron 3.5 Lightning streamed 195 tokens per second on the 36 GB Mac and 241 on the 64 GB Mac, and read a 32k prompt in 9.0 and 6.7 seconds. Both found the planted fact at 120k tokens.

The limit is capacity. TensorFold keeps a model inside 70% of the Mac's RAM by default and reserves each stream's context at load time, so the window you ask for has to fit next to the weights. Gemma 4 31B at 4-bit carries about 17 GiB of weights, and TensorFold sizes each 128k-token stream at about 11 GiB. Our Gemma 4 31B runs did not start on either Mac under the stock settings, even on the 64 GB model where the arithmetic says it should fit, so we are investigating before we report a Mac Gemma number. Two smaller observations from the Mac runs: TensorFold on macOS kept printing chat-template tokens after a correct answer (the model still found the needle; the leak is an engine bug rather than a model error), and the 64 GB Mac's aggregate throughput topped out at four users.

The 1.2.9 engine update: no speed change on stock Qwen 3.8 27B

We re-measured the stock Qwen 3.8 27B MLX 4-bit recipe after upgrading the engine from TensorFold 1.2.7 to 1.2.9 with the DFlash2 drafter, on the 64 GB Mac Studio M5 Max with the 40-core GPU: 300 output tokens, greedy, the same two prompts (a coding task and a prose explanation) we used for the 1.2.7 table above. Code came in at 134.6 tokens per second and prose at 89.7 and 89.8 across two runs, against 133.3 and 87.4 on 1.2.7. That is within run noise, so the upgrade is a compatibility release for this recipe, not a speedup; we publish the check because readers who saw the 1.2.7 numbers deserve to know whether to expect more.

Where the Mac Studio fits

  • One person, one strong model, fast replies: the Mac is excellent. Quiet, compact, nothing else to manage.
  • Long documents: fine for one stream at 64k to 128k if you size memory for it. Buy more memory than the model's file size suggests.
  • Many sub-agents at once: the GB10 boxes carried more parallel work in our tests, and their 128 GB leaves more room for each agent's context.
192 GB unified memory

What 192 GB of Unified Memory Is Actually Good For

The Ryzen AI Max+ PRO 495 holds 192 GB, and AMD's announcement says up to 160 GB can be given to the GPU. We used that room for GLM 5.3 Flash in Project Maya's Maya-S format, an aggressive IQ2_XXS quantization (about 2 bits per weight) that is still a 98 GB file, with space left for long context. The full-precision model is far larger than any desktop box; this is what it takes to run it on one. Measured with our earlier single-stream harness, it decoded 28.7 tokens per second on code and 22.9 on prose for one user.

The weakness is reading speed. It took 50.8 seconds to start answering a 9,252-token prompt and 156.8 seconds for 27,053 tokens, about 173 to 182 tokens per second of prefill. A coding agent that re-reads a large context on every turn would spend most of its time waiting. Smaller models do much better on the same box: Qwen 3.8 Flash Next on the halogen-flash-server engine reached 89.5 tokens per second for one user and read prompts at about 2,230 tokens per second.

On quality, the result surprised us. Even at about 2 bits per weight, GLM 5.3 Flash scored 0.957 accuracy in agentic mode and 0.925 in code-first mode on our CMMC RAG battery (one seed so far), within three points of the same model at FP8 (0.986).

Best uses for 192 GB

  • Running one very large mixture-of-experts model, heavily quantized, with room left over for long context, for a single user who can wait for long prompts.
  • Keeping several mid-size models resident at once (a coder, a reasoner, an embedder) so switching costs nothing.
  • Batch work where answers matter more than latency: overnight document review, classification, summarization.

It is not the right box for interactive coding agents on big repositories or for many simultaneous users. Our own DGX Spark vs Strix Halo vs Gorgon Halo write-up covers the earlier generation of this comparison.

Local LLM + RAG for compliance

Local LLMs on Real Compliance Work: Our CMMC RAG Battery

Speed means nothing if the answers are wrong. We grade models on a CMMC retrieval battery built only from public sources: NIST SP 800-171 and 800-172 with their assessment procedures, plus the DoD CMMC assessment and scoping guides for Levels 1 to 3. The model has a search tool over that corpus and must answer specific questions with citations; trap questions check that it refuses to invent requirements. The scores below are accuracy across three seeds unless noted, all on the same frozen index; the GB10 rows ran on the same class of box this page tests. Grading is deterministic and rule-based (quoted requirement text, citation ids and refusal checks), with no AI model acting as judge.

ModelWhere it ranAgentic accuracyNotes
GLM 5.3 Flash (FP8)Data-center GPUs0.986Reference: the same model at higher precision
Qwen 3.8 27B1x GB100.978Best single-box result
GLM 5.3 Flash + DFlash24x GB10 cluster0.971Cluster keeps full quality
Qwen 3.8 27B (Q4_K_M)Single GPU, 4-bit0.9684-bit costs little accuracy
DeepSeek V4 Flash 07314x GB10 cluster0.968Accurate and quick to answer
GLM 5.3 Flash, Maya-S192 GB Ryzen AI box0.957 (1 seed)About 2 bits per weight, still close to FP8
gpt-oss-120b1x GB100.889Weaker on multi-hop questions
Gemma 4 26B A4B1x GB100.839Rarely chains searches
gpt-oss-20b (MXFP4)vLLM, single GPU0.810Fastest, least accurate of these
Nemotron 3.5 Bolt, Lightning; Gemma 4 31B1x GB10PendingRun with the re-tests

The pattern is consistent: models that search more than once before answering (Qwen 3.8, GLM 5.3, DeepSeek V4) handle multi-part compliance questions far better than models that answer after one search. For a private CMMC assistant, that matters more than raw speed. Petronella Technology Group, Inc. builds these assistants for defense contractors on hardware the client controls; see our CMMC compliance services.

Need a private compliance assistant that keeps CUI on your hardware? Talk to Petronella Technology Group, Inc.
Community questions

Common Local LLM Questions, Answered With Our Numbers

Is the DGX Spark really about three times faster than a Mac Studio at reading prompts?

Not in our tests with the same engine. With Nemotron 3.5 Lightning on TensorFold 1.2.7, the 64 GB Mac Studio read a 32k prompt at 4,877 tokens per second and the 36 GB model at 3,636. The matching single-Spark Lightning row is in the re-run. The published comparisons we reviewed, such as ComputingForGeeks and Presenc, ran different engines on each machine, which is the variable we removed.

How long does a coding agent wait to read a 32k codebase?

On one Spark: 5.4 seconds with gpt-oss-20b, 13.4 seconds with Nemotron 3.5 Bolt and 26 seconds with Qwen 3.8 27B. On a Mac Studio M5 Max with Nemotron 3.5 Lightning: 6.7 to 9.0 seconds. On the 192 GB Ryzen box with GLM 5.3 Flash: about three minutes.

Is the advertised context window real?

For the models we tested, yes, when the server is configured for it: every model that passed the gate recalled a fact planted in the middle of a 120k-token document, and Bolt did it at 250k. The catch is memory: the window has to be reserved per stream, and on smaller machines you trade streams for context.

Does speculative decoding still help with long context and many users?

For Qwen 3.8 27B, clearly yes: 3.2x for one user and about 2x at eight users, with no change to prefill time. The Gemma 4 31B drafter comparison is in the re-run.

Which machine should I buy?

Which Machine Should I Buy? A Decision Guide From Our Own Fleet

Five machine families, one honest answer each. Every row links to the page where we measured it, and every "when not to" is a case we actually hit. No prices appear anywhere on this page; what any configuration costs is a conversation, not a table.

If you needBuyWhy, from our dataWhen NOT to
One person, fast replies, silent desk, no Linux adminMac Studio M5 Max (64 GB if you can)Qwen 3.8 27B with the DFlash2 drafter: 133 to 135 tok/s code, 87 to 90 prose, versus 48.7 on one Spark with the same drafter familyMany sub-agents, or long prompts all day: aggregate topped out at four users on the Mac, and the Spark reads prompts faster
The cheapest 128 GB of unified memory, and a normal PCStrix Halo (Ryzen AI Max+ 395)x86 Windows or Linux laptop or mini PC that also runs everything else; on the same llama.cpp build our GB10 was only 17 to 41 percent faster at decodeLong-prompt agent work: it reads prompts 1.8 to 3.8x slower than GB10 in our same-build test
Models that do not fit 128 GBGorgon Halo (Ryzen AI Max+ PRO 495, 192 GB)Ran GLM 5.3 Flash as a 98 GB roughly-2-bit quant at 0.957 CMMC RAG accuracy; 128 GB boxes cannot hold itInteractive agents: prefill near 180 tok/s means a big prompt takes minutes to read
CUDA, fast prefill, concurrency, clusteringDGX Spark / MSI GB10, one to start, four for a large MoEPrefill near 10,000 tok/s on four units with the Pico engine, 16 users at 258 aggregate on one box with Bolt, clusters over 200 Gb linksA single user whose model fits a Mac: the Mac is faster per box and quieter
Fastest per user for models that fit VRAM, inside an existing desktopRTX PRO 6000 or RTX 5090 class cardsQwen 3.8 27B at 80.1 tok/s on one RTX PRO 6000 against 24.4 on one GB10; gpt-oss-120b at 179 against 35128 GB-class context per stream, or cheap clustering; also check your board and power before adding a card

Full reasoning for each family: the GB10 cluster scaling deep dive (sibling deep dives for Mac Studio, Mac mini, Strix Halo, Gorgon Halo and the RTX cards are being published in the same window and will be linked from the benchmark hub as they land).

Want this decision made for your workload? Ask us to size it
Buying questions

The Nine Questions Buyers Ask Us, Answered

These are the questions that came through our consulting line in the two weeks before this page was written, answered the way we answer them on a call: with our numbers, and with the honest caveats.

Question 1

Can I add a 96 GB RTX PRO 6000 Max-Q next to my RTX 5090 in an older desktop?

Yes, if each card runs its own job: LLM inference on the 96 GB card, image generation on the 5090. On an older board built for two cards (an older Intel generation (the i9-9900K with a Z390 chipset)), the two CPU slots run x8/x8 on PCIe 3.0, and one model per card loses only about 1 to 2 percent at x8 Gen3, because your model loads from NVMe through the chipset at roughly 3.9 GB/s anyway, slower than the x8 slot's 7.9 GB/s. The CPU's age barely matters once the model sits in VRAM. Two things not to do: never split one model across mismatched cards over PCIe 3.0 (tensor parallel over that link is slow and layer splits hurt prefill), and never spill into system RAM.

Question 2

Why is a second GPU fine on that old Intel board but discouraged on on a newer Ryzen desktop (a 9950X3D-class build)?

The board decides this, not the CPU. Many Z390 and X570 boards, and workstation boards generally, were built for x8/x8. Most consumer AM5 boards are not: our X870 board has one Gen5 x16 CPU slot, and a second card lands in an x4 slot at Gen4 x4 at best (about 7.9 GB/s), possibly behind the chipset sharing bandwidth with the network cards and USB. Read the board manual's slot wiring page before you buy the second card; the CPU is irrelevant to the answer.

Question 3

What power supply and circuit do two big cards need?

An RTX 5090 draws up to 575-600 W at board level and NVIDIA recommends a 1,000 W PSU for it alone (their guidance, linked in Sources); the RTX PRO 6000 Max-Q is capped at 300 W and the Workstation Edition at 600 W. A 5090 plus a Max-Q plus an older desktop CPU lands near 1,100 to 1,150 W at full load, which an 850 W unit is not (it only works with the 5090 capped near 300 W). A 1,200 W ATX 3.x unit works, with the 5090 capped at 450-500 W for headroom, which costs a few percent. At the wall, a 15 A / 120 V circuit is 1,800 W peak and about 1,440 W continuous by the 80 percent rule, and this box draws roughly 1,250 to 1,300 W flat out, so share the circuit with little else and check your UPS: many desktop units top out at 900 to 1,000 W.

Question 4

Max-Q or RTX 5090 for image generation?

The 5090 is faster, because its power limit is nearly double. The Max-Q wins the other three rounds: 96 GB of VRAM against 32, a 2-slot blower that stacks cleanly in a multi-card build, and 300 W that a shared circuit can actually feed. For a box that does LLM inference first and image generation second, we would put the jobs on the card that suits each.

Question 5

Should our office buy a big GPU when we already have unified-memory boxes?

Buy the GPU for long-context agent work, not for model size. Unified memory holds big models but reads long prompts slowly: our 192 GB box peaked near 2,200 tok/s of prefill on its best small model, and the four-unit Spark cluster reads about five times faster with the right engine (about 10,000 tok/s). If your production inference already runs somewhere else, a home or office fallback box rarely justifies the spend; we tell clients that honestly.

Question 6

Mac Studio Ultra with 256 GB or 512 GB?

By capacity, 256 GB holds a GLM 5.3 Flash class model (about 190 to 200 GB at 4-bit) in one quiet, low-power box, and 512 GB holds the largest open models with long context. But look at what your work actually is: our four-unit GB10 cluster ran DeepSeek V4.1 Flash at 74 tok/s single-user and 214 aggregate at eight users with prefill around 6,000 to 7,000 tok/s, and GLM 5.3 Flash at 47.8 tok/s on two units and 77.2 on four (the latter from the recipe author's published figures). We have not measured an Ultra, so we make no Ultra speed claims. When the Mac wins: one silent desktop, macOS workstation use, and single-user chat with very large models, with no cluster to cable or manage.

Question 7

Is the Mac mini good for local LLMs?

A Mac mini with an M4 chip and 16 GB holds small models only: in our Apple Silicon matrix, an 8B model at 4-bit fits and answers at about 19 tok/s, a 14B is tight, and 20B-class models are edge cases. It is a good always-on box for small models and drafting work, and it is not a 27B-class machine at that memory. The 32 GB tier is where the MoE daily drivers start.

Question 8

What is the best model for a 32 GB card?

As of 2026-11-17, still Qwen 3.8 27B. It has the best ecosystem: an Apache 2.0 drafter (DFlash2), a measured third-party recipe on the RTX 5090, and native TensorFold support. A newer dense 33B (Nanbeige4.3-33B, released 2026-10-24) claims better agentic coding on its own card (vendor numbers: DeepSWE 65.5 against 42.5), but those are the vendor's figures, not ours; the dense 33B is slower per token, and its drafters are community-only. We also ruled out a same-speed fine-tune and two other releases that did not beat it on our battery. Our 27B rows: 80.1 tok/s on one RTX PRO 6000, 0.978 CMMC RAG accuracy on one GB10.

Question 9

Bottom line: when do I pick each machine over a DGX Spark?

Mac Studio: silent desktop, macOS, the best single-user speed per box in our data (133 to 135 tok/s code on the 64 GB M5 Max), no Linux admin; not for many users or long-prompt agents. Strix Halo: the cheapest 128 GB, a general-purpose PC; slower prefill and a younger software stack. Gorgon Halo 192 GB: for models that do not fit 128 GB (GLM 5.3 Flash at roughly 2 bits, 22.9 to 28.7 tok/s). Spark and GB10: CUDA, NVFP4, the vLLM and SGLang engines, prefill speed, concurrency, and clustering over 200 Gb links. RTX cards: the fastest per user when the model fits VRAM, inside a desktop you already own.

What we will not claim

What the Best Local LLM Benchmarks Do Not Tell You

We do not publish tokens per watt or tokens per dollar on this page. Power was not measured on every machine under the same load, and prices change weekly, so either figure would be a guess presented as a measurement. We also do not rank machines against each other when they ran different models; the verdict table says what to run on each box, and the same-model comparisons are labeled as such. Numbers marked as earlier-harness results were measured before this suite and will be replaced as we re-run them. These are our own measurements on our own hardware and are not yet part of our public leaderboard, and each one is a single configuration of engine, checkpoint and settings, so a different engine version can move them.

Questions we get

Best Local LLM: Frequently Asked Questions

What is the best local LLM right now?

It depends on your hardware. On one DGX Spark class box, Nemotron 3.5 Bolt on TensorFold was the best all-rounder we measured (72 tok/s for one user, 258 tok/s across 16 users, recall at 250k tokens). On a Mac Studio M5 Max, Nemotron 3.5 Lightning was fastest at 195 to 241 tok/s for one user. For compliance question answering, Qwen 3.8 27B and GLM 5.3 Flash scored highest.

DGX Spark or Mac Studio for local LLMs?

The Mac Studio M5 Max decodes faster for one user because it has more memory bandwidth. The DGX Spark carries more parallel work and holds more context per stream in its 128 GB. One person wanting fast replies: Mac Studio. Many agents or long documents at once: Spark.

Is a 192 GB machine worth it over 128 GB?

Only if your models crowd a 128 GB box. Our 192 GB Ryzen AI Max+ PRO 495 ran GLM 5.3 Flash as a 98 GB, roughly 2-bit quant at 0.957 CMMC RAG accuracy, but read prompts at only about 173 tokens per second, so long prompts take minutes.

Does a second or fourth DGX Spark make a local LLM faster?

It lets you run bigger models (DeepSeek V4.1 Flash measured 74 tok/s for one user on four units). For more users on a model that already fits one box, separate replicas usually beat one split model.

What context length did you test?

Every server ran a 128k-token window where memory allowed. Prefill was measured from 1k to 120k tokens, and to 250k for Nemotron 3.5 Bolt. Agent capacity was tested with unique 32k-token requests.

Does speculative decoding help local LLMs?

Yes for dense models: the DFlash2 drafter made Qwen 3.8 27B 3.2 times faster for one user and about twice as fast at eight users on one Spark. It does not speed up prompt reading.

Can a local LLM handle CMMC compliance questions?

With retrieval over the right documents, yes. On our CMMC RAG battery, Qwen 3.8 27B, GLM 5.3 Flash and DeepSeek V4 Flash scored 0.96 to 0.99 accuracy. Smaller models that search only once scored lower.

Which machine should I buy for local LLMs?

It depends on the job. One person, fast replies, silent: Mac Studio M5 Max. Cheapest 128 GB and a normal PC: Strix Halo. Models too big for 128 GB: the 192 GB Gorgon Halo box. CUDA, prefill speed and many agents: DGX Spark class, one box to start and four for a large MoE. Fastest per user when the model fits VRAM: an RTX card in a desktop you already own. The decision table above links the numbers behind every row.

Why are some results marked pending?

We discard any run whose quality gate or setup was wrong rather than publish it. The pending rows are being re-run with the same method and will be added with their dates.

About the author

Craig Petronella

Craig Petronella founded Petronella Technology Group, Inc. in 2002 and leads its private AI and compliance work. He is a CMMC Registered Practitioner (CMMC-RP), Cisco CCNA, Certified Wireless Network Expert (CWNE) and North Carolina Digital Forensic Examiner (License 604180-DFE), and the author of 16 books. His team builds and runs the GPU fleet behind these benchmarks.

Keep comparing

More Hardware Deep Dives From Our Benchmark Lab

Every page below uses the same prompts, the same quality gate and the same methodology, so the numbers line up across machines.

Sources

Sources and References

Run the Right Local LLM on the Right Hardware

We can benchmark the model you plan to deploy on the hardware you are considering, at your context lengths and user counts, before you buy. Our team builds private AI systems that keep sensitive data on equipment you control.

Prefer to read first? Explore our AI services.

Measured on the Petronella Technology Group, Inc. fleet, November 2026. Every number on this page traces to a stored result file whose engine version, checkpoint and launch flags were read from the running server.