Nemotron-3.5-Lightning-30B-A3Bbenchmark results

Measured speed and quality

Speed (single user)

Source: fleet-bench/2026-10-02-best-model-per-tier-results.md

Aggregate throughput

Source: fleet-bench/2026-10-02-best-model-per-tier-results.md

Prefill and latency

Source: fleet-bench/2026-10-02-best-model-per-tier-results.md

Power and efficiency

Source: fleet-bench/2026-10-02-best-model-per-tier-results.md

Other measurements

Source: ai11/study-2026-10-16-fp8-fp16-use-cases.md

What it is

Nemotron-3.5-Lightning-30B-A3B is a mixture-of-experts model built for speed on small unified-memory machines. We measured the NVFP4 checkpoint on one GB10 unit of the DGX Spark class, served by vLLM nightly with the vendor DGX Spark recipe (DSpark k3 spec decode). In our battery it is the fastest single-user model on one GB10, and it also holds the highest user ceiling of the tier, but it pays for that speed with the weakest quality scores of the tier 1 MoE group. That trade is the whole story of this page: pick it when raw tokens per second matter more than nuance, and pick something else when the answers feed compliance, tool calling or long-document work.

The model accepts a long served context and clears the 120k needle test, but it failed a stricter long-context recall gate in the same campaign, and its long-context question battery score is low. Read the Measured speed and Quality sections before committing it to anything that ingests large documents.

Where it runs best

Every speed record we hold for this model comes from one GB10 unit (DGX Spark class, NVFP4 quantization, vLLM nightly). We do not have a verified multi-unit tensor-parallel configuration for this model in the snapshot, so any claim about two-unit or larger deployments would be speculation and we do not make it here.

On the Strix Halo side we have one quality record: on ai11 (Minisforum MS-S1 MAX, Ryzen AI Max+ PRO with Radeon 8065S graphics and a large unified memory pool), the model ran under llama.cpp Vulkan with a UD-Q4_K_XL quantization and scored 37 correct of 40 on the MMLU-Pro battery. That places it among the best 4-bit picks of the ai11 sweep for general knowledge, which is a better relative result than its GB10 quality battery suggests, likely because MMLU-Pro exercises recall rather than tool discipline.

Practical guidance by configuration:

• One GB10 unit, single user: this is the configuration the model is tuned for. If your workload is one person drafting prose or code against a local server and you want the fastest tokens per second available on the box, this is the fastest option we measured.
• One GB10 unit, small team: the model sustains aggregate throughput across many concurrent users better than almost anything else on the platform, so a team of a dozen or more people sharing one box will feel less queueing here than with slower single-user models.
• Strix Halo (ai11 class): a reasonable general-knowledge pick in 4-bit quantization, but we have no measured speed numbers for it on that hardware, so we cannot tell you how it compares on tokens per second.
• Multi-unit GB10 clusters: unmeasured for this model. If you need a two-unit or eight-unit configuration, look at the models with verified records for those layouts.
• H200 or DGX Station class hardware: we have no records for this model on either family, so this page makes no recommendation there.

Measured speed

All figures below come from the 2026-10-02 tier 1 battery on one GB10 unit, clocks locked, engine freshly restarted from a rebooted unit, thinking mode off for the speed rows. A superseded 09-23 survey number and a withheld 09-25 battery are excluded; the 10-02 re-measure replaces both.

Single-user decode is where this model wins outright. Prose answers decoded at 90.2 tokens per second for one user, and code answers were faster still at 114.2 tokens per second. For a sense of scale, the dense 27B reference on the same hardware decoded prose at roughly a quarter of that pace, and the strongest quality rival among the MoE models decoded prose at about a third of it. If your benchmark is "I typed a prompt and I am watching the answer stream," nothing else we measured on one GB10 comes close.

Concurrency is the second win. At 32 concurrent users the aggregate decode throughput reached 608.0 tokens per second. The measured ceiling, the largest user count that still held at least 10 tokens per second per user with a p95 time to first token of 10 seconds or less, was 64 users, at which point aggregate throughput reached 795.4 tokens per second. That is the highest ceiling of the tier 1 MoE models alongside Gemma-4, and it means one box can serve a mid-size office before anyone notices a queue.

Prefill holds up well across context sizes: 6,859 tokens per second at an 8k context, 6,431 at 32k, and 4,632 at 128k. The 128k figure is roughly double what the quality-leading rival managed, so on long-document ingestion this model front-loads far less waiting, even though it answers long-document questions less accurately (see the next section).

Efficiency is strong: 6.3 tokens per joule at the ceiling, measured at the wall over the engine's own PDU outlets. Wall power was light for the class, but the per-watt figure is the one we record, and it is among the best in the tier.

What these speeds mean in practice depends on the shape of your traffic:

• One writer, one prompt at a time: the difference between 90 and 24 tokens per second is the difference between an answer that streams past as fast as you can read it and one you wait for. For interactive drafting, code completion in an editor, or a chat assistant where the user watches the tokens arrive, this model changes the feel of the product, not just the benchmark number.
• Small bursts of parallel work: if your application fans a request out into several sub-queries, or a few staff members hit the server at once, the aggregate figure is what you feel. The engine held its shape as concurrency rose, and it reached the same user ceiling as the best model in the field while delivering more aggregate tokens per user at the low end of the ladder.
• A waiting room of many users: at the measured ceiling the aggregate throughput more than triples the single-user number. If you are sizing a shared departmental server, this model turns one box into a service that a whole floor can use without anyone watching a spinner, provided the work is the kind the model answers well (see the next section, because accuracy is the price).
• Long documents: prefill at the 128k context size still ran at thousands of tokens per second, which means a very large document is chewed through in well under a minute even at the largest size we measured. If your pipeline feeds whole contracts or manuals and the answer is a short extraction, the waiting is mostly on the model's reasoning, not on reading the file.

Quality and accuracy

This is the honest part: on our dgx1 quality battery, Nemotron-3.5-Lightning scored the weakest of the tier 1 MoE models, and the gap is not marginal. Tool calling: 29 of 38, against 37 or 38 for the stronger models in the tier. Agentic completion: 14 of 15, which is actually excellent, but agentic fabrication ran at roughly one run in six in that suite, the worst fabrication rate of the tier. Long-context question answering: 10 of 26, against 24 to 26 for the models that lead that column, and that is the score that should make you think twice before pointing this model at large documents. The CMMC closed-book column was 3 of 35 (seed runs 3, 4 and 3), and code was 7 of 10, with writing 2 of 3. The tool-v2 battery scored 23 with no hallucinated answers at all, so when it does answer a tool question it does not invent, it just passes less often.

One nuance from the campaign: the model cleared the 7-of-8 speed gate with a thinking-mode recheck, but an earlier 09-25 battery was withheld after a 120k needle gate failure. The 10-02 clean re-measure stands on its own, but the episode is a fair warning that long-context behavior on this model is the fragile axis.

On ai11 with the UD-Q4_K_XL quantization the picture brightens for general knowledge: 37 of 40 correct on the 60-item MMLU-Pro battery (and 18 of 20 on the second battery), tied for best among the 4-bit picks of that sweep. If your use is question answering rather than tool orchestration, the quality story is less one-sided than the GB10 battery alone suggests.

Our summary for an IT lead: use it for fast drafting, summarization and chat where speed is the product, and route tool-heavy workflows, compliance document work and long-context analysis to a model with stronger records on those columns.

How we ran it

The GB10 records come from the 2026-10-02 tier 1 battery of the fleet benchmark program. Hardware was one GB10 unit of the DGX Spark class. The engine was vLLM nightly serving the NVFP4 checkpoint with the vendor DGX Spark recipe (DSpark k3 spec decode), started from a fresh reboot of the unit with a burn verify, so no stale clock latch could contaminate the run. Clocks were locked at the battery's standard state for the tier 1 comparison.

Method follows the campaign's fixed battery. A gate ran first to confirm the engine configuration was valid. Decode speed was measured in a steady closed loop at 1, 4, 8, 16 and 32 concurrent users, three shuffled repetitions, 512-token outputs, temperature 0, thinking mode off, with prose and code prompts measured separately. The user ceiling is the largest user count still holding at least 10 tokens per second per user with a p95 time to first token of 10 seconds or less. Prefill was measured at a single request against 8k, 32k and 128k contexts, with the 128k point only when the served context allowed it. Power figures are mean wall AC watts over the measurement window, read from the PDU outlets mapped to the engine's own unit, excluding switches. Quality ran on the dgx1 battery at vendor sampling with the tool, agentic, CMMC, code, writing and long-context batteries, plus the tool-v2 set with a hallucination readout.

The Strix Halo record comes from the 2026-10-16 ai11 study: llama.cpp Vulkan serving a UD-Q4_K_XL quantization on the Minisforum MS-S1 MAX with a large unified memory allocation. The study recorded accuracy batteries, not per-model speed, so no speed number is claimed for that configuration.

Caveats we want on the record: the 09-25 speed battery was withheld after the 120k needle gate failure, and a 09-23 single-unit survey figure was superseded by the 10-02 clean re-measure. Neither number appears anywhere on this page. Every figure shown comes from the 10-02 tier 1 results or the 10-16 ai11 study, read directly from the stored result files.

When to pick it

Pick Nemotron-3.5-Lightning when speed per user is the binding constraint and the task tolerates a quality floor. Concretely:

• A single developer or analyst pair-programming or drafting locally who wants the fastest streaming answers on one GB10: the 90.2 prose and 114.2 code decode rates are the fastest single-user figures we measured on the platform.
• A small office sharing one box for chat and summarization: the 64-user ceiling at 795.4 aggregate tokens per second means most teams will never see a queue, and the 6.3 tokens per joule keeps the power bill reasonable.
• Long-document ingestion on a budget of patience: prefill at 128k runs at 4,632 tokens per second, far ahead of the quality rivals, so first tokens arrive quickly even on very large inputs. Accept the low long-context answer accuracy, or use the model to retrieve and draft rather than to judge.

Do not pick it when: your workflow is tool-calling or agentic and a wrong tool argument is expensive (29 of 38 and a worst-in-tier fabrication rate argue for the stronger models); the task is compliance document recall or long-context QA (10 of 26 says pick a model that leads that column); or you need a verified multi-unit configuration, where we have no records for this model.

The honest comparison in one sentence: it is the sprinter of the tier 1 MoE field, unmatched over a single-user dash and strong across a crowd, while the quality crown on the same hardware belongs to Gemma-4 and the dense reference.

A few deployment notes drawn from the measured configurations:

• Start from the vendor recipe. Our record is specifically vLLM nightly with the DSpark k3 spec decode configuration that NVIDIA publishes for DGX Spark. That recipe is part of why the single-user numbers are so high; a plain serving configuration without the spec decode settings will not reproduce them, so treat the engine line on the model card above as part of the specification, not a footnote.
• Pin your expectations to the same conditions. Our speed rows ran with thinking mode off and fixed 512-token outputs, which is a fair cross-model comparison but a different shape from a production chatbot that thinks at length. If your product enables extended reasoning, expect per-answer latency to be dominated by the reasoning tokens, not the decode rate.
• Verify the long-context path before you ship it. The campaign withheld an entire battery after a 120k needle gate failure before the clean re-measure, and the long-context question score stayed low even in the clean run. Put your own document set in front of the model and check the answers before you point a document pipeline at it.
• Watch the fabrication rate in agentic flows. One fabrication per six agentic runs is the worst we measured in the tier, which matters whenever the model's output drives a subsequent action. If your workflow chains tool calls, either add verification between steps or pick a model with a stronger tool and agentic record.
• Consider the Strix Halo alternative for knowledge work. If you have an ai11-class machine and your workload is answering questions rather than driving tools, the same model in a 4-bit quantization under llama.cpp Vulkan scored at the top of its class on the MMLU-Pro battery there. The choice of hardware and engine changes the quality profile as much as the speed.

Where to go next

• LLM benchmarks hub
• Leaderboards
• How we measure
• Model tier list
• Private AI solutions
• Private LLM guide
• CMMC compliance

FAQ