Qwen3.8-27Bbenchmark results
This page holds every number we measured for this model: single-user prose speed of 162.3 tok/s on 1x GB300 · NVFP4 (modelopt_mixed, nvidia checkpoint), aggregate throughput up to 4,278 tok/s on 1x GB300 · BF16 (bf16 KV) and verified context of 243,392 tokens.
Last measured 2026-10-13Real throughput, latency and accuracy numbers, measured by Petronella Technology Group, Inc. on our own fleet. Last measured 2026-10-13.
At a glance
| Hardware needed | 1x GB300 (DGX Station GB300, dgx1), 1x RTX PRO 2000 Blackwell 16 GB (dgx1 GPU0), 1x GB10 (DGX Spark class), 4x H200 NVL (ptg-rtx, TP4), 1x RTX PRO 6000 Blackwell Max-Q (h200monsta), 1x ai11 (Minisforum MS-S1 MAX, Ryzen AI Max+ PRO 495 / Radeon 8065S, 192 GB unified, 176 GiB GTT) |
| Engine | vLLM, llama.cpp, vLLM nightly, llama.cpp (dense), MTP 3 |
| Quantization | BF16 (bf16 KV), FP8 e4m3 dynamic, NVFP4 (modelopt_mixed, nvidia checkpoint), IQ3_S GGUF with MTP head, NVFP4, FP8, Q6_K |
| Best single-user speed | 162.3 tok/s |
| Best aggregate (8 users) | 568.8 tok/s |
| Verified context | 243,392 tokens |
Checked, not assumed
How speed holds up as users are added
| Configuration | 1 user | 8 users | 16 users | 32 users |
|---|---|---|---|---|
| 1x GB300 · BF16 (bf16 KV) | 90 tok/s | 1,842 tok/s | ||
| 1x GB300 · FP8 e4m3 dynamic | 111.2 tok/s | 2,510.5 tok/s | ||
| 1x GB300 · NVFP4 (modelopt_mixed, nvidia checkpoint) | 162.3 tok/s | 3,552.3 tok/s | ||
| 1x GB10 · NVFP4 | 24.4 tok/s | 316.6 tok/s | ||
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · seo shape seqs32 | 264.4 tok/s | 331.6 tok/s | ||
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · seo shape seqs64 bt32768 | 230 tok/s | 259.5 tok/s | ||
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · tp1 | 80.1 tok/s | 568.8 tok/s | 943.2 tok/s |
Aggregate tokens per second across all users at once. One user is a single stream.
Prefill by prompt length
| 8K prompt | 32K prompt | 128K prompt | |
|---|---|---|---|
| 1x GB300 · BF16 (bf16 KV) | 24,558 tok/s | 24,215 tok/s | 18,411 tok/s |
Every measured number
Speed (single user)
| Configuration | decode 1u prose | decode 1u code |
|---|---|---|
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · BF16 (bf16 KV) | 90 tok/s | |
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · FP8 e4m3 dynamic | 111.2 tok/s | |
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · NVFP4 (modelopt_mixed, nvidia checkpoint) | 162.3 tok/s | |
| 1x GB10 (DGX Spark class) · vLLM nightly · NVFP4 | 24.4 tok/s | 32.3 tok/s |
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 80.1 tok/s |
Source: fleet-bench/2026-09-25-dgx1-best-model-eval.md, fleet-bench/2026-09-23-dgx1-dense-qwen38-27b-fp8.json, fleet-bench/2026-09-23-dgx1-dense-qwen38-27b-nvfp4.json, fleet-bench/2026-10-02-best-model-per-tier-results.md, fleet-bench/2026-09-23-h200monsta-qwen38-27b-concurrency.json
Aggregate throughput
| Configuration | agg decode 8u | agg decode 32u | ceiling users | peak agg decode |
|---|---|---|---|---|
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · BF16 (bf16 KV) | 1,842 tok/s | 4,278 tok/s | ||
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · FP8 e4m3 dynamic | 2,510.5 tok/s | |||
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · NVFP4 (modelopt_mixed, nvidia checkpoint) | 3,552.3 tok/s | |||
| 1x GB10 (DGX Spark class) · vLLM nightly · NVFP4 | 316.6 tok/s | 32 users | ||
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 264.4 tok/s | 331.6 tok/s | ||
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 259.5 tok/s | 64 users | 270.9 tok/s | |
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 568.8 tok/s | 32 users | 1,358 tok/s |
Source: fleet-bench/2026-09-25-dgx1-best-model-eval.md, fleet-bench/2026-09-23-dgx1-dense-qwen38-27b-fp8.json, fleet-bench/2026-09-23-dgx1-dense-qwen38-27b-nvfp4.json, fleet-bench/2026-10-02-best-model-per-tier-results.md, fleet-bench/2026-09-21-h200monsta-qwen38-27b-seo-shape-seqs32.json, fleet-bench/2026-09-21-h200monsta-qwen38-27b-seo-shape-seqs64-bt32768.json, fleet-bench/2026-09-23-h200monsta-qwen38-27b-concurrency.json
Prefill and latency
| Configuration | prefill 8k | prefill 32k | prefill 128k |
|---|---|---|---|
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · BF16 (bf16 KV) | 24,558 tok/s | 24,215 tok/s | 18,411 tok/s |
Source: fleet-bench/2026-09-25-dgx1-best-model-eval.md
Context
| Configuration | context verified |
|---|---|
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · BF16 (bf16 KV) | 243,392 tokens |
| 1x GB10 (DGX Spark class) · vLLM nightly · NVFP4 | 243,392 tokens |
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 131,072 tokens |
Source: fleet-bench/2026-09-25-dgx1-best-model-eval.md, fleet-bench/2026-10-02-best-model-per-tier-results.md, fleet-bench/2026-09-23-h200monsta-qwen38-27b-concurrency.json
Quality
| Configuration | cmmc rag strict pct | toolcall pass |
|---|---|---|
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · BF16 (bf16 KV) | 38 count | |
| 4x H200 NVL (ptg-rtx, TP4) · vLLM · FP8 | 90 % | |
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 0.971 % |
Source: fleet-bench/2026-09-25-dgx1-best-model-eval.md, fleet-bench/2026-10-02-best-model-per-tier-results.md, fleet-bench/2026-10-03-cmmc-rag-v3-results.md
Power and efficiency
| Configuration | tok per joule |
|---|---|
| 1x GB10 (DGX Spark class) · vLLM nightly · NVFP4 | 2.39 tok/J |
Source: fleet-bench/2026-10-02-best-model-per-tier-results.md
Other measurements
| Configuration | Closed-book CMMC items correct of 35 (vendor sampling) | HumanEval pass@1 (vendor sampling) | Aggregate decode at 4 concurrent streams (deployed config, MTP, 4 slots, 32k q8 KV) | Prefill tok/s (deployed config, unspecified input size) | Prefill tok/s on a 28,305-token prompt | CMMC RAG v3 code-first strict, mean over 3 seeds | CMMC RAG v3 frozen strict, mean over 3 seeds | CMMC RAG v3 closed-book strict, 1 seed | TTFT p50, 8 users (8,645-token SEO prompt) | Prompt tokens, median (padded SEO shape) | TTFT p50, 16 users | Aggregate decode at 16 users | Aggregate decode at 48 users (queued past max_num_seqs 32) | Per-stream decode median, 1 user | MMLU-Pro correct of 40 (60-item battery) | HumanEval correct of 20 (60-item battery) | lowest of the measured generation range | highest of the measured generation range | prefill speed |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1x GB300 (DGX Station GB300, dgx1) · vLLM · BF16 (bf16 KV) | 5 count | 94.5 % | |||||||||||||||||
| 1x RTX PRO 2000 Blackwell 16 GB (dgx1 GPU0) · llama.cpp · IQ3_S GGUF with MTP head | 87.8 tok/s | 600 tok/s | 496 tok/s | ||||||||||||||||
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 0.925 rate | 0.875 rate | 0.258 rate | ||||||||||||||||
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 11.447 s | 8,646 tokens | |||||||||||||||||
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 46.956 s | 230 tok/s | |||||||||||||||||
| 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) · vLLM · FP8 | 943.2 tok/s | 1,190.2 tok/s | 82.9 tok/s | ||||||||||||||||
| 1x ai11 (Minisforum MS-S1 MAX, Ryzen AI Max+ PRO 495 / Radeon 8065S, 192 GB unified, 176 GiB GTT) · llama.cpp (dense), MTP 3 · Q6_K | 38 of 40 | 20 of 20 | 15 tok/s | 22 tok/s | 162 tok/s |
Source: fleet-bench/2026-09-25-dgx1-best-model-eval.md, fleet-bench/2026-09-28-dgx1-huihui-qwen38-abliterated.md, fleet-bench/2026-10-03-cmmc-rag-v3-results.md, fleet-bench/2026-09-21-h200monsta-qwen38-27b-seo-shape-seqs32.json, fleet-bench/2026-09-21-h200monsta-qwen38-27b-seo-shape-seqs64-bt32768.json, fleet-bench/2026-09-23-h200monsta-qwen38-27b-concurrency.json, ai11/RESULTS-2026-10-13-model-sweep.md
What it is
Qwen3.8-27B is a dense 27-billion-parameter model from Alibaba's Qwen family. Unlike the mixture-of-experts models in this benchmark set, every one of its parameters runs on every token, which makes it heavier to serve than a same-class MoE model but steadier to reason about: there is no expert-routing lottery, and its behavior does not change when you scale the number of units. We use it internally as the dense reference point whenever we score a new MoE model on the fleet, and that role is why it has more measured configurations than almost anything else on these pages.
The headline finding from our fleet data is simple and a little surprising: with retrieval wired in, this model punches at the level of models several times its size, and it is the best tool-calling and long-context dense model we have measured on a single GB10 unit. Without retrieval, its compliance recall is genuinely weak. That split, good when grounded, weak closed-book, is the single most important thing to understand about this model before you deploy it.
The family name appears in our records under several checkpoints: the stock Qwen3.8-27B in NVFP4, FP8 and BF16 quantizations, a DavidAU community fusion build in Q6_K that ran on the AMD Strix Halo box, and an abliterated Huihui variant we serve for authorized red-team work only. All of them are the same architecture, so the speed and quality pattern carries across the quantizations even though the absolute numbers differ.
Where it runs best
The records cover four hardware families, and the honest ranking for this model looks like this.
DGX Station GB300 (dgx1) is where it is fastest. Single-user prose decode measured at 162.3 tok/s with the NVFP4 quantization, 111.2 tok/s with FP8, and 90 tok/s in full BF16. Aggregate throughput at 32 concurrent users reached 3,552.3 tok/s on NVFP4 and 2,510.5 tok/s on FP8, and the BF16 configuration scaled to a peak aggregate of 4,278 tok/s at the top of the ladder. If you have a Station-class box, this model is fast enough to serve a whole team from one card, and the NVFP4 checkpoint is the fastest of the three quants by a clear margin.
One RTX PRO Blackwell Max-Q GPU on the h200monsta server is the solid mid-tier. The FP8 build on a single card decoded at 80.1 tok/s for one user and peaked at 1,358.0 tok/s aggregate at 32 users, holding 32 users at the ceiling before the engine's sequence cap queued anything. On the real SEO-shaped workload, an 8,646-token prompt with long outputs, the same service managed 264.4 tok/s aggregate at 8 users, and pushing to 64 users with a larger block size only reached 270.9 tok/s at the cost of minute-scale time to first token. Read that as: short prompts are this model's home turf, and very long prompts on a single card will make callers wait.
A single GB10 unit is where it is slow, and this is the honest weak spot. On one DGX Spark-class unit it decoded at 24.4 tok/s for prose and 32.3 tok/s for code, reaching 316.6 tok/s aggregate at 32 users. That is roughly a quarter of the speed of the fastest MoE models on the same hardware, which exchanged about four times Nemotron's speed for the best quality in the one-unit tier. It also passed 7 of 8 gate runs plus a recheck, so one gate item needed a second look. If you need interactive chat on one Spark unit, pick a Flash-class model; this one is the careful, patient option.
AMD Strix Halo (ai11) is the slowest measured configuration. The DavidAU fusion build in Q6_K generated at 15 to 22 tok/s with prefill of 162 tok/s, but it posted the highest quality score in that sweep: 38 of 40 on the MMLU-Pro battery and 20 of 20 on HumanEval. It is a patient-jobs model there, best for overnight or background work where correctness outranks response time.
The RTX PRO 2000 (16 GB) deployment is a special case. The abliterated Huihui variant in IQ3_S with MTP speculative decoding served 4 concurrent slots at 87.8 tok/s aggregate with prefill of 600 tok/s, and prefilled a 28,305-token prompt at 496 tok/s without running out of memory. That is a purpose-built deployment for the security team's authorized offensive-testing work on our own infrastructure, never a general-purpose alias, and it is included here because it shows the model family fits a 16 GB card when quantized hard.
Measured speed
The verified numbers, configuration by configuration:
- DGX Station GB300, NVFP4, vLLM: 162.3 tok/s single-user prose decode, 3,552.3 tok/s aggregate at 32 users.
Fastest single-stream of the three quants we measured on this box.
- DGX Station GB300, FP8 e4m3 dynamic, vLLM: 111.2 tok/s single-user, 2,510.5 tok/s aggregate at 32 users.
- DGX Station GB300, BF16 with bf16 KV cache, vLLM: 90 tok/s single-user prose, 1,842 tok/s aggregate at 32
users, peaking at 4,278 tok/s at the top of the ladder. The ceiling was not reached because the engine's max-num-seqs capped by the engine, not by the model. Prefill measured 24,558 tok/s at 8k prompts, 24,215 tok/s at 32k and 18,411 tok/s at 128k, which makes long-document ingestion one of this model's real strengths on the GB300.
- One GB10 (DGX Spark class), NVFP4, vLLM nightly with MTP3: 24.4 tok/s prose and 32.3 tok/s code single-user,
316.6 tok/s aggregate at 32 users, which is also the ceiling. Efficiency measured 2.39 tok/J at the ceiling.
- One RTX PRO Blackwell Max-Q GPU of the four-GPU h200monsta server (the others were busy), FP8, vLLM with TP=1: 80.1 tok/s for one
user, 568.8 tok/s aggregate at 8 users, 943.2 at 16, 1,358.0 at 32, and 1,190.2 at 48 queued past the 32-slot cap. Throughput peaks at 32 users and drops when you queue beyond it; per-stream speed falls roughly halved in the queued 48-user run.
- AMD Strix Halo, DavidAU fusion Q6_K, llama.cpp with MTP 3: 15 to 22 tok/s generation, 162 tok/s prefill.
- RTX PRO 2000 Blackwell 16 GB, abliterated IQ3_S GGUF, llama.cpp with MTP: 87.8 tok/s aggregate across 4 slots,
600 tok/s prefill.
The pattern to take away: this is a dense model, so its speed tracks the memory bandwidth of the hardware. On datacenter-class cards it is genuinely fast, and its prefill is excellent; on bandwidth-limited desktop-class units it pays the full dense parameter cost on every token and lands at the back of the tier.
Quality and accuracy
Quality is where this model earns its place, and where the closed-book caveat matters most.
With retrieval, it ties the best model we test. On our CMMC RAG v3 battery, the FP8 build on the single FP8 build scored 0.971 strict in agentic mode, averaged over 3 seeds, statistically tied with GLM-5.3-Flash. The same records show 0.925 in code-first mode and 0.875 in frozen-passage mode, both over 3 seeds. A separate reference run from the v3 builder on the four-GPU H200 NVL server with TP4 scored 90 percent agentic strict, consistent with the ladder. In the same battery, gpt-oss-120b fell well behind and the 4 to 8B models fell well behind in agentic mode, so the 27B dense is in a different league once the model is allowed to drive retrieval itself.
Closed-book, it is weak, and we will not pretend otherwise. On the vendor dgx1 battery it answered 5 of 35 closed-book CMMC items correctly. The GB10 records note closed-book recall effectively absent at 1 to 4 of 35. The same pattern shows in the RAG ladder: 0.258 closed-book strict on one seed. If a workload depends on the model knowing compliance content from memory alone, this is the wrong model. With your own documents retrieved into context, that weakness mostly stops mattering, which is exactly how we run it.
Tool calling and long context are its best traits. On the dgx1 battery in BF16 it passed all 38 tool-call items, all 15 agent items, and all 26 long-context items, with verified context of the verified context depth confirmed by three needle passes. HumanEval came to 94.5 percent. On the single GB10 it was likewise the best tool, agent and long-context model of the one-unit tier, with 26 of 26 long-context items. On the Strix Halo sweep it was the highest scorer of the entire sweep, 38 of 40 MMLU-Pro and 20 of 20 HumanEval, just slow.
Hallucination in tool calls is not zero. The dgx1 battery recorded a nonzero rate of hallucinated calls on the multi-turn tool corpus in BF16. That matches several peers on the same battery, but it is a reason to keep tool schemas tight and to prefer the code-first wiring for high-stakes lookups, where fabrications drop to near zero for every model tested.
How we ran it
Every configuration in the tables was measured on our own fleet, from stored result files that name the exact engine build and flags. The GB300 runs used vLLM, one GPU, with BF16, FP8 and NVFP4 checkpoints measured on 2026-09-23 and 2026-09-25, including a concurrency ladder up to the engine sequence limit and a needle test at the verified context depth. The single GB10 run used vLLM nightly with MTP3 speculative decoding, the same speedgate battery as every other model on the one-unit tier: a gate first, then closed-loop decode at 1, 4, 8, 16 and 32 users with 512-token outputs at temperature zero, prose and code separately, and a ceiling defined as the most users who all hold at least 10 tok/s each. Quality batteries ran at vendor sampling. The single-card h200monsta runs used the live FP8 service with TP=1, max_num_seqs 32 and MTP speculative decoding, re-measured on 2026-09-23 with the 09-21 run reproduced within noise. The Strix Halo run used unsloth llama.cpp with the MTP draft head under the Gufo toolbox on the 2026-10-13 sweep. Where a result was superseded, for example the SEO-shape runs at two sequence settings on the h200monsta card, we report the newer ladder and label the long-prompt runs as not comparable to the small-prompt one.
When to pick it
Pick Qwen3.8-27B when three things are true at once.
1. Your workload is grounded: the model reads your documents, your codebase or your compliance corpus through retrieval, and the quality bar is high. This is the model that matched the much larger flagship in agentic retrieval while fitting comfortably on one datacenter GPU in FP8. 2. You need long context that actually works. Verified needle retrieval at the record's verified context depth on the GB300 and the best long-context score on one GB10 unit make it a safe choice for whole-document analysis, contract review and log triage. 3. You can accept the closed-book limits. Anything that requires the model to know regulation content from memory will disappoint at 5 of 35, and on a single Spark unit you trade about three quarters of the speed of a Flash-class MoE model for quality.
Do not pick it as the interactive chat model on a single GB10; pick it as the batch, agent and long-context model there. Do not pick it for closed-book compliance Q&A anywhere; wire in retrieval first. On a GB300 or a single RTX PRO Blackwell Max-Q card on h200monsta, it is an easy default.
How each configuration was served
| Configuration | Hardware | Units | Engine | Quant | Measured | Notes |
|---|---|---|---|---|---|---|
| 1x GB300 · BF16 (bf16 KV) | 1x GB300 (DGX Station GB300, dgx1) | 1 | vLLM | BF16 (bf16 KV) | 2026-09-25 | |
| 1x GB300 · FP8 e4m3 dynamic | 1x GB300 (DGX Station GB300, dgx1) | 1 | vLLM | FP8 e4m3 dynamic | 2026-09-23 | |
| 1x GB300 · NVFP4 (modelopt_mixed, nvidia checkpoint) | 1x GB300 (DGX Station GB300, dgx1) | 1 | vLLM | NVFP4 (modelopt_mixed, nvidia checkpoint) | 2026-09-23 | Fastest single-stream of the three Qwen3.8-27B quants on the GB300. |
| 1x RTX PRO 2000 Blackwell 16 GB · IQ3_S GGUF with MTP head | 1x RTX PRO 2000 Blackwell 16 GB (dgx1 GPU0) | 1 | llama.cpp | IQ3_S GGUF with MTP head | 2026-09-28 | UNCENSORED abliterated model for the security team's authorized red-team testing on our own infrastructure only; never a gateway alias. Runs on dgx1's RTX PRO 2000 GPU0, the GB300 is untouched. |
| 1x GB10 · NVFP4 | 1x GB10 (DGX Spark class) | 1 | vLLM nightly | NVFP4 | 2026-10-02 | |
| 4x H200 NVL · FP8 | 4x H200 NVL (ptg-rtx, TP4) | 4 | vLLM | FP8 | 2026-10-02 | Reference row from the v3 builder (ptg-rtx/H200, seed 21). |
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · cmmc rag v3 | 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) | 1 | vLLM | FP8 | 2026-10-03 | CMMC-RAG v3 ladder row; Qwen3.8-27B FP8 on h200monsta GPU 0. |
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · seo shape seqs32 | 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) | 1 | vLLM | FP8 | 2026-09-21 | |
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · seo shape seqs64 bt32768 | 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) | 1 | vLLM | FP8 | 2026-09-21 | |
| 1x RTX PRO 6000 Blackwell Max-Q · FP8 · vLLM · tp1 | 1x RTX PRO 6000 Blackwell Max-Q (h200monsta) | 1 | vLLM | FP8 | 2026-09-23 | Re-measurement of the 2026-09-21 run on the same live vLLM service (TP=1 on GPU 0, max-num-seqs 32, MTP spec-decode 2, prefix caching). Supersedes the 09-21 run (values reproduced within noise). The other three GPUs were busy, so this is one RTX PRO 6000 card, not four. |
| 1x ai11 · Q6_K | 1x ai11 (Minisforum MS-S1 MAX, Ryzen AI Max+ PRO 495 / Radeon 8065S, 192 GB unified, 176 GiB GTT) | 1 | llama.cpp (dense), MTP 3 | Q6_K | 2026-10-13 | Highest score in the sweep but slow: a patient-jobs dense model. |
FAQ
Is Qwen3.8-27B fast enough for interactive use on a single DGX Spark?
What hardware does it need to run well?
How accurate is it on compliance work?
How does it compare with the mixture-of-experts models on a DGX Spark?
What is its verified context length?
How many concurrent users can it serve?
Which quantization should we run?
Does it hallucinate tool calls?
Can it run on a small workstation GPU?
How does it handle long prompts?
Run this model on your own hardware
Call Penny at 919-348-4912 and describe your fleet, or ask us to benchmark a model on the hardware you already own.