GB10 (DGX Spark) vs H200 for LLM Inference:The Same Models, Measured Side by Side

The GB10 and the H200 sit at opposite ends of the private-inference ladder we run for clients. The GB10 is a desk-side appliance: one small box with unified memory that an individual team can own outright, run on a normal office circuit and move between offices. The H200 is a data-center accelerator; our pool is four of them on NVLink, and it serves production traffic for our own agents every day. Between those two sits everything from a single workstation card to a rack of pods, which is why the benchmark hub exists as a set of pages rather than a single table.

For this comparison we only used configurations where the same model was measured on both tiers with our standard battery: a fixed prompt set for single-user prose and code decoding, a concurrency ladder that walks user counts up to the engine's ceiling, prefill timing at three prompt lengths, long-context needle tests, tool-calling suites and our retrieval-grounded compliance set. Every speed number is gated: the engine must pass arithmetic and needle checks before any throughput is recorded. The full rules, the gate details and the conclusions we have retracted after auditing ourselves are on the methodology page, and the complete ladders behind every table here are on the leaderboards page.

Two honest caveats before the numbers. First, the quantizations differ by family: the GB10 results below use NVFP4 or MXFP4 checkpoints because that is what fits and runs well on that class of hardware, while the H200 results mostly use FP8. Quantization is part of the configuration, so the comparison is "the best configuration we have measured on each tier for this model," not a like-for-like precision test. Second, concurrency ceilings differ: the GB10 appliances saturate somewhere between six and sixty-four users depending on the model, while the H200 pool was laddered to much higher user counts. We report each tier at its own measured ceiling and at matched user counts where both exist.

Want this comparison run against your own workloads? Schedule a test on our fleet

Three patterns matter. First, the gap is real but model-dependent: Qwen3.8-27B runs about three times faster per stream on one H200 than on one GB10, while DeepSeek-V4.1-Flash nearly closes the gap entirely, because that model is a sparse mixture-of-experts whose per-token work is small relative to the memory traffic both tiers can supply. If the model you want is DeepSeek-class, a GB10 pod is a much less painful downgrade than the headline gap suggests.

Second, speculative decoding changes the ranking on the GB10 side. GLM-5.3-Flash on the four-unit pod reaches 44.9 tok/s per stream with the community DFlash2 recipe, which is respectable for a model of that size, but on the H200 pool the same model reaches 238 tok/s with plain FP8 serving, more than five times faster, without any speculative recipe. On the H200 the hardware does the work; on the GB10 the recipe does part of the work. Both are legitimate, but they age differently: a recipe tied to one checkpoint version can break on the next update, while raw FP8 serving survives it.

Third, note what the two-GPU numbers say about scaling. GLM-5.3-Flash on a two-H200 node runs 160.0 tok/s per stream, and on a two-GB10 pod runs 24.1 tok/s, yet a single GB10 running smaller models beats both of those configurations when the model fits: Nemotron-3.5-Lightning-30B-A3B on one GB10 runs 90.2 tok/s for prose and 114.2 tok/s on code prompts. Matching the model to the hardware tier matters more than buying more of the smaller unit.

For reading speed rather than generation speed, these numbers translate directly: at 80 tok/s a page of prose arrives in a couple of seconds; at 24 tok/s you are watching text type out. For chat assistants and coding copilots, where the user reads while the model writes, everything from 30 tok/s upward on the GB10 feels usable, and everything on the H200 feels instant.

Aggregate throughput is the number that sizes a deployment, and it is where the two tiers stop being comparable in kind. One GB10 appliance serving gpt-oss-120b tops out at 198.0 tok/s across its sixteen-user ceiling; a single H200 serving the same model sustains 1,118.1 tok/s at thirty-two users and peaked at 1,511.2 tok/s in our ladder. That is not a tuning gap we could close with a better engine: the H200 simply has more memory bandwidth to feed concurrent streams.

The practical reading is a head-count rule. A GB10 appliance comfortably serves a small team: one user editing code, a handful of chat sessions, a nightly batch job. Past roughly its ceiling, per-user speed collapses because the queues fill; on gpt-oss-120b, aggregate throughput actually falls from 290.5 tok/s at thirty-two requested streams to 198.0 tok/s at the sixteen-user ceiling. A four-way H200 pool, by contrast, kept absorbing users: our GLM-5.3-Flash production service ran 2,254 tok/s aggregate at thirty-two users and reached 5,245 tok/s at its peak, which is why it runs our own voice and console agents all day.

There is one more number worth naming: the H200 pool's serving headroom is not just speed, it is verified context under load. The four-way GLM-5.3-Flash configuration held a verified context of 510,997 tokens while serving concurrent traffic, and its KV cache capacity was measured at 3,329,280 tokens. The GB10 appliances verified between roughly one hundred twenty thousand and two hundred forty-three thousand tokens of usable context per configuration, which is plenty for document work and not enough for whole-repository or whole-case-file sessions.

On prefill the H200 is in a different class. The four-way GLM-5.3-Flash configuration read an eight-thousand-token prompt at more than twenty thousand tokens per second and, unusually, did not slow down as prompts grew: it sustained roughly the same prefill rate at the 32k and 128k lengths, around twenty-two thousand tokens per second. That flatness matters for real work, because it means time-to-first-token stays predictable as sessions grow. The GB10 appliances prefill fastest on short prompts, up to about seven and a half thousand tokens per second on the best one-unit configurations, and then fall off as prompt length grows, sharply so on models with small active expert counts: Gemma-4-26B-A4B dropped from 6,247 tok/s at 8k to 784 tok/s at 128k.

On verified context the H200 pool wins by a wide margin where it matters most. The GLM-5.3-Flash run found needles at a verified 510,997 tokens; the largest verified contexts on the GB10 side are the Qwen3.8-27B run at 243,392 tokens and the Gemma-4 run at 243,271 tokens, with gpt-oss-120b verified at 120,755 tokens on one GB10. For compliance work where an entire policy library, or an entire engagement folder, needs to be in one prompt, the H200 tier is the only one of the two that has demonstrated half a million tokens.

The latency side of prefill shows up in the time-to-first-token measurements too. On one H200, gpt-oss-120b returned the first token of a three-thousand-token judged prompt in a median of 0.904 seconds for a single user and 7.99 seconds with thirty-two users queued; the median prompt in that test was 3,098 tokens. On the GB10 appliances we did not record the same TTFT statistic for that model, so we report what we have rather than filling the gap with a guess.

Fast wrong answers are worse than slow right ones, so both tiers run the same quality battery where the models overlap. On our retrieval-grounded compliance suite, the strict-scoring results were close between the tiers for the same model. Qwen3.8-27B scored 0.971 strict on one H200 and 0.925 on the code-first variant, while its GB10 NVFP4 run recorded an agentic strict score of 0.735 under the stricter agentic protocol. gpt-oss-120b scored 76 percent strict on one H200 and 0.889 code-first, against 77 percent strict on one GB10, essentially the same. On the four-unit H200 pool, GLM-5.3-Flash scored 0.964 strict with a 0.928 code-first mean, and on a two-GB10 pod with the DFlash2 recipe it scored 0.978 on the agentic strict protocol, which makes GLM-5.3-Flash the standout quality-per-watt choice on the GB10 tier.

Tool calling tells a similar story. On four H200s, GLM-5.3-Flash passed 23 of 23 tool-call cases with 85.2 percent overall accuracy, a 0.917 agentic completion rate and zero hallucinated tool calls, and passed 38 of 38 on the deeper tool suite. On the GB10 side, the DeepSeek-V4.1-Flash TensorFold recipes passed 20 of 20 tool-call cases on both engine variants. The pattern we keep seeing: at current checkpoint quality, the small-model tier loses to the big tier on ceiling, not on competence. A GB10 pod does the compliance-grounded and tool-driven work we ask of it; it just cannot do it for forty people at once.

One caution from the same data: quantization is a quality variable, not a constant. The DeepSeek-V4.1-Flash EXL3 recipe on a two-GB10 pod reached 91 of 93 correct on the agentic compliance set for one seed, and the native TensorFold variant of the same checkpoint reached 87 of 93. Ten points of score moved with a serving-recipe change, no hardware change at all, which is why every card on our pages names the engine and the quant next to the number.

The GB10 appliances are efficient in a way the H200 pool is not. Nemotron-3.5-Lightning on one GB10 measured 6.3 tokens per joule, and Qwen3.6-35B-A3B measured 3.62 tokens per joule on the same class of hardware; the eight-unit MiMo-V2.6-Pro-RL pod drew a measured 1,169 watts across all eight units while decoding. The GB10 runs on a normal office circuit, needs no rack, and can sit next to the compliance data it works on. The H200 pool is a data-center asset: our baseline measurement of a single H200-class card recorded copy bandwidth of 1,468 GB/s and read bandwidth of 1,633.1 GB/s at 300 watts for one card, before the other three cards, the NVLink fabric and the cooling that keeps them fed.

That difference in placement is often the deciding factor, not the benchmark table. A two-person development team that handles sensitive client material can put a GB10 under a desk this week and keep every token inside their own boundary. A firm that needs to serve an entire department, or that wants to run an always-on agent fleet like ours, needs the H200 tier's memory bandwidth and its verified half-million-token context, and needs somewhere to put it.

If you are choosing for one person, the honest answer is that a single GB10 running the right model is enough: Nemotron-3.5-Lightning at 90.2 tok/s per stream is faster to read than most people read, and its 795.4 tok/s aggregate at the sixty-four-user ceiling means the same box absorbs a team spike. If you are choosing for a department or an agent fleet, the H200 pool is the only one of the two that held 2,254 tok/s of aggregate decode at thirty-two users with 510,997 tokens of verified context, and that difference is structural.

The model pages carry the per-model detail behind this comparison. Start with GLM-5.3-Flash, which is measured on both tiers and is our quality pick on each, and gpt-oss-120b, whose single-H200 numbers show what one data-center card does for one model. For the mixture-of-experts models that make the GB10 tier shine, see Nemotron-3.5-Lightning and DeepSeek-V4.1-Flash; for the dense small model, see Qwen3.8-27B. The leaderboards page holds the full concurrency ladders, and the hub ties every family together on the LLM benchmarks overview.