DGX Spark Shutdown Under Load: Our 8-Unit GB10 Diagnosis

A DGX Spark shutdown under load is one of the most confusing failures you can hit on NVIDIA GB10 hardware. The unit does not crash, reboot or log an error. It simply turns off, drops to a few watts of standby draw at the outlet, and stays off until someone removes and restores its power. On September 25 and 26 we saw this happen repeatedly on our own eight-unit GB10 pod, four NVIDIA DGX Spark systems and four MSI EdgeXpert MS-C931 systems, every time at the same moment: the start of a very long prompt. This post is our field report. It covers how we instrumented the problem, what we ruled out, the GPU clock lock that has kept every unit up since, what that lock costs in speed, and a second finding from the same week: a 238 versus 262 GB/s split in GB10 memory bandwidth between identical units, and the one-minute habit that fixes it.

We run this pod for model testing and for the self-hosted LLM benchmarks we publish, so we measure before we change anything. Every number below comes from telemetry we saved during the tests. Where a statement is our interpretation rather than a reading, we say so. Running eight units as one pod also keeps the cabling simple: one DGX Spark cluster cable links any two units over their QSFP112 ports.

Update, October 2026

We kept testing after this post went up, and some of what we wrote in September needs a follow-up. The short version:

• Separating the power bricks helped, but it did not fix the DGX Spark shutdown. With the bricks spaced apart, default clocks produced 1 power-off in 233 starts of a 131,000-token prefill across the full pod, against 2 in 22 on one hot unit while the bricks were still taped in pairs. The one loss had the same signature as every earlier one.
• A higher clock lock held up on our pod. A 2,300 MHz ceiling ran 448 long-prompt starts with no power-off, including a 62-minute soak with all eight units loaded, and a 2,400 MHz ceiling ran 224 with none. Since September 30 our pod runs nvidia-smi -lgc 300,2300, which buys back about 1.3 percent of single-user decode and 2.6 to 3.2 percent of prefill compared with the 2,200 MHz lock.
• A new failure mode: after a power loss, and in one case after a routine reboot, a GB10 can come back stuck near 611 MHz under load, with SW Power Cap: Active reported. Soft reboots did not clear it. Removing AC power completely did.
• Bricks must match units. Some of our MSI EdgeXpert units had been running on NVIDIA Spark power bricks. Every unit is now back on its own maker's adapter.
• After we rearranged the rack, all four DGX Spark units powered off at default clocks in the same 35-second window, so the clock lock stays. Run alone with the rack idle, two of them survived the same test and one powered off again, so the cause is still open.
• With every unit at full speed, our eight-unit pod matched the published MiMo-V2.6-Pro-RL speeds on the published extraction workload, and the move did not change our two-, four- and eight-unit scaling results.

The full data, what we changed, and what is still unresolved are in the October 2026 re-test section near the end of this post. The checklist and the open questions below are updated to match.

Key takeaways

• The symptom: a GB10 unit powers itself off within about 8 to 11 seconds of a 131,000-token prefill starting, most often when it is already hot from serving. The OS logs nothing, and the on-disk journal loses the last 6 to 14 seconds before the power-off.
• Not the usual suspects: it was not host memory exhaustion, not the OS thermal trip, not the power strip, and not average wall power. A unit survived three minutes at 204 to 212 W and then powered off at 145 W and 169 W in other runs.
• The workaround: locking the GPU clock with nvidia-smi -lgc 300,2200 at boot. On one unit, one day, the same test sequence powered off 2 of 2 times at default clocks and 0 of 6 times with the lock (Fisher exact test, one-sided, p = 0.036). With the lock on all seven running units, 66 long-input starts under full pod load produced 0 power-offs. Since September 30 we lock at 2,300 MHz instead (see the October update).
• The cost: about 1 to 3 percent of single-user decode speed on the diagnosis unit, nothing at 32 users, and long-prompt prefill that is as fast or faster once the unit is warm.
• The bandwidth split: identical units measured 262 GB/s or 238 to 242 GB/s of GPU read bandwidth. The cause was fragmented free memory left behind by earlier engine launches. Rebooting before launching restored 262 GB/s and raised decode by 4 to 5 percent.
• Still open: the root cause is best supported, not proven, and the lock is a workaround until NVIDIA ships a platform fix. The power bricks were taped together in pairs during these tests. Separating them in a later re-test reduced the power-offs but did not end them.

What a DGX Spark shutdown under load looks like

If you have searched for a DGX Spark shutdown, you have probably read forum threads that describe the same pattern we saw. The unit is off, not rebooting. There is no kernel panic, no out-of-memory killer message, no thermal shutdown line and no power button event. The last lines in the journal are routine. At the outlet, the unit draws about 4 W, the standby level.

Our first four losses came between the afternoon of September 25 and the early hours of September 26, during a model comparison that kept all eight units busy at once: Spark 1 twice, Spark 2 once and MSI 3 once. Three of them came at or near the start of a 131,000-token prefill, and the MSI loss came inside a 120,000-token needle test. The eight GB10 units together were drawing 1,013 to 1,361 W at the wall at those moments. The first of them is the incident we describe in our companion post, DGX Spark Firmware Update: What We Learned on 8 GB10 Units, which records it without a cause. This post is the diagnosis.

Two details made the problem hard to read from the unit itself. First, the on-disk journal on a unit that loses power is missing its final seconds: in our two induced losses, the saved journal ended 6.8 and 13.5 seconds before the moment of death, and the saved container log ended 26 to 29 seconds early. Anyone reading those logs afterward would conclude the unit died earlier, and more quietly, than it did. Second, the GB10 platform gives the operating system very little to look at. There is no BMC with an event log, and the power delivery path is not visible to Linux in the way a server power supply is.

How we instrumented a unit that erases its own evidence

The fix for the evidence problem was to stop trusting the unit's own disk. For the diagnosis runs on one DGX Spark (we call it Spark 2 here), we streamed telemetry over SSH to a separate workstation, so the record survived the power-off:

• memory, memory pressure (PSI), all seven ACPI thermal zones and the process count every 0.5 seconds;
• nvidia-smi every 250 milliseconds: instantaneous and average GPU power, GPU temperature, the T.limit thermal margin, SM clock, and the clock-event (throttle) reasons;
• a live follow of the kernel and system journal, and of the inference engine's container log;
• a network ping every 0.5 seconds, so we knew the moment of death to within half a second;
• wall power per unit from a metered, switched PDU (the PDU controller refreshes about every 30 seconds, so this is an average, not a transient reading).

The workload was the one that had failed in the contest: NVIDIA's Nemotron-3.5-Lightning-30B-A3B in NVFP4 on vLLM, with 85 percent of memory given to the engine, a 262,144-token context window, and prefill tests at 32,000 and 131,000 tokens. We did not reduce the workload to make the problem go away. The goal was a fix that keeps the full configuration.

What we ruled out

Each of these was a reasonable first guess. The telemetry excluded all of them.

• Host memory exhaustion. At both induced losses the available memory was flat at 9.17 GB and 9.40 GB, memory pressure was 0.00, and swap was untouched, sampled until under 1.5 seconds before death. The September 25 losses showed the same headroom in our metrics history.
• A burst of process spawns. We had seen a steady stream of AppArmor denial lines in the logs. They run at 95 to 100 per minute on an idle unit with no model loaded, and the process count stayed flat through every run. They come from a resident service, not from the engine.
• The operating system thermal trip. All seven ACPI zones have a single critical trip point at 104.8 C. The last readings before the two losses were 93.3 C and 96.2 C, and the same unit survived a reading of 98.1 C in another run. A kernel critical trip also performs an orderly shutdown, which would leave log lines.
• The PDU, the circuit or the voltage. The outlet relay stayed on and the voltage was steady. After each loss the outlet read 4.06 to 4.39 W, the same signature as every earlier loss.
• Too much average power. Spark 2 held 204 to 212 W at the wall for three minutes while decoding for 64 users, running a 20-thread CPU burn and three 131,000-token prefills together, and it survived. The two losses happened at 169 W and 145 W.

That last point matters. If the trigger were sustained power or heat as the OS sees it, the hardest run would have been the one to fail. It was not.

Reproducing the DGX Spark shutdown on purpose

We ran nine scenarios on Spark 2 with the exact launch configuration from the contest. Cold engines, eight 131,000-token prefills back to back, a 12-minute heat soak at 64 users, and the maximum-power run above all survived at default clocks. What killed the unit was a short sequence we could repeat: one minute of single-user decode, one minute at 32 users, two 32,000-token prefills, then a 131,000-token prefill.

• At default clocks that sequence ended in a power-off 2 of 2 times, about 11 seconds and about 8 seconds into the first 131,000-token prefill.
• With the GPU clock locked at 300 to 2,200 MHz, the same sequence ran 6 times on a fresh engine (12 prefills at 131,000 tokens and 12 at 32,000) with no power-off and no clock-event reason in any of 5,458 telemetry samples.
• Across all default-clock runs that day, 2 of 22 starts of a 131,000-token prefill ended in a power-off. With the lock, 0 of 15.

The one-sided Fisher exact test for 2 of 2 against 0 of 6 gives p = 1/28, or 0.036. That is one engine on one day, so we treat it as strong evidence, not proof.

The telemetry from the last three seconds before each loss shows what the lock takes away. At default clocks, the GPU drew between 57 and 102 W and swung by up to 45 W from one 250 ms sample to the next. The SM clock hopped between 1,872 and 2,431 MHz. The T.limit thermal margin jumped between minus 7 and plus 13 C between samples. With the lock, the same workload peaked at 83.9 W, the clock stayed between 2,177 and 2,200 MHz, and the thermal margin never dropped below 2 C.

Our reading of the evidence

This part is inference. The trip appears to be a fast event: power steps and hotspot spikes on the order of 100 milliseconds at the uncapped boost clock, on a chassis that is already hot. None of the sensors the OS can read reaches a limit, and the averaged wall power was lower than in runs that survived. So the protection that turns the unit off is most likely acting on something the OS cannot see, such as instantaneous current in the USB-C power delivery path or an internal hotspot. The unit goes to a controlled soft-off rather than losing its supply: the outlet kept reading about 4 W, and a Spark that loses its DC input powers back on by itself when the input returns, which these units did not do. Removing the top boost clock bins removes the transient, which would explain why the lock works while memory use and average power are unchanged.

That matches reports from other owners. A thread on the NVIDIA developer forum, Spark abruptly shuts down, describes the same symptom. NVIDIA's forum moderator there says that shutdowns reported by some DGX Spark and GB10 users are being investigated, that they can be triggered by both thermal and current-related reasons, and that capping the GPU clock with nvidia-smi -lgc is a practical mitigation the community has found, because it reduces the power transients that appear to trigger them. We reached the same workaround independently, with the measurements below. An MSI EdgeXpert owner reports in another forum thread that swapping the power adapter with a working unit did not move the fault; that owner's unit was eventually replaced. For our diagnosis unit, fwupd reported no newer firmware.

The workaround: lock the GPU clock with nvidia-smi -lgc

The change is one command, run as root after the NVIDIA persistence daemon is up:

nvidia-smi -lgc 300,2200

The -lgc option locks the GPU clock to a minimum and maximum in MHz. The lock does not survive a reboot, so we install it as a systemd oneshot service ordered after nvidia-persistenced.service, with ExecStart=/usr/bin/nvidia-smi -lgc 300,2200, ExecStop=/usr/bin/nvidia-smi -rgc to restore default clocks, and RemainAfterExit=yes, wanted by multi-user.target. The lock does not restart any container or service, so it can be applied to a unit that is already serving.

Verifying the lock is harder than applying it. The obvious check is nvidia-smi -q -d CLOCK, but on driver 580.178.04 it shows nothing different when the lock is set, and NVML has no call that reads the locked range back. We verify three ways: the service is enabled and active; the journal for the current boot contains the line GPU clocks set to "(gpuClkMin 300, gpuClkMax 2200)"; and a 20-second matrix-multiply burn peaks at 2,177 to 2,210 MHz instead of the 2,398 to 2,405 MHz the same burn reaches at default clocks. A peak one clock step above 2,200 MHz is normal, so our pass line is 2,250 MHz. We rebooted one unit to confirm the service re-applies the lock at boot.

October 2026: we now run the same service with nvidia-smi -lgc 300,2300. With that ceiling the same burn test peaks at about 2,288 MHz, so the pass line moves to 2,350 MHz. The same burn test is also how we catch the stuck-at-611 MHz state described in the October re-test section.

We are adding the service file, an installer with check, verify and uninstall modes, and the memory check described below to our open-source gb10-cluster-bench repository, alongside the method and raw data from our other GB10 tests.

Does the clock lock hold with the whole pod loaded?

A single-unit test cannot recreate the condition of the contest, where every unit was hot at once. So after rolling the lock out to all seven running units (one MSI unit was powered off and waiting for hands-on attention at the data center), we loaded all seven together for two phases of about 20 minutes each. In the first phase, five units each ran the single-unit Nemotron sequence while the other two ran GLM-5.3-Flash split across both units (tensor parallel size 2). In the second phase, three units ran Nemotron and two pairs ran GLM-5.3-Flash. The GLM runs included 120,000-token needle tests.

• 66 long-input starts: 63 prefills at 131,040 tokens and 3 needles at 120,000 tokens, plus 54 prefills at 32,000 tokens.
• 0 power-offs, 0 failed requests, 0 missed pings.
• The eight GB10 units together drew up to 1,111.5 W at the wall, inside the 1,013 to 1,361 W range recorded at the earlier losses.
• No GPU went above 87.5 W instantaneous and no SM clock sample went above 2,190 MHz. The hottest ACPI zone reached 96.1 C, against the 104.8 C trip point.

As a rough check on chance: at the default-clock loss rate we measured on the diagnosis unit (2 in 22 long prefills), the probability of 48 Nemotron long prefills passing with no loss is about (20/22) to the 48th power, roughly 1 percent. That assumes independent starts with equal risk, and the rack was hotter than in the diagnosis, so we read it as conservative.

What the clock lock costs

A clock cap sounds expensive. On this workload it mostly is not, because at default clocks a warm GB10 already spends time in thermal and power slowdown.

• Single-user decode: on Spark 2, 91.0 to 92.1 tokens per second at default clocks and 89.2 median with the lock, a 2 to 3 percent loss. Under full pod load, Spark 2 measured 89.4 to 90.2, a 1 to 3 percent loss.
• 32-user decode: 614.6 to 618.8 tokens per second at default clocks and 619.0 median with the lock, no loss.
• 131,000-token prefill: 4,040 to 4,414 tokens per second at default clocks after decode heat, and 4,488 median with the lock. Only a cold first prefill is faster at default clocks (4,612 to 4,795), about 3 to 7 percent.
• One exception: Spark 1 measured 7 to 9 percent lower at one user than its default-clock figure from the day before. We believe part of that gap is the unit and the measurement window, not the lock: Spark 1 ran with a thermal margin of 0 to 1 C, and the earlier figure used longer windows. The bandwidth finding below likely explains much of the unit-to-unit spread.

What we do not know yet

We want to be plain about the limits of this work. This list was updated in October 2026; what changed since September is marked.

• The root cause is best supported, not proven. We have a strong statistical difference on one unit, a clean loaded confirmation on seven, and a mechanism that fits the telemetry. We do not have component-level proof, and we cannot see inside the embedded controller or the power delivery firmware.
• The power bricks (updated October 2026). During the September tests the 240 W USB-C power bricks at the data center were bundled in pairs. We have since separated them and re-tested. The power-offs became rare at default clocks but did not stop, so brick heat looks like a contributor, not the whole cause. We also found that some MSI units had been running on NVIDIA Spark bricks; each unit is now on its own maker's adapter, and we have not yet isolated what that change alone does.
• The Spark power-offs after the rack move (new, open). After we rearranged the rack, all four DGX Spark units powered off at default clocks within 35 seconds of each other, while the four MSI units, which reached similar temperatures, did not. Heat explains part of it but not all of it. Run one at a time with the rest of the rack idle, two of those units survived and one, the hottest of the three, powered off again; the fourth was not tested alone. We have not found the root cause yet.
• The stuck-at-611 MHz state (new, open). A full AC removal clears it, and we believe the power delivery negotiation is involved, but we have not proven what triggers it.
• Fan behavior (new, open). The MSI units run 6 to 8 C hotter than the DGX Spark units at the same clock and power. We could not read fan speeds on either type, so we cannot yet say whether the fan policy or the airflow is the limit.
• The lock is a workaround. A platform fix has to come from NVIDIA and the system makers. We have drafted a support case for NVIDIA with this telemetry and added the October data to it. It has not been sent yet.
• Coverage gaps (updated October 2026): clock ceilings of 2,300 and 2,400 MHz are now tested; the September gap is closed. The MSI unit that was powered off during the September rollout was recovered and has run every test since. Tensor parallel runs across two, four and eight units after the rack move are in the October section. Still untested after the move: all eight units at the 2,300 MHz lock under the kill sequence, Spark 2 alone, and the MSI units alone.

GB10 memory bandwidth: why eight identical units split into two tiers

During the loaded confirmation, three units decoded at about 89 to 90 tokens per second for one user while two others sat at 84 to 87, on the same model, launch settings and clock lock. The obvious suspects were a weak unit or a configuration drift. We checked both. All eight units matched on kernel, driver, VBIOS, firmware versions, clock lock, CPU governor, swap, container runtime, engine image, model files (identical SHA-256 hashes) and speculative decoding acceptance rate. Under load they ran the same SM clock, GPU power and temperature.

What differed was GPU memory bandwidth. A simple GPU read probe on a 4 GiB buffer split the pod into two tiers: 262 to 263 GB/s on three units and 238 to 242 GB/s on the other five. The split reproduced across three rounds, and it was the same whether the units ran one at a time or all at once, so it was not rack heat or a shared resource. For context, NVIDIA lists the DGX Spark at 273 GB/s of memory bandwidth, and our earlier post on DGX Spark memory bandwidth and what 400G really means covers how that number relates to cluster networking.

Decode speed followed the bandwidth closely. Across the eight unit averages, the correlation between read bandwidth and 32-user decode was 0.88. The fast tier averaged 610 tokens per second at 32 users against 591 for the rest, 3.2 percent faster on a 9.8 percent bandwidth gap. Prefill, which is compute bound, was flat on every unit within 1.5 percent.

The cause: engine launches fragment the free memory

The fast tier was always the most recently booted units, which first looked like an uptime effect. Our tests point elsewhere. On GB10 the GPU and CPU share one pool of Grace Blackwell unified memory, and the NVIDIA driver backs GPU allocations with ordinary kernel pages that cannot be moved, taking the smallest free blocks first. The read bandwidth of a buffer rises with the size of the physical blocks behind it. In our per-chunk probe, buffers built from 2 MiB blocks read at about 237 GB/s median, from 16 MiB blocks about 250 GB/s, and from 32 MiB blocks about 258 GB/s.

A freshly booted unit has almost no small free blocks in the part of memory that holds unmovable pages. One inference engine run leaves 12 to 15 GiB of them behind, and a second run about 28 GiB. The next engine to start takes those fragments first for its model weights, and because those pages cannot move, we expect that engine to keep its placement for as long as it runs. Two things did not cause fragmentation in our tests: reading 191 GB of model files through the page cache, and a unit left idle after a reboot for the 42 minutes we watched. So we believe the correlation with uptime was really a correlation with the number of engine launches since boot.

Why block size matters to the GPU is our inference: most likely the driver maps larger physical runs with larger GPU pages, which cuts address translation overhead, or larger runs spread better across memory channels. We could not separate the two from user space.

What fixed it and what did not

• Reboot before launching: works. One MSI unit went from 239.3 to 262.3 GB/s after a soft reboot, and Nemotron decode from 85.1 to 88.7 tokens per second at one user (4.2 percent) and from 592.8 to 622.5 at 32 users (5.0 percent), with the same speculative decoding acceptance. A soft reboot took 38 to 52 seconds on the Sparks and that unit, and about 1.5 to 2.2 minutes on two other MSI units.
• Relaunching the engine without a reboot: loses ground. A second launch on the same boot measured 1 to 2.5 percent lower at 32 users on three units.
• Dropping caches and compacting memory: no effect. drop_caches plus compact_memory left bandwidth unchanged on three units. Our explanation: kernel compaction moves pages into free space in movable regions, while these small holes sit inside unmovable regions.
• The cma=128M boot parameter: no effect. A DGX OS 7.6.0 slowdown thread on the NVIDIA forum suggests it. In an A/B test against a freshly rebooted control unit it gave no benefit to bandwidth or decode, so we removed it. The slowdown in that thread is a different and larger fault, and the kho=off kernel hotfix discussed there is already on all eight of our units.

We found no boot parameter or kernel setting that measured better, so we made no fleet-wide configuration change for bandwidth. The fix is a habit.

How to check a unit before a benchmark

You can see the fragmentation directly. As root, /proc/pagetypeinfo lists free blocks by size for each page type. Add up the free memory in the Unmovable rows of the Normal zone in every block size below the largest. If that total is under about 2 GiB, the unit is fresh and the next engine will get large blocks. If it is well above that, reboot before launching for full bandwidth. Our read-only check script does exactly this for a list of hosts, and after the final round of reboots all eight units read 261.8 to 263.0 GB/s.

A long-running inference server started right after a boot stays fast, because its memory placement is fixed at launch. What costs 3 to 5 percent is relaunching an engine on a unit that has already run one. For benchmarks that compare units or configurations, that difference is large enough to change a ranking, so we now reboot every participating unit first and record the check result next to every number we publish.

A GB10 stability and benchmark checklist

1. Stream telemetry off the unit for any long-context run. The unit's own journal and container logs lose their final seconds at a power-off.
2. Install the GPU clock lock at boot on every unit, and verify it with a burn test after every reboot, since nvidia-smi -q will not show it. We now use a 2,300 MHz ceiling. Separating the power bricks did not make default clocks safe on our pod, so keep the lock even with good brick spacing.
3. After any power loss or unexpected reboot, run the burn test again and compare the peak SM clock with what the lock should give (about 2,288 MHz at a 2,300 MHz ceiling). A unit that peaks near 611 MHz is in the stuck power-cap state: shut it down cleanly, remove AC power for about 15 seconds, then power it back on. A soft reboot does not clear it. In a tensor parallel group, one stuck unit slows every unit in the group.
4. Do not bundle power bricks together; give each one its own airflow. Separating ours reduced the power-offs but did not end them. Record wall power per unit if you can.
5. Match each power brick to its unit. Use the adapter that shipped with that maker's system, and label bricks and leads so they go back to the same unit after any rack work.
6. After any rack change, re-derive which switched PDU outlet feeds which unit by loading one unit at a time and watching the outlet readings, before any remote power cycle. After our rack move the outlet labels no longer matched the units.
7. Check firmware with fwupdmgr get-updates on every unit. As of September 30, 2026 all eight of our units reported the latest firmware available through LVFS.
8. Reboot every unit before launching an engine that should run at full speed, and before any benchmark that compares units. Check fragmentation first if you are not sure.
9. For tensor parallel runs, reboot every participating unit. The slowest unit sets the pace of the group.
10. Keep swap and out-of-memory policy identical across units, and keep firmware in step. Our guide to growing a DGX Spark cluster from two units to a switched fabric covers the networking side.
11. Record the clock setting, uptime and bandwidth check with every result, so a later reader can tell a hardware difference from a state difference.

What this means if you run or are buying GB10 hardware

Craig Petronella's rule for our benchmark work is to publish strengths and weaknesses alike, so here are both.

The strengths held up. A single GB10 unit served a 30-billion-parameter mixture-of-experts model at about 580 to 620 tokens per second across 32 users and prefilled 131,000-token prompts at about 4,440 to 4,500 tokens per second with the lock on. The 128 GB of unified memory fits models and context windows that most single workstation GPUs cannot. All eight units matched on every setting we checked and measured the same bandwidth once their memory state was controlled, which is what you want from a cluster.

The weaknesses are real. At default clocks, a hot unit can switch itself off at the start of a long prompt, with no log to tell you why. The platform gives the operating system little visibility into power delivery. Performance varies by 3 to 5 percent depending on how many engines ran since the last boot. None of this is a reason to avoid the hardware, but it is a reason to run it the way you would run any production system: with a documented configuration, off-box monitoring and a tested recovery path. If you are comparing platforms, our RTX PRO 6000 Blackwell versus GB10 benchmark shows where each one fits.

Petronella Technology Group, Inc. designs, builds and supports private AI clusters on NVIDIA hardware for organizations that need their data to stay on their own systems. If you are planning a DGX Spark or MSI EdgeXpert deployment, or you have units that are shutting down under load, contact us and we will share what we have learned.

October 2026 re-test: bricks apart, higher clock locks and a rack move

Between September 29 and October 1 we separated the power bricks, re-ran the same kill sequence that had powered units off, tested clock ceilings above 2,200 MHz, looked for a fan or firmware fix for the hotter MSI units, and then physically rearranged the rack and tested again. The method did not change: telemetry streamed off each unit to a separate workstation, Nemotron-3.5-Lightning-30B-A3B at the same launch settings as before, and every unit rebooted right before its engine started. The kill sequence is an 8-minute heat soak at 64 users, then repeating cycles of single-user decode, 32-user decode, two 32,000-token prefills and two 131,000-token prefills.

Separating the power bricks reduced the power-offs but did not end them

On September 29 the eight 240 W USB-C bricks were untaped and spaced at least an inch apart. On September 30 we ran the kill sequence at default clocks, first on the three units that had tripped before, then on all eight at once.

• Across 233 starts of a 131,000-token prefill at default clocks, one unit powered off: MSI 4, about 20 seconds into its first long prefill, with all eight units loaded and the pod drawing up to 1,416 W at the wall in that phase.
• The signature was the same as every earlier loss: about 4 W at the outlet with the relay on, and nothing in the operating system. In the last three seconds the GPU's instantaneous power swung between 55 and 92 W from one 250 ms sample to the next while the SM clock hopped between 2,067 and 2,327 MHz.
• For comparison, before the bricks were separated one hot unit powered off in 2 of 22 long-prefill starts. The conditions were not identical (one hot unit then, a full pod now), so the two rates are not a precise comparison, but the drop is large.

Our reading: brick heat was a contributor, not the whole cause. The remaining trigger still looks like the boost-clock transient on a hot unit.

Clock ceilings of 2,300 and 2,400 MHz

We then ran the same full-pod kill sequence with higher ceilings, and measured speed by switching the clock live on one running engine per unit, so heat and time drift cancel out between settings.

• 2,300 MHz: 448 long-prefill starts with no power-off, no failed request and no missed ping, including a 62-minute soak with all eight units loaded. Compared with the 2,200 MHz lock: single-user decode up 1.3 to 1.4 percent, prefill from 8,000 to 131,000 tokens up 2.6 to 3.2 percent, wall power up 2 to 5 percent.
• 2,400 MHz: 224 long-prefill starts with no power-off. The DGX Spark units gained 2.6 percent single-user decode and 5 to 6 percent prefill for about 9 percent more wall power. The MSI units gained almost nothing on 131,000-token prefill (0.3 percent) because they reached their thermal limit and slowed down.
• Default clocks: about 1 percent faster than 2,400 MHz on short prefill, for 21 to 23 percent more wall power than the 2,200 MHz lock, and the only setting that powered a unit off.
• 32-user decode did not move at any setting (616 to 625 tokens per second per unit). It is bound by memory bandwidth, not the SM clock, so a server with many users gains nothing from higher clocks.

Counting alone cannot prove that 2,300 or 2,400 MHz is safer than default clocks at a loss rate this low: with 0 losses in 448 and 0 in 224 starts, the 95 percent upper bounds on the per-start loss rate are about 0.7 and 1.3 percent, both above the 0.4 percent we observed at default. What decided it was the thermal margin. At 2,300 MHz every DGX Spark kept a positive margin to its thermal limit with no thermal slowdown, and the worst MSI sat at minus 3 C. At 2,400 MHz and at default, the hottest units ran at minus 4 to minus 6 C with hardware slowdown events, which is the condition the MSI 4 loss happened in. Since September 30 all eight units run the 2,300 MHz lock.

Why the MSI units run hotter, and what we could not change

At the same GPU clock and the same GPU power, the four MSI EdgeXpert units ran 6 to 8 C hotter than the four DGX Spark units. Position mattered as well: two DGX Spark units with identical firmware ran 2 to 5 C hotter than the other two. We looked for a software fix and did not find one.

• Firmware: as of September 30, fwupd reported the latest firmware available through LVFS on all eight units, and the versions were identical within each brand. MSI's download page for this system offers only the DGX OS image, with no separate BIOS or firmware package. There was nothing to flash. Our GB10 firmware update post covers how we updated all eight.
• Fan control: the MSI EdgeXpert BIOS 1.8.0 has no fan settings, and Linux exposes no fan speed sensor on the MSI units. A community-built kernel driver for the embedded controller loaded on one MSI unit, but every request to the controller timed out, so we got no fan speed data. We sent no write to the controller and changed no fan behavior.

That leaves placement and the clock lock as the levers we control.

A new failure mode: a GB10 stuck near 611 MHz

During an overnight eight-unit tensor parallel run on September 30, one MSI unit ran its GPU at about 611 MHz under full load all night, drawing about 77 W at the wall against 120 to 133 W on the other units. That unit had lost AC power earlier that afternoon. A second MSI unit fell into the same state after a routine soft reboot during the night. Three soft reboots did not clear it on the first unit, and nvidia-smi reported SW Power Cap: Active at idle on both.

In a tensor parallel group every layer waits for the slowest unit, so one stuck unit quietly set the pace for all eight. We treat the absolute numbers from that night as a lower bound and are not publishing them as representative.

What cleared it: a clean shutdown, the unit's switched outlet turned off for about 15 seconds, then power back on. After that, the burn test peaked at 2,275 and 2,288 MHz at 95 to 96 TFLOPS, normal for the 2,300 MHz lock. Our inference is that a full power removal forces the USB-C power delivery to renegotiate; we have not proven that. How to detect it: a 40-second matrix-multiply burn peaked at 611 to 617 MHz with about 16 W of GPU power on the stuck units, against 2,184 to 2,281 MHz at 90 to 94 W on the healthy ones. Do not rely on the flag alone: one recovered unit still showed SW Power Cap: Active at idle while running at full speed under load.

Matching power bricks to units

We learned that some of the MSI EdgeXpert units had been running on NVIDIA DGX Spark power bricks. During the rack work on October 1 every unit was moved back to its own maker's adapter. That means part of the September 30 MSI data was measured on the other maker's bricks, and the before-and-after comparison below mixes a placement change with a power supply change. We cannot attribute any difference to either one alone.

The rack move

On October 1 we rearranged the pod to four units per shelf, with the bricks at least two inches apart. Before any remote power action we re-derived which switched PDU outlet feeds which unit, by loading one unit at a time with a matrix-multiply burn and reading the per-outlet power. The outlet labels no longer matched the units. Cycling the wrong outlet could cut power to a unit in the middle of a test, so this check now comes first after any rack work.

The 200G fabric came through the move intact: all sixteen 200G links came back at full rate, and RDMA bandwidth and NCCL all-reduce bandwidth matched the pre-move baseline within 1 percent.

After the move: all four DGX Spark units powered off at default clocks

The first post-move test asked whether the pod could now run without the clock lock. We ran six cycles of the same kill sequence at default clocks on all eight units. In the sixth cycle all four DGX Spark units powered off within 35 seconds of each other, each a few seconds into a 32,000-token prefill. None reached that cycle's 131,000-token request. The signature was the same as before: about 4 W at the outlet with the relay on and nothing in the operating system. Each unit was recovered automatically by cycling only its own outlet, and each came back with the lock re-applied at boot. The four MSI units ran the whole test without a power-off. Under our rule for this test, any default-clock power-off keeps the lock, so the 2,300 MHz lock stays.

Heat explains part of this, but not all of it:

• For heat: before the move the DGX Spark units were the coolest in the pod; the two coolest held a thermal margin of 0 to plus 1 C with no thermal slowdown. After the move they ran as hot as the MSI units: GPU peaks of 88 to 89 C, margins of minus 6 to minus 7 C, and 1.4 to 6.2 percent of loaded samples in thermal slowdown. Their temperature at the start of each prefill block climbed from 70 to 72 C in the first cycle to 78 to 83 C in the sixth, while the MSI units stayed at about 70 to 76 C. Two of the Spark units lost 6 to 9 percent of their 131,000-token prefill speed against the same test before the move.
• Against heat alone: two MSI units reached 88 to 90 C with margins of minus 6 to minus 7 C in the same test and did not power off, while one DGX Spark unit died 2 seconds into a 32,000-token request, having started that prefill block at 83 C. In our pod the Spark units trip at temperatures the MSI units survive.

Our reading is that the new arrangement made the DGX Spark units hotter, most likely through shared shelf air, and that the Spark units have less tolerance for it. That is a placement mistake on our side to correct, and the root cause of the trip itself is still open.

Each DGX Spark alone at default clocks: two survived, one did not

To separate shared shelf heat from a per-unit cause, we ran the same six-cycle kill sequence at default clocks on one DGX Spark at a time, with the other seven units idle.

• Spark 3 and Spark 4 each survived all 6 cycles, 24 long-prefill starts each. In the full-pod test the same units, on the same bricks, had powered off.
• Spark 1 powered off alone in cycle 5, about 2 seconds into a 32,000-token request, with the rest of the rack idle. That was the night's fifth power-off, the limit we set in advance, so testing stopped there: Spark 2 alone and the MSI-alone control runs were not done.
• Under identical load Spark 1 was the hottest DGX Spark: it started each prefill block 6 to 8 C warmer than Spark 3, and its hottest temperature sensor peaked at 97.1 C against 93.8 C on Spark 3. It had also powered off twice in September at its old position.

Our reading: for Spark 3 and Spark 4 the new shelf environment with the whole pod loaded is the cause. Spark 1 points at the unit itself or its slot on the shelf, so on the next visit we will swap Spark 1 and Spark 3 to separate the two. The root cause of the trip itself is still open.

The 2,300 MHz lock after the move

We did not run the planned test of all eight units at the 2,300 MHz lock after the move, because the night reached its power-off limit first. The lock still got a long workout. The four DGX Spark units ran every later scaling and model run at the 2,300 MHz lock, about 8.5 hours in total, including 131,000-token prefills on every engine and 257,000-token prefills, with 0 power-offs. The four MSI units ran the same work at default clocks, also with 0 power-offs. That was real inference work, not the kill sequence, so it does not replace the skipped test; the September 30 result at 2,300 MHz (0 power-offs in 448 long-prefill starts) remains our reference.

Scaling after the move: two, four and eight units

We re-ran our scaling series with Qwen3.8-Flash-Next in NVFP4 after the move, with the DGX Spark units at the 2,300 MHz lock and the MSI units at default clocks. One unit alone is not possible with this model, because the 126 GB checkpoint does not fit a single unit, so the series is two, four and eight units. Against the pre-move series, the move changed nothing measurable in throughput.

• Single-user decode: about 35 tokens per second on two units, about 42 on four, and 45.8 on eight.
• One 131,000-token prompt: 2,750 to 2,858 tokens per second of prefill on two units, 3,137 to 3,193 on four, and 3,179 on eight. Eight units add nothing over four for one long prompt.
• 64 users: two four-unit groups running side by side delivered about 1,297 tokens per second combined (649.1 plus 647.7), against 778.6 for one eight-unit group and 949.8 for four two-unit replicas. A two-unit pair saturates at 64 users: its 95th-percentile time to first token reached 55 to 57 seconds.
• DGX Spark versus MSI: the same speed in the same layout, within 1 to 3 percent. The MSI units drew 13 to 17 percent more wall power, partly because they ran at default clocks while the DGX Spark units ran at the 2,300 MHz lock.

The eight-unit re-run with every unit at full speed

We re-ran MiMo-V2.6-Pro-RL, a 535 GB checkpoint split by tensor parallelism across all eight units, this time with no unit stuck at 611 MHz (the DGX Spark units at the 2,300 MHz lock, the MSI units at default clocks). On the published workload, four long CSV-to-JSON extraction tasks run one request at a time, which is the best case for speculative decoding, our pod now matches or comes close to the published figures:

• Aevonix vLLM with DFlash: 68.79 tokens per second, against 68.13 published.
• SGLang with DFlash-8: 98.4 tokens per second for a single user against 104.6 published, and 310.8 tokens per second in aggregate at 8 concurrent users against 306.7 published.
• 257,000-token prompts prefilled at 1,124 to 1,447 tokens per second, about 3 to 4 minutes per prompt.

Prose is much slower. On a mixed benchmark with SGLang and DFlash-8, a single user got 44.3 tokens per second on prose and 34.3 on narrative, against 80.3 to 100.7 on format, JSON, coding and math tasks. For prose at 4 to 8 users, SGLang without speculation beat SGLang with DFlash-8: 83.5 against 63.7 tokens per second at 4 users and 112.1 against 90.6 at 8. Our reading is that draft acceptance is low on prose and the extra verification work crowds the batch. The September 30 night, paced by the unit stuck at 611 MHz, had reached only 0.54 to 0.70 times the published speeds.

What this adds for GB10 owners and buyers

The strengths held: with the 2,300 MHz lock, our pod ran 448 long-prompt starts under full load without a power-off, the lock now costs less than the 2,200 MHz setting did, and the 200G fabric survived a physical rearrangement with no loss of bandwidth. With every unit at full speed, eight units matched the published MiMo-V2.6-Pro-RL speeds on extraction work, and the DGX Spark and MSI units ran at the same speed in every layout. The weaknesses are also clearer. Default clocks are still not safe on our pod even with the bricks spaced, the DGX Spark units proved sensitive to how they are placed, a unit can come back from a power event at about a quarter of its clock until AC power is fully removed, and we could not read fan speeds on either unit type. Eight units add little over four for one user or one long prompt, the MSI units drew 13 to 17 percent more wall power in our setup (partly a clock difference), and on the large model prose ran at about a third to a half of the single-user speed of structured output. Our draft support case for NVIDIA now includes the October data; it has not been sent yet.

Related reading

• DGX Spark Cluster Cable (QSFP112 400G DAC)
• DGX Spark Cluster: 4 to 8 Nodes, Switch vs Ring
• What Cable Do I Need to Connect Two DGX Sparks?
• DGX Spark Memory Bandwidth: 273 GB/s, and What 400G Adds
• From Two DGX Sparks to a Switched Fabric: Cluster Topology

FAQ

Why does my DGX Spark shut down under load?

In our tests the shutdowns happened within about 8 to 11 seconds of a very long prefill starting on a unit that was already hot, at default GPU clocks. Memory, OS thermal limits, the PDU and average power were ruled out by telemetry. Our best-supported explanation is a fast power or hotspot transient at the top boost clocks that the platform's own protection acts on, but that is not proven at the component level.

How do I stop a DGX Spark from shutting down under load?

The workaround that worked for us is locking the GPU clock at boot with nvidia-smi -lgc 300,2200, run from a systemd oneshot service after nvidia-persistenced. On our pod it took the failing sequence from 2 of 2 power-offs to 0 of 6, and seven units then ran 66 long-input starts under full load with no power-off. Since September 30, 2026 we use a 2,300 MHz ceiling (nvidia-smi -lgc 300,2300), which ran 448 long-prompt starts with no power-off and is slightly faster.

How do I check that the nvidia-smi clock lock is active?

On driver 580.178.04, nvidia-smi -q -d CLOCK does not show the lock. Check that the service is active, that the journal for the current boot shows the GPU clocks set line, and run a short GPU burn: a unit locked at 2,200 MHz peaks near 2,200 MHz, while default clocks reach about 2,400 MHz. With the 2,300 MHz lock we use since September 30, 2026, the burn peaks at about 2,288 MHz.

Does locking the GPU clock slow down a DGX Spark?

Slightly. On our diagnosis unit single-user decode dropped 1 to 3 percent, 32-user decode did not change, and long prefill was as fast or faster once the unit was warm. Only a cold first prefill was 3 to 7 percent faster at default clocks.

What is the real GB10 memory bandwidth?

NVIDIA lists 273 GB/s for the DGX Spark. Our GPU read probe measured 262 to 263 GB/s on freshly booted units and 238 to 242 GB/s on units that had already run an inference engine since boot.

Why is one of my identical GB10 units slower than the others?

In our pod the slower units were the ones that had run engines since their last boot. Each engine run leaves small free memory blocks that the next engine uses first, which lowers GPU read bandwidth and decode speed by about 3 to 5 percent. A reboot before launching restored full speed.

Does cma=128M or compact_memory fix GB10 memory bandwidth?

Not in our tests. Neither the cma=128M boot parameter nor drop_caches with compact_memory improved bandwidth or decode speed. Only a reboot did.

Could the power adapter cause a DGX Spark to power off?

The bricks are a contributor but not the whole story in our pod. Our bricks were taped together in pairs during the September tests. With them separated, power-offs at default clocks dropped to 1 in 233 long-prompt starts across the full pod, but did not stop. Some of our MSI EdgeXpert units had also been running on NVIDIA DGX Spark bricks; each unit is now on its own maker's adapter. One MSI EdgeXpert owner reports on the NVIDIA forum that a brick swap did not fix their unit.

Does separating the DGX Spark power bricks stop the shutdowns?

Not in our tests. Spacing the bricks at least an inch apart made the power-offs rare at default clocks, 1 in 233 long-prompt starts with all eight units loaded, but one unit still powered off with the usual signature. After a later rack move, all four DGX Spark units powered off at default clocks in the same test. We keep the GPU clock lock on every unit.

Can I lock the GB10 GPU clock higher than 2,200 MHz?

On our pod, yes. A 2,300 MHz ceiling ran 448 long-prompt starts with no power-off, including a 62-minute full-pod soak, and a 2,400 MHz ceiling ran 224 with none. Against 2,200 MHz, 2,300 MHz gave about 1.3 percent more single-user decode and 2.6 to 3.2 percent more prefill for 2 to 5 percent more wall power. We chose 2,300 MHz because it kept the most thermal margin of the faster settings.

Why is my GB10 GPU stuck at about 611 MHz under load?

We saw this on two MSI EdgeXpert units, one after a power loss and one after a routine reboot. nvidia-smi showed SW Power Cap: Active, and a GPU burn test peaked near 611 MHz instead of about 2,288 MHz under a 2,300 MHz lock. Soft reboots did not fix it. A clean shutdown, removing AC power for about 15 seconds, and powering back on did.

Get the Private AI Infrastructure Blueprint

Reference build sheets, a hardening checklist for open-weight model hosting and a data-sovereignty map for running AI on your own hardware.

Get the free guide