DGX Spark Shutdown Under Load: Our 8-Unit GB10 Diagnosis

A DGX Spark shutdown under load is one of the most confusing failures you can hit on NVIDIA GB10 hardware. The unit does not crash, reboot or log an error. It simply turns off, drops to a few watts of standby draw at the outlet, and stays off until someone removes and restores its power. On September 25 and 26 we saw this happen repeatedly on our own eight-unit GB10 pod, four NVIDIA DGX Spark systems and four MSI EdgeXpert MS-C931 systems, every time at the same moment: the start of a very long prompt. This post is our field report. It covers how we instrumented the problem, what we ruled out, the GPU clock lock that has kept every unit up since, what that lock costs in speed, and a second finding from the same week: a 238 versus 262 GB/s split in GB10 memory bandwidth between identical units, and the one-minute habit that fixes it.

We run this pod for model testing and for the self-hosted LLM benchmarks we publish, so we measure before we change anything. Every number below comes from telemetry we saved during the tests. Where a statement is our interpretation rather than a reading, we say so.

Key takeaways

• The symptom: a GB10 unit powers itself off within about 8 to 11 seconds of a 131,000-token prefill starting, most often when it is already hot from serving. The OS logs nothing, and the on-disk journal loses the last 6 to 14 seconds before the power-off.
• Not the usual suspects: it was not host memory exhaustion, not the OS thermal trip, not the power strip, and not average wall power. A unit survived three minutes at 204 to 212 W and then powered off at 145 W and 169 W in other runs.
• The workaround: locking the GPU clock with nvidia-smi -lgc 300,2200 at boot. On one unit, one day, the same test sequence powered off 2 of 2 times at default clocks and 0 of 6 times with the lock (Fisher exact test, one-sided, p = 0.036). With the lock on all seven running units, 66 long-input starts under full pod load produced 0 power-offs.
• The cost: about 1 to 3 percent of single-user decode speed on the diagnosis unit, nothing at 32 users, and long-prompt prefill that is as fast or faster once the unit is warm.
• The bandwidth split: identical units measured 262 GB/s or 238 to 242 GB/s of GPU read bandwidth. The cause was fragmented free memory left behind by earlier engine launches. Rebooting before launching restored 262 GB/s and raised decode by 4 to 5 percent.
• Still open: the root cause is best supported, not proven, and the lock is a workaround until NVIDIA ships a platform fix. The power bricks were taped together in pairs during these tests, which we now treat as an open factor.

What a DGX Spark shutdown under load looks like

If you have searched for a DGX Spark shutdown, you have probably read forum threads that describe the same pattern we saw. The unit is off, not rebooting. There is no kernel panic, no out-of-memory killer message, no thermal shutdown line and no power button event. The last lines in the journal are routine. At the outlet, the unit draws about 4 W, the standby level.

Our first four losses came between the afternoon of September 25 and the early hours of September 26, during a model comparison that kept all eight units busy at once: Spark 1 twice, Spark 2 once and MSI 3 once. Three of them came at or near the start of a 131,000-token prefill, and the MSI loss came inside a 120,000-token needle test. The eight GB10 units together were drawing 1,013 to 1,361 W at the wall at those moments. The first of them is the incident we describe in our companion post, DGX Spark Firmware Update: What We Learned on 8 GB10 Units, which records it without a cause. This post is the diagnosis.

Two details made the problem hard to read from the unit itself. First, the on-disk journal on a unit that loses power is missing its final seconds: in our two induced losses, the saved journal ended 6.8 and 13.5 seconds before the moment of death, and the saved container log ended 26 to 29 seconds early. Anyone reading those logs afterward would conclude the unit died earlier, and more quietly, than it did. Second, the GB10 platform gives the operating system very little to look at. There is no BMC with an event log, and the power delivery path is not visible to Linux in the way a server power supply is.

How we instrumented a unit that erases its own evidence

The fix for the evidence problem was to stop trusting the unit's own disk. For the diagnosis runs on one DGX Spark (we call it Spark 2 here), we streamed telemetry over SSH to a separate workstation, so the record survived the power-off:

• memory, memory pressure (PSI), all seven ACPI thermal zones and the process count every 0.5 seconds;
• nvidia-smi every 250 milliseconds: instantaneous and average GPU power, GPU temperature, the T.limit thermal margin, SM clock, and the clock-event (throttle) reasons;
• a live follow of the kernel and system journal, and of the inference engine's container log;
• a network ping every 0.5 seconds, so we knew the moment of death to within half a second;
• wall power per unit from a metered, switched PDU (the PDU controller refreshes about every 30 seconds, so this is an average, not a transient reading).

The workload was the one that had failed in the contest: NVIDIA's Nemotron-3.5-Lightning-30B-A3B in NVFP4 on vLLM, with 85 percent of memory given to the engine, a 262,144-token context window, and prefill tests at 32,000 and 131,000 tokens. We did not reduce the workload to make the problem go away. The goal was a fix that keeps the full configuration.

What we ruled out

Each of these was a reasonable first guess. The telemetry excluded all of them.

• Host memory exhaustion. At both induced losses the available memory was flat at 9.17 GB and 9.40 GB, memory pressure was 0.00, and swap was untouched, sampled until under 1.5 seconds before death. The September 25 losses showed the same headroom in our metrics history.
• A burst of process spawns. We had seen a steady stream of AppArmor denial lines in the logs. They run at 95 to 100 per minute on an idle unit with no model loaded, and the process count stayed flat through every run. They come from a resident service, not from the engine.
• The operating system thermal trip. All seven ACPI zones have a single critical trip point at 104.8 C. The last readings before the two losses were 93.3 C and 96.2 C, and the same unit survived a reading of 98.1 C in another run. A kernel critical trip also performs an orderly shutdown, which would leave log lines.
• The PDU, the circuit or the voltage. The outlet relay stayed on and the voltage was steady. After each loss the outlet read 4.06 to 4.39 W, the same signature as every earlier loss.
• Too much average power. Spark 2 held 204 to 212 W at the wall for three minutes while decoding for 64 users, running a 20-thread CPU burn and three 131,000-token prefills together, and it survived. The two losses happened at 169 W and 145 W.

That last point matters. If the trigger were sustained power or heat as the OS sees it, the hardest run would have been the one to fail. It was not.

Reproducing the DGX Spark shutdown on purpose

We ran nine scenarios on Spark 2 with the exact launch configuration from the contest. Cold engines, eight 131,000-token prefills back to back, a 12-minute heat soak at 64 users, and the maximum-power run above all survived at default clocks. What killed the unit was a short sequence we could repeat: one minute of single-user decode, one minute at 32 users, two 32,000-token prefills, then a 131,000-token prefill.

• At default clocks that sequence ended in a power-off 2 of 2 times, about 11 seconds and about 8 seconds into the first 131,000-token prefill.
• With the GPU clock locked at 300 to 2,200 MHz, the same sequence ran 6 times on a fresh engine (12 prefills at 131,000 tokens and 12 at 32,000) with no power-off and no clock-event reason in any of 5,458 telemetry samples.
• Across all default-clock runs that day, 2 of 22 starts of a 131,000-token prefill ended in a power-off. With the lock, 0 of 15.

The one-sided Fisher exact test for 2 of 2 against 0 of 6 gives p = 1/28, or 0.036. That is one engine on one day, so we treat it as strong evidence, not proof.

The telemetry from the last three seconds before each loss shows what the lock takes away. At default clocks, the GPU drew between 57 and 102 W and swung by up to 45 W from one 250 ms sample to the next. The SM clock hopped between 1,872 and 2,431 MHz. The T.limit thermal margin jumped between minus 7 and plus 13 C between samples. With the lock, the same workload peaked at 83.9 W, the clock stayed between 2,177 and 2,200 MHz, and the thermal margin never dropped below 2 C.

Our reading of the evidence

This part is inference. The trip appears to be a fast event: power steps and hotspot spikes on the order of 100 milliseconds at the uncapped boost clock, on a chassis that is already hot. None of the sensors the OS can read reaches a limit, and the averaged wall power was lower than in runs that survived. So the protection that turns the unit off is most likely acting on something the OS cannot see, such as instantaneous current in the USB-C power delivery path or an internal hotspot. The unit goes to a controlled soft-off rather than losing its supply: the outlet kept reading about 4 W, and a Spark that loses its DC input powers back on by itself when the input returns, which these units did not do. Removing the top boost clock bins removes the transient, which would explain why the lock works while memory use and average power are unchanged.

That matches reports from other owners. A thread on the NVIDIA developer forum, Spark abruptly shuts down, describes the same symptom. NVIDIA's forum moderator there says that shutdowns reported by some DGX Spark and GB10 users are being investigated, that they can be triggered by both thermal and current-related reasons, and that capping the GPU clock with nvidia-smi -lgc is a practical mitigation the community has found, because it reduces the power transients that appear to trigger them. We reached the same workaround independently, with the measurements below. An MSI EdgeXpert owner reports in another forum thread that swapping the power adapter with a working unit did not move the fault; that owner's unit was eventually replaced. For our diagnosis unit, fwupd reported no newer firmware.

The workaround: lock the GPU clock with nvidia-smi -lgc

The change is one command, run as root after the NVIDIA persistence daemon is up:

nvidia-smi -lgc 300,2200

The -lgc option locks the GPU clock to a minimum and maximum in MHz. The lock does not survive a reboot, so we install it as a systemd oneshot service ordered after nvidia-persistenced.service, with ExecStart=/usr/bin/nvidia-smi -lgc 300,2200, ExecStop=/usr/bin/nvidia-smi -rgc to restore default clocks, and RemainAfterExit=yes, wanted by multi-user.target. The lock does not restart any container or service, so it can be applied to a unit that is already serving.

Verifying the lock is harder than applying it. The obvious check is nvidia-smi -q -d CLOCK, but on driver 580.178.04 it shows nothing different when the lock is set, and NVML has no call that reads the locked range back. We verify three ways: the service is enabled and active; the journal for the current boot contains the line GPU clocks set to "(gpuClkMin 300, gpuClkMax 2200)"; and a 20-second matrix-multiply burn peaks at 2,177 to 2,210 MHz instead of the 2,398 to 2,405 MHz the same burn reaches at default clocks. A peak one clock step above 2,200 MHz is normal, so our pass line is 2,250 MHz. We rebooted one unit to confirm the service re-applies the lock at boot.

We are adding the service file, an installer with check, verify and uninstall modes, and the memory check described below to our open-source gb10-cluster-bench repository, alongside the method and raw data from our other GB10 tests.

Does the clock lock hold with the whole pod loaded?

A single-unit test cannot recreate the condition of the contest, where every unit was hot at once. So after rolling the lock out to all seven running units (one MSI unit was powered off and waiting for hands-on attention at the data center), we loaded all seven together for two phases of about 20 minutes each. In the first phase, five units each ran the single-unit Nemotron sequence while the other two ran GLM-5.3-Flash split across both units (tensor parallel size 2). In the second phase, three units ran Nemotron and two pairs ran GLM-5.3-Flash. The GLM runs included 120,000-token needle tests.

• 66 long-input starts: 63 prefills at 131,040 tokens and 3 needles at 120,000 tokens, plus 54 prefills at 32,000 tokens.
• 0 power-offs, 0 failed requests, 0 missed pings.
• The eight GB10 units together drew up to 1,111.5 W at the wall, inside the 1,013 to 1,361 W range recorded at the earlier losses.
• No GPU went above 87.5 W instantaneous and no SM clock sample went above 2,190 MHz. The hottest ACPI zone reached 96.1 C, against the 104.8 C trip point.

As a rough check on chance: at the default-clock loss rate we measured on the diagnosis unit (2 in 22 long prefills), the probability of 48 Nemotron long prefills passing with no loss is about (20/22) to the 48th power, roughly 1 percent. That assumes independent starts with equal risk, and the rack was hotter than in the diagnosis, so we read it as conservative.

What the clock lock costs

A clock cap sounds expensive. On this workload it mostly is not, because at default clocks a warm GB10 already spends time in thermal and power slowdown.

• Single-user decode: on Spark 2, 91.0 to 92.1 tokens per second at default clocks and 89.2 median with the lock, a 2 to 3 percent loss. Under full pod load, Spark 2 measured 89.4 to 90.2, a 1 to 3 percent loss.
• 32-user decode: 614.6 to 618.8 tokens per second at default clocks and 619.0 median with the lock, no loss.
• 131,000-token prefill: 4,040 to 4,414 tokens per second at default clocks after decode heat, and 4,488 median with the lock. Only a cold first prefill is faster at default clocks (4,612 to 4,795), about 3 to 7 percent.
• One exception: Spark 1 measured 7 to 9 percent lower at one user than its default-clock figure from the day before. We believe part of that gap is the unit and the measurement window, not the lock: Spark 1 ran with a thermal margin of 0 to 1 C, and the earlier figure used longer windows. The bandwidth finding below likely explains much of the unit-to-unit spread.

What we do not know yet

We want to be plain about the limits of this work.

• The root cause is best supported, not proven. We have a strong statistical difference on one unit, a clean loaded confirmation on seven, and a mechanism that fits the telemetry. We do not have component-level proof, and we cannot see inside the embedded controller or the power delivery firmware.
• The power bricks were taped together in pairs. During all of these tests the 240 W USB-C power bricks at the data center were bundled in pairs. Paired bricks likely run hotter, so a brick's own over-temperature or over-current protection is a candidate contributor that we have not excluded. We plan to separate them and re-test, first at default clocks and then at higher lock ceilings.
• The lock is a workaround. A platform fix has to come from NVIDIA and the system makers. We have drafted a support case for NVIDIA with this telemetry.
• Coverage gaps: clock ceilings above 2,200 MHz are untested, tensor parallel runs across four and eight units have not yet run under the lock, and one MSI unit was powered off during the rollout.

GB10 memory bandwidth: why eight identical units split into two tiers

During the loaded confirmation, three units decoded at about 89 to 90 tokens per second for one user while two others sat at 84 to 87, on the same model, launch settings and clock lock. The obvious suspects were a weak unit or a configuration drift. We checked both. All eight units matched on kernel, driver, VBIOS, firmware versions, clock lock, CPU governor, swap, container runtime, engine image, model files (identical SHA-256 hashes) and speculative decoding acceptance rate. Under load they ran the same SM clock, GPU power and temperature.

What differed was GPU memory bandwidth. A simple GPU read probe on a 4 GiB buffer split the pod into two tiers: 262 to 263 GB/s on three units and 238 to 242 GB/s on the other five. The split reproduced across three rounds, and it was the same whether the units ran one at a time or all at once, so it was not rack heat or a shared resource. For context, NVIDIA lists the DGX Spark at 273 GB/s of memory bandwidth, and our earlier post on DGX Spark memory bandwidth and what 400G really means covers how that number relates to cluster networking.

Decode speed followed the bandwidth closely. Across the eight unit averages, the correlation between read bandwidth and 32-user decode was 0.88. The fast tier averaged 610 tokens per second at 32 users against 591 for the rest, 3.2 percent faster on a 9.8 percent bandwidth gap. Prefill, which is compute bound, was flat on every unit within 1.5 percent.

The cause: engine launches fragment the free memory

The fast tier was always the most recently booted units, which first looked like an uptime effect. Our tests point elsewhere. On GB10 the GPU and CPU share one pool of Grace Blackwell unified memory, and the NVIDIA driver backs GPU allocations with ordinary kernel pages that cannot be moved, taking the smallest free blocks first. The read bandwidth of a buffer rises with the size of the physical blocks behind it. In our per-chunk probe, buffers built from 2 MiB blocks read at about 237 GB/s median, from 16 MiB blocks about 250 GB/s, and from 32 MiB blocks about 258 GB/s.

A freshly booted unit has almost no small free blocks in the part of memory that holds unmovable pages. One inference engine run leaves 12 to 15 GiB of them behind, and a second run about 28 GiB. The next engine to start takes those fragments first for its model weights, and because those pages cannot move, we expect that engine to keep its placement for as long as it runs. Two things did not cause fragmentation in our tests: reading 191 GB of model files through the page cache, and a unit left idle after a reboot for the 42 minutes we watched. So we believe the correlation with uptime was really a correlation with the number of engine launches since boot.

Why block size matters to the GPU is our inference: most likely the driver maps larger physical runs with larger GPU pages, which cuts address translation overhead, or larger runs spread better across memory channels. We could not separate the two from user space.

What fixed it and what did not

• Reboot before launching: works. One MSI unit went from 239.3 to 262.3 GB/s after a soft reboot, and Nemotron decode from 85.1 to 88.7 tokens per second at one user (4.2 percent) and from 592.8 to 622.5 at 32 users (5.0 percent), with the same speculative decoding acceptance. A soft reboot took 38 to 52 seconds on the Sparks and that unit, and about 1.5 to 2.2 minutes on two other MSI units.
• Relaunching the engine without a reboot: loses ground. A second launch on the same boot measured 1 to 2.5 percent lower at 32 users on three units.
• Dropping caches and compacting memory: no effect. drop_caches plus compact_memory left bandwidth unchanged on three units. Our explanation: kernel compaction moves pages into free space in movable regions, while these small holes sit inside unmovable regions.
• The cma=128M boot parameter: no effect. A DGX OS 7.6.0 slowdown thread on the NVIDIA forum suggests it. In an A/B test against a freshly rebooted control unit it gave no benefit to bandwidth or decode, so we removed it. The slowdown in that thread is a different and larger fault, and the kho=off kernel hotfix discussed there is already on all eight of our units.

We found no boot parameter or kernel setting that measured better, so we made no fleet-wide configuration change for bandwidth. The fix is a habit.

How to check a unit before a benchmark

You can see the fragmentation directly. As root, /proc/pagetypeinfo lists free blocks by size for each page type. Add up the free memory in the Unmovable rows of the Normal zone in every block size below the largest. If that total is under about 2 GiB, the unit is fresh and the next engine will get large blocks. If it is well above that, reboot before launching for full bandwidth. Our read-only check script does exactly this for a list of hosts, and after the final round of reboots all eight units read 261.8 to 263.0 GB/s.

A long-running inference server started right after a boot stays fast, because its memory placement is fixed at launch. What costs 3 to 5 percent is relaunching an engine on a unit that has already run one. For benchmarks that compare units or configurations, that difference is large enough to change a ranking, so we now reboot every participating unit first and record the check result next to every number we publish.

A GB10 stability and benchmark checklist

1. Stream telemetry off the unit for any long-context run. The unit's own journal and container logs lose their final seconds at a power-off.
2. Install the GPU clock lock at boot on every unit, and verify it with a burn test after every reboot, since nvidia-smi -q will not show it.
3. Do not bundle power bricks together; give each one its own airflow. We have not proven the bricks matter, but separating them costs nothing. Record wall power per unit if you can.
4. Reboot every unit before launching an engine that should run at full speed, and before any benchmark that compares units. Check fragmentation first if you are not sure.
5. For tensor parallel runs, reboot every participating unit. The slowest unit sets the pace of the group.
6. Keep swap and out-of-memory policy identical across units, and keep firmware in step. Our guide to growing a DGX Spark cluster from two units to a switched fabric covers the networking side.
7. Record the clock setting, uptime and bandwidth check with every result, so a later reader can tell a hardware difference from a state difference.

What this means if you run or are buying GB10 hardware

Craig Petronella's rule for our benchmark work is to publish strengths and weaknesses alike, so here are both.

The strengths held up. A single GB10 unit served a 30-billion-parameter mixture-of-experts model at about 580 to 620 tokens per second across 32 users and prefilled 131,000-token prompts at about 4,440 to 4,500 tokens per second with the lock on. The 128 GB of unified memory fits models and context windows that most single workstation GPUs cannot. All eight units matched on every setting we checked and measured the same bandwidth once their memory state was controlled, which is what you want from a cluster.

The weaknesses are real. At default clocks, a hot unit can switch itself off at the start of a long prompt, with no log to tell you why. The platform gives the operating system little visibility into power delivery. Performance varies by 3 to 5 percent depending on how many engines ran since the last boot. None of this is a reason to avoid the hardware, but it is a reason to run it the way you would run any production system: with a documented configuration, off-box monitoring and a tested recovery path. If you are comparing platforms, our RTX PRO 6000 Blackwell versus GB10 benchmark shows where each one fits.

Petronella Technology Group, Inc. designs, builds and supports private AI clusters on NVIDIA hardware for organizations that need their data to stay on their own systems. If you are planning a DGX Spark or MSI EdgeXpert deployment, or you have units that are shutting down under load, contact us and we will share what we have learned.

Related reading

• DGX Spark Cluster Cable (QSFP112 400G DAC)
• DGX Spark Cluster: 4 to 8 Nodes, Switch vs Ring
• What Cable Do I Need to Connect Two DGX Sparks?
• DGX Spark Memory Bandwidth: 273 GB/s, and What 400G Adds
• From Two DGX Sparks to a Switched Fabric: Cluster Topology

FAQ

Why does my DGX Spark shut down under load?

In our tests the shutdowns happened within about 8 to 11 seconds of a very long prefill starting on a unit that was already hot, at default GPU clocks. Memory, OS thermal limits, the PDU and average power were ruled out by telemetry. Our best-supported explanation is a fast power or hotspot transient at the top boost clocks that the platform's own protection acts on, but that is not proven at the component level.

How do I stop a DGX Spark from shutting down under load?

The workaround that worked for us is locking the GPU clock at boot with nvidia-smi -lgc 300,2200, run from a systemd oneshot service after nvidia-persistenced. On our pod it took the failing sequence from 2 of 2 power-offs to 0 of 6, and seven units then ran 66 long-input starts under full load with no power-off.

How do I check that the nvidia-smi clock lock is active?

On driver 580.178.04, nvidia-smi -q -d CLOCK does not show the lock. Check that the service is active, that the journal for the current boot shows the GPU clocks set line, and run a short GPU burn: a locked unit peaks near 2,200 MHz, while default clocks reach about 2,400 MHz.

Does locking the GPU clock slow down a DGX Spark?

Slightly. On our diagnosis unit single-user decode dropped 1 to 3 percent, 32-user decode did not change, and long prefill was as fast or faster once the unit was warm. Only a cold first prefill was 3 to 7 percent faster at default clocks.

What is the real GB10 memory bandwidth?

NVIDIA lists 273 GB/s for the DGX Spark. Our GPU read probe measured 262 to 263 GB/s on freshly booted units and 238 to 242 GB/s on units that had already run an inference engine since boot.

Why is one of my identical GB10 units slower than the others?

In our pod the slower units were the ones that had run engines since their last boot. Each engine run leaves small free memory blocks that the next engine uses first, which lowers GPU read bandwidth and decode speed by about 3 to 5 percent. A reboot before launching restored full speed.

Does cma=128M or compact_memory fix GB10 memory bandwidth?

Not in our tests. Neither the cma=128M boot parameter nor drop_caches with compact_memory improved bandwidth or decode speed. Only a reboot did.

Could the power adapter cause a DGX Spark to power off?

It is possible and we have not ruled it out. Our power bricks were taped together in pairs during these tests, which likely makes them run hotter, and a re-test with the bricks separated is planned. One MSI EdgeXpert owner reports on the NVIDIA forum that a brick swap did not fix their unit.

Get the AI Security Guide

Free, practical, and specific to regulated environments. We will email it to you.

Send it to me

No spam. Unsubscribe anytime.