DGX Spark Firmware Update: What We Learned on 8 GB10 Units

We ran a DGX Spark firmware update across all eight GB10 systems in our lab on September 25, 2026: four NVIDIA DGX Spark Founders Edition units and four MSI EdgeXpert units. We did not plan it for that day. We were in the middle of a long benchmark program when one of the machines switched itself off under load, and when we then read the firmware on every unit we found at least four different firmware combinations across the eight. So we stopped every test, updated the whole cluster as one set, verified the result with our own tools, and then measured what the update did to real performance. This post is the field report: what we found, the exact commands we used, what went wrong along the way, and a checklist you can run on your own GB10 hardware tonight.

The short version is good news with one surprise. The update itself went cleanly on all eight units. A synthetic GPU burn test then read about 11% lower on every unit, which looked alarming, but real LLM serving on the same unit, same container and same settings got 4% to 8% faster. We also checked our cluster against the community thread on the NVIDIA developer forum about slowdowns after the DGX OS 7.6.0 update, and found one unit that was missing the hotfix NVIDIA staff pointed to in that thread.

Key takeaways

• Check firmware on every unit before you trust a benchmark. Our eight GB10s had two embedded controller versions on the MSI side, three different readings on the Spark side, and one Spark reporting a placeholder value of 0x00000001 for its embedded controller, SoC firmware and USB-C power delivery controller.
• fwupd plus apt did the whole job. Every firmware update came from the Linux Vendor Firmware Service through fwupdmgr, every OS package through apt. No manual driver work was needed, and none of the eight needed a full power drain for the embedded controller update to take effect.
• A synthetic burn is not a serving benchmark. Our fp16 burn dropped from 88.6 to 91.4 TFLOPS before the update to 77.6 to 81.3 TFLOPS after. Qwen3.8-27B decode on the same unit went from 23.1 to 25.0 tokens per second for one user and from 303 to 315 tokens per second at 32 users.
• Check for the kho=off hotfix. Seven of our eight units had it. One MSI unit did not, and that unit had been the first node of every four-unit and eight-unit test we ran after the September 21 OS update.
• Update as a set, canary first, and verify independently. Updating half a cluster recreates exactly the mixed-firmware state that makes benchmark results hard to compare.

Why we stopped benchmarking to run a DGX Spark firmware update

Petronella Technology Group, Inc. runs a GB10 cluster in our own datacenter rack as part of our LLM benchmark program. The eight units share a 200 Gb/s fabric through a switch, and we use them to test which models run well on one, two, four and eight GB10 systems before we recommend hardware to clients. A benchmark is only as good as the uniformity of the machines under it. When you split a model across four units with tensor parallelism, the slowest unit sets the pace for all of them.

Part way through a model comparison on September 25, after one unit had shut itself off (the next section), we read the firmware versions of every unit with fwupdmgr get-devices. This is what we found, before any update. Spark 1 was down when we first read the others, so its values come from its update job's own pre-update readout:

GB10 cluster firmware before the update, September 25, 2026 (fwupd readings)

Unit	Embedded controller	SoC firmware	USB-C PD controller
Spark 1	0x03000302	0x0200980f	0x00000516
Spark 2	0x03000302	0x0200980f	0x00000516
Spark 3	0x03000508	0x02009b0b	0x00000001
Spark 4	0x00000001	0x00000001	0x00000001
MSI 1	10700	10800	522
MSI 2	10700	10800	522
MSI 3	10800	10900	522
MSI 4	10800	10900	522

Three things stand out. First, the two vendors use different version formats, so you compare Sparks with Sparks and MSI units with MSI units. Second, within each vendor the units were not on the same release: two MSI units were one embedded controller release and one SoC release behind the other two, and the Sparks were spread across three states. Third, Spark 4 reported 0x00000001 for all three components. That is a placeholder, not a version. It told us nothing about what the unit was actually running. The fwupd history on Spark 3 shows the same 0x00000001 as the "previous version" of components it updated on September 21, so a unit reading 0x00000001 has simply not had that component written through fwupd yet, or fwupd cannot read it. Either way, you cannot compare it with anything.

The GPU VBIOS strings differed too: 9A.0B.25.00.00 on four units, 9A.0B.2D.00.00 on three, and 9A.0B.0F.00.16 on Spark 4. The kernel (7.0.0-1019-nvidia) and the NVIDIA driver (580.178.04) were already identical on all eight, because every unit had taken the DGX OS 7.6.0 update on or before September 21. That is a useful lesson on its own. The OS update had been applied everywhere, but firmware had not kept pace, and nothing on the desktop or in nvidia-smi made that obvious.

Our clock checks had not shown a problem. At the start of every session we run a 15-second fp16 matrix burn on all eight units to catch a GPU clock that has latched low. That afternoon, at 13:02 Eastern, all eight ran between 88.6 and 91.4 TFLOPS at 2,164 to 2,268 MHz, with no locked clocks. So the firmware mix was not showing up as an obvious defect. But it was uncontrolled, and a benchmark that compares a four-unit result with a two-unit result has to assume the units are equivalent. We could no longer assume that.

The unit that switched itself off under load

The event that started all of this came at 17:18 that afternoon. Spark 1 was running a long-context prefill test, a prompt of about 131,000 tokens, when it went dark. We did not catch it from the benchmark logs. Our run monitor watched log lines and progress markers, and a dead machine whose client simply hangs produces neither. We caught it from the wall power trace on the rack's switched power distribution unit, which records watts per outlet every 30 seconds or so:

Spark 1 wall power at its PDU outlet, September 25, 2026 (Eastern time)

Time	Watts	What was running
17:05:23	155.5	Throughput and prefill tests
17:10:10	159.7	Same
17:15:01	158.3	Same
17:17:14	161.2	Same
17:17:45	179.6	First 131,000-token prefill had just started
17:18:15	4.36	Unit off
17:18:46 onward	4.2 to 4.5	Unit still off

The outlet was still live and the unit was drawing about 4.4 watts of standby power, so this was not a tripped breaker or a pulled cord. The machine had turned itself off. When we read its system journal after recovery, the previous boot simply stopped at 17:17:33 in the middle of routine log lines. There was no shutdown sequence, no kernel panic, no thermal message and no power button event in the final hour. From the operating system's point of view, the lights just went out.

We recovered it by power cycling its outlet on the PDU, with Craig Petronella's approval, and it booted normally at 18:22, about 65 minutes after it went down. We cannot tell you the root cause, and we are not going to guess. It was running the older embedded controller and SoC firmware at the time, and its draw had just jumped to 179.6 watts as a very long prefill began, but one event is not a pattern. What we can tell you is what to look for: a GB10 that has shut itself off shows standby-level draw at the outlet, and its journal ends without any of the messages a clean shutdown or a crash would leave. The commands to check that are in the checklist below.

It did not stay a single event. Later that evening and overnight, with the new firmware installed, Spark 1 powered off the same way a second time, and Spark 2 and MSI 3 did too, each at or near the start of a very long input: a 131,000-token prefill sweep or, on MSI 3, a 120,000-token retrieval test. So the firmware update did not prevent it. We instrumented the problem, ruled out the usual suspects and found a workaround, a GPU clock lock, and we cover that work in our companion post, DGX Spark Shutdown Under Load: Our 8-Unit GB10 Diagnosis.

The practical lesson for anyone running GB10 units unattended: watch wall power, not just logs. A per-outlet power reading is the one signal that does not depend on the machine being alive to report it.

How we ran the DGX Spark firmware update on eight units

At about 18:25 Eastern, Craig stopped all GB10 testing for a firmware and OS update window. Here is the procedure we followed. It is deliberately conservative, and it is the same procedure we would use on a client's cluster.

Step 1: Stop everything and save the evidence

Every benchmark and inference container on the cluster was stopped. Before removing any container we saved its logs with docker logs, because removing a container deletes the evidence of why it did what it did. Then every unit was checked again: the update job refused to touch a unit if any container other than a known infrastructure tunnel was still running. Firmware updates and a live model server do not mix. NVIDIA's own update guide says the same thing in plainer terms: close all running applications, have a recovery plan, and schedule updates in a maintenance window.

We made one mistake here worth sharing. The command that stopped and removed the benchmark containers used a name filter, and on one MSI unit that filter also matched a stopped service container that was not part of the benchmark. We had its full specification recorded, so we recreated it exactly, but the lesson is simple: remove containers by exact name, never by pattern.

Step 2: Update in groups, canary first

We split the eight units into three groups: the four MSI units, Spark 1 and 2, and Spark 3 and 4. In each group one unit went first as the canary. Only when the canary came back healthy did the rest of the group proceed, two at a time. On each unit the job did the same thing, in this order:

1. Record the starting state verbatim: kernel, driver and VBIOS, OS and DGX OTA version, every network address, every fwupd device version and the count of pending packages.
2. Refresh firmware metadata and list what was pending: sudo fwupdmgr refresh --force, then fwupdmgr get-updates.
3. Apply OS packages non-interactively while keeping existing configuration files: sudo apt-get update and sudo DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold full-upgrade. The job was forbidden from building, installing or removing an NVIDIA kernel module by hand, and would stop without rebooting if apt reported a DKMS build failure.
4. Apply firmware: sudo fwupdmgr update -y --no-reboot-check. On GB10 most of these are UEFI capsule updates staged to disk and written during the next boot.
5. Reboot, then poll every 30 seconds. A GB10 has no baseboard management controller, so if a unit had not returned within 20 minutes the job was to stop and ask for a person. It was never allowed to power cycle anything on its own.
6. Verify: every firmware device on its new version, nothing pending in fwupd or apt, every network address and link unchanged, fabric MTU still 9000, the GPU visible in nvidia-smi, and no failed systemd units.

The canary choice taught us something. Our MSI canary turned out to be already current: it had installed its embedded controller and SoC updates back in August, so it only exercised the OS package half of the job. It passed, which was fine, but it proved less than we intended. Pick a canary that needs every change the rest of its group needs.

What each unit actually needed

• Spark 1 and Spark 2: embedded controller 0x03000302 to 0x03000508 and SoC firmware 0x0200980f to 0x02009b0b. The SoC capsule also moved the GPU VBIOS from 9A.0B.25.00.00 to 9A.0B.2D.00.00. Reboots took about 10.5 minutes (Spark 1) and 9.5 minutes (Spark 2).
• Spark 3: only the USB-C power delivery controller, 0x00000001 to 0x00000516. Its reboot took 2 minutes 41 seconds.
• Spark 4: all three, from the 0x00000001 placeholder to 0x03000508, 0x02009b0b and 0x00000516, and its VBIOS moved from 9A.0B.0F.00.16 to 9A.0B.2D.00.00. Its reboot took about 11 minutes.
• MSI 1 and MSI 2: embedded controller 10700 to 10800 and SoC firmware 10800 to 10900, with the same VBIOS change. Reboots took about 10 minutes each.
• MSI 3 and MSI 4: no firmware; both had been current since August. OS packages only, with reboots of about one minute.

Every unit also had between 23 and 35 OS packages pending, including, on Spark 3 and Spark 4, DGX tooling such as dgx-dashboard and dgx-oobe (plus nvidia-dgx-telemetry on Spark 3). Ubuntu's phased updates held back seven to nine packages per unit where we counted (desktop, audit and apparmor packages, mostly), which is Ubuntu deliberately rolling a release out to a percentage of machines at a time. We installed those explicitly on most units so the cluster would be uniform. One unit still lists nine phased packages that Ubuntu has not released to it yet, which is harmless.

Two readings in the version files surprised us and are normal. Some units list several DGX_OTA_VERSION lines in /etc/dgx-release, one per OTA they have taken; a poster in the forum thread discussed below explains that the updater appends a line each time. And the ConnectX-7 network adapter firmware and the Samsung NVMe firmware had no releases offered on LVFS, so they did not change; the ConnectX-7 firmware read 28.45.4028 on all eight.

The flash itself is the scary part for most owners, so it is worth saying plainly: the capsule updates are slow on reboot, and on the units that took embedded controller and SoC updates the machine was unreachable for 9 to 11 minutes. Do not pull power during that time. If you have no remote console, that silence is normal.

Step 3: Verify independently, do not trust the updater

Each unit's job wrote its own before and after report, and every report ended in a match. We did not stop there. A separate read-only script, run by our coordinator rather than by the update jobs, then queried all eight units for the facts that matter. At 19:03 Eastern, the independent readback showed:

GB10 cluster after the update, independent verification, September 25, 2026

Item	DGX Spark units (4)	MSI EdgeXpert units (4)
Embedded controller	0x03000508 on all four	10800 on all four
SoC firmware	0x02009b0b on all four	10900 on all four
USB-C PD controller	0x00000516 on all four	522 on all four
GPU VBIOS	9A.0B.2D.00.00	9A.0B.2D.00.00
Kernel	7.0.0-1019-nvidia	7.0.0-1019-nvidia
NVIDIA driver	580.178.04	580.178.04
Firmware updates pending	0	0
Fabric links	addresses kept, MTU 9000	addresses kept, MTU 9000

We also re-ran our memory parity check, which confirms that every unit has the same swap size and the same out-of-memory handling. That check exists because of an earlier lesson on this same cluster: one unit alone ran an out-of-memory daemon and alone had no swap, and it kept dying first in large multi-unit runs in a way that looked like an ordinary memory problem. Parity checks catch that class of difference in seconds. After this update, all eight matched.

One more thing the verification caught: after its reboot, a container on Spark 2 with a restart policy of "unless stopped" came back up on its own. Our benchmark preflight refused to start while it was running, and we set its restart policy so it cannot return. If you run anything with an automatic restart policy on a GB10, expect it to come back after an update reboot.

What the update did to performance

With the cluster uniform again, we re-ran the same 15-second fp16 burn on all eight units. Every unit came back roughly 11% lower:

• Before: 88.6 to 91.4 TFLOPS at 2,164 to 2,268 MHz.
• After: 77.6 to 81.3 TFLOPS at 2,236 to 2,340 MHz, the same 91 to 94 watts, and no clock latch on any unit.

That is a strange result: equal or higher clocks, the same power, and less arithmetic done. The kernel, the driver and the burn container image did not change during the window, so the prime suspect is the new SoC and embedded controller firmware. On GB10, nvidia-smi -q reports no power limit and no memory clock (both read N/A on GB10, which uses unified LPDDR5x memory shared with the CPU), so neither can be inspected that way.

A burn test is not what anyone buys a GB10 for, though. So we ran the check that matters: the same model, on the same unit, with the same container image, the same serving flags and the same test harness as before the update. The model was Qwen3.8-27B in NVFP4 on a single DGX Spark, measured with prose prompts at 1 to 32 concurrent users, three repetitions each, with run-to-run variation of 2.2% or less.

Qwen3.8-27B NVFP4 decode throughput on one DGX Spark, before and after the firmware update (tokens per second, aggregate)

Concurrent users	Before	After	Change
1	23.1	25.0	+8.2%
4	82.8	86.3	+4.2%
8	142.8	149.0	+4.3%
16	222.0	232.0	+4.5%
32	303.2	315.0	+3.9%

Serving got faster at every load level, by 4% to 8%. The 11% burn drop did not carry into real work at all.

Why a slower burn and faster serving can both be true

The two tests stress different parts of the chip. A dense fp16 matrix multiply is compute bound: it keeps the GPU's math units busy and barely touches memory bandwidth. Generating tokens from a large language model is the opposite. For every token, the GPU has to read the model's active weights from memory, and on GB10 that memory is 128GB of unified LPDDR5x at 273 GB/s according to NVIDIA's DGX Spark specifications. That bandwidth, not raw compute, sets the ceiling on decode speed. We covered the arithmetic in our post on DGX Spark memory bandwidth and on our Grace Blackwell unified memory page.

So our working explanation is that the new firmware changed the balance of compute and boost behavior without hurting the memory path, or that the burn is sensitive to something serving does not exercise. It is a hypothesis, not a finding, and the limits of what we measured are spelled out below. What we can say with confidence is the practical point: if you judge a firmware update by a synthetic burn alone, you can talk yourself out of an update that makes your real workload faster. Measure the workload you actually run.

The code prompt set, run in the same session, moved the same way: decode rose 4% to 7% at every load level. Prefill, which is compute bound like the burn, did not share the gain. It was flat at 8,192 tokens of input and slightly slower at the two longest inputs, which fits the working explanation above but does not prove it.

Qwen3.8-27B NVFP4 on one DGX Spark, code prompts and prefill, before and after the firmware update (tokens per second)

Test	Before	After	Change
Code decode, 1 user	30.9	32.1	+3.9%
Code decode, 4 users (aggregate)	109.3	115.4	+5.6%
Code decode, 8 users (aggregate)	184.6	197.4	+6.9%
Code decode, 16 users (aggregate)	294.2	307.1	+4.4%
Code decode, 32 users (aggregate)	402.7	419.0	+4.0%
Prefill, 8,192-token input	2,400	2,409	+0.4%
Prefill, 32,768-token input	2,114	2,054	-2.8%
Prefill, 131,040-token input	1,302	1,217	-6.5%

Prefill here is one request at a time with the prefix cache defeated, median of three runs per input length.

Here is what we did not measure. We planned two follow-up checks on the burn result: the same burn in a second container image, and a GPU memory bandwidth probe. Neither ran before the cluster went back to benchmark work, and we have no bandwidth reading from before the update. So our own data cannot say whether the firmware left the memory path alone; the explanation above rests on the serving and prefill numbers only. The next day we did measure GPU read bandwidth on all eight updated units, and it showed something else entirely: units on identical firmware split into two tiers, 262 to 263 GB/s and 238 to 242 GB/s, and the split followed each unit's memory state since its last boot, not its firmware. That finding and the reboot habit that fixes it are in the companion post, DGX Spark Shutdown Under Load: Our 8-Unit GB10 Diagnosis.

The NVIDIA forum thread on DGX OS 7.6.0 slowdowns

If you own a DGX Spark and searched for "dgx spark slow" this month, you may have found a thread on the NVIDIA developer forum titled "Upgrade to 7.6.0 / kernel 7.0.0-1019-nvidia system wide slowdown 50/60%". It is worth reading in full. Here is an accurate summary as of September 25, 2026, with credit to the people who did the work.

The thread was started on September 21 by Digital_David, who reported a 50% to 60% slowdown in prefill and decode after updating a single Spark to DGX OS 7.6.0 with kernel 7.0.0-1019-nvidia. His smoke test on Qwen3.8-Flash-Next showed prefill falling from 1,022 to 631 tokens per second and decode from 37.1 to 15.7 tokens per second, with Dynamo bytecode transform times roughly doubling and torch.compile time rising from 29 to 46 seconds. His first post also listed kho=off among his settings. jahnclawdmonet worked through the evidence methodically, ruling out CPU frequency scaling and a change in reported device memory. mashie reported model loads taking 50% longer on an upgraded cluster than on one that had not been upgraded.

Digital_David then posted a resolution he described as three compounding issues:

1. Memory allocation (CMA). The 7.0.0-1019-nvidia kernel config leaves CmaTotal at 0 kB. His fix added cma=128M kho=off to the kernel command line in GRUB.
2. GPU clock throttling. He saw the SM clock drop to about 676 MHz under load with software power capping active, and overrode it with sudo nvidia-smi -lgc 2418,2418, later resetting to the default with sudo nvidia-smi -rgc once the third issue was fixed.
3. A 30 W power fallback. He saw the ConnectX-7 driver log "insufficient power on the PCIe slot (27W)" and concluded that the USB-C power delivery and embedded controller had latched into a 30 W fallback that a normal reboot does not clear. His fix was a cold drain: shut down, unplug the power brick, hold the power button for 15 to 20 seconds, wait a full minute, then reconnect and boot.

After all three he reported prefill of 10,402 tokens per second, against what he gave as an original baseline of 9,420 in a different test from his first smoke test, and decode of 41.3 tokens per second. Other posters added important context. hoesing said he thought the power delivery issue had long since been fixed by PD firmware updates and had not seen it in months on MSI and Founders Edition units. vedcsolution reported that the CMA fix alone did not solve their problem, then posted a correction that their remaining failure reproduced on the older kernel and was a CUDA graph and vLLM issue on their side, while noting that the OTA removed the older kernel's driver modules, so booting the old kernel needs an extra package. azampatti reported running the latest release with no issues.

The one NVIDIA staff reply in the thread, from moderator aniculescu, reads in full: "It sounds like you did not update your system to include the latest hotfix which adds the kho=off parameter. Please make sure you are updated to the latest version." Digital_David replied that his system had been current as of the day before and that his workaround most likely did the same thing.

Keep the distinction clear when you act on the thread. The three-part diagnosis is one owner's careful field work on one system. NVIDIA's reply in the thread points to a hotfix that adds kho=off. In the Linux kernel documentation, KHO is Kexec HandOver, a mechanism for preserving memory regions across a kexec that is enabled with a kho=on boot parameter. On our units the hotfix arrives as a package named nvidia-spark-grub-kho, described as "NVIDIA Spark GRUB KHO configuration", which adds the parameter to the GRUB configuration.

What we found when we checked our cluster against the thread

The kho=off hotfix: one unit was missing it

We checked every unit read-only with grep -o kho=off /proc/cmdline and dpkg -l nvidia-spark-grub-kho. Seven of eight had the package and the parameter; on those seven it had been installed on September 21 alongside the 7.6.0 update. MSI 1 had neither. It took the same OS update and simply did not get the hotfix package, and we do not yet know why.

That matters more than it sounds. MSI 1 had been the first node, rank 0, of every four-unit and eight-unit run we did after September 21. In a tensor-parallel run, every unit waits for the slowest. So every multi-unit number we measured in that period now carries a label, "measured before the kho hotfix on one node", until we re-measure it. Craig approved installing the hotfix, and we measured the same GLM-5.3-Flash four-unit configuration before and after on that node so both results exist.

We installed the package on MSI 1 that evening. After a 60-second reboot, /proc/cmdline showed kho=off. We then re-ran GLM-5.3-Flash across the four MSI units with the same container image, launch command and environment as the run before the fix. We saw no consistent effect from the hotfix on our workload. Prose decode went from 27.0 to 27.7 tokens per second for one user and from 159.2 to 164.6 at 32 users. Code decode went from 39.1 to 38.4 for one user and from 228.2 to 216.4 at 32 users. Across all load levels, decode moved between about -7% and +9%, in both directions, so we do not read it as an effect of the hotfix. Prefill was no clearer: 812 and 823 tokens per second at 8,192 tokens of input, 1,350 and 1,659 at 32,768, and 1,139 and 820 at 131,040, a pattern too inconsistent to credit to the hotfix either way. The two runs were about five hours apart, with a reboot of MSI 1 and other engines on the same units in between, so this is a practical before and after, not a controlled isolation. For our multi-unit numbers, the "before the kho hotfix" label is a footnote, not a correction. We did not measure torch.compile times or model load times, where the thread also reports slowdowns, so this does not rule out an effect there.

If you own a GB10, run the two commands above now. It takes five seconds and it checks for the hotfix NVIDIA staff pointed to.

CmaTotal is 0 kB on all eight units

Every one of our eight units reports CmaTotal: 0 kB in /proc/meminfo, including the seven with the hotfix. Those seven serve models normally, and the Qwen3.8-27B result above was measured on one of them. We have not added cma=128M and have not needed it for our workloads so far. So on our cluster, a zero CMA reading by itself is not a sign of trouble. If you are seeing the memory ballooning described in the thread, that is a different situation, and the thread is the place to start.

GPU clocks: no latch before or after

Our 15-second burn is designed to catch exactly the clock-latch symptom in the thread, and it found no latched unit either before or after the update, with SM clocks at 2.2 to 2.3 GHz under load. Two units showed "SW Power Cap: Active" in nvidia-smi -q while idle, which on its own is not diagnostic. We did not lock clocks with nvidia-smi -lgc. If you experiment with it, note that it overrides the default boost behavior, and reset it afterwards with sudo nvidia-smi -rgc as the thread describes.

The "insufficient power on the PCIe slot (27W)" message is routine on our units

This is the one we most want other owners to know. The ConnectX-7 message insufficient power on the PCIe slot (27W) appears in the kernel log of every boot on all eight of our units (four lines per boot on the units we sampled, and in their boot logs back to 2025), through months of normal performance. So on our hardware it is not by itself evidence of the 30 W fallback described in the thread. If you see it, do not panic and do not start pulling power bricks on that basis alone. Check performance first. If a unit is genuinely slow, with low clocks under load and poor decode speed, then the cold drain procedure in the thread is a reasonable next step. None of our eight needed it.

A DGX Spark firmware update checklist you can run

These are the commands we use. The first group is read-only and safe to run on a busy machine. The second group changes the system.

Read-only checks (safe anytime)

• Firmware versions: fwupdmgr get-devices. Compare the embedded controller, the UEFI device firmware entries (SoC and USB-C PD) and the platform key across every unit of the same brand.
• Pending firmware: fwupdmgr get-updates (run sudo fwupdmgr refresh first if the metadata is old).
• kho hotfix: grep -o kho=off /proc/cmdline and dpkg -l nvidia-spark-grub-kho. No output from the first, or no ii line from the second, means the hotfix is not active.
• CMA reading: grep CmaTotal /proc/meminfo.
• Kernel, driver, VBIOS and clocks: uname -r and nvidia-smi --query-gpu=driver_version,vbios_version,clocks.sm,power.draw --format=csv. Run the second one again under load.
• DGX OS version: grep OTA_VERSION /etc/dgx-release. Several lines is normal; the last one is the current OTA.
• Power messages: sudo dmesg | grep -i "insufficient power", then check older boots with journalctl -k -b -1 | grep -i "insufficient power" to see whether it is new.
• Unexpected power-off: journalctl --list-boots, then journalctl -b -1 -n 50 to read the end of the previous boot. A clean shutdown leaves shutdown messages; a crash usually leaves a panic or error. Neither is what we saw.
• Pending OS packages: apt list --upgradable.
• Anything still running: docker ps, plus any Podman, Ollama or other service you run.

The update itself

Stop your inference containers and services first. Save their logs if you care why they behaved as they did. Then, on one unit:

1. sudo apt update
2. sudo apt full-upgrade (NVIDIA's guide uses dist-upgrade, which is the same operation)
3. sudo fwupdmgr refresh --force
4. sudo fwupdmgr update
5. sudo reboot, and wait. Units taking embedded controller and SoC capsules were unreachable for 9 to 11 minutes on our cluster. Do not interrupt power.
6. Re-run the read-only checks. If an embedded controller still shows its old version after the reboot, it may need a full power cycle to take effect: shut down, unplug the power adapter, wait, reconnect and boot.

Then do the next unit, and do not start benchmarking or serving across the cluster until every unit reads the same. For DGX Spark Founders Edition units, NVIDIA's OS and component update guide strongly recommends the DGX Dashboard for updates and documents the same manual apt and fwupdmgr commands for advanced users. It also notes that devices from other manufacturers might have different update procedures. On our MSI EdgeXpert units, fwupd offered MSI's own LVFS releases for the MS-C931 platform, and the same commands worked. The Linux Vendor Firmware Service documentation and the fwupd project explain how those releases reach your machine.

GB10 cluster maintenance: the lessons we are keeping

1. Check versions across the whole cluster before trusting any benchmark. We now record the firmware table and the burn result at the start of every benchmark session, and every result file carries them.
2. Update as a set, with a canary. A partial update is worse than none, because it recreates the mixed state you were trying to leave. And pick a canary that needs every change.
3. Verify independently. The updater's own report said everything matched, and it was right. We still checked with a separate read-only script, because the one time it is wrong is the time it matters.
4. Measure real serving before and after. Our burn said 11% slower; our serving benchmark said 4% to 8% faster. Only one of those is what users feel.
5. Keep a human approval gate on privileged batch changes. Every privileged step in this update ran only under an explicit, time-limited approval that Craig recorded for that specific batch. A request in a chat window does not unlock it, and when the approval expires the batch cannot run.
6. Watch wall power per outlet. It caught the one failure our log-based monitor missed.
7. Make routine maintenance code, not improvisation. For this one-off window, each unit was updated by its own AI agent worker, running on a model we host on our own hardware, following the fixed written procedure above, one machine per worker, with the canary and parallel limits enforced by the harness rather than by the agent. For the routine version, we built deterministic, scheduled maintenance code with no AI model making decisions. A read-only collector checks every unit daily. The applier runs only inside a weekly window, only with a live approval, and only when every unit in the cluster is idle, meaning no containers, no GPU processes and no model loaded. If any one unit is busy or unreachable, the whole cluster waits for the next window. It never power cycles anything; if a unit does not come back, it stops and asks for a person. It is running in report-only mode today.

None of this is exotic. It is the same discipline that good patch management brings to servers and workstations, applied to AI hardware that most organizations still treat like a desktop appliance. If your GB10 systems handle regulated data, maintenance records matter for compliance too: NIST SP 800-171 Revision 2 has a whole maintenance family, starting with requirement 3.7.1, perform maintenance on organizational systems.

Running GB10 hardware? Where to go next

If you are building or growing a GB10 cluster, the same hardware lessons apply whichever brand you run. Two units connect directly with one cable, three can form a switchless ring, and four or more usually go through a switch, which is how our eight are wired. The details are in our guides to going from two Sparks to a switched fabric, switch versus ring for four or more nodes and the three-Spark ring wiring guide.

Pairing two GB10 units or building a three-node ring? We stock the 0.5m QSFP112 400G direct-attach cable at $159 with free US shipping, in stock and shipping in 1 to 3 business days. It fits the DGX Spark Founders Edition and every OEM GB10 build, including the MSI EdgeXpert cluster cable page for MSI owners. Order the DGX Spark cluster cable. A 2-pack is $299 and a 3-pack ring kit is $435.

For context on price: NVIDIA lists the DGX Spark Founders Edition at an MSRP of $4,699, following a February 2026 price change. Actual prices from resellers and OEMs vary with memory supply and market conditions, so get a current quote. An eight-unit cluster is a real investment, and an hour of maintenance discipline protects it.

Our GB10 benchmark results, including the post-update numbers as they land, are published in our LLM benchmark program and the benchmark leaderboards. If you are weighing a GB10 against Apple silicon, read our comparisons of the Mac Studio M5 Ultra vs NVIDIA DGX Spark and the DGX Spark vs Mac mini for local AI. If you have outgrown a desk-side cluster, our DGX Station GB300 page and the DGX Station GB300 power efficiency results cover the next step up.

Want this kind of rigor on your own AI hardware? Petronella Technology Group, Inc. designs, deploys and maintains private AI systems, from a single Spark to a private AI cluster, with the same secure AI infrastructure practices we use in our own rack. Book a consultation with Craig Petronella or contact our team.

Related reading

• DGX Spark Cluster Cable (QSFP112 400G DAC)
• DGX Spark Cluster: 4 to 8 Nodes, Switch vs Ring
• From Two DGX Sparks to a Switched Fabric: Cluster Topology
• What Cable Do I Need to Connect Two DGX Sparks?
• 3 DGX Spark Ring: Port-by-Port Wiring Guide

FAQ

How do I update the firmware on a DGX Spark?

NVIDIA recommends the DGX Dashboard for Founders Edition units. From a terminal, the equivalent is sudo apt update, sudo apt full-upgrade, sudo fwupdmgr refresh, sudo fwupdmgr update, then a reboot. Stop inference containers first. On our units the reboot took up to about 11 minutes while firmware capsules were written.

How do I check whether my DGX Spark has the kho=off hotfix?

Run grep -o kho=off /proc/cmdline and dpkg -l nvidia-spark-grub-kho. If the first prints nothing or the second shows no installed package, the hotfix is not active. On our cluster, seven of eight units had it after the DGX OS 7.6.0 update and one MSI EdgeXpert unit did not.

Did the DGX Spark firmware update make our GB10 units slower?

A synthetic fp16 burn test read about 11% lower on all eight units after the update. Real LLM serving got faster: Qwen3.8-27B on the same unit went from 23.1 to 25.0 tokens per second for one user and from 303 to 315 tokens per second at 32 users.

Why would a burn test drop while LLM serving gets faster?

A dense fp16 burn is limited by compute. LLM token generation is limited by memory bandwidth, because every token reads the model weights from the GB10's 273 GB/s unified memory. That is our working explanation, not a proven one: we did not measure memory bandwidth before the update.

Is the insufficient power on the PCIe slot (27W) message a problem?

Not by itself on our hardware. It appears in the kernel log of all eight of our GB10 units, and on the units we sampled it also appears in boot logs from 2025 with normal performance. Check real performance before acting on it.

Do MSI EdgeXpert units update the same way as a DGX Spark?

On our units, yes. fwupd offered MSI's own releases for the MS-C931 platform through LVFS, and the same apt and fwupdmgr commands worked. The version numbers use a different format, so compare MSI units with MSI units.

What does fwupdmgr version 0x00000001 mean on a DGX Spark?

It is a placeholder, not a real version. One of our Sparks reported 0x00000001 for its embedded controller, SoC and USB-C PD firmware, and fwupd history on another unit shows the same value as the previous version of components it later updated. After the update, the unit reported normal versions.

Why did a DGX Spark turn itself off under load?

The first unit dropped from about 180 watts to 4.4 watts of standby draw during a long prefill test, and its journal ended with no shutdown, panic or thermal message. A power cycle of its outlet brought it back. It happened again after the firmware update, on that unit and two others, each time at or near the start of a very long input. Our diagnosis and the GPU clock lock that stopped it on our units are in our companion post, DGX Spark Shutdown Under Load: Our 8-Unit GB10 Diagnosis.

Get the AI Security Guide

Free, practical, and specific to regulated environments. We will email it to you.

Send it to me

No spam. Unsubscribe anytime.