Four DGX Spark units need a switch. Each GB10 workstation has exactly two QSFP ports, so a fourth unit cannot have a direct cable to every other unit, and NVIDIA supports up to three units on direct cables and up to four through a switch. People still ask whether a 4 DGX Spark cluster can skip the switch and run as a ring, usually because a switch they wanted is out of stock. This post answers that question with what NVIDIA documents, what owners have tried on the NVIDIA developer forum, and what we measured on our own switched GB10 fabric: what a ring of four actually does, what a switch costs you in bandwidth, and the exact parts list for each build.
We are Petronella Technology Group, Inc. We sell the 0.5 m QSFP112 cable that every GB10 cluster uses, one per node whether you build a ring or a switched fabric, and we run six GB10 workstations on two MikroTik switches in our lab. The measurements below are ours and are published with commands in our free GB10 cluster guide.
Four node cables: two 2-packs, $598 shipped
The short answer: switch for four, ring for three
- Two units: one cable, Port to Port. NVIDIA's "Connect Two Sparks" playbook.
- Three units: three cables in a ring, each unit using both ports. Three nodes with three cables is a full mesh, so every pair has a direct link. NVIDIA's "Connect Three DGX Spark in a Ring Topology" playbook. Our 3-pack ring kit is built for it. The port-by-port ring wiring guide shows which cage each cable goes in.
- Four units: a 200G-capable switch and one cable per unit. NVIDIA's "Connect Multiple DGX Spark through a Switch" playbook, written for four systems.
NVIDIA's clustering documentation puts the limits in one sentence: the Sync Cluster Assistant "supports up to three DGX Spark systems connected directly through cables, and up to four systems when using a switch." Everything after this section is about why the line sits at three, and what happens when people cross it anyway.
Why four GB10 units cannot form a direct mesh
For every pair of four nodes to have its own cable, each node needs three links. A GB10 has two QSFP ports. Wire four units in a loop and each unit reaches its two neighbors directly, but the two units across the loop from each other share no cable. We tried exactly this before our switch arrived: four units wired as four point-to-point links, and the two non-adjacent units had no route to each other.
That missing link matters more on a GB10 than it would on an ordinary server, because cluster traffic is RDMA over Converged Ethernet (RoCE), not plain IP. As a forum member explained on a six-node ring thread, RoCE needs directly connected interfaces from a layer 2 point of view. The owner on that thread confirmed it with raw ib_write_bw tests that bypassed NCCL and vLLM entirely: between any two nodes that were not physical neighbors, the RDMA connection failed at the verbs layer with "Failed to modify QP to RTR," even though IP routing between them worked.
There is a second, quieter cost. The ConnectX-7 connects to the GB10 chip over two PCIe Gen 5 x4 links, per NVIDIA's documentation, and both QSFP ports share them. One cable already carries the full 200 Gb/s across its two PCIe halves (we measured 98.04 + 98.04 = 196.08 Gb/s on a single direct cable). In a ring, a node's two ports face two different neighbors, and anything relayed for a non-adjacent pair has to share that same budget with the node's own traffic.
What owners have actually tried with switchless four-node and larger rings
The NVIDIA developer forum has three threads worth reading before you decide, and we summarize them here without adding claims of our own.
- "4-node DGX Spark cluster without a switch" (thread 368726, about 5,800 views). The proposal was to avoid switch latency by building a true four-node full mesh out of optical splitting, or out of NVIDIA's H-shaped 2 x 200G QSFP56 cross-connect optical cables. The thread did not produce a tested build, and one participant noted the cross-connect cables cost more than a switch. The original poster said the goal was lower latency and a quieter office, not cost savings.
- "6-Node DGX Spark Ring Topology: NCCL Fails on Non-Adjacent Node Pairs" (thread 377435). Six units in a switchless ring, direct neighbors working, non-adjacent pairs failing at the RDMA layer. The owner eventually got all six connected by combining
NCCL_IB_MERGE_NICS=0with a subnet-aware routing option from a patched NCCL build, and reported a throughput gain of only about 7 percent. Another member's reply was blunt: if you want more than three Sparks, get a switch. - "4 Sparks with switch vs ring?" (thread 383218, September 2026). The asker could not find a MikroTik CRS804 in stock. The reply: a switch gives every Spark a direct 200 Gb/s path, a ring adds hops for non-adjacent traffic that compete with forwarded traffic, several users have reported four-node rings working, and a switch is still the recommendation for more than two Sparks.
So a four-node ring is possible, but it takes a patched NCCL, careful per-peer routing, and rank placement you control, and you still lose direct paths for two of the six node pairs. That is a research project, not a build you hand to a team that needs the cluster to work on Monday.
What a switch actually costs in bandwidth
The fear behind most ring questions is that a switch will slow the fabric down. On our hardware it barely does. These numbers come from ib_write_bw runs documented in our guide's validation matrix:
- Direct cable, one PCIe half: 111.86 Gb/s.
- Through a MikroTik CRS812, one PCIe half, DGX Spark pair: 109.11 Gb/s. The switch costs about 0.75 Gb/s per half, 0.7 percent.
- Through the switch, both halves, MSI EdgeXpert pair: 98.0 + 98.0 = 196 Gb/s per node, with no PFC and no ECN configured.
- Three pairs at once on one switch: 109.31 + 109.22 + 109.27 = 327.8 Gb/s aggregate, and a DGX Spark pair landed within 0.1 Gb/s of an MSI pair.
The other half of the comparison is what four nodes buy you in software. Tensor parallel needs the model's attention head count to divide by the number of ranks, and a maintainer of the spark-vllm-docker project notes on the three-node thread that powers of two (2, 4, 8) are the practical rule. Four is a power of two; three is not. On our four-node switched fabric, Qwen3.8-Flash-Next with tensor parallel 4 plus expert parallel ran 39.6 / 57.0 / 76.9 tok/s single stream on prose, code and math, against 35.3 / 36.0 / 53.2 tok/s on two nodes: 21 to 54 percent faster per stream. GLM-5.3-Flash with tensor parallel 4 came up with a 2,246,948-token KV cache. Both results, with dates and ranges, are on our LLM benchmark page.
What NVIDIA's switch playbook requires
NVIDIA's "Connect Multiple DGX Spark through a Switch" playbook is short, and it answers most of the questions people post after the fact:
- Switch: "QSFP switch with at least 4 QSFP56-DD ports (at least 200Gbps each)." If the switch has 400G ports, breakout cables can split each one into two 200G ports.
- Cables: one cable per Spark. "Full bandwidth can be achieved with just one QSFP cable."
- Same port everywhere: use the same ConnectX-7 port on every Spark "for easier network configuration and avoiding NCCL test failures." The playbook uses the second port, the one further from the Ethernet jack, whose interfaces are
enp1s0f1np1andenP2p1s0f1np1. - Check the speed: "The link speed might not default to 200Gbps with auto-negotiation." Run
sudo ethtool enp1s0f1np1 | grep Speedon every node; if it reads 100000Mb/s, set 200G manually on the switch port and turn auto-negotiation off. - One bridge: put every Spark-facing port in a single bridge on the switch, because "some switches have restriction that Hardware offloading can only be enabled on one bridge."
- Keep management separate: use the RJ45 Ethernet or Wi-Fi for management and internet, and keep the ConnectX-7 network for workload traffic.
Order node cables for your switch build
Which switch, if the CRS804 is out of stock
Forum owners often point four-Spark builders at the MikroTik CRS804 DDQ, and the September thread above exists because it was hard to find. We built on the MikroTik CRS812-8DS-2DQ-2DDQ-RM. It has 2 x 400G QSFP56-DD, 2 x 200G QSFP56, 8 x 50G SFP56 and 2 x 10G RJ45 ports. Each QSFP56-DD port breaks out to two 200G legs, so one switch serves four nodes on breakouts, or six if you also use the two native 200G ports. We ran it on RouterOS 7.24.1. MikroTik pricing changes, so check it directly; we do not quote it.
Three settings decide whether it works, and each one cost us a session. Our switched-fabric write-up and the guide's switch document have the full configuration:
- Force the breakout speed. The QSFP56-DD ports do not auto-negotiate 2 x 200G. Set
auto-negotiation=no speed=200G-baseCR4on the lane-master interfaces (lanes 1 and 5 of each DD port). The legs came up at 200 Gbps with FEC91 within about ten seconds. - Jumbo frames on the switch ports. Set
l2mtu=9216 mtu=9000on every fabric port, not only on the hosts, and prove it end to end withping -M do -s 8972. - One hardware-offloaded bridge. This is the same point NVIDIA's playbook makes. When we put the fabric in a second bridge, the switch CPU forwarded it and throughput fell from 196 Gb/s to 4.85 Gb/s. Back in one bridge with
hw-offload=yes, it returned to line rate.
Parts list: four nodes on a switch versus a switchless ring
Both builds use the same node cable: the 0.5 m QSFP112 passive DAC built to the NVIDIA-approved Amphenol NJAAKK0006 / Luxshare LMTQF022-SD-R spec. Prices below are ours for the cable and what we paid for breakouts; the switch is priced separately.
Four nodes, one switch (the supported build)
- Four node cables: two 2-packs at $299 each, $598.
- Two QSFP56-DD to 2 x QSFP56 breakout cables, one per DD port. We used NADDOD Q2Q56-400G-CU1, MikroTik-coded, at $138 each.
- One 200G-capable switch, such as the MikroTik CRS812-8DS-2DQ-2DDQ-RM or CRS804 DDQ.
- Fabric total in cables and breakouts: $874, plus the switch.
Four nodes, switchless ring (the experiment)
- Four node cables: two 2-packs, $598.
- A patched NCCL build with per-peer subnet-aware routing,
NCCL_IB_MERGE_NICS=0, and rank placement you control, per the six-node forum thread. - No direct path for two of the six node pairs, and time to maintain the patches.
The cables are the same in both builds, so starting with a ring and moving to a switch later wastes nothing: the four cables move straight onto the switch. The same holds if you start with a three-node ring and add a fourth unit. For five to eight nodes, our guide's bill of materials covers two-switch layouts with a 200G inter-switch link.
Our recommendation
If you have three units, build the ring. It is fully supported, it needs three cables and no switch, and every pair has a direct link. If you have four units, or know you will, buy a switch. Four is where tensor parallel works cleanly, the switch penalty we measured is under one percent, and NVIDIA's playbook covers the configuration. Build a four-node ring only if the ring itself is what you want to study, and plan on patched NCCL.
Whichever you choose, prove the fabric before you trust it. Run ib_write_bw one PCIe half at a time and expect about 109 to 111 Gb/s per half. If a brand-new link is far below that, power the units fully off and unplug them for a minute, which forum owners report clears a first-plug slowdown, then update the OS and firmware on every node.
Quick answers
Can four DGX Sparks be clustered without a switch? Not in a supported way. A four-node ring leaves two pairs without a direct link, and RoCE between them needs a patched NCCL and extra routing.
What is the maximum number of DGX Sparks without a switch? Three, wired as a ring with three cables. NVIDIA supports up to four through a switch.
How much speed does the switch cost? About 0.7 percent per PCIe half in our tests: 109.11 Gb/s through a MikroTik CRS812 against 111.86 Gb/s direct.
What do four nodes cost in cables? Four node cables ($598 as two 2-packs) plus two breakout cables on a QSFP56-DD switch.
Frequently asked questions
Can you cluster four DGX Sparks without a switch?
Not as a supported configuration. Each GB10 unit has two QSFP ports, so four units cabled in a ring leave two pairs with no direct link. RoCE needs a direct layer 2 path, and routed RDMA between non-adjacent nodes fails unless you run patched NCCL and extra configuration. NVIDIA supports up to three units on direct cables and up to four through a switch.
What switch do I need for four DGX Sparks?
NVIDIA's switch playbook calls for a QSFP switch with at least four QSFP56-DD ports of at least 200 Gb/s each, one cable per Spark, and optional breakout cables that split a 400G port into two 200G ports. Petronella Technology Group, Inc. runs a MikroTik CRS812-8DS-2DQ-2DDQ-RM, which serves four nodes on its two QSFP56-DD ports broken out to 2 x 200G each.
How much bandwidth does a switch cost a DGX Spark cluster?
Very little. We measured 109.11 Gb/s per PCIe half through a MikroTik CRS812 against 111.86 Gb/s on a direct cable, about 0.7 percent, and 196 Gb/s per node with both halves running, with no PFC or ECN tuning.
How many cables does a four-node DGX Spark cluster need?
Four node cables, one per unit, plus the switch and, on a switch with 400G QSFP56-DD ports, one breakout cable per pair of nodes. Two 2-packs of the 0.5 m QSFP112 cluster cable cost $598 with free US shipping.
Is four DGX Sparks better than three?
For tensor parallel inference, usually yes. Tensor parallel needs the attention head count to divide by the number of ranks, so four works where three generally does not. On our four-node switched fabric, Qwen3.8-Flash-Next ran 21 to 54 percent faster per stream than on two nodes.
Why does my four-node DGX Spark link only show 100G through the switch?
NVIDIA's switch playbook notes that auto-negotiation may not default to 200 Gb/s. Check with ethtool on every node, and if it reads 100000Mb/s, set the switch port to 200G manually with auto-negotiation off. On a MikroTik CRS812 breakout port this means forcing 200G-baseCR4 on the lane-master interfaces.
Get the cables, or get help with the build
Our 0.5 m QSFP112 cluster cable is in stock and fits every GB10 workstation: the Dell Pro Max GB10, ASUS Ascent GX10, HP ZGX Nano, Lenovo ThinkStation PGX, MSI EdgeXpert, Gigabyte AI TOP ATOM and Acer Veriton GN100, and brands mix freely in one cluster. A single cable is $159, a 2-pack is $299 and the 3-pack ring kit is $435, with free shipping to US addresses and shipping in 1 to 3 business days. For five or more cables, a purchase order, or help mapping ports, breakouts and the switch for a four-node or larger build, call Penny at 919-348-4912 or use the quote form on the cable page. The background math on why clustering is a capacity play is in our bandwidth explainer.
Buy DGX Spark cluster cables, free US shipping
Sources: NVIDIA, Connect Multiple DGX Spark through a Switch (switch requirements, same-port rule, link speed check, single bridge); NVIDIA DGX Spark User Guide, ConnectX-7 networking (two PCIe Gen 5 x4 links, three direct and four switched); Petronella Technology Group, Inc., GB10 cluster guide (measurements, switch configuration, bill of materials); NVIDIA developer forum threads 368726, 377435, 383218 and 365296.
Free, practical, and specific to regulated environments. We will email it to you.
No spam. Unsubscribe anytime.
