H100 and H200 NVLink Cluster Scaling: Maximum Configurations and Performance

The physical mechanism for connecting NVLink lanes between GPUs depends on the form factor. In the NVL (NVLink) configurations, GPUs are connected using NVLink bridges: physical connectors that sit atop the GPU modules and carry the NVLink lanes between them. In SXM configurations, GPUs connect through NVSwitch chips on the baseboard. This difference has profound consequences for topology, bandwidth distribution, and maximum NVLink domain size.

H100 NVL: Two GPUs, One Bridge, Full Bandwidth

The H100 NVL is a PCIe form factor card that pairs two H100 GPUs connected by an NVLink bridge. In this configuration, each GPU dedicates all 18 of its NVLink 4.0 links to a single peer GPU through the bridge. This means the two GPUs share 900 GB/s of bidirectional bandwidth between them.

Because all 18 links terminate at the same peer, the H100 NVL pair achieves the theoretical maximum point-to-point NVLink bandwidth. There is no bandwidth sharing with other GPUs. For workloads that only require two-GPU parallelism (such as inference with model sizes between 80 GB and 188 GB, where a single GPU's memory is insufficient but two GPUs suffice), this topology is optimal. The NVLink domain size is 2 GPUs.

Beyond the NVLink pair, each H100 NVL GPU also has a PCIe Gen 5 x16 connection to the host system, providing 64 GB/s bidirectional to CPU memory and the rest of the server fabric. Communication with GPUs in other NVL pairs must traverse PCIe, InfiniBand, or Ethernet, all of which operate at significantly lower bandwidth than the NVLink connection.

H200 NVL: Same Bridge Topology, More Memory

The H200 NVL retains the identical NVLink bridge topology as the H100 NVL. Two GPUs are connected through an NVLink bridge, with all 18 lanes per GPU dedicated to the single peer connection. The bandwidth between the pair remains 900 GB/s bidirectional.

The critical difference is memory: each H200 GPU carries 141 GB of HBM3e (up from 94 GB HBM3 on the H100 NVL), bringing the total per pair to 282 GB. Memory bandwidth also increases from 3.35 TB/s to 4.8 TB/s per GPU. For inference workloads, this means models up to approximately 270 GB (accounting for KV cache and activation memory overhead) can fit within a single NVL pair, a substantial improvement over the H100 NVL's effective capacity.

For training workloads, the larger memory per GPU reduces the minimum degree of tensor parallelism required for a given model size. A model that previously needed 4-way tensor parallelism across 4 H100 GPUs (requiring NVSwitch or cross-pair communication) might fit in 2-way tensor parallelism on an H200 NVL pair, keeping all tensor-parallel communication within the NVLink domain.

Double Bridge Configurations: Splitting NVLink Lanes

The concept of double bridges introduces an important tradeoff. When NVLink lanes are split across two bridges to connect a GPU to two different peers simultaneously, each bridge carries approximately half the available lanes. With 18 total NVLink 4.0 links per GPU, a double-bridge configuration allocates roughly 9 links per bridge, providing approximately 450 GB/s bidirectional to each of two peers.

This lane-splitting is not commonly used in the H100 NVL product line. NVIDIA's standard H100 NVL product dedicates all 18 links to a single peer. However, the principle of lane splitting is fundamental to understanding why NVSwitch exists and why direct bridge topologies cannot scale to 8 GPUs.

Consider the mathematical constraint: an H100 GPU has 18 NVLink lanes. To directly connect to 7 other GPUs (as in an 8-GPU fully connected mesh), each connection would receive at most 2 to 3 lanes, providing only 100 to 150 GB/s per peer. This severe bandwidth reduction per connection makes direct bridging impractical for large NVLink domains. The solution is NVSwitch.

The theoretical maximum cluster size for H100 and H200 systems is bounded by the scale-out network, not by NVLink. NVLink defines the intra-node (or intra-pair) domain, while InfiniBand or Ethernet handles inter-node communication. NVIDIA's reference architectures and real-world deployments define practical limits.

DGX H100 (8x SXM): Up to 32,768 GPUs

The DGX H100 node contains 8 H100 SXM5 GPUs connected via NVSwitch, plus 8 ConnectX-7 InfiniBand adapters providing 400 Gb/s (50 GB/s) each. The standard scale-out fabric is NVIDIA Quantum-2 InfiniBand at 400 Gb/s NDR.

NVIDIA's DGX SuperPOD reference architecture specifies scalable unit pods. A single SuperPOD rack group contains 32 DGX H100 nodes (256 GPUs). Multiple SuperPODs connect through spine InfiniBand switches. The largest validated SuperPOD configuration is the Eos supercomputer at 576 DGX H100 nodes (4,608 GPUs).

In practice, Meta's Research SuperCluster (RSC) demonstrated 16,384 H100 GPUs connected via InfiniBand, and their subsequent Grand Teton cluster scaled to 24,576 H100 GPUs. The theoretical limit for a fat-tree InfiniBand topology with Quantum-2 switches is approximately 32,768 GPUs, constrained by switch port count and the number of tiers in the fabric.

H100 NVL: Pair-Based Scaling

H100 NVL clusters scale by populating servers with multiple NVL pairs. Common server configurations include 4 GPUs (2 pairs), 8 GPUs (4 pairs), or 10 GPUs (5 pairs), depending on the OEM platform. Each pair has its own NVLink domain of 2 GPUs, and cross-pair communication uses PCIe-based interconnects or network adapters.

Since the NVLink domain is only 2 GPUs, the InfiniBand or Ethernet network handles a larger share of the communication burden compared to SXM clusters. The maximum cluster size is again limited by the network fabric rather than NVLink. Practically, NVL-based clusters can scale to thousands of GPUs, but training throughput per GPU is lower than equivalent SXM clusters for workloads that benefit from TP greater than 2.

The primary use case for large H100 NVL clusters is inference at scale, where the 2-GPU NVLink domain is sufficient for serving most models, and the higher per-GPU memory (94 GB on H100 NVL, 141 GB on H200 NVL) reduces the number of GPUs needed per model replica.

DGX H200: SXM Topology with Expanded Memory

The DGX H200 uses the same 8-GPU NVSwitch topology as the DGX H100 but with H200 GPUs providing 141 GB HBM3e each. Total node memory increases from 640 GB to 1,128 GB. The scale-out network remains 8x ConnectX-7 at 400 Gb/s.

For cluster scaling, the DGX H200 offers an interesting advantage: because each GPU has 76% more memory than the H100 SXM (141 GB vs 80 GB), the same model can be trained with fewer GPUs or with a lower degree of tensor parallelism. A model that required TP=8 on H100 (using all 8 GPUs in the NVLink domain for tensor parallelism) might work with TP=4 on H200, freeing the remaining 4 GPUs for increased data parallelism. This can improve cluster utilization and training efficiency.

The maximum cluster size for DGX H200 follows the same InfiniBand fabric constraints as DGX H100. NVIDIA's reference architecture supports the same SuperPOD configurations, with the cluster's theoretical maximum remaining in the range of 32,768 GPUs.

Tensor parallelism (TP) splits individual weight matrices across multiple GPUs. For a transformer model, this means each GPU holds a shard of every layer's weights, and the GPUs must perform an all-reduce operation after each matrix multiplication to combine partial results. The communication volume per layer is proportional to the hidden dimension size, and this communication happens on the critical path, meaning it cannot be hidden behind computation.

The practical maximum TP degree is equal to the NVLink domain size. Going beyond the NVLink domain forces tensor-parallel all-reduce operations across InfiniBand, which reduces throughput by the bandwidth ratio (approximately 18x). Research from NVIDIA, Microsoft, and Google consistently shows that TP should never exceed the NVLink domain size for optimal performance.

For models larger than the TP-limited capacity of the NVLink domain, pipeline parallelism (PP) distributes different layers across different nodes. PP communication volume is much smaller (only boundary activations between pipeline stages), making it tolerable over InfiniBand. The combination of TP within the NVLink domain and PP across nodes is the standard approach for training large language models on Hopper-class hardware.

Expert parallelism (EP), used in mixture-of-experts models, introduces a third dimension. EP distributes different experts across GPUs, with all-to-all communication routing tokens to the appropriate expert. This communication pattern benefits from large NVLink domains because the all-to-all traffic can stay within NVLink when TP and EP are co-located in the same domain.

Sizing a GPU cluster requires mapping your model's parameter count, training data volume, and time-to-train target to specific hardware configurations. The interconnect topology determines which configurations are feasible and which will bottleneck on communication.

Step 1: Calculate Memory Requirements

For mixed-precision training (FP16 weights, FP32 optimizer states), the minimum GPU memory per parameter is approximately 18 to 20 bytes when using Adam optimizer. This accounts for 2 bytes for FP16 weights, 4 bytes for FP32 master weights, 4 bytes for FP32 first moment, 4 bytes for FP32 second moment, and 2 to 4 bytes for gradients, plus activation memory that depends on batch size and sequence length.

A 70B parameter model requires approximately 1.26 to 1.4 TB of GPU memory for training. A single DGX H100 node provides 640 GB, so a minimum of 3 nodes (24 GPUs) is needed. A DGX H200 node provides 1,128 GB, so 2 nodes (16 GPUs) may suffice with activation checkpointing.

Step 2: Choose Parallelism Strategy

Based on the memory requirement and NVLink domain size, determine the parallelism configuration:

• Model fits in 1 GPU: Use data parallelism only. Any form factor works.
• Model fits in NVLink domain (2 GPUs for NVL, 8 for SXM): Use TP within domain, DP across nodes.
• Model exceeds NVLink domain: Use TP within domain + PP across nodes + DP for replicas. SXM strongly preferred.
• 1T+ parameter model: Full 3D parallelism (TP + PP + DP) mandatory. SXM required for competitive throughput. Consider waiting for Blackwell NVL72 if timeline permits.

Step 3: Estimate Training Time

Training time in GPU-hours can be estimated using the scaling law approximation: approximately 6 * N * D / (throughput per GPU), where N is parameter count and D is token count. For an H100 SXM at peak utilization (roughly 40% of theoretical 989 TFLOPS FP16, or about 396 TFLOPS effective), a 70B model trained on 2T tokens requires approximately 700,000 GPU-hours.

On a 512-GPU DGX H100 cluster (64 nodes), this corresponds to approximately 57 days of continuous training, assuming 95% cluster uptime and 85% GPU utilization (accounting for communication overhead). On a 1,024-GPU cluster, this drops to approximately 30 days.

Step 4: Network Fabric Design

For clusters with more than one node, the InfiniBand fabric design is critical. NVIDIA recommends a rail-optimized topology for DGX clusters: each of the 8 GPUs in a node connects to a dedicated InfiniBand rail, and each rail uses a separate leaf switch. This ensures that GPU 0 on every node communicates through the same switch, GPU 1 through another switch, and so on. This topology optimizes for the data-parallel all-reduce pattern, where GPUs at the same position across nodes exchange gradients.

For clusters over 256 GPUs, a two-tier (leaf-spine) InfiniBand fabric is typical. For clusters over 2,048 GPUs, a three-tier fat-tree is required to maintain full bisection bandwidth. The network fabric cost can represent 20% to 30% of the total cluster cost, making it a significant factor in the total cost of ownership.

Petronella Technology Group designs and deploys complete GPU cluster solutions, from single-node AI development systems to multi-rack InfiniBand-connected clusters. Our team handles the complete stack: hardware procurement, rack and power infrastructure, InfiniBand fabric design, NVLink topology validation, NVIDIA Base Command configuration, and distributed training software optimization.