How to Benchmark GPU Interconnects for AI Clusters

A GPU cluster can show excellent per-device performance and still fail to deliver expected training or simulation throughput. The usual cause is not the accelerator. It is the path between accelerators. Knowing how to benchmark GPU interconnects means measuring that path as a system: physical topology, protocol behavior, collective communication, contention, and the workload patterns that expose them.

For institutional AI, scientific computing, and engineering simulation environments, this distinction is material. A nominal interconnect specification describes theoretical capacity under defined conditions. A benchmark must establish the usable communication envelope under the conditions your models, schedulers, and storage paths actually create.

Define the Decision Before Measuring the Fabric

A benchmark without an architectural question produces numbers without consequence. Before selecting tools or launching jobs, define the decision the measurement will support. You may be validating a new GPU node design, comparing NVLink or NVSwitch domains against PCIe-connected accelerators, qualifying GPU Direct RDMA across nodes, or diagnosing poor scaling in a distributed training workload.

Each decision requires a different emphasis. Large-message bandwidth matters when all-reduce operations exchange substantial gradient tensors. Small-message latency can dominate tightly synchronized simulation codes or inference pipelines that coordinate frequently. Bidirectional throughput matters where simultaneous send and receive traffic is normal. Collective performance matters more than point-to-point results whenever the production workload is built around data parallelism, tensor parallelism, or distributed numerical methods.

The core principle is simple: benchmark the communication primitive your application uses, at the message sizes it produces, across the topology it will inhabit. A peak link-rate figure is not a substitute for that discipline.

Establish the Topology as a Measurable Baseline

GPU interconnect benchmarking begins with topology discovery, not with a bandwidth test. Record the GPU-to-GPU paths, GPU-to-NIC affinity, CPU socket attachment, PCIe root-complex arrangement, NUMA domains, and available high-speed fabric. A system with eight identical GPUs may contain materially different communication paths between pairs.

Within a node, accelerators may communicate through NVLink, an NVSwitch fabric, PCIe switches, shared root complexes, or paths that traverse one or more CPU sockets. Across nodes, the relevant path expands to include the NIC, PCIe attachment, network switch tier, routing policy, and remote GPU affinity. The slowest or most contended segment often determines observed collective behavior.

Capture the topology before tuning anything. Identify asymmetric paths, oversubscribed PCIe switches, NICs that are distant from their assigned GPUs, and communication routes that cross NUMA boundaries. This evidence turns a disappointing result into an architectural diagnosis rather than a generic performance complaint.

Clock state and platform configuration also belong in the baseline. GPU application clocks, CPU frequency governors, power limits, BIOS PCIe settings, IOMMU behavior, firmware levels, driver versions, CUDA runtime versions, and network stack settings can each alter results. For comparative testing, hold these variables constant and document them. A benchmark that cannot be reproduced is unsuitable for capacity planning.

Verify Peer Access and Data Paths

Before running collectives, verify that direct GPU peer access is available where expected. A failed peer-to-peer path may silently route transfers through host memory, producing numbers that look plausible but do not represent the intended architecture.

For multi-node systems, test whether GPU Direct RDMA is active and whether the NIC selected by the runtime is physically local to the GPU. A GPU attached to one CPU socket and a NIC attached to another can incur avoidable traversal through inter-socket links. In a small benchmark, the penalty may seem tolerable. At scale, repeated communication through that path can become the limiting resource.

Measure Point-to-Point Behavior First

Point-to-point tests establish the basic characteristics of each path. Measure unidirectional bandwidth, bidirectional bandwidth, and latency for every relevant GPU pair. Do not limit testing to an average across the node. The minimum, maximum, and distribution of pairwise performance reveal whether the fabric behaves uniformly enough for the intended communication pattern.

Run tests over a range of payload sizes. Small transfers expose launch overhead, protocol overhead, and latency. Medium transfers reveal where throughput begins to saturate. Large transfers show sustainable bandwidth and whether the link reaches a stable plateau. A single 1 GB transfer can be useful, but it says little about a workload issuing frequent 64 KB or 4 MB exchanges.

Record at least the following measurements:

• Latency at small payload sizes, including both median and tail behavior where the tool permits it.

• Unidirectional and bidirectional bandwidth at saturation-relevant payload sizes.

• Pairwise variance across all GPU paths in the node.

• Host-staged versus direct peer-to-peer transfer behavior when both are possible.

• Cross-node GPU-to-GPU results for each relevant GPU and NIC affinity combination.

The comparison between unidirectional and bidirectional bandwidth is particularly instructive. Some paths scale efficiently in both directions, while others reveal contention in PCIe switches, root complexes, memory controllers, or network links. This is not a minor implementation detail. Distributed workloads routinely generate simultaneous traffic.

Benchmark Collectives, Not Just Links

Most AI and HPC applications do not communicate as isolated sender-receiver pairs. They execute all-reduce, all-gather, reduce-scatter, broadcast, and all-to-all operations. The collective library selects algorithms and transport paths according to topology, message size, process placement, and runtime configuration. That behavior must be measured directly.

Use a collective communications benchmark with the same process model planned for production: one process per GPU where appropriate, realistic rank counts, and the intended node count. Test across message sizes that correspond to model gradients, activation sharding, domain decomposition boundaries, or checkpoint coordination. Measure both algorithmic bandwidth and bus bandwidth, while being explicit about which is reported. They answer different questions and should not be compared casually.

All-reduce is often the first test for data-parallel training, but it is not sufficient for every architecture. Large language model training may depend heavily on all-gather and reduce-scatter. Mixture-of-experts workloads can make all-to-all traffic decisive. Multi-physics simulations may involve irregular nearest-neighbor exchanges that synthetic all-reduce does not represent. The benchmark suite should reflect this communication profile rather than defaulting to the most familiar collective.

Scale the test deliberately: one GPU, then the local GPU group, then one node, then multiple nodes, followed by the planned production scale where feasible. Track efficiency as resources are added. If aggregate bandwidth rises while per-GPU effective throughput collapses, the system may be exposing network oversubscription, poor rank placement, or a collective algorithm poorly matched to the fabric.

Introduce Contention as a First-Class Condition

Clean-room benchmarks are necessary, but production systems are not clean rooms. Test communication under realistic contention: simultaneous jobs, storage traffic, checkpoint writes, shared network paths, and host CPU activity. A fabric that performs well in isolation may degrade sharply when multiple GPU groups communicate through the same switch tier.

This is where jitter becomes as important as averages. Training steps synchronize at the slowest participating rank. A small group of latency outliers can reduce overall step throughput even when mean bandwidth remains strong. Capture percentile results and correlate them with node, GPU, NIC, switch port, and process placement. Engineering intelligence emerges from locating the source of variance, not merely observing it.

Validate With the Real Workload

Synthetic benchmarks characterize the fabric. They do not certify the application. The final stage is a controlled workload-level experiment using a representative model or simulation case. Keep the dataset, precision mode, batch strategy, software build, and node configuration fixed. Then compare scaling efficiency, iteration or timestep duration, communication overlap, accelerator utilization, and tail latency as GPU count increases.

Profile whether communication is hidden beneath computation or exposed on the critical path. A slower nominal link can be adequate if the workload overlaps transfers effectively. Conversely, a high-bandwidth fabric may offer limited benefit to a compute-bound model with infrequent synchronization. This is the essential trade-off: infrastructure should be judged by delivered workload performance, not by isolated peak metrics.

For AI training, observe step-time decomposition rather than GPU utilization alone. High utilization can coexist with inefficient scaling if devices wait at synchronization points between kernels. For simulation, examine communication-to-computation ratio across mesh size and decomposition strategy. The correct answer may be a different rank mapping, larger local problem size, or a revised parallelization method rather than an immediate hardware change.

Turn Results Into an Architectural Record

A useful benchmark report states the environment, topology, test method, message sizes, rank mapping, runtime parameters, observed results, and known limitations. It distinguishes theoretical link capacity from measured pairwise throughput, collective throughput, and application-level scaling. It also identifies the bottleneck with enough specificity to guide remediation.

Possible remediation may include changing GPU-NIC affinity, correcting process pinning, enabling a direct data path, revising PCIe or BIOS configuration, rebalancing network topology, changing collective settings, or redesigning the node architecture. These interventions differ substantially in cost and permanence. The benchmark should make that choice defensible.

At ELDEF Technology, this methodology is treated as part of computational architecture validation, not as a final acceptance test. Interconnect performance is a property of the complete system design, from accelerator placement to runtime behavior and workload mathematics.

The most valuable benchmark result is therefore not the largest bandwidth number. It is a clear, repeatable explanation of where communication time goes, what changes it, and whether the fabric will remain credible as the computational estate grows.