Nccl on Alessandro Sangiorgi — GPU Performance Engineer

Wiring Two DGX Sparks: One Script to Detect, Configure, and Validate the RoCE Link

Sun, 28 Jun 2026 00:00:00 +0000

You plug the ConnectX-7 stacking cable into two DGX Sparks, assign a couple of IPs, and… now what? Before you trust that link with any distributed workload, you want to know three things are true:

The link runs at line rate (~100 Gbps/rail), not pinned at ~13 Gbps by a firmware-throttle trap.
RDMA latency is in the single-digit microseconds.
NCCL collectives actually ride the RoCE fabric — not a silent TCP fallback that looks like it works but runs 50× slower.

Every one of those can fail silently, reporting a plausible-but-wrong number. And before you can even test them, you have to answer a surprisingly annoying question: which network device is the live one? On a fresh Spark the answer is not obvious, and it is not the same name on both boxes.

The Number Nobody Published: Small-Message NCCL Latency Between Two DGX Sparks

Thu, 25 Jun 2026 00:00:00 +0000

If you go looking for performance numbers on two NVIDIA DGX Sparks wired together over the ConnectX-7 stacking cable, you’ll find the same thing everywhere: bandwidth. NCCL bus bandwidth of ~24 GB/s. RDMA ib_write_bw of ~190 Gbps aggregate. iperf throughput. Forum threads about getting from a throttled 13 Gbps back up to line rate.

What you will not find — anywhere, as of this writing — is the number that actually matters for a latency-bound distributed workload: how long does a small-message collective take? Not a 1 GB all-gather that’s bottlenecked on bandwidth, but a few-kilobyte all-reduce whose cost is pure round-trip latency, the kind that lands on the critical path of tightly-coupled parallel execution and gets paid once per layer, per step, with no overlap.