If you go looking for performance numbers on two NVIDIA DGX Sparks wired together over the ConnectX-7 stacking cable, you’ll find the same thing everywhere: bandwidth. NCCL bus bandwidth of ~24 GB/s. RDMA ib_write_bw of ~190 Gbps aggregate. iperf throughput. Forum threads about getting from a throttled 13 Gbps back up to line rate.
What you will not find — anywhere, as of this writing — is the number that actually matters for a latency-bound distributed workload: how long does a small-message collective take? Not a 1 GB all-gather that’s bottlenecked on bandwidth, but a few-kilobyte all-reduce whose cost is pure round-trip latency, the kind that lands on the critical path of tightly-coupled parallel execution and gets paid once per layer, per step, with no overlap.
Bandwidth tells you whether you can move a model’s worth of data. Small-message latency tells you whether a tightly-coupled parallel strategy is even worth attempting between two boxes. The two are unrelated, and only one of them is published. So I measured the other one.
The setup
Two NVIDIA GB10 (DGX Spark) units, each with an onboard ConnectX-7, connected directly by the bundled QSFP stacking cable — no switch. The link runs RoCE (RDMA over Converged Ethernet), so the “InfiniBand” verbs and NCCL NET/IB transport ride on top of an Ethernet link layer.
One architectural detail worth knowing up front, because it explains every per-rail number below: each QSFP port on the GB10 is two 100G MACs behind two PCIe Gen5 x4 links. The driver confirms it:
mlx5_core ...: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
So a single stream tops out around ~100 Gbps; you only approach 200 Gbps by loading both rails. For latency this is irrelevant — a small message uses one rail and never touches the bandwidth ceiling — but it’s why the bandwidth baseline below reads ~109 Gbps rather than ~200.
The firmware on these units is 28.45.4028. That’s notable: it’s the same version several people report shipping in a throttled state (~13 Gbps instead of ~100). Mine were not throttled. More on that trap later.
Baseline 1: bandwidth (so we know the link is healthy)
# server
ib_write_bw -d rocep1s0f1 -i 1 -F --report_gbits
# client
ib_write_bw -d rocep1s0f0 -i 1 -F --report_gbits 10.0.0.1
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 109.23 109.21 0.208300
~109 Gbps on one rail. Healthy — right at the per-rail PCIe limit, no throttle. This is the part everyone already publishes, and it’s only here to establish that the link is in its good state before we measure latency on it.
Baseline 2: the RDMA latency floor
Before NCCL, the raw verbs floor — the lowest latency the fabric can physically deliver, with no collective library on top. This one is also rarely shown (people quote “~1.5 µs” in prose, but I couldn’t find an actual ib_write_lat table for two Sparks):
# server
ib_write_lat -d rocep1s0f1 -i 1 -F
# client
ib_write_lat -d rocep1s0f0 -i 1 -F 10.0.0.1
#bytes #iter t_min[usec] t_max[usec] t_typical[usec] t_avg[usec]
2 1000 1.95 2.62 2.01 2.01
~2.0 µs for a 2-byte RDMA write. That’s the hardware floor. Everything above it — the gap between 2 µs and what NCCL actually delivers — is software: collective library overhead, kernel launch, the proxy thread, and the GPU↔NIC handoff.
The headline: small-message NCCL all-reduce
This is the figure that wasn’t out there. Two nodes, one GPU each, all_reduce_perf from nccl-tests over the RoCE link, forced onto the native IB transport:
mpirun -np 2 -H 10.0.0.1,10.0.0.2 \
--mca oob_tcp_if_include 10.0.0.0/30 \
--mca btl_tcp_if_include 10.0.0.0/30 \
-x NCCL_NET_PLUGIN=none -x NCCL_IB_MERGE_NICS=1 \
-x NCCL_IB_HCA=rocep1s0f1,rocep1s0f0 \
-x NCCL_SOCKET_IFNAME=enp1s0f1np1,enp1s0f0np0 \
./nccl-tests/build/all_reduce_perf -b 16K -e 32K -f 2 -g 1
NCCL confirms it picked the right transport — NET/IB ... rocep1s0f1:1/RoCE, not a socket fallback — and the result:
# size count type redop time algbw busbw #wrong
# (B) (us) (GB/s) (GB/s)
16384 4096 float sum 40.79 0.40 0.40 0
32768 8192 float sum 43.42 0.75 0.75 0
~40 µs for a 16 KB all-reduce, ~43 µs at 32 KB, between two DGX Sparks over RoCE. Correct results (#wrong 0), and genuinely on the IB path. The busbw column (~0.5 GB/s) looks alarming until you remember these are tiny messages: this regime is entirely latency-bound, so bus bandwidth is meaningless here. The number that matters is the time column.
So the chain, end to end:
| Layer | Latency | What it is |
|---|---|---|
RDMA write (ib_write_lat) | ~2 µs | hardware/fabric floor |
| NCCL all-reduce, 16 KB | ~40 µs | what a real collective costs |
| NCCL all-reduce, 32 KB | ~43 µs | still latency-bound |
Why 40 µs and not 2 µs?
A 20× gap between the verbs floor and the NCCL collective is larger than I expected, and it’s worth being honest about where it comes from. Two ranks, one all-reduce, but NCCL still has to: launch the collective kernel, hand off through its proxy thread, stage the buffer between GPU and NIC, do the actual RoCE round-trip, and do the reduction. Between two hosts over RoCE — as opposed to two GPUs on one NVLink fabric, where you’d see single-digit microseconds — tens of microseconds for a small-message all-reduce is, on reflection, normal.
One contributor stands out in the logs:
... Connected all rings, use ring PXN 0 GDR 0
GDR 0 — GPUDirect RDMA is off. The NIC isn’t DMA-ing straight into GPU memory; the data is staged through host memory on the way to the wire. On a unified-memory part like GB10 the GDR story is nuanced anyway, but it’s plausibly a chunk of that 40 µs, and the obvious lever to pull if you wanted to chase it lower. I’ll leave that for a follow-up.
Why this number is the one that matters
Bandwidth decides whether you can fit and feed a workload across two boxes. Small-message latency decides whether a tightly-coupled one will fly or die. If your parallel strategy puts a collective on the critical path N times per step — once per layer, say — then you’re paying N × 40 µs of pure, un-overlappable synchronization, every step. Whether that’s negligible or fatal depends entirely on how much compute sits between the collectives, and that’s a budgeting decision you can only make if you know the 40 µs.
The reason it’s worth writing down: with bandwidth at ~190 Gbps aggregate, the cable is wildly over-provisioned for the volume a small collective moves (a 16 KB all-reduce is ~microseconds of wire time at line rate). The binding constraint between two Sparks isn’t bandwidth at all — it’s this ~40 µs latency floor and how many times you have to pay it. That inverts the usual mental model, and it’s exactly the figure the bandwidth-only benchmarks leave out.
The traps that silently corrupt the result
Half the value of doing this yourself is discovering how many ways the measurement quietly lies to you. In rough order of how much time each one can cost:
1. The firmware throttle. Plenty of units ship pinned at ~13–16 Gbps despite negotiating a 200G link. The fix is a ConnectX-7 firmware update plus a full NIC power-drain — not just a reboot. Always run ib_write_bw first; if it’s not ~100 Gbps/rail, every latency number you take afterward is measured on a crippled link. (Mine, on 28.45.4028, were fine — so the version alone isn’t a reliable tell. Measure.)
2. NCCL silently falling back to TCP. The default NCCL stack will happily run your “RoCE” benchmark over plain sockets and report a busbw of ~1–3 GB/s — and you’ll think that’s the RDMA number. Force the native path with NCCL_NET_PLUGIN=none and NCCL_IB_MERGE_NICS=1, and read the debug log to confirm it says NET/IB, not NET/Socket.
3. OpenMPI picking the wrong network. Both boxes have a docker0 bridge on the same 172.17.x subnet, plus Wi-Fi, plus the rails. OpenMPI’s out-of-band layer sees the overlapping docker0 subnet on both hosts, concludes they’re directly reachable, and faceplants with cannot find a corresponding process entry for that peer. Pin it to the link:
--mca oob_tcp_if_include 10.0.0.0/30 --mca btl_tcp_if_include 10.0.0.0/30
4. The two-rail naming asymmetry. The single cable lights up two rails, and the port numbers differ on each end — one box exposes ...f1 ports, the other ...f0. So NCCL_IB_HCA and NCCL_SOCKET_IFNAME have to list both names and let each node match its own. Easy to stare at for a while.
5. The PCIe power warning. The driver logs Detected insufficient power on the PCIe slot (27W). It didn’t visibly hurt my numbers, but it’s the kind of thing that would explain an otherwise-inexplicable underperformance, so it’s worth noting it’s there.
There’s also the usual yak-shaving — nccl.h and mpi.h not on the include path, nccl-tests needing MPI_HOME pointed at the right prefix — but those are mechanical. The five above are the ones that change the number without changing anything you’d notice.
Caveats
This is one configuration, honestly reported. Single rail (latency doesn’t need both), GDR off, a specific NCCL/firmware/driver stack, a 16–32 KB sweep. A different NCCL version, GDR enabled, or a larger message set would move it — and I’d genuinely like to see someone reproduce it and disagree. But as a starting point, here’s the claim I’m willing to put a number on:
Between two DGX Sparks over the ConnectX-7 RoCE link, a small-message all-reduce costs about 40 microseconds, on a ~2 µs RDMA floor, with ~190 Gbps of bandwidth you’ll almost never be limited by.
If you’re deciding whether a latency-coupled multi-node strategy is viable on a pair of Sparks, that 40 µs — times however many collectives sit on your critical path — is the calculation nobody handed you. Now you have it.