Posts tagged "Bringup"

You cabled two DGX Sparks together with the ConnectX-7 stacking cable. Now: is the link actually at line rate, or silently firmware-throttled? Are your collectives on RDMA, or quietly falling back to TCP? Which of the FOUR RoCE devices is even the live one — and why is it named differently on each box? Here is a single self-contained bash script that detects every port from sysfs, installs the tooling, and validates the link end-to-end (bandwidth, latency, and a real NCCL all-reduce over RoCE) with green/red gates — plus the traps it encodes. Measured on two GB10 Sparks: 109 Gbps/rail, 1.45 µs RDMA, 29.5 µs small-message all-reduce.