Posts tagged "Dgx-Spark"

You cabled two DGX Sparks together with the ConnectX-7 stacking cable. Now: is the link actually at line rate, or silently firmware-throttled? Are your collectives on RDMA, or quietly falling back to TCP? Which of the FOUR RoCE devices is even the live one — and why is it named differently on each box? Here is a single self-contained bash script that detects every port from sysfs, installs the tooling, and validates the link end-to-end (bandwidth, latency, and a real NCCL all-reduce over RoCE) with green/red gates — plus the traps it encodes. Measured on two GB10 Sparks: 109 Gbps/rail, 1.45 µs RDMA, 29.5 µs small-message all-reduce.

The DGX Spark setup is a phone-style hotspot-and-wizard flow that pulls down new software and firmware over Wi-Fi. On one of my two units that update left me staring at 'Unable to mount root fs on unknown-block(0,0)'. Here's the out-of-box experience, and exactly how I recovered the bricked one — including where my recovery diverged from NVIDIA's official guide.

Every dual-DGX-Spark writeup benchmarks bandwidth — GB/s of busbw, ~190 Gbps of RDMA. None of them publish the one figure that actually decides whether a latency-bound multi-node workload is viable: the small-message NCCL all-reduce latency over the ConnectX-7 RoCE link. So I measured it on two GB10 Sparks: ~2 µs RDMA write floor, and ~40 µs for a 16–32 KB all-reduce. Plus the firmware, NCCL, and OpenMPI traps that silently wreck the result.