https://contact.alessandrosangiorgi.net/tags/benchmarks/Recent content in Benchmarks on Alessandro Sangiorgi — GPU Performance EngineerHugoenThu, 25 Jun 2026 00:00:00 +0000https://contact.alessandrosangiorgi.net/posts/dgx-spark-nccl-collective-latency/Thu, 25 Jun 2026 00:00:00 +0000https://contact.alessandrosangiorgi.net/posts/dgx-spark-nccl-collective-latency/<p>If you go looking for performance numbers on two NVIDIA DGX Sparks wired together over the ConnectX-7 stacking cable, you&rsquo;ll find the same thing everywhere: <strong>bandwidth</strong>. NCCL bus bandwidth of ~24 GB/s. RDMA <code>ib_write_bw</code> of ~190 Gbps aggregate. iperf throughput. Forum threads about getting from a throttled 13 Gbps back up to line rate.</p> <p>What you will <em>not</em> find — anywhere, as of this writing — is the number that actually matters for a latency-bound distributed workload: <strong>how long does a small-message collective take?</strong> Not a 1 GB all-gather that&rsquo;s bottlenecked on bandwidth, but a few-kilobyte all-reduce whose cost is pure round-trip latency, the kind that lands on the critical path of tightly-coupled parallel execution and gets paid once per layer, per step, with no overlap.</p>