Posts tagged "Networking"

You cabled two DGX Sparks together with the ConnectX-7 stacking cable. Now: is the link actually at line rate, or silently firmware-throttled? Are your collectives on RDMA, or quietly falling back to TCP? Which of the FOUR RoCE devices is even the live one — and why is it named differently on each box? Here is a single self-contained bash script that detects every port from sysfs, installs the tooling, and validates the link end-to-end (bandwidth, latency, and a real NCCL all-reduce over RoCE) with green/red gates — plus the traps it encodes. Measured on two GB10 Sparks: 109 Gbps/rail, 1.45 µs RDMA, 29.5 µs small-message all-reduce.

Every dual-DGX-Spark writeup benchmarks bandwidth — GB/s of busbw, ~190 Gbps of RDMA. None of them publish the one figure that actually decides whether a latency-bound multi-node workload is viable: the small-message NCCL all-reduce latency over the ConnectX-7 RoCE link. So I measured it on two GB10 Sparks: ~2 µs RDMA write floor, and ~40 µs for a 16–32 KB all-reduce. Plus the firmware, NCCL, and OpenMPI traps that silently wreck the result.

Google blocked netlink socket bind in Android 11 (targetSDK 30), breaking ARP table access for apps. I wrote a JNI library that bypasses the restriction by sending RTNetlink dump requests without binding — until SELinux closed the door for good. Ubiquiti wanted to buy it for WiFiMan; I open-sourced it instead.