<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Systems on Alessandro Sangiorgi — Software Engineer: Systems, Performance, Security</title><link>https://contact.alessandrosangiorgi.net/tags/systems/</link><description>Recent content in Systems on Alessandro Sangiorgi — Software Engineer: Systems, Performance, Security</description><generator>Hugo</generator><language>en</language><lastBuildDate>Sun, 27 Sep 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://contact.alessandrosangiorgi.net/tags/systems/index.xml" rel="self" type="application/rss+xml"/><item><title>Codex, Qwen3.8, and a Chat Template That Refused to Bend</title><link>https://contact.alessandrosangiorgi.net/posts/qwen3-8-vllm-responses-api-system-message-chat-template/</link><pubDate>Sun, 27 Sep 2026 00:00:00 +0000</pubDate><guid>https://contact.alessandrosangiorgi.net/posts/qwen3-8-vllm-responses-api-system-message-chat-template/</guid><description>&lt;p&gt;I run &lt;a href="https://github.com/openai/codex"&gt;Codex CLI&lt;/a&gt; against a locally hosted &lt;strong&gt;unsloth/Qwen3.8-27B-NVFP4&lt;/strong&gt;, served by vLLM on an RTX 5090. It&amp;rsquo;s wired up as a Codex profile (&lt;code&gt;qwen38.config.toml&lt;/code&gt;) pointing at &lt;code&gt;http://10.8.0.34:8000/v1&lt;/code&gt;. It had been working. Then one session it started failing, reliably, with this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-json" data-lang="json"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;{&lt;span style="color:#7ee787"&gt;&amp;#34;error&amp;#34;&lt;/span&gt;:{&lt;span style="color:#7ee787"&gt;&amp;#34;message&amp;#34;&lt;/span&gt;:&lt;span style="color:#a5d6ff"&gt;&amp;#34;System message must be at the beginning.&amp;#34;&lt;/span&gt;,&lt;span style="color:#7ee787"&gt;&amp;#34;type&amp;#34;&lt;/span&gt;:&lt;span style="color:#a5d6ff"&gt;&amp;#34;BadRequestError&amp;#34;&lt;/span&gt;,&lt;span style="color:#7ee787"&gt;&amp;#34;param&amp;#34;&lt;/span&gt;:&lt;span style="color:#79c0ff"&gt;null&lt;/span&gt;,&lt;span style="color:#7ee787"&gt;&amp;#34;code&amp;#34;&lt;/span&gt;:&lt;span style="color:#a5d6ff"&gt;400&lt;/span&gt;}}
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The interesting part wasn&amp;rsquo;t that it failed — it&amp;rsquo;s &lt;em&gt;when&lt;/em&gt; it failed. The very first prompt of a session always went through fine. The break only showed up on the &lt;strong&gt;second&lt;/strong&gt; turn, right after the model made a tool call (a shell command) and Codex sent the result back. First turn: 200 OK. First tool round-trip: 400. Every time.&lt;/p&gt;</description></item><item><title>Serving DeepSeek-V4-Flash-0731 in Tensor Parallel Across Two DGX Sparks</title><link>https://contact.alessandrosangiorgi.net/posts/deepseek-v4-flash-0731-tensor-parallel-dgx-spark/</link><pubDate>Fri, 31 Jul 2026 00:00:00 +0000</pubDate><guid>https://contact.alessandrosangiorgi.net/posts/deepseek-v4-flash-0731-tensor-parallel-dgx-spark/</guid><description>&lt;p&gt;DeepSeek published &lt;strong&gt;DeepSeek-V4-Flash-0731&lt;/strong&gt; today — the official release superseding the preview, &amp;ldquo;only re-post-trained&amp;rdquo; on the same architecture, but with a large jump in agentic capability:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;Benchmark&lt;/th&gt;
 &lt;th style="text-align: right"&gt;V4-Flash-0731&lt;/th&gt;
 &lt;th style="text-align: right"&gt;V4-Flash (Preview)&lt;/th&gt;
 &lt;th style="text-align: right"&gt;V4-Pro (Preview)&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;Terminal Bench 2.1&lt;/td&gt;
 &lt;td style="text-align: right"&gt;82.7&lt;/td&gt;
 &lt;td style="text-align: right"&gt;61.8&lt;/td&gt;
 &lt;td style="text-align: right"&gt;72.1&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Cybergym&lt;/td&gt;
 &lt;td style="text-align: right"&gt;76.7&lt;/td&gt;
 &lt;td style="text-align: right"&gt;38.7&lt;/td&gt;
 &lt;td style="text-align: right"&gt;52.7&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;DeepSWE&lt;/td&gt;
 &lt;td style="text-align: right"&gt;54.4&lt;/td&gt;
 &lt;td style="text-align: right"&gt;7.3&lt;/td&gt;
 &lt;td style="text-align: right"&gt;12.8&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Toolathlon-Verified&lt;/td&gt;
 &lt;td style="text-align: right"&gt;70.3&lt;/td&gt;
 &lt;td style="text-align: right"&gt;49.7&lt;/td&gt;
 &lt;td style="text-align: right"&gt;55.9&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;DSBench-Hard&lt;/td&gt;
 &lt;td style="text-align: right"&gt;59.6&lt;/td&gt;
 &lt;td style="text-align: right"&gt;25.8&lt;/td&gt;
 &lt;td style="text-align: right"&gt;31.1&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Notably it &lt;strong&gt;beats V4-Pro-Preview on every one of those&lt;/strong&gt;, despite a much smaller activated parameter count — and it comes with a speculative-decoding module attached out of the box. I have two &lt;a href="https://contact.alessandrosangiorgi.net/posts/dgx-spark-nccl-collective-latency/"&gt;GB10 DGX Sparks wired together over ConnectX-7&lt;/a&gt; for exactly this kind of thing, so this post is the tensor-parallel deployment recipe: what&amp;rsquo;s actually in the checkpoint, whether it fits, the launch flags, and the trap that will wedge a fresh multi-node deployment if you skip it.&lt;/p&gt;</description></item><item><title>Unboxing Two DGX Sparks, and Recovering One From a Kernel Panic on First Boot</title><link>https://contact.alessandrosangiorgi.net/posts/dgx-spark-kernel-panic-recovery/</link><pubDate>Sun, 28 Jun 2026 00:00:00 +0000</pubDate><guid>https://contact.alessandrosangiorgi.net/posts/dgx-spark-kernel-panic-recovery/</guid><description>&lt;p&gt;I recently took delivery of &lt;strong&gt;two NVIDIA DGX Sparks&lt;/strong&gt; (the GB10 units I later wired together for the &lt;a href="https://contact.alessandrosangiorgi.net/posts/dgx-spark-nccl-collective-latency/"&gt;small-message NCCL latency benchmark&lt;/a&gt;). The out-of-box experience is genuinely slick — right up until one of the two bricked itself on the very first software update and dropped me into a kernel panic. This post is the other half of the story: the setup flow, and exactly how I brought the dead one back.&lt;/p&gt;</description></item><item><title>Wiring Two DGX Sparks: One Script to Detect, Configure, and Validate the RoCE Link</title><link>https://contact.alessandrosangiorgi.net/posts/dgx-spark-roce-link-validation/</link><pubDate>Sun, 28 Jun 2026 00:00:00 +0000</pubDate><guid>https://contact.alessandrosangiorgi.net/posts/dgx-spark-roce-link-validation/</guid><description>&lt;p&gt;You plug the ConnectX-7 stacking cable into two DGX Sparks, assign a couple of IPs, and&amp;hellip; now what? Before you trust that link with any distributed workload, you want to &lt;em&gt;know&lt;/em&gt; three things are true:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The link runs at line rate (~100 Gbps/rail), not pinned at ~13 Gbps by a firmware-throttle trap.&lt;/li&gt;
&lt;li&gt;RDMA latency is in the single-digit microseconds.&lt;/li&gt;
&lt;li&gt;NCCL collectives actually ride the RoCE fabric — not a silent TCP fallback that &lt;em&gt;looks&lt;/em&gt; like it works but runs 50× slower.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Every one of those can fail &lt;em&gt;silently&lt;/em&gt;, reporting a plausible-but-wrong number. And before you can even test them, you have to answer a surprisingly annoying question: &lt;strong&gt;which network device is the live one?&lt;/strong&gt; On a fresh Spark the answer is not obvious, and it is not the same name on both boxes.&lt;/p&gt;</description></item><item><title>The Number Nobody Published: Small-Message NCCL Latency Between Two DGX Sparks</title><link>https://contact.alessandrosangiorgi.net/posts/dgx-spark-nccl-collective-latency/</link><pubDate>Thu, 25 Jun 2026 00:00:00 +0000</pubDate><guid>https://contact.alessandrosangiorgi.net/posts/dgx-spark-nccl-collective-latency/</guid><description>&lt;p&gt;If you go looking for performance numbers on two NVIDIA DGX Sparks wired together over the ConnectX-7 stacking cable, you&amp;rsquo;ll find the same thing everywhere: &lt;strong&gt;bandwidth&lt;/strong&gt;. NCCL bus bandwidth of ~24 GB/s. RDMA &lt;code&gt;ib_write_bw&lt;/code&gt; of ~190 Gbps aggregate. iperf throughput. Forum threads about getting from a throttled 13 Gbps back up to line rate.&lt;/p&gt;
&lt;p&gt;What you will &lt;em&gt;not&lt;/em&gt; find — anywhere, as of this writing — is the number that actually matters for a latency-bound distributed workload: &lt;strong&gt;how long does a small-message collective take?&lt;/strong&gt; Not a 1 GB all-gather that&amp;rsquo;s bottlenecked on bandwidth, but a few-kilobyte all-reduce whose cost is pure round-trip latency, the kind that lands on the critical path of tightly-coupled parallel execution and gets paid once per layer, per step, with no overlap.&lt;/p&gt;</description></item><item><title>Blocking WiFi De-Auth Attacks in the Kernel with eBPF and XDP</title><link>https://contact.alessandrosangiorgi.net/posts/ebpf-xdp-wifi-deauth-defense/</link><pubDate>Tue, 07 Apr 2026 00:00:00 +0000</pubDate><guid>https://contact.alessandrosangiorgi.net/posts/ebpf-xdp-wifi-deauth-defense/</guid><description>&lt;p&gt;802.11 de-authentication attacks are one of the oldest and cheapest WiFi denial-of-service techniques. An attacker sends forged de-auth management frames to disconnect clients from an access point — &lt;code&gt;aireplay-ng&lt;/code&gt; sends 128 per attack command (64 directed at the AP, 64 at the client). Despite being a known problem for over two decades, the main defense is still 802.11w (Protected Management Frames), which requires both AP and client support and still degrades under high-rate floods.&lt;/p&gt;</description></item></channel></rss>