Serving DeepSeek-V4-Flash-0731 in Tensor Parallel Across Two DGX Sparks

DeepSeek published DeepSeek-V4-Flash-0731 today — the official release superseding the preview, “only re-post-trained” on the same architecture, but with a large jump in agentic capability:

Notably it beats V4-Pro-Preview on every one of those, despite a much smaller activated parameter count — and it comes with a speculative-decoding module attached out of the box. I have two GB10 DGX Sparks wired together over ConnectX-7 for exactly this kind of thing, so this post is the tensor-parallel deployment recipe: what’s actually in the checkpoint, whether it fits, the launch flags, and the trap that will wedge a fresh multi-node deployment if you skip it.

What’s actually in the checkpoint

config.json reports "quant_method": "fp8", and if you stop there you’ll assume a uniform FP8 model. Reading the safetensors headers directly tells a different story:

The tell is the expert tensor shape: layers.0.ffn.experts.0.w1.weight is I8 [2048, 2048] with a scale of [2048, 128]. 2048 bytes → 4096 4-bit values, and 4096/128 gives a 32-element group per scale — that’s MXFP4 with E8M0 exponent scales, packed into int8 because safetensors has no native 4-bit dtype. The overwhelming majority of parameters live in the experts, so the effective model is mostly 4-bit with an 8-bit attention path, not the flat FP8 the config implies. That’s also why 155.4 GiB across 48 shards is small enough to be interesting for two boxes at all — the same model in BF16 would be well over 300 GiB.

It also ships the speculative module already trained in: three mtp.{0,1,2} layers with a Markov head (mtp.2.markov_head.markov_w1/w2), matching dspark_target_layer_ids: [40, 41, 42] and dspark_block_size: 5 in the config. Nothing extra to fetch or attach — --speculative-config just needs to point at the method the checkpoint already carries.

Does it fit?

Each GB10 has 121 GiB of unified memory — CPU and GPU share the same pool, so --gpu-memory-utilization is a fraction of the host’s memory, not a separate VRAM budget. Split across two nodes with --tensor-parallel-size 2, the actual load numbers from a real bring-up:

Model loading took 79.17 GiB and 214.5 seconds     (per node)
Available KV cache memory: 10.04 GiB
GPU KV cache size: 1,412,421 tokens
Maximum concurrency for 1,000,000 tokens/request: 1.41x

At --gpu-memory-utilization 0.78 (≈ 94.4 GiB budget per box), ~79 GiB goes to sharded weights, leaving ~15 GiB for KV cache and activations — enough for a full 1M-token context window with light concurrency headroom. It fits, but there isn’t much slack, which matters for the trap below.

The launch

Worker (rank 1) starts first, headless; head (rank 0) drives the rendezvous. Both nodes need the RoCE interface names from the link validation post — and remember PCI enumeration can name the same physical port differently on each box, so don’t hardcode one name and copy it across.

# worker (10.0.0.2), headless
vllm serve /models/deepseek-v4-flash \
  --served-model-name deepseek-v4-flash \
  --tensor-parallel-size 2 --pipeline-parallel-size 1 \
  --nnodes 2 --node-rank 1 --headless \
  --master-addr 10.0.0.1 --master-port 25000 \
  --kv-cache-dtype fp8_ds_mla --max-model-len 1000000 \
  --gpu-memory-utilization 0.78 --max-num-seqs 6 \
  --max-num-batched-tokens 8192 --enable-chunked-prefill --enable-prefix-caching \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
  --distributed-executor-backend mp --enable-flashinfer-autotune

# head (10.0.0.1) — same flags, drop --headless, add --node-rank 0 --host 0.0.0.0 --port 8888

Environment on both sides carries the RoCE fabric config from the earlier posts (NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME, NCCL_IB_HCA, NCCL_NET=IB), plus VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 for the 1M context window.

Decode throughput and draft acceptance

With k=5 (num_speculative_tokens=5) DSpark speculation, single in-flight request, temperature 0, no competing traffic:

vLLM’s own spec-decode telemetry for that run, aggregated across the windows it covers:

Accepted: 1626 tokens, Drafted: 2390 tokens  ->  68.0% overall draft acceptance
Per-position acceptance rate: ~0.84-0.92 (position 1) down to ~0.19-0.74 (position 5)
Mean acceptance length: 3.3-5.2 tokens per step, depending on content

The shape is intuitive once you see it: structured, repetitive content (JSON, code) is easy for a small draft model to predict correctly, so acceptance stays high through most of the 5 draft positions. Free-form prose reasoning is the hardest case — it’s the least predictable text, so the draft disagrees with the target more often and throughput drops accordingly. That’s a property of the content, not a bug to chase.

The trap: cold kernels under multi-node tensor parallel

This is the one worth taking seriously before you trust a fresh deployment. Triton and TileLang kernels here compile lazily, per shape-key, per process — prefill chunk size, context bucket, sparse-attention top-k width, MoE expert-load histogram. Under two-node TP the ranks run in lockstep: if one rank stops mid-step to JIT-compile a cold kernel, its peer can end up waiting inside the next NCCL collective for a partner that isn’t coming yet. A models-endpoint health check proves nothing here — it happens before any real generation shape has been touched, so an API that answers /v1/models may not have compiled a single decode kernel a real request will need.

The fix is a real generation, not a ping, and specifically a decode long enough and concurrent enough to touch the shapes that matter — sequential short requests alone won’t reach the kernels that only show up under a concurrent long-context batch:

warm() {  # warm <chars> <max_tokens> [prompt]
  python3 - "$1" "$2" "${3-}" <<'PY'
import json, sys, time, urllib.request
chars, maxtok, prompt = int(sys.argv[1]), int(sys.argv[2]), sys.argv[3]
prompt = prompt or ('warm the kernels with a long prompt. ' * 1200)[:chars]
body = json.dumps({"model": "deepseek-v4-flash",
    "messages": [{"role": "user", "content": prompt}],
    "max_tokens": maxtok, "temperature": 0}).encode()
t0 = time.time()
d = json.load(urllib.request.urlopen(urllib.request.Request(
    "http://127.0.0.1:8888/v1/chat/completions", data=body,
    headers={"Content-Type": "application/json"}), timeout=590))
print("%.1fs completion=%s" % (time.time() - t0, d["usage"]["completion_tokens"]))
PY
}

# 1. ramped sequential — warms prefill shapes at increasing context sizes
for size in 2000 6000 12000 24000 48000; do warm "$size" 20; done

# 2. concurrent burst — reaches shapes only a multi-request batch produces
for i in 1 2 3 4; do warm 400 150 & done; wait

# 3. concurrent x long context — the combination that catches the rest
for i in 1 2 3; do warm 12000 40 & done; wait

# 4. sustained decode — a real generation, not a max_tokens cap that
#    lets the model say "OK" and return after two tokens
warm 400 700 "Write a detailed 600-word explanation of how a B-tree \
stays balanced during insertion. Do not stop early."

Run all four phases, single request then concurrent, before trusting a multi-node deployment with real traffic — and before wiring up any automatic recovery. On a health-check design specifically: a generation probe (not just the models endpoint) that a supervisor polls periodically is a reasonable pattern for catching a wedge automatically, but the recovery it triggers should re-run this warm-up before declaring the stack healthy again — otherwise a restart just hands back an unwarmed engine that can wedge the same way on its first real burst.

Other traps worth knowing about

• Don’t push --gpu-memory-utilization too close to 1.0 on unified memory. Unlike a discrete-GPU box, this is taking memory away from the host OS, not a separate VRAM pool — push it too far and you can starve the kernel itself, not just the model.
• The speculative config’s method name has to match what the checkpoint actually carries — dspark here, not a generic eagle/medusa/ngram value — or the draft head never engages and you silently lose the whole benefit of the module DeepSeek shipped for you.
• --tokenizer-mode and --reasoning-parser both need the deepseek_v4 value — the encoding scheme and the <think>...</think> reasoning format are model-specific, and defaults will parse neither correctly.

Takeaway

The checkpoint is a drop-in for anyone already running the DSpark-shaped preview — identical config, identical tensor layout, same mixed FP8/FP4 format, just better-trained weights — so if you already have a two-Spark tensor-parallel deployment working, swapping in 0731 is a weights-only change. If you’re starting from scratch, the two things that will actually cost you time are the ones a /v1/models check can’t see: whether the mixed-precision math is what you think it is, and whether the kernels a real request needs have ever actually been compiled.