Posts tagged "Llm-Serving"

Codex CLI against a self-hosted Qwen3.8-27B-NVFP4 on vLLM worked fine for exactly one turn, then started throwing a raw 400 the moment a tool call came back: 'System message must be at the beginning.' Here's the trail from a stale config comment, through vLLM's Responses API internals, into the model's own Jinja chat template, and the upstream GitHub issue that told me not to patch vLLM at all — but to override the template instead.

DeepSeek shipped the official V4-Flash-0731 release today, folding its speculative-decoding module into the base checkpoint and beating V4-Pro-Preview on agentic benchmarks despite a far smaller activated parameter count. Here's what's actually inside the checkpoint (mixed FP8/FP4, not the plain FP8 config.json advertises), whether 155 GiB fits across two 121 GiB unified-memory GB10 boxes, the vLLM tensor-parallel launch recipe over the RoCE link, and the JIT-warm-up trap that will silently wedge a multi-node deployment on its first real request.