I run Codex CLI against a locally hosted unsloth/Qwen3.8-27B-NVFP4, served by vLLM on an RTX 5090. It’s wired up as a Codex profile (qwen38.config.toml) pointing at http://10.8.0.34:8000/v1. It had been working. Then one session it started failing, reliably, with this:
{"error":{"message":"System message must be at the beginning.","type":"BadRequestError","param":null,"code":400}}
The interesting part wasn’t that it failed — it’s when it failed. The very first prompt of a session always went through fine. The break only showed up on the second turn, right after the model made a tool call (a shell command) and Codex sent the result back. First turn: 200 OK. First tool round-trip: 400. Every time.
Red herring #1: wire_api
My qwen38.config.toml had this, with a comment that turned out to be my own past self trying to remember why:
[model_providers.vllm]
name = "vllm-local"
base_url = "http://10.8.0.34:8000/v1" # the bridge, NOT vLLM directly
wire_api = "responses" # ← required now
“Required now” was a clue I’d left myself and then ignored. I tried switching it to wire_api = "chat" on the theory that the newer Responses API translation layer was the thing misbehaving. Codex refused outright:
Error: `wire_api = "chat"` is no longer supported.
How to fix: set `wire_api = "responses"` in your provider config.
More info: https://github.com/openai/codex/discussions/7782
So that’s not adjustable — current Codex only speaks the Responses API to custom providers. Back to "responses", and back to square one.
Red herring #2: the model catalog’s multi_agent_version
My models-qwen.json (the Codex model-catalog entry for this profile) had "multi_agent_version": "v2" set. Multi-agent handoffs are exactly the kind of thing that could inject an extra system-ish message into a request, so I nulled it out and retried:
"multi_agent_version": null,
Same error, same trigger point (first tool round-trip). Ruled out.
It’s not a bridge
The # the bridge, NOT vLLM directly comment sent me looking for a translation proxy that doesn’t exist. docker ps cleared that up fast:
CONTAINER ID IMAGE COMMAND STATUS PORTS
c83cdb22bab0 vllm-qwen3.8-5090 "vllm serve --model…" Up 8 days 0.0.0.0:8000->8000/tcp
10.8.0.34:8000 is vLLM. No bridge. vLLM 0.28.0 ships a native /v1/responses route now, so my own note from a few weeks back describing a “bridge” was simply stale — written back when I’d needed one, before vLLM grew Responses API support itself. That explained why patching a translation layer was never going to fix anything: there wasn’t one to patch.
With the mystery middle-tier gone, the actual traceback was sitting right there in docker logs:
ERROR [hf.py:797] An error occurred in `transformers` while applying chat template
Traceback (most recent call last):
File ".../vllm/renderers/hf.py", line 789, in safe_apply_chat_template
plain = tokenizer.apply_chat_template(
...
File ".../transformers/utils/chat_template_utils.py", line 479, in raise_exception
raise jinja2.exceptions.TemplateError(message)
jinja2.exceptions.TemplateError: System message must be at the beginning.
INFO: 10.8.0.34:53982 - "POST /v1/responses HTTP/1.1" 400 Bad Request
Not a bridge bug. Not a Codex bug. vLLM’s own renderer, rejected by the model’s own Jinja chat template, on the request built from the tool-call turn.
Into vLLM’s Responses API internals
vLLM builds the message list for /v1/responses in vllm/entrypoints/openai/responses/utils.py, via construct_input_messages():
def construct_input_messages(
*,
request_instructions: str | None = None,
request_input: str | list[ResponseInputOutputItem],
prev_msg: list[ChatCompletionMessageParam] | None = None,
prev_response_output: list[ResponseOutputItem] | None = None,
):
messages: list[ChatCompletionMessageParam] = []
if request_instructions:
messages.append({"role": "system", "content": request_instructions})
if prev_msg is not None:
# Filter out system messages from previous conversation -- per the
# OpenAI spec, instructions should NOT carry over across responses.
messages.extend(m for m in prev_msg if m.get("role") != "system")
...
# Append the new input.
input_messages = construct_chat_messages_with_tool_call(request_input)
messages.extend(input_messages)
return messages
Read on its own, this looks correct: the top-level instructions field always gets prepended first, and system messages from prior turns are deliberately stripped (per the Responses API spec, instructions don’t carry across turns). But it never inspects request_input for a system or developer-role item that isn’t first. Whatever role Codex tags an item with inside input, it gets passed straight through, wherever it happens to sit in that array. On the second turn, Codex includes a context item (its own per-turn environment refresh, sent alongside the tool result) tagged as system/developer — and it’s not the first item in that array.
That’s a perfectly valid request under the OpenAI Chat Completions schema, which doesn’t restrict system to position 0. It’s the model’s own template that’s stricter.
The actual guard
Qwen3.8’s chat_template.jinja (same shape as Qwen3.5/3.6’s) computes how many leading, contiguous system/developer messages it’s willing to merge, then raises for any straggler:
{%- set sysns = namespace(count=0, text='') %}
{%- for message in messages %}
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + render_content(message.content, false, true)|trim %}
{%- set sysns.count = sysns.count + 1 %}
{%- endif %}
{%- endfor %}
{%- set num_sys = sysns.count %}
...
{%- for message in messages %}
{%- if loop.index0 >= num_sys %}
{%- if message.role == "system" or message.role == "developer" %}
{{- raise_exception('System message must be at the beginning.') }}
{%- elif message.role == "user" %}
...
num_sys is the length of the run of system/developer messages starting at index 0. Anything with that role appearing after the run ends is fatal. On turn one, Codex’s input is [system, user] — one leading system message, num_sys = 1, nothing after it, fine. On turn two, the list looks roughly like [user, assistant(tool_call), tool(output), system(context refresh), ...] — the system item is real, but it’s not leading, so num_sys = 0 and the template blows up the moment it hits that item later in the loop.
Checking whether a newer vLLM just fixes this
Before touching anything, I checked whether this was already fixed upstream — no point patching something a pip install -U vllm solves for free. It’s not fixed, and it’s deliberately not fixed.
vllm-project/vllm#41114 is the same error on the same model family, same root cause: a second system-role message landing mid-conversation. A community PR, #44505, did exactly what I was about to write by hand — hoist any stray system/developer messages up into the leading block before handing the conversation to the template. It was closed without merging. From maintainer DarkLight1337:
I think it would be way easier to simply pass a custom chat template that doesn’t have this restriction, rather than trying to modify vLLM logic to handle all the cases.
The PR author agreed after re-reading the original report — the mid-conversation system message really is present in the sender’s payload, not something vLLM’s own preprocessing introduces, so “fixing” it centrally in vLLM means guessing at every client’s conventions forever. There’s an open docs-only PR, #57522, that formalizes the guidance: consolidate on the client/proxy side first, and fall back to a model-specific --chat-template override when you can’t. pip index versions vllm showed 0.30.0 available against my installed 0.28.0 — upgrading wouldn’t have touched any of this.
That settled it. The template guard is intentional and the maintainers’ answer is “swap the template,” not “wait for a patch.”
The fix: a locally overridden chat template
vLLM lets you pass --chat-template to serve a different Jinja file than the one baked into the model repo. I pulled the original out of the HF cache (bind-mounted from the container onto the host, so no docker cp gymnastics needed), and changed exactly one branch: instead of raising on a late system/developer message, render it inline as an ordinary system turn — the same thing templates like ChatML and Llama 3 already do without complaint.
{%- if message.role == "system" or message.role == "developer" %}
- {{- raise_exception('System message must be at the beginning.') }}
+ {{- '<|im_start|>system\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "user" %}
Everything else in the template — the leading-block merge, the reasoning-effort injection, the tool-call formatting — stays untouched. It’s the narrowest change that turns a hard rejection into the same permissive behavior most other chat templates already have.
I dropped the edited copy at volumes/chat_templates/qwen3.8-permissive.jinja on the host and recreated the container with one extra bind mount and one extra flag — image, GPU allocation, ports, and every other vllm serve argument stayed exactly as they were:
docker stop vllm-qwen3.8 && docker rm vllm-qwen3.8
docker run -d \
--name vllm-qwen3.8 \
--restart unless-stopped \
--gpus all \
--ipc private \
--shm-size 2g \
-p 8000:8000 \
-v /home/asangior/sangiorgisrl/vllm-try/volumes/vllm:/root/.cache/vllm \
-v /home/asangior/sangiorgisrl/vllm-try/volumes/hf:/root/.cache/huggingface \
-v /home/asangior/sangiorgisrl/vllm-try/volumes/chat_templates:/root/.cache/chat_templates \
vllm-qwen3.8-5090 \
--model unsloth/Qwen3.8-27B-NVFP4 \
--tensor-parallel-size 1 \
--max-model-len 114688 \
--kv-cache-dtype fp8 \
--enforce-eager \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--gpu-memory-utilization 0.93 \
--max-num-seqs 8 \
--chat-template /root/.cache/chat_templates/qwen3.8-permissive.jinja
Verifying it actually fixed the right thing
Rather than trust it and go back to a full Codex session, I reproduced the exact failing shape directly against /v1/responses — a system message dropped in after a user/assistant exchange, which is precisely what used to blow up:
curl -s -X POST http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "unsloth/Qwen3.8-27B-NVFP4",
"instructions": "You are a helpful assistant.",
"input": [
{"role": "user", "content": "say hi in one word"},
{"role": "assistant", "content": "hi"},
{"role": "system", "content": "reminder: be brief"},
{"role": "user", "content": "say bye in one word"}
]
}'
Before the template swap: 400, System message must be at the beginning. After: status: completed, with a normal reasoning + message output ending in "bye". Back in an actual Codex session, tool calls now round-trip past the second turn without incident.
Takeaways
- A stale comment in your own config is still a lie.
# the bridge, NOT vLLM directlycost me the first chunk of debugging time chasing a proxy that no longer existed — vLLM had grown native/v1/responsessupport since I wrote that note. - Read the actual server logs before the client-side error. The client only ever saw a generic 400; the real, specific Jinja traceback was sitting in
docker logsthe whole time. - A strict chat template is a model-author decision, not a vLLM bug — even when it produces a 400 from correctly-shaped OpenAI-schema requests. vLLM’s own maintainers declined to paper over it centrally, on the reasonable grounds that they’d be guessing at every client’s message-injection conventions forever.
- Check upstream before you patch downstream. I had a working patch to vLLM’s
construct_input_messages()ready to apply — a fix that would’ve needed re-applying on every image rebuild. Five minutes on the vLLM issue tracker showed the maintainers had already rejected that exact approach and pointed at the right lever:--chat-template. --chat-templateis the sanctioned escape hatch. One line changed in a copy of the model’s own template, mounted in alongside the existing volumes, survives vLLM upgrades untouched since it never lives inside the vLLM package itself.