[Feature] Add Qwen3.5-4B chat template support

Mark commented

2026-04-25 19:36:25 +00:00

Owner

Problem

Qwen3.5-4B (unsloth/Qwen3.5-4B-GGUF) uses a chat template that differs from the generic "llama" template currently applied to all models. The model supports:

System role: <|im_start|>system\n...<|im_end|>\n
User role: <|im_start|>user\n...<|im_end|>\n
Assistant role: <|im_start|>assistant\n...<|im_end|>\n
Thinking mode: <|im_start|>assistant\n<|thinking|>...<|/thinking|>\n<|im_end|>\n

Currently, vllm_mm applies a generic "llama" template to all models, which produces incorrect prompt formatting for Qwen3.5-4B and likely degrades output quality.

Evidence

The model's GGUF metadata (extracted via strings on the Q4_K_M file) shows:

chat_template = "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"

Proposed Solution

Add a new chat_template field to the model configuration that allows per-model template selection:

Backend: Add chat_template column to models table (SQLite schema already supports this)
Frontend: Add template selector in model add/edit dialog with options:

llama (default, current behavior)
qwen3 (for Qwen3.5-4B and similar)
gemma4 (for Gemma 4 models — see related work)

API: Include chat_template in /api/models and /api/models/{id} responses
Inference: Pass selected template to vLLM via --chat-template argument or Jinja2 string

Acceptance Criteria

User can select "qwen3" template when adding/editing Qwen3.5-4B
Template is persisted in database and returned by API
vLLM backend receives correct --chat-template argument on model load
Chat completions via /v1/chat/completions use the correct template
Existing models without explicit template continue using "llama" (backward compatible)

Gemma 4 chat template: https://github.com/vllm-project/vllm/issues/13885
vLLM chat template docs: https://docs.vllm.ai/en/latest/serving/chat_template.html

## Problem Qwen3.5-4B (unsloth/Qwen3.5-4B-GGUF) uses a chat template that differs from the generic "llama" template currently applied to all models. The model supports: - **System role**: `<|im_start|>system\n...<|im_end|>\n` - - **User role**: `<|im_start|>user\n...<|im_end|>\n` - - **Assistant role**: `<|im_start|>assistant\n...<|im_end|>\n` - - **Thinking mode**: `<|im_start|>assistant\n<|thinking|>...<|/thinking|>\n<|im_end|>\n` - Currently, vllm_mm applies a generic "llama" template to all models, which produces incorrect prompt formatting for Qwen3.5-4B and likely degrades output quality. ## Evidence The model's GGUF metadata (extracted via `strings` on the Q4_K_M file) shows: ``` chat_template = "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}" ``` ## Proposed Solution Add a new `chat_template` field to the model configuration that allows per-model template selection: 1. **Backend**: Add `chat_template` column to `models` table (SQLite schema already supports this) 2. 2. **Frontend**: Add template selector in model add/edit dialog with options: 3. - `llama` (default, current behavior) - - `qwen3` (for Qwen3.5-4B and similar) - - `gemma4` (for Gemma 4 models — see related work) - 3. **API**: Include `chat_template` in `/api/models` and `/api/models/{id}` responses 4. 4. **Inference**: Pass selected template to vLLM via `--chat-template` argument or Jinja2 string 5. ## Acceptance Criteria - [ ] User can select "qwen3" template when adding/editing Qwen3.5-4B - [ ] - [ ] Template is persisted in database and returned by API - [ ] - [ ] vLLM backend receives correct `--chat-template` argument on model load - [ ] - [ ] Chat completions via `/v1/chat/completions` use the correct template - [ ] - [ ] Existing models without explicit template continue using "llama" (backward compatible) - [ ] ## Related - Gemma 4 chat template: https://github.com/vllm-project/vllm/issues/13885 - - vLLM chat template docs: https://docs.vllm.ai/en/latest/serving/chat_template.html