BUG: UI has no field for mmproj/vision model configuration #36

New issue

Closed

opened 2026-09-04 23:39:05 +00:00 by Mark · 1 comment

Mark commented

2026-09-04 23:39:05 +00:00

Owner

Summary

The frontend UI (http://localhost:5173, Vite dev server, "VLLM Model Manager") has no UI affordance for multimodal (vision) model configuration. There is no field in the add/edit-model dialog for mmproj (vision projector) GGUF files, even though the backend already supports mmprojFile settings and llama.cpp requires --mmproj for vision inference.

What works

Multimodal inference works only when bypassing both the UI and the backend proxy: directly POSTing OpenAI-format requests with content: [{type:"text",...}, {type:"image_url", image_url:{url:"data:image/png;base64,..."}}] to a manually-spawned llama-server listening on a known port (e.g., 18000) on the host.

Verified with:

Model: unsloth/gemma-4-E4B-it-GGUF (Q4_K_M) + mmproj-F16.gguf (990 MB)
Request: 64x64 red PNG base64 + "What color is this image? One short answer."
Response: "Red" (2 completion tokens, 284 prompt tokens)

Same setup confirmed working for unsloth/Qwen3.5-4B-GGUF (Q4_K_M, 2.74 GB) + mmproj-F16.gguf (672 MB).

What's missing

Frontend form: src/components/dialogs/add-edit-model-dialog.tsx does not expose mmprojFile. Users cannot register multimodal models through the UI even though the backend API supports it.
Validation schema: src/lib/validations/model.ts does not include mmprojFile in the llama.cpp settings interface.
API client: src/lib/api.ts LlamaCppSettingsApi interface omits mmproj_file.

Workaround

Manually register models via curl: POST /api/models with settings.mmprojFile: "mmproj-F16.gguf". Backend will record the path but, per the separate bug filed against the llama.cpp start handler, the start endpoint currently does not pass --mmproj to llama-server — so even with the setting saved, it would not be applied at start time.

Expected

Add mmprojFile field to the llama.cpp model registration form. On submit, include it in the settings object of the POST/PUT body. After start, the backend's llama.cpp subprocess command must include --mmproj <path> so vision inference actually works.

Files

src/components/dialogs/add-edit-model-dialog.tsx (form)
src/lib/validations/model.ts (schema)
src/lib/api.ts (LlamaCppSettingsApi)
Backend: backend/app/services/backends/llamacpp_backend.py (subprocess command builder)

## Summary The frontend UI (`http://localhost:5173`, Vite dev server, "VLLM Model Manager") has no UI affordance for multimodal (vision) model configuration. There is no field in the add/edit-model dialog for `mmproj` (vision projector) GGUF files, even though the backend already supports `mmprojFile` settings and llama.cpp requires `--mmproj` for vision inference. ## What works Multimodal inference works **only** when bypassing both the UI and the backend proxy: directly POSTing OpenAI-format requests with `content: [{type:"text",...}, {type:"image_url", image_url:{url:"data:image/png;base64,..."}}]` to a manually-spawned `llama-server` listening on a known port (e.g., 18000) on the host. Verified with: - Model: `unsloth/gemma-4-E4B-it-GGUF` (Q4_K_M) + `mmproj-F16.gguf` (990 MB) - Request: 64x64 red PNG base64 + "What color is this image? One short answer." - Response: `"Red"` (2 completion tokens, 284 prompt tokens) Same setup confirmed working for `unsloth/Qwen3.5-4B-GGUF` (Q4_K_M, 2.74 GB) + `mmproj-F16.gguf` (672 MB). ## What's missing 1. **Frontend form**: `src/components/dialogs/add-edit-model-dialog.tsx` does not expose `mmprojFile`. Users cannot register multimodal models through the UI even though the backend API supports it. 2. **Validation schema**: `src/lib/validations/model.ts` does not include `mmprojFile` in the llama.cpp settings interface. 3. **API client**: `src/lib/api.ts` `LlamaCppSettingsApi` interface omits `mmproj_file`. ## Workaround Manually register models via curl: `POST /api/models` with `settings.mmprojFile: "mmproj-F16.gguf"`. Backend will record the path but, per the separate bug filed against the llama.cpp start handler, the start endpoint currently does not pass `--mmproj` to llama-server — so even with the setting saved, it would not be applied at start time. ## Expected Add `mmprojFile` field to the llama.cpp model registration form. On submit, include it in the `settings` object of the POST/PUT body. After start, the backend's llama.cpp subprocess command must include `--mmproj <path>` so vision inference actually works. ## Files - `src/components/dialogs/add-edit-model-dialog.tsx` (form) - `src/lib/validations/model.ts` (schema) - `src/lib/api.ts` (`LlamaCppSettingsApi`) - Backend: `backend/app/services/backends/llamacpp_backend.py` (subprocess command builder)

Mark added the

triage-approved

label

2026-09-14 14:04:18 +00:00

Mark commented

2026-09-18 02:08:26 +00:00

Author

Owner

Verified on main (HEAD 9aa91ec) — the UI/backend chain this issue describes is already wired end-to-end.

Frontend (no changes needed):

src/components/dialogs/add-edit-model-dialog.tsx exposes a "Multimodal (MMproj)" toggle that reveals mmprojPath and mmprojSha256 inputs (lines 647–696).
src/lib/validations/model.ts includes both as optional top-level fields on hfModelCreateSchema (lines 126–128).
src/types/vllm.ts carries mmprojPath? on LlamaCppSettings (line 78) and on HFModel (lines 185–186).
src/lib/api.ts LlamaCppSettingsApi includes mmproj_path?: string | null (line 48).

Wire path (create + update):

src/hooks/useModels.ts:168 includes mmproj_path in the settings payload sent to POST /api/models.
src/hooks/useModels.ts:217–218, 253–254 forwards top-level mmproj_path/mmproj_sha256 alongside multimodal.

Backend (no changes needed):

backend/app/services/backends/llamacpp_backend.py:51 maps mmprojPath → mmproj_path, and lines 174–178 emit --mmproj {data_path}/gguf/{mmproj_path} to the llama-server subprocess.
backend/app/database.py:50–51, 104–105 and backend/app/repositories/model_repository.py:94–95, 120–123 persist mmproj_path/mmproj_sha256 on the model record.

Verification: backend pytest 221 passed / 2 skipped, frontend vitest 77 passed, tsc --noEmit clean.

This work was originally shipped in commits 2d36127 (M005/S01+S02: Add multimodal fields to schemas/db/types/frontend + --mmproj launch flag), 504d8d5 (#13 #23 persist multimodal fields on create/update), 1e410f4 and 8251fb1 (regression tests), and 7636dae (#37 proxy passes multimodal through to backend vision handler).

Closing as completed; the bug is fixed on main. Reopen if a real-world regression appears.

Verified on `main` (HEAD `9aa91ec`) — the UI/backend chain this issue describes is already wired end-to-end. **Frontend (no changes needed):** - `src/components/dialogs/add-edit-model-dialog.tsx` exposes a "Multimodal (MMproj)" toggle that reveals `mmprojPath` and `mmprojSha256` inputs (lines 647–696). - `src/lib/validations/model.ts` includes both as optional top-level fields on `hfModelCreateSchema` (lines 126–128). - `src/types/vllm.ts` carries `mmprojPath?` on `LlamaCppSettings` (line 78) and on `HFModel` (lines 185–186). - `src/lib/api.ts` `LlamaCppSettingsApi` includes `mmproj_path?: string | null` (line 48). **Wire path (create + update):** - `src/hooks/useModels.ts:168` includes `mmproj_path` in the settings payload sent to `POST /api/models`. - `src/hooks/useModels.ts:217–218, 253–254` forwards top-level `mmproj_path`/`mmproj_sha256` alongside `multimodal`. **Backend (no changes needed):** - `backend/app/services/backends/llamacpp_backend.py:51` maps `mmprojPath` → `mmproj_path`, and lines 174–178 emit `--mmproj {data_path}/gguf/{mmproj_path}` to the `llama-server` subprocess. - `backend/app/database.py:50–51, 104–105` and `backend/app/repositories/model_repository.py:94–95, 120–123` persist `mmproj_path`/`mmproj_sha256` on the model record. **Verification:** backend `pytest` 221 passed / 2 skipped, frontend `vitest` 77 passed, `tsc --noEmit` clean. This work was originally shipped in commits `2d36127` (M005/S01+S02: Add multimodal fields to schemas/db/types/frontend + `--mmproj` launch flag), `504d8d5` (`#13 #23` persist multimodal fields on create/update), `1e410f4` and `8251fb1` (regression tests), and `7636dae` (`#37` proxy passes multimodal through to backend vision handler). Closing as completed; the bug is fixed on `main`. Reopen if a real-world regression appears.

Mark closed this issue

2026-09-18 02:08:26 +00:00