BUG: no embedding model concept — embeddings routed only to active chat model #38

New issue

Closed

opened 2026-09-05 07:35:30 +00:00 by Mark · 1 comment

Mark commented

2026-09-05 07:35:30 +00:00

Owner

Summary

vllm_mm has no concept of model type (chat vs embedding). The proxy's /v1/embeddings endpoint routes to whatever model is currently is_active=True, which is expected to be a chat model. There is no way to register, load, or route to a dedicated embedding model. llama.cpp's llama-server supports --embedding mode at startup for embedding-only models, but vllm_mm never wires this in.

The VRAM profiler already has a can_fit(required_gb) method intended to check if a secondary embedding model can fit alongside the chat model — but nothing in the request flow ever calls it.

What's missing

Model type field: models table (SQLite) has no model_type column or equivalent. All registered models are implicitly chat models.
Embedding-specific settings: llamacpp_backend.py build_subprocess_command does not include --embedding, --pooling, or any embedding-mode flags.
Endpoint routing: proxy.py:927-944 (/v1/embeddings) only looks at is_active=True. There's no concept of an "active embedding model" separate from the "active chat model."
Concurrent loading UI: Users cannot register an embedding-only model (e.g., nomic-embed-text-v1.5-Q8_0.gguf ~130MB) and have it stay loaded alongside the chat model. The UI's add-edit-model-dialog.tsx has no field for this.

What works today

POST /v1/embeddings works against whichever chat model is currently is_active=True, IF that model supports embeddings. llama-server can do this for embedding-capable chat models, but with degraded behavior since the model wasn't optimized for it.

Workaround

For now, the only embedding path is:

Register a chat model whose HF repo also has a GGUF in ggml-org (e.g., Qwen3-Embedding or BGE).
Start it in chat mode.
Hit /v1/embeddings against the active model.

This doesn't satisfy the use case of having a small dedicated embedding model (e.g., nomic-embed, BGE-small) loaded alongside a chat model on the same GPU.

Why it matters

RAG pipelines need embeddings, not chat completions.
A small dedicated embedding model (130-300MB) running in parallel with a chat model is a common pattern.
On P4 (7.6GB VRAM), running a 4B chat model + small embedding model fits comfortably and avoids the cost of switching models for every RAG query.

Files

backend/app/services/vram_profiler.py:64-87 — can_fit exists, unused
backend/app/routers/proxy.py:927-944 — embeddings endpoint, single-model
backend/app/services/backends/llamacpp_backend.py — build_subprocess_command, no --embedding flag
backend/app/services/process_manager.py — port allocator (_find_free_port supports concurrent), but no model-type discrimination
src/components/dialogs/add-edit-model-dialog.tsx — UI, no model type field
src/lib/validations/model.ts — schema, no model type field

Issue #35 (llama.cpp backend start endpoint never spawns llama-server) — embedding work depends on this being fixed first.
Issue #37 (proxy layer does not support multimodal/vision requests) — same pattern, different modality.

## Summary vllm_mm has no concept of model type (chat vs embedding). The proxy's `/v1/embeddings` endpoint routes to whatever model is currently `is_active=True`, which is expected to be a chat model. There is no way to register, load, or route to a dedicated embedding model. llama.cpp's llama-server supports `--embedding` mode at startup for embedding-only models, but vllm_mm never wires this in. The VRAM profiler already has a `can_fit(required_gb)` method intended to check if a secondary embedding model can fit alongside the chat model — but nothing in the request flow ever calls it. ## What's missing 1. **Model type field**: `models` table (SQLite) has no `model_type` column or equivalent. All registered models are implicitly chat models. 2. **Embedding-specific settings**: `llamacpp_backend.py` `build_subprocess_command` does not include `--embedding`, `--pooling`, or any embedding-mode flags. 3. **Endpoint routing**: `proxy.py:927-944` (`/v1/embeddings`) only looks at `is_active=True`. There's no concept of an "active embedding model" separate from the "active chat model." 4. **Concurrent loading UI**: Users cannot register an embedding-only model (e.g., `nomic-embed-text-v1.5-Q8_0.gguf` ~130MB) and have it stay loaded alongside the chat model. The UI's `add-edit-model-dialog.tsx` has no field for this. ## What works today `POST /v1/embeddings` works against whichever chat model is currently `is_active=True`, IF that model supports embeddings. llama-server can do this for embedding-capable chat models, but with degraded behavior since the model wasn't optimized for it. ## Workaround For now, the only embedding path is: 1. Register a chat model whose HF repo also has a GGUF in `ggml-org` (e.g., Qwen3-Embedding or BGE). 2. Start it in chat mode. 3. Hit `/v1/embeddings` against the active model. This doesn't satisfy the use case of having a small dedicated embedding model (e.g., nomic-embed, BGE-small) loaded alongside a chat model on the same GPU. ## Why it matters - RAG pipelines need embeddings, not chat completions. - A small dedicated embedding model (130-300MB) running in parallel with a chat model is a common pattern. - On P4 (7.6GB VRAM), running a 4B chat model + small embedding model fits comfortably and avoids the cost of switching models for every RAG query. ## Files - `backend/app/services/vram_profiler.py:64-87` — `can_fit` exists, unused - `backend/app/routers/proxy.py:927-944` — embeddings endpoint, single-model - `backend/app/services/backends/llamacpp_backend.py` — `build_subprocess_command`, no `--embedding` flag - `backend/app/services/process_manager.py` — port allocator (`_find_free_port` supports concurrent), but no model-type discrimination - `src/components/dialogs/add-edit-model-dialog.tsx` — UI, no model type field - `src/lib/validations/model.ts` — schema, no model type field ## Related - Issue #35 (llama.cpp backend start endpoint never spawns llama-server) — embedding work depends on this being fixed first. - Issue #37 (proxy layer does not support multimodal/vision requests) — same pattern, different modality.

Mark added the

triage-approved

label

2026-09-14 14:04:18 +00:00

Mark removed the

triage-approved

label

2026-09-18 02:27:04 +00:00

Mark referenced this issue from a commit

2026-09-18 03:12:55 +00:00

Fix #38: introduce embedding model concept — dedicated /v1/embeddings routing

Mark closed this issue

2026-09-18 03:12:55 +00:00

Mark commented

2026-09-18 03:14:09 +00:00

Author

Owner

Closing as fixed by commit ea16ebf9 ("Fix #38: introduce embedding model concept — dedicated /v1/embeddings routing"), now on origin/main.

The fix introduces a model_type enum (chat | embedding) on the models table with a back-compat migration (backend/migrations/add_model_type.py), wires --embedding and --pooling flags into LlamaCppBackend.build_subprocess_command, routes /v1/embeddings to prefer a dedicated embedding model with chat-model fallback, and exposes a Model Type selector in the add/edit model dialog. Backend tests pass (96 passed, 2 skipped across test_backends.py / test_model_repository.py / test_proxy.py).

Note: the repo has 77 pre-existing ruff errors at HEAD (86 on origin/main before this fix) — none introduced by this commit (the diff net-fixes 9 ruff violations). The kilo-gsd Lint gate tripped on the baseline dirt, not on this change. Filed as Mark/vllm_mm#43 (lint cleanup) so future feature dispatches land cleanly.

Closing as fixed by commit ea16ebf9 ("Fix #38: introduce embedding model concept — dedicated /v1/embeddings routing"), now on origin/main. The fix introduces a `model_type` enum (chat | embedding) on the `models` table with a back-compat migration (`backend/migrations/add_model_type.py`), wires `--embedding` and `--pooling` flags into `LlamaCppBackend.build_subprocess_command`, routes `/v1/embeddings` to prefer a dedicated embedding model with chat-model fallback, and exposes a Model Type selector in the add/edit model dialog. Backend tests pass (96 passed, 2 skipped across test_backends.py / test_model_repository.py / test_proxy.py). Note: the repo has 77 pre-existing ruff errors at HEAD (86 on origin/main before this fix) — none introduced by this commit (the diff net-fixes 9 ruff violations). The kilo-gsd Lint gate tripped on the baseline dirt, not on this change. Filed as Mark/vllm_mm#43 (lint cleanup) so future feature dispatches land cleanly.

Mark referenced this issue

2026-09-18 03:15:55 +00:00

Fix the lint on vllm_mm #43