BUG: llama.cpp backend start endpoint never spawns llama-server #35

New issue

Closed

opened 2026-09-04 23:39:04 +00:00 by Mark · 0 comments

Mark commented

2026-09-04 23:39:04 +00:00

Owner

Summary

POST /api/models/{id}/start for llama.cpp backend models never spawns llama-server. The endpoint sets status to loading and the model sits there forever until the 600s timeout fires. The UI's Start button is therefore non-functional for any llama.cpp model registered on this backend.

Reproduction

Register a llama.cpp model via POST /api/models (e.g., unsloth/Qwen3.5-4B-GGUF with ggufFile: Qwen3.5-4B-Q4_K_M.gguf, backend: llama.cpp).
POST /api/models/{id}/start with {} body.
Observe: backend logs show HEAD https://huggingface.co/... + GET ...xet-read-token/... succeed (file verified).
Then: silence. No llama-server process is spawned. No error.
GET /api/models/{id} returns status: "loading" indefinitely.
ps aux | grep llama-server inside the backend container shows zero processes.

Expected

After the HEAD verifies the GGUF, the backend should spawn llama-server with the appropriate flags (-m, --host 0.0.0.0, --port <next free>, --n-gpu-layers, --cache-type-k, --cache-type-v, -c, -b, --threads) and the model record transitions to running once the health check responds.

Actual

The HEAD request verifies the file. No subprocess is created. No error is returned to the caller. Status stuck on loading.

Workaround

Manually start llama-server inside the backend container (e.g., podman exec -d vllm_mm_backend_1 setsid llama-server -m ... --host 0.0.0.0 --port 18000 -c 8192 -b 256 --threads 8 --n-gpu-layers 999 --cache-type-k q4_0 --cache-type-v q4_0 ...). Inference works at http://localhost:18000/v1/chat/completions directly, bypassing the backend API entirely.

Environment

Host: unstable (Tesla P4, Pascal sm_61)
Backend image: localhost/vllm_mm_backend:latest
Compose: Dockerfile (single-Dockerfile path, not Dockerfile.cuda12 / Dockerfile.pascal)
Models tested: unsloth/Qwen3.5-4B-GGUF, unsloth/gemma-4-E4B-it-GGUF — same bug for both

Notes

The vLLM path has a separate, known bug (CUDA 13 dropped Pascal — vllm not ready after 600s). This issue is specifically about the llama.cpp start path not spawning the subprocess at all.

## Summary `POST /api/models/{id}/start` for `llama.cpp` backend models never spawns `llama-server`. The endpoint sets status to `loading` and the model sits there forever until the 600s timeout fires. The UI's Start button is therefore non-functional for any llama.cpp model registered on this backend. ## Reproduction 1. Register a llama.cpp model via `POST /api/models` (e.g., `unsloth/Qwen3.5-4B-GGUF` with `ggufFile: Qwen3.5-4B-Q4_K_M.gguf`, `backend: llama.cpp`). 2. `POST /api/models/{id}/start` with `{}` body. 3. Observe: backend logs show `HEAD https://huggingface.co/...` + `GET ...xet-read-token/...` succeed (file verified). 4. Then: silence. No `llama-server` process is spawned. No error. 5. `GET /api/models/{id}` returns `status: "loading"` indefinitely. 6. `ps aux | grep llama-server` inside the backend container shows zero processes. ## Expected After the HEAD verifies the GGUF, the backend should spawn `llama-server` with the appropriate flags (`-m`, `--host 0.0.0.0`, `--port <next free>`, `--n-gpu-layers`, `--cache-type-k`, `--cache-type-v`, `-c`, `-b`, `--threads`) and the model record transitions to `running` once the health check responds. ## Actual The HEAD request verifies the file. No subprocess is created. No error is returned to the caller. Status stuck on `loading`. ## Workaround Manually start `llama-server` inside the backend container (e.g., `podman exec -d vllm_mm_backend_1 setsid llama-server -m ... --host 0.0.0.0 --port 18000 -c 8192 -b 256 --threads 8 --n-gpu-layers 999 --cache-type-k q4_0 --cache-type-v q4_0 ...`). Inference works at `http://localhost:18000/v1/chat/completions` directly, bypassing the backend API entirely. ## Environment - Host: unstable (Tesla P4, Pascal sm_61) - Backend image: `localhost/vllm_mm_backend:latest` - Compose: `Dockerfile` (single-Dockerfile path, not `Dockerfile.cuda12` / `Dockerfile.pascal`) - Models tested: `unsloth/Qwen3.5-4B-GGUF`, `unsloth/gemma-4-E4B-it-GGUF` — same bug for both ## Notes The vLLM path has a separate, known bug (CUDA 13 dropped Pascal — `vllm not ready after 600s`). This issue is specifically about the `llama.cpp` start path not spawning the subprocess at all.