vllm_MM

A multi-backend model management UI and API for running LLMs locally via vLLM and llama.cpp. Built to make switching between inference engines, managing model configs, and monitoring runtime status actually usable.

Supports both GPU-accelerated inference (vLLM, llama.cpp CUDA) and CPU-only fallbacks.

---

Features

• Dual backends: vLLM for throughput, llama.cpp for flexibility (GGUF, speculative decoding, CPU offload)
• Web UI: Add, edit, start, stop, and monitor models from a browser
• OpenAI-compatible API: /v1/chat/completions, /v1/models — drop-in replacement
• Real-time metrics: Token/s, VRAM usage, progress streaming
• Model presets: One-click configurations for common setups
• TurboQuant integration: 3-bit KV cache compression for fitting larger models in limited VRAM

---

Quick Start

# Clone
git clone <repo-url> && cd vllm_MM

# Build & start both frontend and backend
podman-compose up -d

# Frontend: http://localhost:5300
# API:      http://localhost:8020

Requirements

• NVIDIA GPU with CUDA 12.4+ (for GPU inference)
• Podman + podman-compose (Docker also works)
• 12GB+ VRAM recommended for 4B-9B models at decent context
  • With TurboQuant: 26B models fit in 12GB at 32k context

---

TurboQuant: 3-Bit KV Cache

This fork integrates AmesianX/TurboQuant into llama.cpp, enabling 3-bit KV cache compression (~3.06 bpw vs FP16's 16 bpw). This is a 5x reduction in KV cache memory.

Why this matters

A 26B parameter model with 32k context normally needs:

• Model weights: ~10 GB
• KV cache (FP16): ~4-6 GB
• Total: 14-16 GB — doesn't fit in 12 GB VRAM

With TurboQuant tbq3:

• Model weights: ~10 GB
• KV cache (tbq3): ~120 MB
• Total: ~11 GB — fits comfortably

Supported cache types

Type	Bits/weight	Notes
f16	16	Default. No compression.
q8_0	8	Standard quantization
tbq3	~3.06	Most aggressive. Base TurboQuant 3-bit.
tbq3_1	~3.12	128-block variant, slightly better accuracy
tbq4	~4.06	TurboQuant 4-bit (less compression, higher quality)
amx3	~3.12	AMX-optimized 3-bit (Intel-specific)

The UI exposes cache_type_k and cache_type_v in the model edit dialog.

Caveats

• SWA (Sliding Window Attention) layers stay at FP16. Some models (Gemma-4, Qwen3) use SWA for a subset of layers. These caches cannot be compressed and still use full precision. For Gemma-4-26B this adds ~850 MB regardless of context size.
• VRAM savings scale with context size. At 4k context, TurboQuant saves ~100 MB. At 32k context, it saves ~4+ GB.

---

Configuration

Environment Variables (backend)

Variable	Default	Description
DATABASE_URL	sqlite+aiosqlite:///./vllm_models.db	Model metadata DB
MODEL_DATA_PATH	/models	Where GGUFs/HF repos are stored
MODEL_BASE_PORT	18000	First port for model processes
MODEL_MAX_PORT	19000	Last port for model processes
NVIDIA_VISIBLE_DEVICES	all	GPU visibility

Default settings (new models)

Backend	Setting	Default
llama.cpp	contextSize	16384
llama.cpp	batchSize	512
llama.cpp	nGpuLayers	-1 (all GPU)
vLLM	maxModelLen	32768
vLLM	gpuMemoryUtilization	0.9

---

Project Structure

├── backend/
│   ├── Dockerfile              # Multi-stage: builds TurboQuant llama.cpp
│   ├── app/
│   │   ├── models/schemas.py   # Pydantic models, presets
│   │   ├── services/backends/  # vLLM & llama.cpp process managers
│   │   └── routers/            # API endpoints, proxy
│   └── vllm_models.db          # SQLite DB (persisted)
├── src/
│   ├── components/dialogs/     # Add/edit model UI
│   ├── lib/validations/model.ts # Zod schemas (cache types, etc.)
│   └── types/vllm.ts           # Frontend type definitions
├── docker-compose.yml
└── README.md

Backend Dockerfile

The backend image compiles llama.cpp from AmesianX/TurboQuant instead of upstream ggml-org/llama.cpp:

# Stage 1: Build llama.cpp with TurboQuant + CUDA
RUN git clone --depth 1 https://github.com/AmesianX/TurboQuant.git /tmp/llama.cpp && \
    cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 && \
    cmake --build build --config Release -j$(nproc)

This is a long build (~60+ minutes on a 4-core machine) due to CUDA template instantiations for all TurboQuant variants.

---

API

Proxy (OpenAI-compatible)

curl http://localhost:8020/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Gemma-4-26B-A4B-IQ2_M",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }'

Management

# List models
curl http://localhost:8020/api/models

# Start/stop a model
curl -X POST http://localhost:8020/api/models/{id}/start
curl -X POST http://localhost:8020/api/models/{id}/stop

# Get logs
curl http://localhost:8020/api/models/{id}/logs

# Get status
curl http://localhost:8020/api/models/{id}/status

---

Known Issues

• docker-compose v1.29.2 is broken — use podman-compose instead
• Gemma-4 models require llama.cpp ≥ fd6ae4c (TurboQuant build is well past this)
• Batch size affects compute buffer memory — large models at high context may need batchSize: 64 instead of 512 to avoid CUDA OOM during graph allocation
• Reasoning starvation — Models with chain-of-thought reasoning can consume the entire token budget on low max_tokens settings. See MODEL_EVALUATION.md

Model Evaluation & Smart Routing

See MODEL_EVALUATION.md for:

• Quality benchmarks across Gemma-4-26B, GPT-OSS-20B, and Qwen3.6-35B
• The "reasoning starvation" issue and mitigation
• Active configuration for Hermes smart routing
• Decision log for selecting GPT-OSS-20B as the routing model

---

License

Same as upstream llama.cpp (MIT)