Mark/vllm_mm

Fork 0

Bug: Subprocess logs not captured - cannot debug model start failures #8

New issue

Closed

opened 2026-05-01 00:39:06 +00:00 by Mark · 1 comment

Mark commented

2026-05-01 00:39:06 +00:00

Owner

Bug: Subprocess logs not captured - can't debug model start failures

Description:
After refactoring to use subprocess-only architecture (no Docker containers for model serving), error debugging is broken. When models fail to start, container logs only show:

llama.cpp not ready after 600s
Worker 0 error: invalid state

The actual subprocess stdout/stderr is never captured or exposed. This makes it impossible to debug why models fail to start.

Steps to reproduce:

Start any model (llama.cpp GGUF or vLLM) via POST /api/models/{id}/start
Endpoint times out after ~60s
Check /api/models/{id}/status shows stuck at 80% "Waiting for server to be ready..."
Container logs only show "not ready" message, not WHY

Expected:

Subprocess stdout/stderr captured to accessible log file
Errors printed to container logs with stack traces
Ability to see what's actually happening inside spawned process

Impact:

Cannot test or use the refactored subprocess-only architecture
Both llama.cpp and vLLM backends appear broken (but we can't see why)

Additional context:

Backend is running in Podman container at port 11434
This is blocking all model serving after the Docker container path was removed

**Bug:** Subprocess logs not captured - can't debug model start failures **Description:** After refactoring to use subprocess-only architecture (no Docker containers for model serving), error debugging is broken. When models fail to start, container logs only show: - `llama.cpp not ready after 600s` - - `Worker 0 error: invalid state` - The actual subprocess stdout/stderr is never captured or exposed. This makes it impossible to debug why models fail to start. **Steps to reproduce:** 1. Start any model (llama.cpp GGUF or vLLM) via POST /api/models/{id}/start 2. 2. Endpoint times out after ~60s 3. 3. Check /api/models/{id}/status shows stuck at 80% "Waiting for server to be ready..." 4. 4. Container logs only show "not ready" message, not WHY 5. **Expected:** - Subprocess stdout/stderr captured to accessible log file - - Errors printed to container logs with stack traces - - Ability to see what's actually happening inside spawned process - **Impact:** - Cannot test or use the refactored subprocess-only architecture - - Both llama.cpp and vLLM backends appear broken (but we can't see why) - **Additional context:** - Backend is running in Podman container at port 11434 - - This is blocking all model serving after the Docker container path was removed

Mark added the

feature-request

label

2026-05-12 23:26:52 +00:00

Mark added

quickfix

and removed

feature-request

labels

2026-05-20 23:39:43 +00:00

Mark removed the

quickfix

label

2026-05-27 05:19:18 +00:00

Mark added the

blocked:needs-decision

label

2026-05-28 06:45:52 +00:00

Mark commented

2026-05-28 06:45:52 +00:00

Author

Owner

⏸ Needs Decision

Blocking factors:

Didn't meet auto-approve criteria — scope=small, risk=medium, spec=aligned

Scope: small Risk: medium Spec: aligned Type: bug
Summary: Fix subprocess stdout/stderr capture in process_manager.py so model start failures produce debuggable logs instead of silent timeouts
Suggestion: Redirect subprocess.Popen stdout/stderr to timestamped log files in a dedicated directory, expose log file paths via the model status API, and add a GET /api/models/{id}/logs endpoint to fetch subprocess output
Rationale: Small scope: the fix is concentrated in process_manager.py (log redirection) plus a lightweight log-serving endpoint and a status response field. Medium risk because it touches the core process lifecycle code where any regression would silently break model serving. Fully aligned with the project's subprocess-only architecture and the explicit AGENTS.md Process Management docstring.

Action: Review each blocking factor above. Remove blocked:needs-decision label to route back to triage.

## ⏸ Needs Decision **Blocking factors:** - **Didn't meet auto-approve criteria** — scope=small, risk=medium, spec=aligned --- **Scope:** small **Risk:** medium **Spec:** aligned **Type:** bug **Summary:** Fix subprocess stdout/stderr capture in process_manager.py so model start failures produce debuggable logs instead of silent timeouts **Suggestion:** Redirect subprocess.Popen stdout/stderr to timestamped log files in a dedicated directory, expose log file paths via the model status API, and add a GET /api/models/{id}/logs endpoint to fetch subprocess output **Rationale:** Small scope: the fix is concentrated in process_manager.py (log redirection) plus a lightweight log-serving endpoint and a status response field. Medium risk because it touches the core process lifecycle code where any regression would silently break model serving. Fully aligned with the project's subprocess-only architecture and the explicit AGENTS.md Process Management docstring. **Action:** Review each blocking factor above. Remove `blocked:needs-decision` label to route back to triage.

Mark added

triage-approved

bug

in-progress

and removed

blocked:needs-decision

labels

2026-05-28 07:18:54 +00:00

Mark removed the

triage-approved

bug

in-progress