GPU memory monitoring + graceful OOM #15

New issue

Closed

opened 2026-09-01 20:40:31 +00:00 by Mark · 2 comments

Mark commented

2026-09-01 20:40:31 +00:00

Owner

On a 6GB card, medium model (~5GB) + alignment model (~300MB) + audio buffer may OOM. Need:

Pre-flight check: estimate VRAM usage from model size + audio length, error early if it won't fit
Runtime monitoring: torch.cuda.memory_allocated() logged at each phase
Graceful degradation: if alignment OOMs, fall back to ASR-only and warn
--max-memory flag to hard-cap VRAM usage (sacrifices model size for safety)

Why: silent CUDA OOM crashes are worse than clean errors. Operators on unknown hardware need guidance.

On a 6GB card, `medium` model (~5GB) + alignment model (~300MB) + audio buffer may OOM. Need: - Pre-flight check: estimate VRAM usage from model size + audio length, error early if it won't fit - Runtime monitoring: `torch.cuda.memory_allocated()` logged at each phase - Graceful degradation: if alignment OOMs, fall back to ASR-only and warn - `--max-memory` flag to hard-cap VRAM usage (sacrifices model size for safety) **Why:** silent CUDA OOM crashes are worse than clean errors. Operators on unknown hardware need guidance.

Mark added the

feature-request

triage-approved

labels

2026-09-01 20:41:35 +00:00

Mark commented

2026-09-05 17:37:38 +00:00

Author

Owner

Launcher note (kilo-gsd-runner t_c9b94f5f):

Attempted dispatch for this issue failed at kilo-gsd.py internal preflight (ruff check . returned non-zero — 7 ruff errors against Mark/whisper-transcribe main).

The runner tried to open a cleanup tracking issue with labels lint-cleanup + blocked:preflight but Forgejo returned HTTP 422 (label names were passed as strings instead of per-repo integer label IDs). No cleanup issue was created on this repo. The runner exited before the kanban block took effect.

Workaround for operators / future runs:

Fix the 7 ruff errors on main (ruff check . --fix should clean most) and commit to main. Kilo-GSD will pick up the held tickets on the next dispatch.
Or: a runner patch is needed to map label names to per-repo IDs before POSTing (currently a known gap in the lint-cleanup branch of kilo-gsd.py).

Sibling whisper-transcribe runs #10/#12/#13/#14 are still in flight and have not yet hit the same wall.

Launcher note (kilo-gsd-runner t_c9b94f5f): Attempted dispatch for this issue failed at kilo-gsd.py internal preflight (`ruff check .` returned non-zero — 7 ruff errors against `Mark/whisper-transcribe` main). The runner tried to open a cleanup tracking issue with labels `lint-cleanup` + `blocked:preflight` but Forgejo returned HTTP 422 (label names were passed as strings instead of per-repo integer label IDs). No cleanup issue was created on this repo. The runner exited before the kanban block took effect. Workaround for operators / future runs: - Fix the 7 ruff errors on main (`ruff check . --fix` should clean most) and commit to `main`. Kilo-GSD will pick up the held tickets on the next dispatch. - Or: a runner patch is needed to map label names to per-repo IDs before POSTing (currently a known gap in the lint-cleanup branch of `kilo-gsd.py`). Sibling whisper-transcribe runs #10/#12/#13/#14 are still in flight and have not yet hit the same wall.

Mark commented

2026-09-06 05:33:10 +00:00

Author

Owner

kilo-gsd.py preflight rejected this issue (attempt 2026-09-06 05:30 UTC): ruff check . returns 133 errors against the working tree. The cause is sibling #6 (Mark/whisper-transcribe#6) mid-Implement on the same checkout, leaving dirty WIP files in src/whisper_transcribe/. Origin/main itself (commit 418c100) is clean — once #6 lands its commits, a fresh spawn will pass.

Note: kilo-gsd.py's cleanup-ticket path tries to POST a new issue with labels=[lint-cleanup, blocked:preflight], which 422s on this repo because those labels are not defined. Once the working tree is clean, a fresh python3 ~/.hermes/scripts/kilo-gsd.py --repo Mark/whisper-transcribe --issue 15 --skip-concurrency should succeed.

kilo-gsd.py preflight rejected this issue (attempt 2026-09-06 05:30 UTC): `ruff check .` returns 133 errors against the working tree. The cause is sibling #6 (Mark/whisper-transcribe#6) mid-Implement on the same checkout, leaving dirty WIP files in `src/whisper_transcribe/`. Origin/main itself (commit 418c100) is clean — once #6 lands its commits, a fresh spawn will pass. Note: kilo-gsd.py's cleanup-ticket path tries to POST a new issue with labels=[`lint-cleanup`, `blocked:preflight`], which 422s on this repo because those labels are not defined. Once the working tree is clean, a fresh `python3 ~/.hermes/scripts/kilo-gsd.py --repo Mark/whisper-transcribe --issue 15 --skip-concurrency` should succeed.

Mark

2026-09-06 05:54:10 +00:00