Multi-language and code-switched audio #14

New issue

Closed

opened 2026-09-01 20:40:24 +00:00 by Mark · 1 comment

Mark commented

2026-09-01 20:40:24 +00:00

Owner

Smoke-tested pure-Spanish and pure-English. What about:

Spanish + English mix (common in US podcasts, sports interviews)
English with technical terms / proper nouns / accents
Heavy accent (Indian English, Australian English)
Multiple languages in one batch (process Spanish + English files in same run)

What needs building:

Verify Whisper handles --lang auto gracefully when language shifts mid-file
Add a --language-hints en,es mode that biases detection but allows fallback
Document known limitations: WhisperX uses one alignment model per detected language — if a file switches, alignment may be off after the switch point
Test: find a real code-switched podcast episode (NPR's Planet Money has some, Lex has some), transcribe, verify quality at the switch points

Smoke-tested pure-Spanish and pure-English. What about: - Spanish + English mix (common in US podcasts, sports interviews) - English with technical terms / proper nouns / accents - Heavy accent (Indian English, Australian English) - Multiple languages in one batch (process Spanish + English files in same run) **What needs building:** - Verify Whisper handles `--lang auto` gracefully when language shifts mid-file - Add a `--language-hints en,es` mode that biases detection but allows fallback - Document known limitations: WhisperX uses one alignment model per detected language — if a file switches, alignment may be off after the switch point - Test: find a real code-switched podcast episode (NPR's `Planet Money` has some, Lex has some), transcribe, verify quality at the switch points

Mark added the

feature-request

triage-approved

labels

2026-09-01 20:41:35 +00:00

Mark

2026-09-05 17:32:46 +00:00

closed this issue
removed the
feature-request

triage-approved
labels

Mark referenced this issue from a commit

2026-09-05 18:05:44 +00:00

Issue #14: extend language-hint selection with mutual-exclusion and validation tests

Mark

2026-09-06 00:38:40 +00:00

reopened this issue
added the
triage-approved
label

Mark commented

2026-09-06 00:38:47 +00:00

Author

Owner

Re-opened: false-positive close.

This issue was closed on 2026-09-05T18:05:42Z without the work being finished. Audit against current origin/main (commit a189437):

Req 1 (--language-hints en,es with bias + fallback): DONE in commits 66b2b40 + 6197c98. Mechanism: pre-call model.detect_language(audio) -> probability map -> select_language_from_hints() picks highest-scoring hint above 0.50 floor, else falls back to unrestricted detection. Documented in src/whisper_transcribe/language_hints.py module docstring.
Req 2 (--lang auto graceful mid-file shifts + WARNING on detected-language change): PARTIAL. The CLI does call detect_language() once at start and falls back gracefully, but does NOT log a WARNING when the detected language changes mid-file (WhisperX itself does not expose mid-file detection -- fundamental limitation per the issue body). Need a diagnostic helper that re-runs detection at intervals OR documents the limitation explicitly.
Req 3 (directory batch mode, per-file independence): NOT IMPLEMENTED. The CLI accepts only a single source via resolve_source(). Need a directory path that walks the input dir and runs each file independently through the existing single-file pipeline.
Req 4 (README usage examples for --language-hints es,en and --lang auto): NOT UPDATED. Current README's Quickstart section does not mention either flag.

Re-adding triage-approved to route back through the pipeline. Workspace is currently dirty with pytest-tempdir residue + an uncommitted pre-dispatch gate update; wrapper will clean that up before dispatching kilo-gsd.

Re-opened: false-positive close. This issue was closed on 2026-09-05T18:05:42Z without the work being finished. Audit against current `origin/main` (commit a189437): - Req 1 (`--language-hints en,es` with bias + fallback): DONE in commits 66b2b40 + 6197c98. Mechanism: pre-call `model.detect_language(audio)` -> probability map -> `select_language_from_hints()` picks highest-scoring hint above 0.50 floor, else falls back to unrestricted detection. Documented in `src/whisper_transcribe/language_hints.py` module docstring. - Req 2 (`--lang auto` graceful mid-file shifts + WARNING on detected-language change): PARTIAL. The CLI does call `detect_language()` once at start and falls back gracefully, but does NOT log a WARNING when the detected language changes mid-file (WhisperX itself does not expose mid-file detection -- fundamental limitation per the issue body). Need a diagnostic helper that re-runs detection at intervals OR documents the limitation explicitly. - Req 3 (directory batch mode, per-file independence): NOT IMPLEMENTED. The CLI accepts only a single source via `resolve_source()`. Need a directory path that walks the input dir and runs each file independently through the existing single-file pipeline. - Req 4 (README usage examples for `--language-hints es,en` and `--lang auto`): NOT UPDATED. Current README's Quickstart section does not mention either flag. Re-adding `triage-approved` to route back through the pipeline. Workspace is currently dirty with pytest-tempdir residue + an uncommitted pre-dispatch gate update; wrapper will clean that up before dispatching kilo-gsd.

Mark referenced this issue from a commit

2026-09-06 00:39:10 +00:00

chore(repo): remove pytest tempdir residue + ignore future occurrences

Mark added the

in-progress

label

2026-09-06 00:39:26 +00:00

Mark referenced this issue from a commit

2026-09-06 00:49:18 +00:00

docs(whisper-transcribe): add multi-language limitations reference

Mark referenced this issue

2026-09-06 00:49:33 +00:00