Mark/whisper-transcribe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Resume from checkpoint on long runs #16

New issue
Closed
opened 2026-09-01 20:40:31 +00:00 by Mark · 3 comments
Mark commented 2026-09-01 20:40:31 +00:00
Owner
Copy link

Lex 3.4h took 60s. But what about a 10h file? If the run crashes at hour 7, we don't want to re-transcribe hours 0-7.

What needs building:

  • Chunk-based progress file (transcribe.state.json) listing which chunks are done
  • On startup, check state file, skip done chunks
  • --resume flag (default ON if state file exists)
  • Cleanup of stale state files (--fresh to ignore and restart)

Tied to: issue #8 (long-file streaming). Both should land together.

Lex 3.4h took 60s. But what about a 10h file? If the run crashes at hour 7, we don't want to re-transcribe hours 0-7. **What needs building:** - Chunk-based progress file (`transcribe.state.json`) listing which chunks are done - On startup, check state file, skip done chunks - `--resume` flag (default ON if state file exists) - Cleanup of stale state files (`--fresh` to ignore and restart) **Tied to:** issue #8 (long-file streaming). Both should land together.
Mark commented 2026-09-06 06:15:48 +00:00
Author
Owner
Copy link

Schema spec for transcribe.state.json landed in docs/state-file.md on branch docs/state-file-schema (commit 217cb08). Branch is local-only; push + PR are the review checkpoint.

The spec covers:

  • Field list (§2.1 top-level, §2.2 per-chunk) — chunk index, start/end offsets, model hash, file mtime, plus per-chunk status state machine (pending → in_progress → done / failed), started_at / completed_at, output_json_path, and segment_count / word_count diagnostics.
  • Model hash (§3) — SHA-256 of the on-disk weights file; resume is rejected on mismatch so re-downloaded snapshots can't silently change output.
  • Atomic write semantics (§4) — write-temp-then-rename with os.fsync and os.replace, same-filesystem temp via tempfile.mkstemp(dir=target.parent), plus per-chunk JSON dump order (chunk JSON first, state rewrite second) so a crash mid-run never leaves a half-written state file.
  • Stale-state detection (§5) — three hard rejects (schema version, source identity + mtime, model + chunk-geometry match) plus two soft warnings (state mtime older than --stale-hours, default 24; in_progress chunk older than 2h is reset to pending and retried).
  • Proposed CLI flags (§6) — --resume / --no-resume (auto-default by state-file presence), --fresh, --stale-hours, --workdir.
  • Acceptance (§7) — explicit list the implementation task (kanban t_9b77a683) builds against.

Field list is stable enough to build against without further design changes; changes after acceptance require bumping schema_version with a migration note.

Schema spec for `transcribe.state.json` landed in [`docs/state-file.md`](https://forgejo.tail018ac4.ts.net/Mark/whisper-transcribe/src/branch/main/docs/state-file.md) on branch `docs/state-file-schema` (commit `217cb08`). Branch is local-only; push + PR are the review checkpoint. The spec covers: - **Field list** (§2.1 top-level, §2.2 per-chunk) — chunk index, start/end offsets, model hash, file mtime, plus per-chunk `status` state machine (`pending` → `in_progress` → `done` / `failed`), `started_at` / `completed_at`, `output_json_path`, and `segment_count` / `word_count` diagnostics. - **Model hash** (§3) — SHA-256 of the on-disk weights file; resume is rejected on mismatch so re-downloaded snapshots can't silently change output. - **Atomic write semantics** (§4) — write-temp-then-rename with `os.fsync` and `os.replace`, same-filesystem temp via `tempfile.mkstemp(dir=target.parent)`, plus per-chunk JSON dump order (chunk JSON first, state rewrite second) so a crash mid-run never leaves a half-written state file. - **Stale-state detection** (§5) — three hard rejects (schema version, source identity + mtime, model + chunk-geometry match) plus two soft warnings (state mtime older than `--stale-hours`, default 24; `in_progress` chunk older than 2h is reset to `pending` and retried). - **Proposed CLI flags** (§6) — `--resume` / `--no-resume` (auto-default by state-file presence), `--fresh`, `--stale-hours`, `--workdir`. - **Acceptance** (§7) — explicit list the implementation task (kanban t_9b77a683) builds against. Field list is stable enough to build against without further design changes; changes after acceptance require bumping `schema_version` with a migration note.
Mark commented 2026-09-06 06:22:19 +00:00
Author
Owner
Copy link

Additive scope filed as #19 (Stale-state cleanup + 7-day threshold + smoke tests) — please absorb into this PR if the Plan phase has not yet finalized, or land as a follow-up if it has.

Additions over the current spec at docs/state-file.md (commit 217cb08):

  1. --stale-hours default changes from 24h -> 168h (7 days). Update §5.4 + §6 in the spec doc, and the CLI default.
  2. Source-audio mtime newer than state-file mtime is a soft warning + --fresh gate, NOT a hard StaleStateError reject. Refactor validate_state() to either return a result object or raise a distinct SourceMtimeChangedWarning. Spec §5.1/§5.2 needs to match.
  3. New --clean-stale flag with --dry-run / --yes / --workdir semantics (full spec in #19 §3).
  4. New tests/test_resume.py covering 4 scenarios (interrupted run, resume, modified source, --fresh) — full spec in #19 §4.

The four scenarios in (4) line up exactly with the kanban task t_e5ac8155 acceptance criteria, so the test file doubles as the verification artifact for the parent decomposer (t_d5c5d159).

If this lands in the same PR as #16, the PR description should reference both #16 and #19. If it lands as a follow-up PR, base off the #16 PR branch, not main.

Additive scope filed as **#19** (Stale-state cleanup + 7-day threshold + smoke tests) — please absorb into this PR if the Plan phase has not yet finalized, or land as a follow-up if it has. Additions over the current spec at docs/state-file.md (commit 217cb08): 1. `--stale-hours` default changes from **24h -> 168h (7 days)**. Update §5.4 + §6 in the spec doc, and the CLI default. 2. **Source-audio mtime newer than state-file mtime** is a soft warning + `--fresh` gate, NOT a hard `StaleStateError` reject. Refactor `validate_state()` to either return a result object or raise a distinct `SourceMtimeChangedWarning`. Spec §5.1/§5.2 needs to match. 3. New `--clean-stale` flag with `--dry-run` / `--yes` / `--workdir` semantics (full spec in #19 §3). 4. New `tests/test_resume.py` covering 4 scenarios (interrupted run, resume, modified source, --fresh) — full spec in #19 §4. The four scenarios in (4) line up exactly with the kanban task t_e5ac8155 acceptance criteria, so the test file doubles as the verification artifact for the parent decomposer (t_d5c5d159). If this lands in the same PR as #16, the PR description should reference both #16 and #19. If it lands as a follow-up PR, base off the #16 PR branch, not main.
Mark commented 2026-09-09 23:41:52 +00:00
Author
Owner
Copy link

Integration summary: #16 resume x #8 long-file streaming

Post-merge composition audit of the two features, requested by kanban card t_02afa75b (fixit board). Both landed on main: #8 as e2d1647 / d43a782 / 418c100, #16 as 217cb08 (spec) + b95dfe3 (impl), plus #19 as 8b2f2afe5d (stale-state cleanup). Audited at origin/main 4a0b976 with a clean tree and ruff check src tests clean.

Where the two features join

The composition point is cli._run_transcription:

resolve_source -> probe_duration (#8) -> plan_chunks (#8)
  -> _resolve_workdir (#16) -> read_state (#16) -> validate_state (#16)
  -> select_pending_chunks (#16)
  -> per chunk: decode_chunk (#8) -> transcribe
                -> atomic_write_chunk_json (#16)
                -> offset_segments(seg, chunk.start_sec) (#8)
                -> mark_done (#16)

The join key is the chunk index: ChunkRange.index from #8's planner against ChunkRecord["index"] in #16's state file, guarded by the chunk_seconds equality check in state.validate_state (state.py:396).

Chunk-index stability: verified

chunking.plan_chunks(duration_sec, chunk_minutes) is a pure function of its two arguments — no wall clock, no random, no ordering dependence. Empirically:

  • plan_chunks(10h, 30min) -> 20 chunks, identical across repeated calls.
  • The hour-7 crash boundary is chunk index 14 (start_sec=25200.0, end_sec=27000.0).
  • A +0.5s ffprobe duration jitter (a real cross-version risk) shifts only the final chunk's end_sec; every non-final (index, start_sec) pair stays byte-identical, so a resume after a duration re-probe still joins correctly on index.
  • The one case that would genuinely break the join — changing --chunk-minutes between runs — is already hard-rejected as chunk_seconds mismatch rather than silently mis-joined. Correct behaviour.

No change needed for index stability.

Three gaps -> #21

  1. No --state-file PATH override. Only --workdir DIR exists; the filename is the fixed STATE_FILENAME constant. Not a one-line addition, since state.py derives every path from a workdir today (state_path, chunk_json_path, cleanup_state, find_stale_state_workspaces).
  2. Default location is a genuine design conflict, not an oversight. The card asks for output-adjacent; the shipped default is source-audio-adjacent (cli._resolve_workdir), which §1 of docs/state-file.md mandates on purpose so --out can move without invalidating resume. Recommendation on #21: keep the shipped default (resume correctness is the whole point of #16) and let --state-file cover the read-only-source-dir case. Flipping it is a behaviour change for existing state files, so it wants an explicit decision.
  3. No combined 10h crash-at-hour-7 test. tests/test_resume.py:614 test_resume_picks_up_after_interrupt has the right shape at the wrong scale (2h / 4 chunks / crash on chunk 3), asserts uniqueness but not contiguity, and never exercises audio_stream.decode_chunk jointly with the resume loop.

All three are tracked on #21 with per-item acceptance criteria. Cross-linked from #8.

## Integration summary: #16 resume x #8 long-file streaming Post-merge composition audit of the two features, requested by kanban card `t_02afa75b` (fixit board). Both landed on `main`: #8 as `e2d1647` / `d43a782` / `418c100`, #16 as `217cb08` (spec) + `b95dfe3` (impl), plus #19 as `8b2f2afe5d` (stale-state cleanup). Audited at `origin/main` `4a0b976` with a clean tree and `ruff check src tests` clean. ### Where the two features join The composition point is `cli._run_transcription`: ``` resolve_source -> probe_duration (#8) -> plan_chunks (#8) -> _resolve_workdir (#16) -> read_state (#16) -> validate_state (#16) -> select_pending_chunks (#16) -> per chunk: decode_chunk (#8) -> transcribe -> atomic_write_chunk_json (#16) -> offset_segments(seg, chunk.start_sec) (#8) -> mark_done (#16) ``` The join key is the **chunk index**: `ChunkRange.index` from #8's planner against `ChunkRecord["index"]` in #16's state file, guarded by the `chunk_seconds` equality check in `state.validate_state` (`state.py:396`). ### Chunk-index stability: verified `chunking.plan_chunks(duration_sec, chunk_minutes)` is a pure function of its two arguments — no wall clock, no `random`, no ordering dependence. Empirically: - `plan_chunks(10h, 30min)` -> 20 chunks, identical across repeated calls. - The hour-7 crash boundary is chunk index 14 (`start_sec=25200.0`, `end_sec=27000.0`). - A +0.5s `ffprobe` duration jitter (a real cross-version risk) shifts **only** the final chunk's `end_sec`; every non-final `(index, start_sec)` pair stays byte-identical, so a resume after a duration re-probe still joins correctly on index. - The one case that would genuinely break the join — changing `--chunk-minutes` between runs — is already hard-rejected as `chunk_seconds mismatch` rather than silently mis-joined. Correct behaviour. No change needed for index stability. ### Three gaps -> #21 1. **No `--state-file PATH` override.** Only `--workdir DIR` exists; the filename is the fixed `STATE_FILENAME` constant. Not a one-line addition, since `state.py` derives every path from a workdir today (`state_path`, `chunk_json_path`, `cleanup_state`, `find_stale_state_workspaces`). 2. **Default location is a genuine design conflict, not an oversight.** The card asks for output-adjacent; the shipped default is source-audio-adjacent (`cli._resolve_workdir`), which §1 of `docs/state-file.md` mandates on purpose so `--out` can move without invalidating resume. Recommendation on #21: keep the shipped default (resume correctness is the whole point of #16) and let `--state-file` cover the read-only-source-dir case. Flipping it is a behaviour change for existing state files, so it wants an explicit decision. 3. **No combined 10h crash-at-hour-7 test.** `tests/test_resume.py:614 test_resume_picks_up_after_interrupt` has the right shape at the wrong scale (2h / 4 chunks / crash on chunk 3), asserts uniqueness but not contiguity, and never exercises `audio_stream.decode_chunk` jointly with the resume loop. All three are tracked on #21 with per-item acceptance criteria. Cross-linked from #8.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
blocked:preflight

Blocked on a precondition (parent issue, upstream PR, or sibling fix)

  
bug

Something is broken and needs fixing

  
feature-request

New feature or capability

  
in-progress

Locked by an active fixit pipeline run

  
quickfix

Fast-path: small, low-risk fix ready for kilo-gsd

  
smoke-tested

Verified end-to-end on 2026-09-01

  
triage-approved

Triaged and approved for implementation

No labels
blocked:preflight
bug
feature-request
in-progress
quickfix
smoke-tested
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/whisper-transcribe#16
No description provided.