Mark/transcriber
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOCAL_WHISPER backend is defined in config but has no implementation #2

New issue
Closed
opened 2026-05-12 16:50:24 +00:00 by Mark · 1 comment
Mark commented 2026-05-12 16:50:24 +00:00
Owner
Copy link

Problem

SpeechBackendType.LOCAL_WHISPER is defined as an enum value and its prerequisite check exists in SpeechBackend.checkPrerequisites() (checks for a model file at filesDir/models/ggml-small.bin), but there is no actual LocalWhisperTranscriptionProvider implementing the TranscriptionProvider interface.

This means:

  1. If a user configures LOCAL_WHISPER in their priority order and the model file exists, SpeechBackend.select() will return it as available
  2. But TranscriptionService.kt has no code path to instantiate or use a local Whisper provider
  3. The enum exists, the config path exists, the prerequisite check exists — but the actual transcription engine is missing

Revised Implementation Spec

Architecture

Create a proper on-device Whisper transcription engine using whisper.cpp via JNI. This gives us on-device transcription with no API costs, working fully offline.

What to Build

1. JNI Bridge Layer (app/src/main/cpp/whisper_jni.cpp)

  • Load GGML model file at init via whisper_init_from_file()
  • Expose native functions:
    • nativeInit(modelPath: String): Boolean
    • nativeTranscribe(audioFile: String, numThreads: Int): String
    • nativeRelease()
  • Compile via CMake + NDK into libwhisper_jni.so for armeabi-v7a, arm64-v8a, x86_64
  • Thread pool for non-blocking transcription

2. Provider Class (LocalWhisperTranscriptionProvider.kt)

  • Implements TranscriptionProvider interface
  • recognize(audioFile: File): TranscriptionResult
    • Convert audio to 16kHz PCM WAV (MediaRecorder usually outputs AMR or AAC)
    • Call JNI transcribe, parse output, return result with confidence and segments
  • Re-init on config change (model path, thread count)
  • Graceful error handling: missing model → error result, JNI crash → fallback to empty result

3. Model Download (ModelDownloadManager.kt)

  • First-launch check: if filesDir/models/ggml-small.bin missing, show download prompt
  • Download from https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin
  • Progress via foreground notification (required for large downloads on Android 13+)
  • Resume on interrupt using Range header
  • WiFi-only preference in settings (avoid cellular data surprise at ~466MB)
  • Checksum verification on completion

4. Wiring

  • TranscriptionService.kt: add SpeechBackendType.LOCAL_WHISPER → LocalWhisperTranscriptionProvider
  • Gradle: add CMake/NDK plugin, whisper.cpp source (git submodule or prebuilt .so)
  • SpeechBackend.checkPrerequisites(): already implemented — verify model exists and .so loads

Files Changed

File Action
app/src/main/cpp/whisper_jni.cpp NEW — JNI bridge
app/CMakeLists.txt NEW — NDK build config
app/src/main/java/.../service/LocalWhisperTranscriptionProvider.kt NEW
app/src/main/java/.../service/ModelDownloadManager.kt NEW
app/build.gradle.kts MODIFY — add NDK/CMake, add whisper dep
app/.../service/TranscriptionService.kt MODIFY — wire new provider
app/.../service/SpeechBackend.kt MODIFY — remove LOCAL_WHISPER from default config until implemented

Fallback Path

If NDK complexity is too high, implement via TensorFlow Lite using a TFLite Whisper model (smaller, but lower accuracy). The JNI path is preferred for full whisper.cpp accuracy and support, but TFLite is a lighter build dependency.

## Problem `SpeechBackendType.LOCAL_WHISPER` is defined as an enum value and its prerequisite check exists in `SpeechBackend.checkPrerequisites()` (checks for a model file at `filesDir/models/ggml-small.bin`), but there is no actual `LocalWhisperTranscriptionProvider` implementing the `TranscriptionProvider` interface. This means: 1. If a user configures `LOCAL_WHISPER` in their priority order and the model file exists, `SpeechBackend.select()` will return it as available 2. But `TranscriptionService.kt` has no code path to instantiate or use a local Whisper provider 3. The enum exists, the config path exists, the prerequisite check exists — but the actual transcription engine is missing ## Revised Implementation Spec ### Architecture Create a proper on-device Whisper transcription engine using **whisper.cpp** via JNI. This gives us on-device transcription with no API costs, working fully offline. ### What to Build **1. JNI Bridge Layer** (`app/src/main/cpp/whisper_jni.cpp`) - Load GGML model file at init via `whisper_init_from_file()` - Expose native functions: - `nativeInit(modelPath: String): Boolean` - `nativeTranscribe(audioFile: String, numThreads: Int): String` - `nativeRelease()` - Compile via CMake + NDK into `libwhisper_jni.so` for armeabi-v7a, arm64-v8a, x86_64 - Thread pool for non-blocking transcription **2. Provider Class** (`LocalWhisperTranscriptionProvider.kt`) - Implements `TranscriptionProvider` interface - `recognize(audioFile: File): TranscriptionResult` - Convert audio to 16kHz PCM WAV (MediaRecorder usually outputs AMR or AAC) - Call JNI transcribe, parse output, return result with confidence and segments - Re-init on config change (model path, thread count) - Graceful error handling: missing model → error result, JNI crash → fallback to empty result **3. Model Download** (`ModelDownloadManager.kt`) - First-launch check: if `filesDir/models/ggml-small.bin` missing, show download prompt - Download from `https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin` - Progress via foreground notification (required for large downloads on Android 13+) - Resume on interrupt using `Range` header - WiFi-only preference in settings (avoid cellular data surprise at ~466MB) - Checksum verification on completion **4. Wiring** - `TranscriptionService.kt`: add `SpeechBackendType.LOCAL_WHISPER` → `LocalWhisperTranscriptionProvider` - Gradle: add CMake/NDK plugin, whisper.cpp source (git submodule or prebuilt .so) - `SpeechBackend.checkPrerequisites()`: already implemented — verify model exists and .so loads ### Files Changed | File | Action | |------|--------| | `app/src/main/cpp/whisper_jni.cpp` | **NEW** — JNI bridge | | `app/CMakeLists.txt` | **NEW** — NDK build config | | `app/src/main/java/.../service/LocalWhisperTranscriptionProvider.kt` | **NEW** | | `app/src/main/java/.../service/ModelDownloadManager.kt` | **NEW** | | `app/build.gradle.kts` | **MODIFY** — add NDK/CMake, add whisper dep | | `app/.../service/TranscriptionService.kt` | **MODIFY** — wire new provider | | `app/.../service/SpeechBackend.kt` | **MODIFY** — remove LOCAL_WHISPER from default config until implemented | ### Fallback Path If NDK complexity is too high, implement via **TensorFlow Lite** using a TFLite Whisper model (smaller, but lower accuracy). The JNI path is preferred for full whisper.cpp accuracy and support, but TFLite is a lighter build dependency.
Mark commented 2026-05-20 04:05:29 +00:00
Author
Owner
Copy link

GSD VERIFY phase failed after 3 attempts.
Conformance failure: Unexpected files modified: .../service/LocalWhisperTranscriptionProvider.kt; Expected files not modified: app/src/main/java/com/voicetranscriber/service/LocalWhisperTranscriptionProvider.kt, app/src/main/java/com/voicetranscriber/service/SpeechBackend.kt, app/src/main/java/com/voicetranscriber/service/TranscriptionService.kt

GSD VERIFY phase failed after 3 attempts. Conformance failure: Unexpected files modified: .../service/LocalWhisperTranscriptionProvider.kt; Expected files not modified: app/src/main/java/com/voicetranscriber/service/LocalWhisperTranscriptionProvider.kt, app/src/main/java/com/voicetranscriber/service/SpeechBackend.kt, app/src/main/java/com/voicetranscriber/service/TranscriptionService.kt
Mark 2026-05-20 20:23:13 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
blocked:children

Blocked by child issues

  
blocked:needs-decision

   
blocked:needs-human

   
blocked:verify-failed

   
bug

Something is broken

  
deferred

Deferred — will revisit later

  
enhancement

New feature or request

  
feature-request

   
in-progress

   
needs-decompose

   
needs-triage

   
quickfix

   
triage-approved

Approved by triage — ready for fixer pipeline

No labels
blocked:children
blocked:needs-decision
blocked:needs-human
blocked:verify-failed
bug
deferred
enhancement
feature-request
in-progress
needs-decompose
needs-triage
quickfix
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/transcriber#2
No description provided.