Mark/transcriber

Fork 0

Add on-device local LLM inference for summarization and topic extraction #127

New issue

Closed

opened 2026-06-03 23:56:02 +00:00 by Mark · 3 comments

Mark commented

2026-06-03 23:56:02 +00:00

Owner

Problem

Summarization (LLMSummarizer) and topic extraction (LLMTopicExtractor) currently require a network LLM endpoint. There is no on-device local option.

Spec principle

Local/offline should be the default workflow, with cloud as an opt-in enhanced experience.

Feasibility

TL;DR: Feasible but non-trivial. ~2-4 weeks of integration work.

Best path: llama-kotlin-android (llama.cpp JNI wrapper) — same pattern as existing sherpa-onnx integration for LOCAL_WHISPER.

Realistic constraints:

Model size: 350M-500M params @ Q4 = ~200-300MB (downloaded separately)
Speed: 2-10 tokens/sec on modern hardware — slow, not instant
APK impact: ~15MB library + model downloaded on demand
Existing LLMSummarizer/LLMTopicExtractor code can be adapted (not rewritten) — they already use a clean LLMConfig abstraction

Implementation notes

Add LocalLLMProvider class analogous to LocalDiarizationProvider
Reuse existing LLMConfig (endpoint, model) — just change the HTTP call to a local JNI call
Consider MediaPipe LLM Inference as an alternative (simpler integration, larger models)
Model delivery: bundle with app assets or download on first use

Labels

feature-request, local-first

## Problem Summarization (`LLMSummarizer`) and topic extraction (`LLMTopicExtractor`) currently require a network LLM endpoint. There is no on-device local option. ## Spec principle Local/offline should be the default workflow, with cloud as an opt-in enhanced experience. ## Feasibility **TL;DR: Feasible but non-trivial. ~2-4 weeks of integration work.** Best path: `llama-kotlin-android` (llama.cpp JNI wrapper) — same pattern as existing sherpa-onnx integration for LOCAL_WHISPER. Realistic constraints: - Model size: 350M-500M params @ Q4 = ~200-300MB (downloaded separately) - Speed: 2-10 tokens/sec on modern hardware — slow, not instant - APK impact: ~15MB library + model downloaded on demand - Existing `LLMSummarizer`/`LLMTopicExtractor` code can be adapted (not rewritten) — they already use a clean `LLMConfig` abstraction ## Implementation notes - Add `LocalLLMProvider` class analogous to `LocalDiarizationProvider` - Reuse existing `LLMConfig` (endpoint, model) — just change the HTTP call to a local JNI call - Consider MediaPipe LLM Inference as an alternative (simpler integration, larger models) - Model delivery: bundle with app assets or download on first use ## Labels feature-request, local-first

Mark referenced this issue

2026-06-03 23:56:33 +00:00

Topic extraction should default to local/keyword-based, with LLM as enhanced option #128

Mark added the

triage-approved

label

2026-06-06 15:15:13 +00:00

Mark commented

2026-06-10 03:33:14 +00:00

Author

Owner

Optimization commit `d78d669` deployed

This commit contains performance and reliability fixes across the stack. Build compiles and all unit tests pass.

Key changes:

R8 minification + shrinkResources enabled in release
BuildConfig.DEBUG guards on Log.d in hot paths
State race condition fix using StateFlow.update
CoroutineScope leak fixes (scope.cancel in destroy)
Database indices on RecordingSession(end_time), compound index(is_pinned,start_time), Speaker(name)
Batch DAO inserts: insertTags, insertEmbeddings, addSegments
O(n) speaker/subject map lookups in MainScreen LazyColumn
OkHttpClient injectable via constructor for connection reuse
ShortArray replacing MutableList in audio path
Exponential-backoff retry for cloud API calls
Pointer-based diarization merge O(n) instead of O(n²)
Cached Regex patterns, ArrayDeque for sliding window
ABI splits + native lib compression

## Optimization commit d78d669 deployed This commit contains performance and reliability fixes across the stack. Build compiles and all unit tests pass. Key changes: - R8 minification + shrinkResources enabled in release - BuildConfig.DEBUG guards on Log.d in hot paths - State race condition fix using StateFlow.update - CoroutineScope leak fixes (scope.cancel in destroy) - Database indices on RecordingSession(end_time), compound index(is_pinned,start_time), Speaker(name) - Batch DAO inserts: insertTags, insertEmbeddings, addSegments - O(n) speaker/subject map lookups in MainScreen LazyColumn - OkHttpClient injectable via constructor for connection reuse - ShortArray replacing MutableList<Short> in audio path - Exponential-backoff retry for cloud API calls - Pointer-based diarization merge O(n) instead of O(n²) - Cached Regex patterns, ArrayDeque for sliding window - ABI splits + native lib compression

Mark commented

2026-06-10 03:45:04 +00:00

Author

Owner

Follow-up fix deployed: `9c75ca7`

Fixes integration issues from the optimization commit:

Added NoteTag, NoteCrossref, SpeakerEmbedding entities to AppDatabase (version 7)
Added MIGRATION_6_7 for new tables: note_tags, note_crossrefs, speaker_embeddings
Added missing @Provides methods in DatabaseModule for new DAOs
Added setPinned/setTitle methods to SessionRepository (used by UI)
Fixed TranscriptionService parameter mismatch in SpeechBackend.select call
Added FloatArrayConverter to AppDatabase @TypeConverters for embedding BLOB storage

Full debug build passes. All unit tests pass.

## Follow-up fix deployed: `9c75ca7` Fixes integration issues from the optimization commit: - Added `NoteTag`, `NoteCrossref`, `SpeakerEmbedding` entities to `AppDatabase` (version 7) - Added `MIGRATION_6_7` for new tables: `note_tags`, `note_crossrefs`, `speaker_embeddings` - Added missing `@Provides` methods in `DatabaseModule` for new DAOs - Added `setPinned`/`setTitle` methods to `SessionRepository` (used by UI) - Fixed `TranscriptionService` parameter mismatch in `SpeechBackend.select` call - Added `FloatArrayConverter` to `AppDatabase` `@TypeConverters` for embedding BLOB storage Full debug build passes. All unit tests pass.

Mark commented

2026-06-10 04:36:55 +00:00

Author

Owner

Offline infrastructure deployed: `79954aa`

Foundation for full sustained offline functionality:

ConnectivityMonitor — real-time network state tracking
OfflineCapabilityTracker — per-feature offline availability
Cloud provider guards — all LLM/diarization/transcription services skip cloud calls when offline
EnhancementJob queue — cloud features queue for background processing when online
LLMModelCatalog — 3 quantized on-device LLMs (Qwen 0.5B, TinyLlama 1.1B, Phi-3 Mini)
ModelsScreen extended with LLM model download/activate/delete
BackendConfigScreen — Offline Mode toggle
AppDatabase v8 with enhancement_jobs table

Build passes, all unit tests pass.

## Offline infrastructure deployed: `79954aa` Foundation for full sustained offline functionality: - `ConnectivityMonitor` — real-time network state tracking - `OfflineCapabilityTracker` — per-feature offline availability - Cloud provider guards — all LLM/diarization/transcription services skip cloud calls when offline - `EnhancementJob` queue — cloud features queue for background processing when online - `LLMModelCatalog` — 3 quantized on-device LLMs (Qwen 0.5B, TinyLlama 1.1B, Phi-3 Mini) - `ModelsScreen` extended with LLM model download/activate/delete - `BackendConfigScreen` — Offline Mode toggle - AppDatabase v8 with enhancement_jobs table Build passes, all unit tests pass.

Mark

2026-06-14 16:29:26 +00:00