[Phase 6.4.3] Document Chunker #94

New issue

Closed

opened 2026-04-07 04:20:04 +00:00 by Mark · 0 comments

Mark commented

2026-04-07 04:20:04 +00:00

Owner

Parent: Global Document Library

Overview

Build document chunking system for efficient embedding and retrieval.

Details

Create substrate/document_chunker.py:

DocumentChunker class:

chunk_document(content, chunk_size=500, overlap=50)
chunk_by_headings(content) - Preserve structure
chunk_code(content) - Split on functions/classes

Output:

DocumentChunk with: chunk_id, content, chunk_index, total_chunks
Preserve metadata (source, language)

Token estimation:

~0.25 tokens per character
Target ~500 tokens per chunk

Acceptance Criteria

Split text into overlapping chunks
Respect code structure (functions/classes)
Preserve markdown headings
Return chunk metadata

## Parent: Global Document Library ## Overview Build document chunking system for efficient embedding and retrieval. ## Details Create `substrate/document_chunker.py`: **DocumentChunker class:** - chunk_document(content, chunk_size=500, overlap=50) - chunk_by_headings(content) - Preserve structure - chunk_code(content) - Split on functions/classes **Output:** - DocumentChunk with: chunk_id, content, chunk_index, total_chunks - Preserve metadata (source, language) **Token estimation:** - ~0.25 tokens per character - Target ~500 tokens per chunk ## Acceptance Criteria - [ ] Split text into overlapping chunks - [ ] Respect code structure (functions/classes) - [ ] Preserve markdown headings - [ ] Return chunk metadata