Mark/mesh-seed

Fork 0

[Model Spec] Analyze for fine-tuning candidates #18

New issue

Closed

opened 2026-03-10 20:51:28 +00:00 by Mark · 2 comments

Mark commented

2026-03-10 20:51:28 +00:00

Owner

Weekly analysis for metabolic optimization.

Functionality

Identify high-volume capabilities (> 1000 calls/month)
Evaluate fine-tune ROI
Trigger distillation when break-even < 3 months
Track specialist model performance

Pipeline

70B → 7B → 1B specialization
A/B test vs general LLM
Promote winners to bone

Files

evolution/model_analysis.py
specs/model-specialization.md

Target

10x cost reduction through specialization

Weekly analysis for metabolic optimization. ## Functionality - Identify high-volume capabilities (> 1000 calls/month) - Evaluate fine-tune ROI - Trigger distillation when break-even < 3 months - Track specialist model performance ## Pipeline - 70B → 7B → 1B specialization - A/B test vs general LLM - Promote winners to bone ## Files - evolution/model_analysis.py - specs/model-specialization.md ## Target - 10x cost reduction through specialization

Mark referenced this issue

2026-03-14 18:59:12 +00:00

[Phase 1] Evolution - Self-Improvement #28

Mark referenced this issue

2026-03-14 19:11:47 +00:00

[Evolution] Weekly championship tournament #15

Mark commented

2026-03-14 19:12:11 +00:00

Author

Owner

Phase Dependency Mapping

Linked to Phase Tracker: #28 (Phase 1: Evolution)

Prerequisites:

#8 (Integration test) ✅ or in-progress
#10 (Intent classification) - performance data
Historical execution data

Blocks:

#15 (Weekly championship) - feeds tournament data
Phase 1 model specialization

Purpose: Identifies when custom fine-tuning beats general models
Priority: P2 (optimization phase of Phase 1)

Acceptance Criteria:

Performance regression detection
Cost-benefit analysis for fine-tuning
Candidate model identification
ROI calculation framework

## Phase Dependency Mapping **Linked to Phase Tracker:** #28 (Phase 1: Evolution) **Prerequisites:** - #8 (Integration test) ✅ or in-progress - #10 (Intent classification) - performance data - Historical execution data **Blocks:** - #15 (Weekly championship) - feeds tournament data - Phase 1 model specialization **Purpose:** Identifies when custom fine-tuning beats general models **Priority:** P2 (optimization phase of Phase 1) **Acceptance Criteria:** - Performance regression detection - Cost-benefit analysis for fine-tuning - Candidate model identification - ROI calculation framework

Mark referenced this issue

2026-03-14 19:12:48 +00:00

[Meta] Create project milestones and organize roadmap #26

Mark referenced this issue

2026-03-14 20:01:09 +00:00

[Phase 0] Foundation - MVP Completion #27

Mark closed this issue

2026-03-14 20:17:30 +00:00

Mark commented

2026-03-14 20:17:46 +00:00

Author

Owner

✅ COMPLETED - Phase 1 Evolution

Implementation Summary:

Analyzes ledger for high-volume capabilities (>1000 calls/month)
ROI calculation with break-even < 3 months threshold
Tracks specialist vs general LLM performance
Recommends distillation: 70B → 7B → 1B specialization
Multiple output formats: Console, JSON, Markdown
Weekly report runner for integration with rumination

Example Output:

Model Specialization Weekly Report
==================================
Capability           | Calls | Cost/call | Break-even
---------------------|-------|-----------|------------
writer_v1           | 2,340 | $0.002    | 2.1 months
research_topic      | 1,890 | $0.003    | 2.8 months

Files Created:

evolution/model_analysis.py (analyzer with ROI calc)
tests/evolution/test_model_analysis.py (new)
specs/model-specialization.md (strategy doc)

Verification:

pytest tests/evolution/test_model_analysis.py -v
# Tests passing (volume, ROI, recommendations)

Closed as part of Phase 1 completion.

## ✅ COMPLETED - Phase 1 Evolution **Implementation Summary:** - Analyzes ledger for high-volume capabilities (>1000 calls/month) - ROI calculation with break-even < 3 months threshold - Tracks specialist vs general LLM performance - Recommends distillation: 70B → 7B → 1B specialization - Multiple output formats: Console, JSON, Markdown - Weekly report runner for integration with rumination **Example Output:** ``` Model Specialization Weekly Report ================================== Capability | Calls | Cost/call | Break-even ---------------------|-------|-----------|------------ writer_v1 | 2,340 | $0.002 | 2.1 months research_topic | 1,890 | $0.003 | 2.8 months ``` **Files Created:** - evolution/model_analysis.py (analyzer with ROI calc) - tests/evolution/test_model_analysis.py (new) - specs/model-specialization.md (strategy doc) **Verification:** ```bash pytest tests/evolution/test_model_analysis.py -v # Tests passing (volume, ROI, recommendations) ``` **Closed as part of Phase 1 completion.**

Mark referenced this issue

2026-03-14 20:17:58 +00:00

[Phase 1] Evolution - Self-Improvement #28

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/mesh-seed#18

No description provided.

Rows
Columns