[Phase 5.2] Closed-Loop Factory Learning

Mark commented

2026-04-01 01:58:03 +00:00

Owner

Overview

Create a feedback loop from championship results back to factory mutation strategies, enabling factories to learn which mutations succeed.

Problem Statement

Current flow:

Factory → Generate Variants → Championship → Promote Winner
                                              ↓
                                           [END - no feedback]

The factory never learns which mutation types work best. It generates variants blindly.

Proposed Solution

5.2.1: Championship Analytics

Create evolution/factory_analytics.py:

class FactoryAnalytics:
    """Track and analyze mutation strategy effectiveness."""
    
    def record_result(
        self,
        variant_id: str,
        mutation_type: str,
        mutation_params: dict,
        championship_score: float,
        promoted: bool
    ):
        """Record championship result for a variant."""
        
    def get_strategy_effectiveness(
        self,
        mutation_type: str
    ) -> dict:
        """Get success rate and avg score for mutation type."""
        
    def recommend_mutations(
        self,
        factory_id: str,
        target_metric: str = "success_rate"
    ) -> list[dict]:
        """Recommend mutation strategies based on historical performance."""

5.2.2: Mutation Strategy Tuning

class AdaptiveMutationRegistry(MutationRegistry):
    """MutationRegistry that learns from results."""
    
    def __init__(self, analytics: FactoryAnalytics):
        self.analytics = analytics
        
    def get_weighted_strategies(
        self,
        factory_id: str
    ) -> list[tuple[str, float]]:
        """Return mutation types weighted by historical success."""
        # Instead of uniform random, bias toward successful strategies

5.2.3: Learning Pipeline

Championship completes
    ↓
FactoryAnalytics.record_result() for each participant
    ↓
Daily: FactoryAnalytics.compute_trends()
    ↓
AdaptiveMutationRegistry adjusts weights
    ↓
Next variant generation uses learned weights

Deliverables

evolution/factory_analytics.py - Analytics class
analytics table in ledger (SQLite)
AdaptiveMutationRegistry - Learning-aware registry
Integration with evolution/championship.py
API endpoint: GET /v1/factory/analytics
Dashboard metrics: mutation success rates

Database Schema

CREATE TABLE factory_analytics (
    id INTEGER PRIMARY KEY,
    factory_id TEXT NOT NULL,
    variant_id TEXT NOT NULL,
    mutation_type TEXT NOT NULL,
    mutation_params JSON,
    championship_score REAL,
    promoted BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_factory_mutation ON factory_analytics(factory_id, mutation_type);

Success Criteria

Analytics persist across restarts
Can query success rate by mutation type
AdaptiveMutationRegistry biases toward successful strategies
Measurable improvement in variant quality over time
Dashboard shows mutation effectiveness

Dependencies

None (foundational)

Estimated Duration

2 weeks

Parent: #62 (Phase 5 Meta)
Uses: capabilities/_shared/base.py (MutationRegistry)

## Overview Create a feedback loop from championship results back to factory mutation strategies, enabling factories to learn which mutations succeed. ## Problem Statement Current flow: ``` Factory → Generate Variants → Championship → Promote Winner ↓ [END - no feedback] ``` The factory never learns which mutation types work best. It generates variants blindly. ## Proposed Solution ### 5.2.1: Championship Analytics Create `evolution/factory_analytics.py`: ```python class FactoryAnalytics: """Track and analyze mutation strategy effectiveness.""" def record_result( self, variant_id: str, mutation_type: str, mutation_params: dict, championship_score: float, promoted: bool ): """Record championship result for a variant.""" def get_strategy_effectiveness( self, mutation_type: str ) -> dict: """Get success rate and avg score for mutation type.""" def recommend_mutations( self, factory_id: str, target_metric: str = "success_rate" ) -> list[dict]: """Recommend mutation strategies based on historical performance.""" ``` ### 5.2.2: Mutation Strategy Tuning ```python class AdaptiveMutationRegistry(MutationRegistry): """MutationRegistry that learns from results.""" def __init__(self, analytics: FactoryAnalytics): self.analytics = analytics def get_weighted_strategies( self, factory_id: str ) -> list[tuple[str, float]]: """Return mutation types weighted by historical success.""" # Instead of uniform random, bias toward successful strategies ``` ### 5.2.3: Learning Pipeline ``` Championship completes ↓ FactoryAnalytics.record_result() for each participant ↓ Daily: FactoryAnalytics.compute_trends() ↓ AdaptiveMutationRegistry adjusts weights ↓ Next variant generation uses learned weights ``` ## Deliverables - [ ] `evolution/factory_analytics.py` - Analytics class - [ ] `analytics` table in ledger (SQLite) - [ ] `AdaptiveMutationRegistry` - Learning-aware registry - [ ] Integration with `evolution/championship.py` - [ ] API endpoint: `GET /v1/factory/analytics` - [ ] Dashboard metrics: mutation success rates ## Database Schema ```sql CREATE TABLE factory_analytics ( id INTEGER PRIMARY KEY, factory_id TEXT NOT NULL, variant_id TEXT NOT NULL, mutation_type TEXT NOT NULL, mutation_params JSON, championship_score REAL, promoted BOOLEAN, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); CREATE INDEX idx_factory_mutation ON factory_analytics(factory_id, mutation_type); ``` ## Success Criteria - [ ] Analytics persist across restarts - [ ] Can query success rate by mutation type - [ ] AdaptiveMutationRegistry biases toward successful strategies - [ ] Measurable improvement in variant quality over time - [ ] Dashboard shows mutation effectiveness ## Dependencies - None (foundational) ## Estimated Duration 2 weeks ## Related - Parent: #62 (Phase 5 Meta) - Uses: `capabilities/_shared/base.py` (MutationRegistry)