[Phase 6.5] Cost & Token Tracking

Mark commented

2026-04-02 20:53:55 +00:00

Owner

Overview

Track LLM token usage and costs per intent for economic sustainability.

Problem Statement

Current mesh has no cost tracking:

Unknown spend per intent
No budget limits
No cost attribution
Can't optimize expensive operations
Economic sustainability unknown

Running mesh blind to costs is unsustainable.

Proposed Solution

6.5.1: Token Tracking in Results

# Intent result with cost info
result:
  intent_id: abc123
  content: "Generated article..."
  tokens:
    input: 150  # Prompt tokens
    output: 619  # Generated tokens
    total: 769
  cost:
    model: qwen2.5:0.5b
    input_cost_usd: 0.00015  # Model-specific pricing
    output_cost_usd: 0.00062
    total_cost_usd: 0.00077
  duration_ms: 4710

6.5.2: Cost Model Registry

# config/cost_models.yaml
models:
  qwen2.5:0.5b:
    input_cost_per_1k_tokens: 0.001  # Local model, minimal cost
    output_cost_per_1k_tokens: 0.001
    source: local
    
  deepseek-coder:6.7b:
    input_cost_per_1k_tokens: 0.002
    output_cost_per_1k_tokens: 0.002
    source: local
    
  gpt-4:
    input_cost_per_1k_tokens: 0.03  # API pricing
    output_cost_per_1k_tokens: 0.06
    source: openai_api
    
  claude-3-opus:
    input_cost_per_1k_tokens: 0.015
    output_cost_per_1k_tokens: 0.075
    source: anthropic_api

6.5.3: Budget Limits

class BudgetManager:
    """Enforce spending limits."""
    
    def check_budget(self, project: str, estimated_cost: float) -> bool:
        """Check if intent can proceed within budget"""
        
    def record_spend(self, project: str, intent_id: str, cost: float):
        """Record actual spend"""
        
    def get_remaining_budget(self, project: str) -> float:
        """Get remaining budget for project"""
        
    def set_budget_limit(self, project: str, limit: float, period: str):
        """Set budget limit (daily, weekly, monthly)"""

6.5.4: Cost Aggregation

-- Cost tracking tables
CREATE TABLE intent_costs (
    intent_id TEXT PRIMARY KEY,
    model TEXT,
    input_tokens INTEGER,
    output_tokens INTEGER,
    input_cost_usd REAL,
    output_cost_usd REAL,
    total_cost_usd REAL,
    created_at TIMESTAMP
);

CREATE TABLE project_budgets (
    project TEXT PRIMARY KEY,
    budget_limit_usd REAL,
    budget_period TEXT,  -- daily, weekly, monthly
    spent_usd REAL,
    period_start TIMESTAMP,
    updated_at TIMESTAMP
);

6.5.5: API Endpoints

endpoints:
  GET /v1/costs?project=X&period=week:
    returns: {total_cost, breakdown_by_model, intent_count}
    
  GET /v1/costs/<intent_id>:
    returns: {tokens, cost_usd, model}
    
  POST /v1/budgets:
    body: {project, limit_usd, period}
    returns: {budget_id}
    
  GET /v1/budgets/<project>:
    returns: {limit, spent, remaining, period}

Deliverables

intent_costs table in ledger
project_budgets table in ledger
config/cost_models.yaml - Model pricing
substrate/budget_manager.py - Budget enforcement
Token counting in LLM execution
Cost recording in intent results
Budget check before intent execution
API endpoints: /v1/costs/*, /v1/budgets/*
Unit tests
Integration tests

Dependencies

None (standalone)

Estimated Duration

1 week

Success Criteria

Tokens counted for every LLM call
Costs recorded per intent
Cost aggregated per project/period
Budget limits enforced
Budget exceeded blocks intents

Parent: #69 (Phase 6 Meta)

## Overview Track LLM token usage and costs per intent for economic sustainability. ## Problem Statement Current mesh has no cost tracking: - Unknown spend per intent - No budget limits - No cost attribution - Can't optimize expensive operations - Economic sustainability unknown Running mesh blind to costs is unsustainable. ## Proposed Solution ### 6.5.1: Token Tracking in Results ```yaml # Intent result with cost info result: intent_id: abc123 content: "Generated article..." tokens: input: 150 # Prompt tokens output: 619 # Generated tokens total: 769 cost: model: qwen2.5:0.5b input_cost_usd: 0.00015 # Model-specific pricing output_cost_usd: 0.00062 total_cost_usd: 0.00077 duration_ms: 4710 ``` ### 6.5.2: Cost Model Registry ```yaml # config/cost_models.yaml models: qwen2.5:0.5b: input_cost_per_1k_tokens: 0.001 # Local model, minimal cost output_cost_per_1k_tokens: 0.001 source: local deepseek-coder:6.7b: input_cost_per_1k_tokens: 0.002 output_cost_per_1k_tokens: 0.002 source: local gpt-4: input_cost_per_1k_tokens: 0.03 # API pricing output_cost_per_1k_tokens: 0.06 source: openai_api claude-3-opus: input_cost_per_1k_tokens: 0.015 output_cost_per_1k_tokens: 0.075 source: anthropic_api ``` ### 6.5.3: Budget Limits ```python class BudgetManager: """Enforce spending limits.""" def check_budget(self, project: str, estimated_cost: float) -> bool: """Check if intent can proceed within budget""" def record_spend(self, project: str, intent_id: str, cost: float): """Record actual spend""" def get_remaining_budget(self, project: str) -> float: """Get remaining budget for project""" def set_budget_limit(self, project: str, limit: float, period: str): """Set budget limit (daily, weekly, monthly)""" ``` ### 6.5.4: Cost Aggregation ```sql -- Cost tracking tables CREATE TABLE intent_costs ( intent_id TEXT PRIMARY KEY, model TEXT, input_tokens INTEGER, output_tokens INTEGER, input_cost_usd REAL, output_cost_usd REAL, total_cost_usd REAL, created_at TIMESTAMP ); CREATE TABLE project_budgets ( project TEXT PRIMARY KEY, budget_limit_usd REAL, budget_period TEXT, -- daily, weekly, monthly spent_usd REAL, period_start TIMESTAMP, updated_at TIMESTAMP ); ``` ### 6.5.5: API Endpoints ```yaml endpoints: GET /v1/costs?project=X&period=week: returns: {total_cost, breakdown_by_model, intent_count} GET /v1/costs/<intent_id>: returns: {tokens, cost_usd, model} POST /v1/budgets: body: {project, limit_usd, period} returns: {budget_id} GET /v1/budgets/<project>: returns: {limit, spent, remaining, period} ``` ## Deliverables - [ ] `intent_costs` table in ledger - [ ] `project_budgets` table in ledger - [ ] `config/cost_models.yaml` - Model pricing - [ ] `substrate/budget_manager.py` - Budget enforcement - [ ] Token counting in LLM execution - [ ] Cost recording in intent results - [ ] Budget check before intent execution - [ ] API endpoints: `/v1/costs/*`, `/v1/budgets/*` - [ ] Unit tests - [ ] Integration tests ## Dependencies - None (standalone) ## Estimated Duration 1 week ## Success Criteria - [ ] Tokens counted for every LLM call - [ ] Costs recorded per intent - [ ] Cost aggregated per project/period - [ ] Budget limits enforced - [ ] Budget exceeded blocks intents ## Related - Parent: #69 (Phase 6 Meta)