feat: implement smart model routing (length + tools + perplexity + OH CRAP escalate) #2

New issue

Closed

opened 2026-05-04 15:50:09 +00:00 by Mark · 0 comments

Mark commented

2026-05-04 15:50:09 +00:00

Owner

Feature Request: Smart Model Routing

Date: 2026-05-04
Status: Proposed

Summary

Implement smart model routing beyond the existing dead-config keys. Current smart_model_routing config is accepted but has no implementation.

Routing Criteria (3-stage)

1. Length Threshold

len(msg) <= max_simple_chars AND word_count <= max_simple_words

2. Tool Complexity Heuristic

No tool calls needed -> cheap model
Complex multi-step / many tools -> primary (powerful) model

3. Perplexity Scoring (optional)

Use a small local model to score message simplicity (cross-entropy). If perplexity > perplexity_threshold -> escalate to primary. Config: perplexity_model: Qwen3.5-0.8B-Q4KM (local, no API cost).

Escalation Path (OH CRAP)

When cheap model fails, automatically retry with primary model:

Crash: retry once with primary
429 / rate limit: retry once with primary
Timeout: retry once with primary
Gibberish detection: perplexity re-check -> escalate

Proposed Config

smart_model_routing:
enabled: true
max_simple_chars: 160
max_simple_words: 28
perplexity_threshold: 2.5
perplexity_model: Qwen3.5-0.8B-Q4KM
cheap_model:
provider: vllm-mm
model: GPT-OSS-20B-Q4KM
escalate_on_error: true

Implementation

File: run_agent.py
Hook: After conversation turn log (~line 10281), before first LLM call
Swap self.model + self.provider if conditions met
Escalation: catch crash/429/timeout, retry with saved primary model

Evidence (Bug Report)

See: issues/smart-model-routing-not-implemented.md

Tags

#hermes #feature #routing #cost-optimization

## Feature Request: Smart Model Routing **Date:** 2026-05-04 **Status:** Proposed ## Summary Implement smart model routing beyond the existing dead-config keys. Current smart_model_routing config is accepted but has no implementation. ## Routing Criteria (3-stage) ### 1. Length Threshold len(msg) <= max_simple_chars AND word_count <= max_simple_words ### 2. Tool Complexity Heuristic No tool calls needed -> cheap model Complex multi-step / many tools -> primary (powerful) model ### 3. Perplexity Scoring (optional) Use a small local model to score message simplicity (cross-entropy). If perplexity > perplexity_threshold -> escalate to primary. Config: perplexity_model: Qwen3.5-0.8B-Q4KM (local, no API cost). ## Escalation Path (OH CRAP) When cheap model fails, automatically retry with primary model: - Crash: retry once with primary - 429 / rate limit: retry once with primary - Timeout: retry once with primary - Gibberish detection: perplexity re-check -> escalate ## Proposed Config smart_model_routing: enabled: true max_simple_chars: 160 max_simple_words: 28 perplexity_threshold: 2.5 perplexity_model: Qwen3.5-0.8B-Q4KM cheap_model: provider: vllm-mm model: GPT-OSS-20B-Q4KM escalate_on_error: true ## Implementation File: run_agent.py Hook: After conversation turn log (~line 10281), before first LLM call Swap self.model + self.provider if conditions met Escalation: catch crash/429/timeout, retry with saved primary model ## Evidence (Bug Report) See: issues/smart-model-routing-not-implemented.md ## Tags #hermes #feature #routing #cost-optimization