LLMLingua-2 Sidecar

Lossy context compression for Hermes conversations, built on LLMLingua-2.

Why

Hermes sessions can blow past the 256K compression threshold inside one cooldown window. The built-in summarizer then runs on an already-over-threshold context. This sidecar sits in front of the summarizer: it token-level compresses the middle of the conversation (everything except the recency buffer, system prompt, and tool calls), so the summarizer sees a smaller, cleaner input. Result: fewer tokens per turn, cheaper sessions, and less pressure on the compression threshold.

Architecture

Hermes session
    │
    ▼
[compression trigger] ──► POST /compress
                              │
                              ▼
                    ┌─────────────────────┐
                    │   FastAPI sidecar    │
                    │                     │
                    │  1. Segment input   │
                    │     (system / tools │
                    │     / recency /     │
                    │     middle)          │
                    │                     │
                    │  2. LLMLingua-2 on  │
                    │     middle only     │
                    │                     │
                    │  3. Return segments │
                    └─────────────────────┘
                              │
                              ▼
                    Hermes summarizer
                    (on compressed middle)

Quick Start

pip install -r requirements.txt
python -m uvicorn app.main:app --host 0.0.0.0 --port 8317

Health check:

curl http://localhost:8317/health

Compress a conversation:

curl -X POST http://localhost:8317/compress \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Very long conversation..."}
    ],
    "target_ratio": 0.3,
    "recency_tail_turns": 6
  }'

Configuration

Env Var	Default	Description
LLMLINGUA_MODEL	microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank	HuggingFace model ID
LLMLINGUA_DEVICE	cpu	cpu or cuda
LLMLINGUA_DEFAULT_RATIO	0.3	Default compression ratio (0.0–1.0)
PORT	8317	Listen port

API

GET /health

Returns model load status and device.

POST /compress

Request body:

{
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "target_ratio": 0.3,
  "recency_tail_turns": 6
}

Response:

{
  "compressed_messages": [...],
  "original_tokens": 150000,
  "compressed_tokens": 45000,
  "ratio": 0.3,
  "segments": {
    "system": 1,
    "tools": 0,
    "recency": 6,
    "middle_compressed": 42
  }
}

Guarantees

• System prompt: never compressed.
• Tool/function calls: never compressed (JSON integrity).
• Code blocks: never compressed (variable names survive).
• Recency tail: last N turns excluded from compression.

Eval Harness

See eval/ for the quality gate. Run real conversation traces through the sidecar, compare downstream answers against uncompressed baselines, and only ship if quality holds.

cd eval/
python run_eval.py --sessions ../test_sessions/*.jsonl --ratio 0.3

License

MIT