CRITICAL: No vector/semantic embedding infrastructure for bidding #123

New issue

Closed

opened 2026-04-11 02:26:12 +00:00 by Mark · 0 comments

Mark commented

2026-04-11 02:26:12 +00:00

Owner

Problem

The mesh claims to support semantic bidding (vector-based matching between intents and skills), but no embedding infrastructure exists in the codebase.

Current State (Investigated 2026-04-10)

What's Missing:

No embedding model integration - No sentence-transformers, Ollama embeddings, or similar
No vector storage - Ledger schema has no embedding columns
No similarity search - No FAISS, HNSW, pgvector, or sqlite-vss
No vector bidding logic - find_eligible_worker_instances() uses exact string matching only

Current Bidding (Broken):

-- ledger.py line 2593
WHERE capability_id = ?  -- Exact match only!

This means:

Intent "write_article" will NOT match skill "write_blog_post"
No semantic similarity between intent descriptions and skill descriptions
Bidding is literal string matching, not vector similarity

Expected Architecture

Intent "write_article" (text)
    ↓
Embedding Model → Vector [0.23, -0.45, 0.89, ...] (384+ dimensions)
    ↓
Vector Similarity Search (cosine distance)
    ↓
Skills with similar vectors (regardless of exact name match)
    ↓
Bid weighted by similarity score

Evidence

Files searched:

substrate/ledger.py - No embedding columns in schema
docker/mesh_node.py - No vector bidding logic
runtime/ - No embedding utilities
Database: sqlite3 ledger.db ".schema" - No vector columns

Code references:

find_eligible_worker_instances() does exact string match
No imports of: sentence-transformers, faiss, hnswlib, pgvector, sqlite-vss

Impact

Issue #118 (No workers) is partially caused by this - workers can't be found via semantic matching
Evolution system cannot function without vector similarity for championship scoring
Routing is literal only, no semantic flexibility
Contract evolution impossible without vector drift tracking

Research Needed

Before implementation, research:

Embedding model choice (local vs API)
Vector database (sqlite-vss vs pgvector vs in-memory)
Similarity threshold strategies
Bid calculation formula using similarity scores
Migration path from string-based to vector-based routing

#118 - Mesh accepts intents but no workers found
#104 - worker_factory bootstrap
#114 - Capabilities should be skill bundles

Status

BLOCKING: This is foundational infrastructure. Without it:

Workers can't be discovered semantically
Evolution can't compare skill variants
Mesh is stuck with literal string matching

Priority: CRITICAL - Must be resolved before mesh can function properly.

## Problem The mesh claims to support semantic bidding (vector-based matching between intents and skills), but **no embedding infrastructure exists** in the codebase. ## Current State (Investigated 2026-04-10) ### What's Missing: 1. **No embedding model integration** - No sentence-transformers, Ollama embeddings, or similar 2. **No vector storage** - Ledger schema has no `embedding` columns 3. **No similarity search** - No FAISS, HNSW, pgvector, or sqlite-vss 4. **No vector bidding logic** - `find_eligible_worker_instances()` uses exact string matching only ### Current Bidding (Broken): ```sql -- ledger.py line 2593 WHERE capability_id = ? -- Exact match only! ``` This means: - Intent "write_article" will NOT match skill "write_blog_post" - No semantic similarity between intent descriptions and skill descriptions - Bidding is literal string matching, not vector similarity ## Expected Architecture ``` Intent "write_article" (text) ↓ Embedding Model → Vector [0.23, -0.45, 0.89, ...] (384+ dimensions) ↓ Vector Similarity Search (cosine distance) ↓ Skills with similar vectors (regardless of exact name match) ↓ Bid weighted by similarity score ``` ## Evidence **Files searched:** - `substrate/ledger.py` - No embedding columns in schema - `docker/mesh_node.py` - No vector bidding logic - `runtime/` - No embedding utilities - Database: `sqlite3 ledger.db ".schema"` - No vector columns **Code references:** - `find_eligible_worker_instances()` does exact string match - No imports of: sentence-transformers, faiss, hnswlib, pgvector, sqlite-vss ## Impact - **Issue #118** (No workers) is partially caused by this - workers can't be found via semantic matching - **Evolution system** cannot function without vector similarity for championship scoring - **Routing** is literal only, no semantic flexibility - **Contract evolution** impossible without vector drift tracking ## Research Needed Before implementation, research: 1. Embedding model choice (local vs API) 2. Vector database (sqlite-vss vs pgvector vs in-memory) 3. Similarity threshold strategies 4. Bid calculation formula using similarity scores 5. Migration path from string-based to vector-based routing ## Related - #118 - Mesh accepts intents but no workers found - #104 - worker_factory bootstrap - #114 - Capabilities should be skill bundles ## Status **BLOCKING:** This is foundational infrastructure. Without it: - Workers can't be discovered semantically - Evolution can't compare skill variants - Mesh is stuck with literal string matching **Priority:** CRITICAL - Must be resolved before mesh can function properly.