INFRA-003: Create true E2E test for full intent execution flow

Mark commented

2026-04-07 18:13:55 +00:00

Owner

Problem

Current tests use mocks. Need unmocked E2E test to verify full flow works.

Scope (MINIMAL)

Create single test file: tests/e2e/test_infrastructure_bidding.py
Test only: bootstrap workers → intent → infrastructure bids → claim
Don't test spawning/execution - that's separate issues

Files

tests/e2e/test_infrastructure_bidding.py (new)

Acceptance Criteria

Test uses real Redis, real ledger (no mocks)
Verifies infrastructure bids on behalf of workers

## Problem Current tests use mocks. Need unmocked E2E test to verify full flow works. ## Scope (MINIMAL) 1. Create single test file: `tests/e2e/test_infrastructure_bidding.py` 2. Test only: bootstrap workers → intent → infrastructure bids → claim 3. Don't test spawning/execution - that's separate issues ## Files - `tests/e2e/test_infrastructure_bidding.py` (new) ## Acceptance Criteria - [ ] Test uses real Redis, real ledger (no mocks) - [ ] Verifies infrastructure bids on behalf of workers

Mark referenced this issue

2026-04-10 01:02:00 +00:00

INFRA-015: Fix test collection errors blocking full suite #112

Mark commented

2026-04-10 01:07:09 +00:00

Author

Owner

Additional Context (2026-04-09)

Current E2E Test Status

From test execution:

17/19 E2E tests passing (89%)
Real AI integration working (qwen3.5:9b, 102s execution)
Tests pass but architecture broken

Test Collection Errors

Issue #112 - 6 test files fail collection:

tests/load/test_auto_scaling.py
tests/load/test_queue_depth.py
tests/production/test_e2e_production.py
tests/production/test_real_ollama.py
tests/test_deployment_worker.py
tests/unit/test_capability_execution.py

This blocks running full 1763 test suite.

Architecture vs Tests

Problem: Tests pass because mesh_node executes work in-process.

Current:

Intent → mesh_node (routes AND executes) → Result

Correct:

Intent → mesh_node (routes) → Worker (executes) → Result

Tests verify execution works, but do not verify architecture is correct.

What E2E Tests Show

✅ Tool registry and execution working
✅ Skill registry and execution working
✅ Real AI integration working
✅ Writer capability produces output

❌ Workers do not spawn as processes
❌ Infrastructure executes instead of routing
❌ Evolution bypasses mesh flow

Issue #98 (INFRA-001) - Infrastructure bidding (root cause)
Issue #102 (INFRA-005) - Worker spawning after bid
Issue #112 - Test collection errors (blocking)
Issue #111 - meta_factory error (capability loading)

Recommendation

Create E2E test that verifies:

Workers spawn as OS processes (not threads)
Workers have unique PIDs
Workers subscribe to backplane independently
Workers emit results via result.committed

This would catch the architecture violation.

## Additional Context (2026-04-09) ### Current E2E Test Status From test execution: - **17/19 E2E tests passing** (89%) - **Real AI integration working** (qwen3.5:9b, 102s execution) - **Tests pass but architecture broken** ### Test Collection Errors **Issue #112** - 6 test files fail collection: - tests/load/test_auto_scaling.py - tests/load/test_queue_depth.py - tests/production/test_e2e_production.py - tests/production/test_real_ollama.py - tests/test_deployment_worker.py - tests/unit/test_capability_execution.py This blocks running full 1763 test suite. ### Architecture vs Tests **Problem:** Tests pass because mesh_node executes work in-process. **Current:** ``` Intent → mesh_node (routes AND executes) → Result ``` **Correct:** ``` Intent → mesh_node (routes) → Worker (executes) → Result ``` Tests verify execution works, but do not verify architecture is correct. ### What E2E Tests Show ✅ Tool registry and execution working ✅ Skill registry and execution working ✅ Real AI integration working ✅ Writer capability produces output ❌ Workers do not spawn as processes ❌ Infrastructure executes instead of routing ❌ Evolution bypasses mesh flow ### Related Issues - **Issue #98** (INFRA-001) - Infrastructure bidding (root cause) - **Issue #102** (INFRA-005) - Worker spawning after bid - **Issue #112** - Test collection errors (blocking) - **Issue #111** - meta_factory error (capability loading) ### Recommendation Create E2E test that verifies: 1. Workers spawn as OS processes (not threads) 2. Workers have unique PIDs 3. Workers subscribe to backplane independently 4. Workers emit results via result.committed This would catch the architecture violation.

Mark commented

2026-05-05 03:41:53 +00:00

Author

Owner

🟡 STATUS UPDATE — E2E Testing Progress

Current state

3/4 E2E tests passing in tests/e2e/
The one failure (test_full_intent_to_result_flow) is a test-side UUID suffix mismatch — _spawn_async_worker uses worker_id-{uuid} but the test looks up the original worker_id
Integration tests: 47/47 pass (real worker spawning, real process tracking)
Unit tests: 127/127 pass

What's covered

test_spawn_failure_no_workers_available ✅
test_claim_rejected_insufficient_bond ✅
test_multiple_intents_sequential ✅
test_worker.py test worker correctly writes results ✅

What's still needed (S04)

Writer execution E2E test with real LLM call (vLLM)
Full intent→bid→claim→spawn→execute→result flow end-to-end
This is tracked as M009/S04 in the milestone plan

## 🟡 STATUS UPDATE — E2E Testing Progress ### Current state - **3/4 E2E tests passing** in `tests/e2e/` - The one failure (`test_full_intent_to_result_flow`) is a test-side UUID suffix mismatch — `_spawn_async_worker` uses `worker_id-{uuid}` but the test looks up the original `worker_id` - Integration tests: **47/47 pass** (real worker spawning, real process tracking) - Unit tests: **127/127 pass** ### What's covered - `test_spawn_failure_no_workers_available` ✅ - `test_claim_rejected_insufficient_bond` ✅ - `test_multiple_intents_sequential` ✅ - `test_worker.py` test worker correctly writes results ✅ ### What's still needed (S04) - Writer execution E2E test with real LLM call (vLLM) - Full intent→bid→claim→spawn→execute→result flow end-to-end - This is tracked as M009/S04 in the milestone plan

Mark commented

2026-05-05 04:20:19 +00:00

Author

Owner

✅ MAJOR PROGRESS

M009 Completed Slices

S01 (Primitive Name Resolution) — ✅ Committed 88af283
S02 (Factory Handoff) — ✅ Committed 705635f
S03 (Writer Execution) — ✅ Committed 250806e, 8826b6f, 8198404
- T01: Refactored primitive_execution to use LLMRegistry
- T02: Added explicit skills to writer template.yaml
- T03: Created E2E writer execution test file

S04 In Progress

Full intent→provision→handoff→execute→result flow
Test file at tests/e2e/test_writer_execution.py (428 lines)
Requires: Redis + vLLM/Ollama running to pass

Current Test Status

174/174 unit + integration tests passing
3/4 E2E tests passing (1 pre-existing UUID suffix mismatch)
New writer E2E test needs runtime infrastructure (Redis + LLM)

## ✅ MAJOR PROGRESS ### M009 Completed Slices - **S01** (Primitive Name Resolution) — ✅ Committed `88af283` - **S02** (Factory Handoff) — ✅ Committed `705635f` - **S03** (Writer Execution) — ✅ Committed `250806e`, `8826b6f`, `8198404` - T01: Refactored primitive_execution to use LLMRegistry - T02: Added explicit skills to writer template.yaml - T03: Created E2E writer execution test file ### S04 In Progress - Full intent→provision→handoff→execute→result flow - Test file at `tests/e2e/test_writer_execution.py` (428 lines) - **Requires:** Redis + vLLM/Ollama running to pass ### Current Test Status - 174/174 unit + integration tests passing - 3/4 E2E tests passing (1 pre-existing UUID suffix mismatch) - New writer E2E test needs runtime infrastructure (Redis + LLM)

Mark commented

2026-05-05 04:42:59 +00:00

Author

Owner

REBUILD AND TEST RUN COMPLETE

Build

mesh-node:latest - Built from Dockerfile.node with M009 changes
mesh-gateway:latest - Built from Dockerfile.gateway
mesh-worker:latest - Built from Dockerfile.worker

Deploy

Containers recreated with fresh images on docker_mesh-network
Both Host0 (mesh-seed) and Host1 (mesh-seed-2) started successfully
Gateway at localhost:8081 healthy - backplane connected

Runtime Verification

Full intent pipeline confirmed working via live gateway test:

POST /intent with writing_task -> accepted (202)
Intent discovered -> bid_manager wins -> claim validated -> worker spawned
Worker execution -> result.committed with success=True
Bond release + reward credited after completion

Test Status

96/96 unit + integration tests passing
Pre-registered writer workers work end-to-end (verified in container logs)
Factory-provisioned writer handoff functional but needs refinement
S04 E2E test needs Redis-based fixture (pending)

## REBUILD AND TEST RUN COMPLETE ### Build - mesh-node:latest - Built from Dockerfile.node with M009 changes - mesh-gateway:latest - Built from Dockerfile.gateway - mesh-worker:latest - Built from Dockerfile.worker ### Deploy - Containers recreated with fresh images on docker_mesh-network - Both Host0 (mesh-seed) and Host1 (mesh-seed-2) started successfully - Gateway at localhost:8081 healthy - backplane connected ### Runtime Verification Full intent pipeline confirmed working via live gateway test: 1. POST /intent with writing_task -> accepted (202) 2. Intent discovered -> bid_manager wins -> claim validated -> worker spawned 3. Worker execution -> result.committed with success=True 4. Bond release + reward credited after completion ### Test Status - 96/96 unit + integration tests passing - Pre-registered writer workers work end-to-end (verified in container logs) - Factory-provisioned writer handoff functional but needs refinement - S04 E2E test needs Redis-based fixture (pending)

Mark added the

feature-request

label

2026-05-12 23:26:58 +00:00

Mark added the

triage-approved

quickfix

labels

2026-05-18 06:13:27 +00:00

Mark added the

in-progress

label