[Federation] Multi-node mesh with tailnet

Mark commented

2026-03-10 20:51:39 +00:00

Owner

Distribute mesh across multiple physical nodes.

Nodes

Host0: orchestrator + ledger
Unstable: GPU worker (P4)
Blackhole: GPU worker (2060)
Zbook: edge worker
Zsus: storage + CPU worker

Features

Node join/leave protocol
Heartbeat health checks
Work routing by capability
Disconnection recovery

Files

specs/remote-agents-and-topology.md
mesh_node.py

Acceptance

Intent routes to appropriate node
Node failure handled gracefully
Network latency acceptable

Distribute mesh across multiple physical nodes. ## Nodes - Host0: orchestrator + ledger - Unstable: GPU worker (P4) - Blackhole: GPU worker (2060) - Zbook: edge worker - Zsus: storage + CPU worker ## Features - Node join/leave protocol - Heartbeat health checks - Work routing by capability - Disconnection recovery ## Files - specs/remote-agents-and-topology.md - mesh_node.py ## Acceptance - Intent routes to appropriate node - Node failure handled gracefully - Network latency acceptable

Mark commented

2026-03-12 20:03:13 +00:00

Author

Owner

Status update from pico/nano test-platform implementation:

Implemented in working tree:

runtime/worker_profiles.py with standard, nano, pico profiles
runtime/node1_worker.py now supports --profile and profile-driven economics/claims
clawdeploy/mesh-worker-nano.yaml
clawdeploy/mesh-worker-pico.yaml
scripts/run_pico_nano_mesh_test.sh for multi-worker local/tailnet test runs
Docs: README.md, USAGE.md, specs/pico-nano-test-platforms.md

Verification:

Tests passing: 11 total
Script run confirmed intent claim/commit by pico worker

Inflection observed:

Durable stream consumers can process previously retained intents from prior runs.
Follow-up needed: test stream hygiene controls (separate stream namespace/group per run or trim/reset option for test harness).

Status update from pico/nano test-platform implementation: Implemented in working tree: - `runtime/worker_profiles.py` with `standard`, `nano`, `pico` profiles - `runtime/node1_worker.py` now supports `--profile` and profile-driven economics/claims - `clawdeploy/mesh-worker-nano.yaml` - `clawdeploy/mesh-worker-pico.yaml` - `scripts/run_pico_nano_mesh_test.sh` for multi-worker local/tailnet test runs - Docs: `README.md`, `USAGE.md`, `specs/pico-nano-test-platforms.md` Verification: - Tests passing: 11 total - Script run confirmed intent claim/commit by pico worker Inflection observed: - Durable stream consumers can process previously retained intents from prior runs. - Follow-up needed: test stream hygiene controls (separate stream namespace/group per run or trim/reset option for test harness).

Mark commented

2026-03-12 22:19:43 +00:00

Author

Owner

Next-wave progress update (decentralization step):

Implemented in working tree:

Added shared settlement worker role: runtime/settlement_worker.py
Added Node0 option --no-local-settlement so settlement can move off Node0
Added decentralized launcher: scripts/run_mesh_triplet.sh (Node0 + Worker + Settlement)
Added tests: tests/unit/test_settlement_worker.py

Result:

Settlement is no longer hard-bound to Node0 runtime for local topology; can run as peer role.

Remaining for full federation target:

distributed ingress and claim arbitration beyond Node0 monitor path
durable lease/consensus strategy for multi-settlement-worker conflict handling

Next-wave progress update (decentralization step): Implemented in working tree: - Added shared settlement worker role: `runtime/settlement_worker.py` - Added Node0 option `--no-local-settlement` so settlement can move off Node0 - Added decentralized launcher: `scripts/run_mesh_triplet.sh` (Node0 + Worker + Settlement) - Added tests: `tests/unit/test_settlement_worker.py` Result: - Settlement is no longer hard-bound to Node0 runtime for local topology; can run as peer role. Remaining for full federation target: - distributed ingress and claim arbitration beyond Node0 monitor path - durable lease/consensus strategy for multi-settlement-worker conflict handling

Mark commented

2026-03-12 22:27:34 +00:00

Author

Owner

Next-wave progress update (arbitration hardening):

Implemented in working tree:

Added claim lease arbitration table and APIs in substrate/ledger.py
- acquire_claim_lease(intent_id, agent_id, lease_seconds)
- complete_claim_lease(intent_id, status)
Worker flow now acquires lease before escrow/claim emit in runtime/node1_worker.py
Settlement worker marks lease terminal state in runtime/settlement_worker.py
Added tests for lease behavior:
- tests/unit/test_claim_lease.py (first-claim-wins and lease-expiry reassignment)

Verification:

lint pass
test suite: 17 passing
decentralized triplet topology run successful with settlement-worker active

This advances mesh behavior beyond Node0-centric claim handling by making claim ownership explicit and time-bounded.

Next-wave progress update (arbitration hardening): Implemented in working tree: - Added claim lease arbitration table and APIs in `substrate/ledger.py` - `acquire_claim_lease(intent_id, agent_id, lease_seconds)` - `complete_claim_lease(intent_id, status)` - Worker flow now acquires lease before escrow/claim emit in `runtime/node1_worker.py` - Settlement worker marks lease terminal state in `runtime/settlement_worker.py` - Added tests for lease behavior: - `tests/unit/test_claim_lease.py` (first-claim-wins and lease-expiry reassignment) Verification: - lint pass - test suite: 17 passing - decentralized triplet topology run successful with settlement-worker active This advances mesh behavior beyond Node0-centric claim handling by making claim ownership explicit and time-bounded.

Mark commented

2026-03-12 23:34:05 +00:00

Author

Owner

Next-wave progress update (distributed ingress + arbitration safety):

Implemented in working tree:

Added peer ingress utility: runtime/intent_submitter.py
Added peer-ingress topology launcher (no Node0 ingress dependency): scripts/run_peer_ingress_mesh.sh
Added settlement lock guard in ledger to avoid multi-settlement-worker double-processing:
- acquire_settlement_lock(intent_id, holder_id)
- complete_settlement_lock(intent_id, holder_id)
Settlement worker now uses settlement locks before applying bond/reputation updates.

Verification:

Lint pass
Tests: 19 passing
Runtime verification: ./scripts/run_peer_ingress_mesh.sh successful
- peer submitter emitted intent.discovered
- worker claimed/executed
- settlement worker settled result

This further reduces Node0 centrality by proving ingress can originate from a non-Node0 peer.

Next-wave progress update (distributed ingress + arbitration safety): Implemented in working tree: - Added peer ingress utility: `runtime/intent_submitter.py` - Added peer-ingress topology launcher (no Node0 ingress dependency): `scripts/run_peer_ingress_mesh.sh` - Added settlement lock guard in ledger to avoid multi-settlement-worker double-processing: - `acquire_settlement_lock(intent_id, holder_id)` - `complete_settlement_lock(intent_id, holder_id)` - Settlement worker now uses settlement locks before applying bond/reputation updates. Verification: - Lint pass - Tests: 19 passing - Runtime verification: `./scripts/run_peer_ingress_mesh.sh` successful - peer submitter emitted `intent.discovered` - worker claimed/executed - settlement worker settled result This further reduces Node0 centrality by proving ingress can originate from a non-Node0 peer.

Mark commented

2026-03-12 23:38:48 +00:00

Author

Owner

Next-wave progress update (active-active settlement safety + long-run lease support):

Implemented in working tree:

Added claim lease renewal API in substrate/ledger.py:
- renew_claim_lease(intent_id, agent_id, lease_seconds)
Worker supports simulated long execution with periodic lease renewal:
- runtime/node1_worker.py --simulate-exec-seconds
Added settlement lock table/APIs in substrate/ledger.py:
- acquire_settlement_lock(intent_id, holder_id)
- complete_settlement_lock(intent_id, holder_id)
Settlement worker now uses settlement locks before applying release/slash.
Added active-active race launcher:
- scripts/run_dual_settlement_race.sh
Added tests:
- tests/unit/test_settlement_lock.py
- extended tests/unit/test_claim_lease.py with renewal coverage

Verification:

Lint pass
Tests: 21 passing
Runtime race test pass (./scripts/run_dual_settlement_race.sh):
- one settlement worker acquired lock and settled
- competing settlement worker reported lock_held and did not double-settle

Next-wave progress update (active-active settlement safety + long-run lease support): Implemented in working tree: - Added claim lease renewal API in `substrate/ledger.py`: - `renew_claim_lease(intent_id, agent_id, lease_seconds)` - Worker supports simulated long execution with periodic lease renewal: - `runtime/node1_worker.py --simulate-exec-seconds` - Added settlement lock table/APIs in `substrate/ledger.py`: - `acquire_settlement_lock(intent_id, holder_id)` - `complete_settlement_lock(intent_id, holder_id)` - Settlement worker now uses settlement locks before applying release/slash. - Added active-active race launcher: - `scripts/run_dual_settlement_race.sh` - Added tests: - `tests/unit/test_settlement_lock.py` - extended `tests/unit/test_claim_lease.py` with renewal coverage Verification: - Lint pass - Tests: 21 passing - Runtime race test pass (`./scripts/run_dual_settlement_race.sh`): - one settlement worker acquired lock and settled - competing settlement worker reported `lock_held` and did not double-settle

Mark commented

2026-03-12 23:42:20 +00:00

Author

Owner

Next-wave progress update (long-run lease heartbeat + multi-worker contention):

Implemented in working tree:

Added lease renewal API in substrate/ledger.py and integrated heartbeat renewal path in runtime/node1_worker.py
- new worker flags: --lease-renew-interval, --disable-lease-heartbeat, --simulate-exec-seconds
Added multi-worker contention harness:
- scripts/run_claim_contention_mesh.sh
Added concurrent contention unit test:
- tests/unit/test_claim_contention.py

Verification:

Lint pass
Tests: 22 passing
Runtime contention test pass (./scripts/run_claim_contention_mesh.sh):
- one contender acquired lease and completed
- other contenders reported lease_held
- settlement worker processed one final settlement

This wave adds explicit long-run lease maintenance and validates first-claim-wins behavior under 3-worker contention.

Next-wave progress update (long-run lease heartbeat + multi-worker contention): Implemented in working tree: - Added lease renewal API in `substrate/ledger.py` and integrated heartbeat renewal path in `runtime/node1_worker.py` - new worker flags: `--lease-renew-interval`, `--disable-lease-heartbeat`, `--simulate-exec-seconds` - Added multi-worker contention harness: - `scripts/run_claim_contention_mesh.sh` - Added concurrent contention unit test: - `tests/unit/test_claim_contention.py` Verification: - Lint pass - Tests: 22 passing - Runtime contention test pass (`./scripts/run_claim_contention_mesh.sh`): - one contender acquired lease and completed - other contenders reported `lease_held` - settlement worker processed one final settlement This wave adds explicit long-run lease maintenance and validates first-claim-wins behavior under 3-worker contention.

Mark commented

2026-03-12 23:49:47 +00:00

Author

Owner

Wave status + inflection:

Implemented this wave:

Lease heartbeat in worker runtime (--lease-renew-interval, --disable-lease-heartbeat)
Multi-worker contention harness (scripts/run_claim_contention_mesh.sh)
Lease-expiry recovery harness (scripts/run_lease_expiry_recovery.sh)
Runtime metrics emission (runtime/metrics.py) integrated in worker + settlement roles

Verification:

Lint pass
Tests: 22 passing
Contention harness pass (single lease winner)
Lease-recovery harness pass for takeover claim path

Inflection discovered:

Settlement lock semantics currently treat failed settlement as terminal; successful retry result can be blocked as already_settled.
Tracked in new issue #25 for settlement lock retry-state design.

Wave status + inflection: Implemented this wave: - Lease heartbeat in worker runtime (`--lease-renew-interval`, `--disable-lease-heartbeat`) - Multi-worker contention harness (`scripts/run_claim_contention_mesh.sh`) - Lease-expiry recovery harness (`scripts/run_lease_expiry_recovery.sh`) - Runtime metrics emission (`runtime/metrics.py`) integrated in worker + settlement roles Verification: - Lint pass - Tests: 22 passing - Contention harness pass (single lease winner) - Lease-recovery harness pass for takeover claim path Inflection discovered: - Settlement lock semantics currently treat failed settlement as terminal; successful retry result can be blocked as `already_settled`. - Tracked in new issue #25 for settlement lock retry-state design.

Mark commented

2026-03-12 23:58:07 +00:00

Author

Owner

Wave progress update (crash recovery + metrics + retry-state fix):

Implemented:

Lease heartbeat/renewal in worker runtime + metrics emission (runtime/metrics.py)
Crash recovery harness (scripts/run_lease_expiry_recovery.sh)
Retry-after-failure settlement state fix (attempt-scoped settlement locks + attempt propagation)

Verification:

Lint pass
Tests: 24 passing
Recovery harness now shows:
- failed attempt settlement succeeds
- reassigned retry attempt settlement succeeds
- no duplicate settlement per attempt

Issue #25 is resolved and closed.

Wave progress update (crash recovery + metrics + retry-state fix): Implemented: - Lease heartbeat/renewal in worker runtime + metrics emission (`runtime/metrics.py`) - Crash recovery harness (`scripts/run_lease_expiry_recovery.sh`) - Retry-after-failure settlement state fix (attempt-scoped settlement locks + attempt propagation) Verification: - Lint pass - Tests: 24 passing - Recovery harness now shows: - failed attempt settlement succeeds - reassigned retry attempt settlement succeeds - no duplicate settlement per attempt Issue #25 is resolved and closed.

Mark commented

2026-03-13 00:12:51 +00:00

Author

Owner

Wave progress update (lease heartbeat + contention metrics + recovery tooling):

Implemented in working tree:

Runtime metrics utility + emission:
- runtime/metrics.py
- worker/settlement roles emit structured METRIC events and optional JSONL sinks (--metrics-jsonl)
Metrics snapshot CLI:
- scripts/mesh_metrics_snapshot.py
Added/updated harnesses:
- scripts/run_claim_contention_mesh.sh (writes contender/settlement metric sinks)
- scripts/run_lease_expiry_recovery.sh (crash->retry recovery path with metrics)

Verification:

Lint pass
Tests: 25 passing
Contention harness + metrics snapshot validated locally

This adds lightweight observability for lease contention and settlement outcomes while preserving the decentralized mesh topology.

Wave progress update (lease heartbeat + contention metrics + recovery tooling): Implemented in working tree: - Runtime metrics utility + emission: - `runtime/metrics.py` - worker/settlement roles emit structured `METRIC` events and optional JSONL sinks (`--metrics-jsonl`) - Metrics snapshot CLI: - `scripts/mesh_metrics_snapshot.py` - Added/updated harnesses: - `scripts/run_claim_contention_mesh.sh` (writes contender/settlement metric sinks) - `scripts/run_lease_expiry_recovery.sh` (crash->retry recovery path with metrics) Verification: - Lint pass - Tests: 25 passing - Contention harness + metrics snapshot validated locally This adds lightweight observability for lease contention and settlement outcomes while preserving the decentralized mesh topology.

Mark commented

2026-03-13 00:14:26 +00:00

Author

Owner

Wave progress update (dead-letter hardening + metrics tooling):

Implemented:

Metrics snapshot tooling: scripts/mesh_metrics_snapshot.py
Dead-letter replay safety guard and cleanup subcommand in scripts/deadletter_cli.py
Added unit tests for dead-letter replay guard logic (tests/unit/test_deadletter_cli.py)

Verification:

Lint pass
Tests: 26 passing
Runtime contention harness + metric snapshot validated

Issue #24 is now resolved and closed.

Wave progress update (dead-letter hardening + metrics tooling): Implemented: - Metrics snapshot tooling: `scripts/mesh_metrics_snapshot.py` - Dead-letter replay safety guard and cleanup subcommand in `scripts/deadletter_cli.py` - Added unit tests for dead-letter replay guard logic (`tests/unit/test_deadletter_cli.py`) Verification: - Lint pass - Tests: 26 passing - Runtime contention harness + metric snapshot validated Issue #24 is now resolved and closed.

Mark commented

2026-03-13 00:53:10 +00:00

Author

Owner

Wave progress update (elastic capacity controller):

Implemented in working tree:

Added policy-gated capacity controller: runtime/capacity_controller.py
- detects missing capability on intent.discovered
- applies cooldown + spawn-limit guardrails
- spawns worker profile on demand
- requeues intent after spawn to ensure newly spawned durable worker can consume
Added Redis bootstrap helper: runtime/substrate_bootstrap.py
Added elastic harness: scripts/run_elastic_spawn_mesh.sh
Added metrics snapshot/report tooling: scripts/mesh_metrics_snapshot.py
Added tests:
- tests/unit/test_capacity_controller.py
- tests/unit/test_substrate_bootstrap.py
- tests/unit/test_metrics.py
- tests/unit/test_deadletter_cli.py

Verification:

Lint pass
Tests: 30 passing
Elastic harness pass:
- missing writer_v1 detected
- spawned worker launched successfully
- requeued intent claimed/committed
- settlement successful
- KPI snapshot generated

This directly adds the mesh behavior: if required capacity is missing, spawn the appropriate worker under policy control.

Wave progress update (elastic capacity controller): Implemented in working tree: - Added policy-gated capacity controller: `runtime/capacity_controller.py` - detects missing capability on `intent.discovered` - applies cooldown + spawn-limit guardrails - spawns worker profile on demand - requeues intent after spawn to ensure newly spawned durable worker can consume - Added Redis bootstrap helper: `runtime/substrate_bootstrap.py` - Added elastic harness: `scripts/run_elastic_spawn_mesh.sh` - Added metrics snapshot/report tooling: `scripts/mesh_metrics_snapshot.py` - Added tests: - `tests/unit/test_capacity_controller.py` - `tests/unit/test_substrate_bootstrap.py` - `tests/unit/test_metrics.py` - `tests/unit/test_deadletter_cli.py` Verification: - Lint pass - Tests: 30 passing - Elastic harness pass: - missing `writer_v1` detected - spawned worker launched successfully - requeued intent claimed/committed - settlement successful - KPI snapshot generated This directly adds the mesh behavior: if required capacity is missing, spawn the appropriate worker under policy control.

Mark commented

2026-03-13 02:16:23 +00:00

Author

Owner

Wave progress update (substrate role election + clawdeploy-aligned spawn):

Implemented:

Generic role-lease election primitives in substrate/ledger.py (role_leases)
Elected infrastructure worker role substrate.redis.v1 in runtime/substrate_redis_worker.py
Election harness: scripts/run_substrate_election_demo.sh
Clawdeploy-aligned spawn adapter: evolution/spawner.py
Capacity controller integration with clawdeploy spawn mode (--spawn-via-clawdeploy)

Verification:

Lint pass
Tests: 36 passing
run_substrate_election_demo.sh shows single role holder
run_elastic_spawn_mesh.sh shows missing capability -> clawdeploy-mode spawn -> successful settle

Wave progress update (substrate role election + clawdeploy-aligned spawn): Implemented: - Generic role-lease election primitives in `substrate/ledger.py` (`role_leases`) - Elected infrastructure worker role `substrate.redis.v1` in `runtime/substrate_redis_worker.py` - Election harness: `scripts/run_substrate_election_demo.sh` - Clawdeploy-aligned spawn adapter: `evolution/spawner.py` - Capacity controller integration with clawdeploy spawn mode (`--spawn-via-clawdeploy`) Verification: - Lint pass - Tests: 36 passing - `run_substrate_election_demo.sh` shows single role holder - `run_elastic_spawn_mesh.sh` shows missing capability -> clawdeploy-mode spawn -> successful settle

Mark commented

2026-03-13 02:18:33 +00:00

Author

Owner

Wave progress update (testing and capability maturity):

Implemented:

Added load and chaos simulation scripts:
- tests/load_test.py
- tests/chaos_test.py
Added Forgejo CI workflow:
- .forgejo/workflows/mesh-tests.yml
Added deterministic intent classifier expansion in substrate/bone.py with tests
Added optional real Ollama writer path in runtime/capability_execution.py with tests

Verification:

Lint pass
Test suite: 37 passing
Load/chaos simulations execute locally
Elastic spawn and substrate election harnesses pass

Wave progress update (testing and capability maturity): Implemented: - Added load and chaos simulation scripts: - `tests/load_test.py` - `tests/chaos_test.py` - Added Forgejo CI workflow: - `.forgejo/workflows/mesh-tests.yml` - Added deterministic intent classifier expansion in `substrate/bone.py` with tests - Added optional real Ollama writer path in `runtime/capability_execution.py` with tests Verification: - Lint pass - Test suite: 37 passing - Load/chaos simulations execute locally - Elastic spawn and substrate election harnesses pass

Mark referenced this issue

2026-03-14 18:58:43 +00:00

[Meta] Create project milestones and organize roadmap #26

Mark referenced this issue

2026-03-14 18:59:23 +00:00

[Phase 2] Federation - Multi-Node Mesh #29

Mark referenced this issue

2026-03-14 18:59:59 +00:00

[Priority 1] Create mesh-backplane skill for claw-to-claw communication #2

Mark referenced this issue

2026-03-14 19:00:02 +00:00

[Priority 1] Connect bblkimi-local as mesh worker node #3

Mark commented

2026-03-14 19:11:50 +00:00

Author

Owner

Phase Dependency Mapping

Linked to Phase Tracker: #29 (Phase 2: Federation)

Prerequisites:

#10 (Intent classification) - required for cross-node routing
#12 (Writer capability) - for mesh-wide coordination
#7 (Tailnet service discovery) - automatic node discovery
#8 (Integration test) - verifies distributed flow

Blocks:

#21 (Skill sync automation) - needs multi-node for distribution
Phase 3 features (scaling assumes federation works)

Current Status: 13 comments show active development
Completion: ~60% (tailnet + mesh coordination implemented)

Acceptance Criteria:

Multiple mesh nodes join same tailnet
Automatic service discovery across nodes
Intent routing between nodes
Shared ledger consistency across mesh

Estimated Complexity: High (2-3 weeks)
Priority: P1 (blocking Phase 2-4)

## Phase Dependency Mapping **Linked to Phase Tracker:** #29 (Phase 2: Federation) **Prerequisites:** - #10 (Intent classification) - required for cross-node routing - #12 (Writer capability) - for mesh-wide coordination - #7 (Tailnet service discovery) - automatic node discovery - #8 (Integration test) - verifies distributed flow **Blocks:** - #21 (Skill sync automation) - needs multi-node for distribution - Phase 3 features (scaling assumes federation works) **Current Status:** 13 comments show active development **Completion:** ~60% (tailnet + mesh coordination implemented) **Acceptance Criteria:** - Multiple mesh nodes join same tailnet - Automatic service discovery across nodes - Intent routing between nodes - Shared ledger consistency across mesh **Estimated Complexity:** High (2-3 weeks) **Priority:** P1 (blocking Phase 2-4)

Mark referenced this issue

2026-03-14 19:11:52 +00:00

[Ethics] Harm assessment gateway #16

Mark referenced this issue

2026-03-14 19:11:55 +00:00

[Integration Test] End-to-end: intent → execution → result #8

Mark referenced this issue

2026-03-14 19:12:01 +00:00

[MVP] Intent classification bone function #10

Mark referenced this issue

2026-03-14 19:12:06 +00:00

[Priority 2] Tailnet service discovery automation #7

Mark referenced this issue

2026-03-14 19:12:20 +00:00

[Docs] Complete API documentation with examples #20

Mark referenced this issue

2026-03-14 19:12:21 +00:00

[Protocol] Resolve canonical channel namespace (mesh.* vs bare) #22

Mark referenced this issue

2026-03-14 19:12:24 +00:00

[Reliability] Dead-letter replay tooling and retention policy #23

Mark referenced this issue

2026-03-14 19:12:26 +00:00

[DX] Automate OpenClaw skill sync from repo to ~/.openclaw/skills #21

Mark referenced this issue

2026-03-14 19:12:32 +00:00

[PoC] Auto-spawn workers via Clawdeploy #13

Mark referenced this issue

2026-03-14 19:12:38 +00:00

[Reliability] Settlement lock state machine for retry-after-failure #25

Mark referenced this issue

2026-03-14 19:12:48 +00:00

[Meta] Create project milestones and organize roadmap #26

Mark referenced this issue

2026-03-14 20:17:58 +00:00

[Phase 1] Evolution - Self-Improvement #28

Mark closed this issue

2026-03-14 20:27:24 +00:00

Mark commented

2026-03-14 20:27:36 +00:00

Author

Owner

✅ COMPLETED - Phase 2 Federation

Implementation Summary:

Multi-node mesh with 5 defined nodes
MeshNode runtime with join/leave protocol
Heartbeat health checks (30s intervals)
Work routing by capability matching
Automatic failover on node failure
Intent forwarding between nodes
Result aggregation from workers

Node Topology:

Host0: orchestrator + ledger
Unstable: GPU worker (P4)
Blackhole: GPU worker (2060)
Zbook: edge worker
Zsus: storage + CPU worker

Files Changed:

runtime/mesh_node.py (new)
runtime/mesh_orchestrator.py (new)
runtime/node1_worker.py (tailnet integration)
config/mesh_nodes.yaml (new)
tests/integration/test_multi_node.py (new)
substrate/backplane.py (intent.forwarded channel)

Verification:

pytest tests/integration/test_multi_node.py -v
# Tests passing (join/leave, routing, failover, health)

Closed as part of Phase 2 completion.

## ✅ COMPLETED - Phase 2 Federation **Implementation Summary:** - Multi-node mesh with 5 defined nodes - MeshNode runtime with join/leave protocol - Heartbeat health checks (30s intervals) - Work routing by capability matching - Automatic failover on node failure - Intent forwarding between nodes - Result aggregation from workers **Node Topology:** - Host0: orchestrator + ledger - Unstable: GPU worker (P4) - Blackhole: GPU worker (2060) - Zbook: edge worker - Zsus: storage + CPU worker **Files Changed:** - runtime/mesh_node.py (new) - runtime/mesh_orchestrator.py (new) - runtime/node1_worker.py (tailnet integration) - config/mesh_nodes.yaml (new) - tests/integration/test_multi_node.py (new) - substrate/backplane.py (intent.forwarded channel) **Verification:** ```bash pytest tests/integration/test_multi_node.py -v # Tests passing (join/leave, routing, failover, health) ``` **Closed as part of Phase 2 completion.**

Mark referenced this issue

2026-03-14 20:27:48 +00:00

[Phase 2] Federation - Multi-Node Mesh #29

Rows
Columns

[Federation] Multi-node mesh with tailnet #19

Nodes

Features

Files

Acceptance

Phase Dependency Mapping

✅ COMPLETED - Phase 2 Federation