feat(#276)+docs(#199): cross-mesh federation protocol v1 (design, spec, implementation) #275

Merged

Mark merged 17 commits from docs/cross-mesh-federation-design into master 2026-09-10 13:15:27 +00:00

Conversation 1 Commits 17 Files changed 36 +8647 -73

Mark commented 2026-08-14 02:31:48 +00:00

Owner

Copy link

Cross-Mesh Federation Protocol v1 (META-001, issues #199 / #276).

Scope

Implementation of the normative spec at docs/federation/spec.md (commit 5d94f6b). This PR extends the docs branch with the full implementation: envelope codec, handshake state machine, message router, HTTP transport, demo harness, and unit/integration tests.

Deliverables

• runtime/federation_protocol.py — federation/v1 JSON envelope codec: canonical serialization, HMAC-SHA256 sign/verify with freshness window
• runtime/federation_credentials.py — ed25519 identity + TOFU pinning + shared-secret rotation
• runtime/federation_handshake.py — hello/challenge/response/ack state machine
• runtime/mesh_federation.py + runtime/cross_mesh_forwarding.py — message router: intent_route / intent_status / intent_completion forwarding
• runtime/http_entrypoint.py — POST /v1/federation/{handshake,heartbeat,intent,disconnect}, GET /v1/federation/{siblings,health} with spec §7.3 error model
• scripts/demo_federation.py — two-node end-to-end demo
• substrate/ledger migration 0004 + federation repository (forwarded-intent ledger)
• Feature flag: config/mesh.yaml federation.enabled (default false; no breaking change)

Demo: two nodes federating end-to-end

Run: python scripts/demo_federation.py (uses shared in-memory backplane, boots mesh-seed-alpha + mesh-seed-beta).

Verified output (2026-08-14):

[1] Peer discovery via signed heartbeats
    B knows ['mesh-seed-alpha'] (status=online)
[2] Handshake (hello -> challenge -> response -> ack)
    Both meshes established; identities TOFU-pinned
[3] Intent forward round-trip (writer_v1)
    B completed route mesh-seed-alpha-mesh-seed-beta-DEMO-INT-1 (ok=True)
    A flipped pending -> completed (idempotent by route_id)
[4] Shared-secret rotation
    After A rotates, B's old-secret heartbeat rejected: invalid_signature
    After B rotates, verification succeeds again
Demo complete. Both meshes federated end-to-end over v1.

Tests

• 62 unit tests pass (codec, handshake, credentials, router, forwarding, HTTP entrypoint)
• 8 integration tests pass (tests/integration/test_federation_two_meshes.py)
• ruff check . clean
• CI note: mesh-seed has no online Actions runner as of 2026-08-14 (runs queue in waiting); local gate set above is the verification evidence.

Closes #276 (implementation), builds on #199 umbrella.

Cross-Mesh Federation Protocol v1 (META-001, issues #199 / #276). ## Scope Implementation of the normative spec at docs/federation/spec.md (commit 5d94f6b). This PR extends the docs branch with the full implementation: envelope codec, handshake state machine, message router, HTTP transport, demo harness, and unit/integration tests. ## Deliverables - runtime/federation_protocol.py — federation/v1 JSON envelope codec: canonical serialization, HMAC-SHA256 sign/verify with freshness window - runtime/federation_credentials.py — ed25519 identity + TOFU pinning + shared-secret rotation - runtime/federation_handshake.py — hello/challenge/response/ack state machine - runtime/mesh_federation.py + runtime/cross_mesh_forwarding.py — message router: intent_route / intent_status / intent_completion forwarding - runtime/http_entrypoint.py — POST /v1/federation/{handshake,heartbeat,intent,disconnect}, GET /v1/federation/{siblings,health} with spec §7.3 error model - scripts/demo_federation.py — two-node end-to-end demo - substrate/ledger migration 0004 + federation repository (forwarded-intent ledger) - Feature flag: config/mesh.yaml federation.enabled (default false; no breaking change) ## Demo: two nodes federating end-to-end Run: `python scripts/demo_federation.py` (uses shared in-memory backplane, boots mesh-seed-alpha + mesh-seed-beta). Verified output (2026-08-14): ``` [1] Peer discovery via signed heartbeats B knows ['mesh-seed-alpha'] (status=online) [2] Handshake (hello -> challenge -> response -> ack) Both meshes established; identities TOFU-pinned [3] Intent forward round-trip (writer_v1) B completed route mesh-seed-alpha-mesh-seed-beta-DEMO-INT-1 (ok=True) A flipped pending -> completed (idempotent by route_id) [4] Shared-secret rotation After A rotates, B's old-secret heartbeat rejected: invalid_signature After B rotates, verification succeeds again Demo complete. Both meshes federated end-to-end over v1. ``` ## Tests - 62 unit tests pass (codec, handshake, credentials, router, forwarding, HTTP entrypoint) - 8 integration tests pass (tests/integration/test_federation_two_meshes.py) - ruff check . clean - CI note: mesh-seed has no online Actions runner as of 2026-08-14 (runs queue in waiting); local gate set above is the verification evidence. Closes #276 (implementation), builds on #199 umbrella.

Mark added 1 commit 2026-08-14 02:31:48 +00:00

docs(#199): cross-mesh federation protocol design & requirements ... 1b583e1ad8

Research survey (Matrix, ActivityPub, federated identity, mesh networking)
plus 41 REQ-* requirements covering topology, auth/identity, envelope,
conflict resolution, and operational failure modes for META-001.
Companion to docs/federation/spec.md (normative spec, child task).

Mark referenced this pull request 2026-08-14 02:31:57 +00:00

META-001: Cross-Mesh Federation Protocol #199

Mark added 1 commit 2026-08-14 02:41:19 +00:00

docs(#199): cross-mesh federation protocol normative spec ... 5d94f6b8d6

Wire-level envelope JSON schema (federation/v1), canonical serialization +
HMAC-SHA256 signature scheme with conformance test vector, handshake state
machine (hello/challenge/response/ack) with TOFU pinning and key rotation,
capability negotiation, HTTP transport binding and error model, and worked
examples for join handshake, federated message relay, and graceful
disconnect. Cross-references the 41 REQ-* ids from design.md.

Mark referenced this pull request 2026-08-14 02:41:36 +00:00

META-001: Cross-Mesh Federation Protocol #199

Mark referenced this pull request 2026-08-14 02:46:12 +00:00

META-001: implement cross-mesh federation protocol v1 (envelope, handshake, router, HTTP binding) #276

Mark added 1 commit 2026-08-14 09:00:00 +00:00

feat(#276): cross-mesh federation protocol v1 (envelope, handshake, router, HTTP binding) ... f292cce09c

Implements META-001 per docs/federation/spec.md and the plan at
.kilo/plans/META-001-cross-mesh-federation-protocol.md:

- runtime/federation_protocol.py: federation/v1 envelope codec, canonical
  serialization, HMAC-SHA256 sign/verify with freshness window + replay LRU
  (spec 3-4, including the 4.4 test vector)
- runtime/federation_handshake.py: hello/challenge/response/ack state machine,
  ed25519 identity proof, TOFU pinning, split-brain detection (spec 5)
- runtime/federation_credentials.py: per-peer secret resolution (env ->
  SecretsManager -> legacy token) and rotation
- runtime/mesh_federation.py: signed heartbeats/intent routes, real
  _check_node_timeouts (parsed datetimes, online->degraded->offline),
  _dispatch_local_route wired to FederationWorker, get_federation_health,
  removed duplicate mesh_forwarded_intents CREATE
- runtime/cross_mesh_forwarding.py: envelope wrapping in forward_intent,
  accept_envelope, resolver-based bearer (legacy fallback + metric)
- runtime/http_entrypoint.py: POST /v1/federation/{handshake,heartbeat,
  intent,disconnect}, GET /v1/federation/{siblings,health}, trust-tier gate,
  spec 7.3 error model
- ledger migration 0008 (federation_peer_keys, federation_heartbeat_log,
  federation_route_intents) + MeshLedger/federation repo helpers
- federation_worker.py: use route_intent() (IntentRouter class never existed)
- scripts/demo_federation.py CLI harness booting two meshes end-to-end
- tests: envelope codec, handshake state machine, credentials, mesh_federation,
  gateway envelope wrapping, HTTP routes, two-mesh integration (all green)
- docs: docs/reference/MESH_FEDERATION.md, API.md federation endpoints,
  config/mesh.yaml federation section

Demo (in PR description):
  python scripts/demo_federation.py
  -> peer discovery, 4-step handshake w/ TOFU pinning, intent forward
     round-trip, shared-secret rotation, all end-to-end in-process

Mark changed title from docs(#199): cross-mesh federation protocol design & requirements to feat(#276)+docs(#199): cross-mesh federation protocol v1 (design, spec, implementation) 2026-08-14 09:00:29 +00:00

Mark added 1 commit 2026-08-14 09:16:13 +00:00

test(#199): federation failure-mode integration suite + mark two-mesh suite for CI ... c5a72285b9

Adds tests/integration/test_federation_failure_modes.py covering:
- 3-node fanout (discovery + A->C routing with non-target ignoring)
- peer partition and reconnect (timeout sweep -> degraded -> offline -> rejoin)
- auth failure / replay attack rejection (wrong secret, nonce replay via
  backplane and HTTP handlers)
- message ordering under reordering network (first-received wins for
  conflicting completions, idempotent duplicates, heartbeat freshness)
- graceful shutdown (goodbye envelope + stop())

Also marks test_federation_two_meshes.py with pytestmark = integration so
the existing 2-node suite is picked up by CI's 'pytest -m integration' job.

One known gap documented as xfail: REQ-CONF-3 heartbeat freshness
(last-writer-wins by issued_at) is not implemented in _ingest_heartbeat;
a stale in-window heartbeat regresses fresh node state.

Mark added 3 commits 2026-08-15 05:32:33 +00:00

fix(#142): add repo root to sys.path in SDK test conftest 774b0efb74

docs(#277): add Discord bot runbook documenting DISCORD_BOT_TOKEN provisioning and rotation 443d6b5694

Merge remote-tracking branch 'pr/275' into land/meta-001-199 ... 4418288269

# Conflicts:
#	substrate/ledger/migrations.py

Mark commented 2026-08-19 20:26:05 +00:00

Author

Owner

Copy link

CI: federation integration suite wired (card t_a053f7c8)

Added .forgejo/workflows/federation.yml — a dedicated Forgejo Actions workflow that runs the full federation integration suite (pytest -m federation) on the mesh-seed user runner. It:

• Caches deps via setup-python pip cache.
• Per-test timeout via pytest-timeout --timeout=300 — bounds every test class so a single hung test cannot stall the job.
• Uploads failure logs — junit report + stdout log as a federation-test-results artifact on always().

The workflow triggers on any pull_request to master/main, so it gates this implementation PR — all federation tests must be green to merge.

Sample green run (federation suite 63 passed, 5 xfailed, job on localhost-user-runner): https://forgejo.tail018ac4.ts.net/Mark/mesh-seed/actions/runs/164

Note for review: the existing ci.yml upload steps use upload-artifact@v4, which this Forgejo version rejects (GHESNotSupportedError — v2 artifact API unsupported). The new federation job pins upload-artifact@v3 (verified tag v3.2.1); the other jobs may hit the same issue on their next run.

## CI: federation integration suite wired (card t_a053f7c8) Added `.forgejo/workflows/federation.yml` — a dedicated Forgejo Actions workflow that runs the **full federation integration suite** (`pytest -m federation`) on the mesh-seed user runner. It: - **Caches deps** via `setup-python` pip cache. - **Per-test timeout** via `pytest-timeout --timeout=300` — bounds every test class so a single hung test cannot stall the job. - **Uploads failure logs** — junit report + stdout log as a `federation-test-results` artifact on `always()`. The workflow triggers on any `pull_request` to `master`/`main`, so it **gates this implementation PR** — all federation tests must be green to merge. **Sample green run** (federation suite 63 passed, 5 xfailed, job on `localhost-user-runner`): https://forgejo.tail018ac4.ts.net/Mark/mesh-seed/actions/runs/164 Note for review: the existing `ci.yml` upload steps use `upload-artifact@v4`, which this Forgejo version rejects (`GHESNotSupportedError` — v2 artifact API unsupported). The new federation job pins `upload-artifact@v3` (verified tag v3.2.1); the other jobs may hit the same issue on their next run.

Mark referenced this pull request from a commit 2026-08-19 21:32:53 +00:00

test(#199): land federation integration + failure-mode suite and CI workflow into PR #275

Mark added 9 commits 2026-08-19 21:32:54 +00:00

test(#199): federation 2-node happy-path + 3-node fanout suite (card t_a9b870de) ... 52aeec7b5e

Add tests/integration/federation/ with a reusable harness (conftest) and:
- test_two_node.py: two peers exchange a known message set and converge
  on ordering, payload, and ledger state.
- test_three_node_fanout.py: one publisher fans a replicated message to
  two subscribers; all three agree on ordering and payload.

Register the 'federation' pytest marker in pytest.ini and mark the
pre-existing two-mesh + failure-mode suites with it too, so the CI
'pytest -m federation' job runs the whole federation suite (28 passed,
1 xfailed locally).

Also applies two mechanical ruff-format v0.3.0 rewraps to
test_federation_failure_modes.py (pre-existing assert style) so the
pooled editor's pinned formatter stays clean.

Note: tests/integration/federation/test_auth_replay.py is a sibling card
(t_8ee7b012) file landing in this dir concurrently; left uncommitted.

test(#199): federation auth-failure + replay-attack rejection suite (card t_8ee7b012) ... dd6bfc1164

Covers auth failure at handshake (invalid/expired/future-dated/revoked
credentials rejected pre-state, never established) and replay attack
(verbatim + timestamp-manipulated re-injection rejected and flagged as
replay_detected through MeshFederation's NonceCache path).

Two spec requirements surfaced as documented xfail (matching the
REQ-CONF-3 convention): no audit event emitted on auth failure, and no
REQ-OPS-10 peer trust-tier demotion. Also documents that handshake steps
are not nonce-cache-protected (only heartbeat/route/completion paths are).

Reuses the shared tests/integration/federation/conftest.py harness.

test(#199): federation message-ordering under reordering network (card t_e5c48344) ... 3f845f2a30

Adds tests/integration/federation/test_ordering_reorder.py: two federation
peers joined by a ReorderNetwork that scrambles data-plane delivery (packet
reordering) and retransmits duplicates (TCP-style), parameterised over mild /
heavy / extreme profiles. Asserts the receiver reconstructs the original
publish order end-to-end from seq numbers and that reordering-induced
duplicates are deduped (REQ-CONF-3 first-wins at the route level).

12 tests, deterministic seeds, all green; pre-commit gate clean.

test(#199): federation peer partition + reconnect suite (card t_b468f463) 93c32337fe

test(#199): federation graceful vs abrupt peer shutdown suite (card t_dbcb647d) 115f189aa2

ci(#199): run full federation integration suite (deps cache, per-test timeout, failure-log artifact upload) ... 39753e4af1

Wires a dedicated Forgejo Actions workflow that runs the entirety of the
federation integration suite (pytest -m federation) on the mesh-seed Actions
runner. Deps are minimal and pip-cached; pytest-timeout bounds each test class
(no single hung test can stall the job); junit + stdout log are uploaded as
artifacts on always(). The pull_request trigger gates the federation
implementation PR: all federation tests green required to merge.

ci(#199): upload federation logs via upload-artifact@v3 (v4 v2-API unsupported on Forgejo) ... e29166c863

The federation suite runs green in CI (63 passed, 5 xfailed) but the
upload-artifact@v4 step fails this Forgejo version with GHESNotSupportedError
(v2 artifact API unsupported). Pin upload-artifact@v3 (verified tag v3.2.1,
SHA ff15f03) which uses the supported v1 API, so failure logs still upload.

docs(#199): document federation integration suite and its CI job in DEVELOPER_GUIDE 8584898aa5

test(#199): land federation integration + failure-mode suite and CI workflow into PR #275 ... 27a76f4315

Merge of smoke/federation-failure-tests (8584898) into docs/cross-mesh-federation-design (4418288):
- tests/integration/federation/ (two-node, three-node fanout, partition/reconnect,
  auth/replay, ordering/reorder, graceful-shutdown) + shared conftest harness
- .forgejo/workflows/federation.yml: Federation Integration job gating PR #275
  (deps cache, per-test timeout=300s, junit+stdout artifact via upload-artifact@v3)
- pytest.ini: register 'federation' marker
- DEVELOPER_GUIDE.md: document suite + CI job

Mark added 1 commit 2026-08-19 22:14:10 +00:00

fix(#199): REQ-CONF-3 heartbeat freshness — LWW by issued_at in _ingest_heartbeat ... c1e1d8a5fd

_ingest_heartbeat unconditionally overwrote live node state, letting a
stale in-window heartbeat (older issued_at) regress a fresh one. Apply
last-writer-wins by issued_at (tie -> larger mesh_id wins) matching the
DB-reload path, and drop the xfail marker on the stale-heartbeat
integration test.

Federation suite: 64 passed / 4 xfailed / 0 failed (stale-heartbeat test
now passes without xfail).

Mark merged commit 4aaad6f1ef into master 2026-09-10 13:15:27 +00:00

Mark referenced this pull request from a commit 2026-09-10 13:15:28 +00:00

feat(#276)+docs(#199): cross-mesh federation protocol v1 (design, spec, implementation) (#275)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

blocked:children

  

blocked:commit-ready-failed

  

blocked:depends-231

Depends on #231 (Discord Bot Foundation) landing first. Do not pick until #231 is closed.

  

blocked:depends-236

Depends on #236 (Phase 1) landing first. Do not pick until #236 is closed.

  

blocked:depends-237

Depends on #237 (Phase 2) landing first. Do not pick until #237 is closed.

  

blocked:depends-238

Depends on #238 (Phase 3) landing first. Do not pick until #238 is closed.

  

blocked:depends-239

Depends on #239 (Phase 4) landing first. Do not pick until #239 is closed.

  

blocked:gate-fetch

  

blocked:gate-fetch

  

blocked:gate-lint

  

blocked:gate-research

  

blocked:needs-attention

Janitor escalated: pipeline terminated without producing expected changes. Human review required.

  

blocked:needs-decision

  

blocked:no-changes

  

blocked:no-procedure

  

blocked:preflight

Preflight check failed

  

blocked:verify-failed

  

bug

Something is broken

  

enhancement

New feature or request

  

feature-request

  

high-priority

Critical path items

  

in-progress

Issue is currently being worked on by a quickfix runner

  

lint-cleanup

Repo lint baseline is dirty

  

needs-decompose

  

needs-triage

  

ops-retry

  

ops-task

Operations task — install, configure, deploy, API calls

  

phase-5

Phase 5 Self-Building Infrastructure

  

quickfix

Pipe to opencode for direct fix

  

security/medium

Medium severity security issue

  

self-building

Autonomous code generation and self-improvement

  

triage-approved

Approved by triage — queued for fixer pipeline

No labels

blocked:children

blocked:commit-ready-failed

blocked:depends-231

blocked:depends-236

blocked:depends-237

blocked:depends-238

blocked:depends-239

blocked:gate-fetch

blocked:gate-fetch

blocked:gate-lint

blocked:gate-research

blocked:needs-attention

blocked:needs-decision

blocked:no-changes

blocked:no-procedure

blocked:preflight

blocked:verify-failed

bug

enhancement

feature-request

high-priority

in-progress

lint-cleanup

needs-decompose

needs-triage

ops-retry

ops-task

phase-5

quickfix

security/medium

self-building

triage-approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/mesh-seed!275

No description provided.