Merge smoke/federation-failure-tests -> master #288

Closed

Mark wants to merge 12 commits from smoke/federation-failure-tests into master

pull from: smoke/federation-failure-tests

merge into: Mark:master

Mark:main

Conversation 0 Commits 12 Files changed 37 +9033 -69

Mark commented 2026-09-12 16:51:01 +00:00

Owner

Copy link

Auto-opened by batch merger to land smoke/federation-failure-tests into master.

Auto-opened by batch merger to land smoke/federation-failure-tests into master.

Mark added 12 commits 2026-09-12 16:51:01 +00:00

docs(#199): cross-mesh federation protocol design & requirements ... 1b583e1ad8

Research survey (Matrix, ActivityPub, federated identity, mesh networking)
plus 41 REQ-* requirements covering topology, auth/identity, envelope,
conflict resolution, and operational failure modes for META-001.
Companion to docs/federation/spec.md (normative spec, child task).

docs(#199): cross-mesh federation protocol normative spec ... 5d94f6b8d6

Wire-level envelope JSON schema (federation/v1), canonical serialization +
HMAC-SHA256 signature scheme with conformance test vector, handshake state
machine (hello/challenge/response/ack) with TOFU pinning and key rotation,
capability negotiation, HTTP transport binding and error model, and worked
examples for join handshake, federated message relay, and graceful
disconnect. Cross-references the 41 REQ-* ids from design.md.

feat(#276): cross-mesh federation protocol v1 (envelope, handshake, router, HTTP binding) ... f292cce09c

Implements META-001 per docs/federation/spec.md and the plan at
.kilo/plans/META-001-cross-mesh-federation-protocol.md:

- runtime/federation_protocol.py: federation/v1 envelope codec, canonical
  serialization, HMAC-SHA256 sign/verify with freshness window + replay LRU
  (spec 3-4, including the 4.4 test vector)
- runtime/federation_handshake.py: hello/challenge/response/ack state machine,
  ed25519 identity proof, TOFU pinning, split-brain detection (spec 5)
- runtime/federation_credentials.py: per-peer secret resolution (env ->
  SecretsManager -> legacy token) and rotation
- runtime/mesh_federation.py: signed heartbeats/intent routes, real
  _check_node_timeouts (parsed datetimes, online->degraded->offline),
  _dispatch_local_route wired to FederationWorker, get_federation_health,
  removed duplicate mesh_forwarded_intents CREATE
- runtime/cross_mesh_forwarding.py: envelope wrapping in forward_intent,
  accept_envelope, resolver-based bearer (legacy fallback + metric)
- runtime/http_entrypoint.py: POST /v1/federation/{handshake,heartbeat,
  intent,disconnect}, GET /v1/federation/{siblings,health}, trust-tier gate,
  spec 7.3 error model
- ledger migration 0008 (federation_peer_keys, federation_heartbeat_log,
  federation_route_intents) + MeshLedger/federation repo helpers
- federation_worker.py: use route_intent() (IntentRouter class never existed)
- scripts/demo_federation.py CLI harness booting two meshes end-to-end
- tests: envelope codec, handshake state machine, credentials, mesh_federation,
  gateway envelope wrapping, HTTP routes, two-mesh integration (all green)
- docs: docs/reference/MESH_FEDERATION.md, API.md federation endpoints,
  config/mesh.yaml federation section

Demo (in PR description):
  python scripts/demo_federation.py
  -> peer discovery, 4-step handshake w/ TOFU pinning, intent forward
     round-trip, shared-secret rotation, all end-to-end in-process

test(#199): federation failure-mode integration suite + mark two-mesh suite for CI ... c5a72285b9

Adds tests/integration/test_federation_failure_modes.py covering:
- 3-node fanout (discovery + A->C routing with non-target ignoring)
- peer partition and reconnect (timeout sweep -> degraded -> offline -> rejoin)
- auth failure / replay attack rejection (wrong secret, nonce replay via
  backplane and HTTP handlers)
- message ordering under reordering network (first-received wins for
  conflicting completions, idempotent duplicates, heartbeat freshness)
- graceful shutdown (goodbye envelope + stop())

Also marks test_federation_two_meshes.py with pytestmark = integration so
the existing 2-node suite is picked up by CI's 'pytest -m integration' job.

One known gap documented as xfail: REQ-CONF-3 heartbeat freshness
(last-writer-wins by issued_at) is not implemented in _ingest_heartbeat;
a stale in-window heartbeat regresses fresh node state.

test(#199): federation 2-node happy-path + 3-node fanout suite (card t_a9b870de) ... 52aeec7b5e

Add tests/integration/federation/ with a reusable harness (conftest) and:
- test_two_node.py: two peers exchange a known message set and converge
  on ordering, payload, and ledger state.
- test_three_node_fanout.py: one publisher fans a replicated message to
  two subscribers; all three agree on ordering and payload.

Register the 'federation' pytest marker in pytest.ini and mark the
pre-existing two-mesh + failure-mode suites with it too, so the CI
'pytest -m federation' job runs the whole federation suite (28 passed,
1 xfailed locally).

Also applies two mechanical ruff-format v0.3.0 rewraps to
test_federation_failure_modes.py (pre-existing assert style) so the
pooled editor's pinned formatter stays clean.

Note: tests/integration/federation/test_auth_replay.py is a sibling card
(t_8ee7b012) file landing in this dir concurrently; left uncommitted.

test(#199): federation auth-failure + replay-attack rejection suite (card t_8ee7b012) ... dd6bfc1164

Covers auth failure at handshake (invalid/expired/future-dated/revoked
credentials rejected pre-state, never established) and replay attack
(verbatim + timestamp-manipulated re-injection rejected and flagged as
replay_detected through MeshFederation's NonceCache path).

Two spec requirements surfaced as documented xfail (matching the
REQ-CONF-3 convention): no audit event emitted on auth failure, and no
REQ-OPS-10 peer trust-tier demotion. Also documents that handshake steps
are not nonce-cache-protected (only heartbeat/route/completion paths are).

Reuses the shared tests/integration/federation/conftest.py harness.

test(#199): federation message-ordering under reordering network (card t_e5c48344) ... 3f845f2a30

Adds tests/integration/federation/test_ordering_reorder.py: two federation
peers joined by a ReorderNetwork that scrambles data-plane delivery (packet
reordering) and retransmits duplicates (TCP-style), parameterised over mild /
heavy / extreme profiles. Asserts the receiver reconstructs the original
publish order end-to-end from seq numbers and that reordering-induced
duplicates are deduped (REQ-CONF-3 first-wins at the route level).

12 tests, deterministic seeds, all green; pre-commit gate clean.

test(#199): federation peer partition + reconnect suite (card t_b468f463) 93c32337fe

test(#199): federation graceful vs abrupt peer shutdown suite (card t_dbcb647d) 115f189aa2

ci(#199): run full federation integration suite (deps cache, per-test timeout, failure-log artifact upload) ... 39753e4af1

Wires a dedicated Forgejo Actions workflow that runs the entirety of the
federation integration suite (pytest -m federation) on the mesh-seed Actions
runner. Deps are minimal and pip-cached; pytest-timeout bounds each test class
(no single hung test can stall the job); junit + stdout log are uploaded as
artifacts on always(). The pull_request trigger gates the federation
implementation PR: all federation tests green required to merge.

ci(#199): upload federation logs via upload-artifact@v3 (v4 v2-API unsupported on Forgejo) ... e29166c863

The federation suite runs green in CI (63 passed, 5 xfailed) but the
upload-artifact@v4 step fails this Forgejo version with GHESNotSupportedError
(v2 artifact API unsupported). Pin upload-artifact@v3 (verified tag v3.2.1,
SHA ff15f03) which uses the supported v1 API, so failure logs still upload.

docs(#199): document federation integration suite and its CI job in DEVELOPER_GUIDE 8584898aa5

Mark closed this pull request 2026-09-21 00:57:12 +00:00

Some checks failed

Hide all checks

Federation Integration / Federation Integration Suite (push) Successful in 3m29s

Details

CI / Lint and Format (pull_request) Failing after 27s

Details

CI / Type Check (pull_request) Successful in 41s

Details

CI / Security Check (pull_request) Failing after 1m17s

Details

CI / Code Quality Checks (pull_request) Successful in 22s

Details

CI / Unit Tests (pull_request) Failing after 52s

Details

CI / Full Test Suite (pull_request) Has been skipped

Details

CI / Integration Tests (pull_request) Successful in 33s

Details

CI / Smoke Tests (pull_request) Failing after 44s

Details

Federation Integration / Federation Integration Suite (pull_request) Successful in 1m38s

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

blocked:children

  

blocked:commit-ready-failed

  

blocked:depends-231

Depends on #231 (Discord Bot Foundation) landing first. Do not pick until #231 is closed.

  

blocked:depends-236

Depends on #236 (Phase 1) landing first. Do not pick until #236 is closed.

  

blocked:depends-237

Depends on #237 (Phase 2) landing first. Do not pick until #237 is closed.

  

blocked:depends-238

Depends on #238 (Phase 3) landing first. Do not pick until #238 is closed.

  

blocked:depends-239

Depends on #239 (Phase 4) landing first. Do not pick until #239 is closed.

  

blocked:gate-fetch

  

blocked:gate-fetch

  

blocked:gate-lint

  

blocked:gate-research

  

blocked:needs-attention

Janitor escalated: pipeline terminated without producing expected changes. Human review required.

  

blocked:needs-decision

  

blocked:no-changes

  

blocked:no-procedure

  

blocked:preflight

Preflight check failed

  

blocked:verify-failed

  

bug

Something is broken

  

enhancement

New feature or request

  

feature-request

  

high-priority

Critical path items

  

in-progress

Issue is currently being worked on by a quickfix runner

  

lint-cleanup

Repo lint baseline is dirty

  

needs-decompose

  

needs-triage

  

ops-retry

  

ops-task

Operations task — install, configure, deploy, API calls

  

phase-5

Phase 5 Self-Building Infrastructure

  

quickfix

Pipe to opencode for direct fix

  

security/medium

Medium severity security issue

  

self-building

Autonomous code generation and self-improvement

  

triage-approved

Approved by triage — queued for fixer pipeline

No labels

blocked:children

blocked:commit-ready-failed

blocked:depends-231

blocked:depends-236

blocked:depends-237

blocked:depends-238

blocked:depends-239

blocked:gate-fetch

blocked:gate-fetch

blocked:gate-lint

blocked:gate-research

blocked:needs-attention

blocked:needs-decision

blocked:no-changes

blocked:no-procedure

blocked:preflight

blocked:verify-failed

bug

enhancement

feature-request

high-priority

in-progress

lint-cleanup

needs-decompose

needs-triage

ops-retry

ops-task

phase-5

quickfix

security/medium

self-building

triage-approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/mesh-seed!288

No description provided.