Mark/mesh-seed

Fork 0

[Reliability] Settlement lock state machine for retry-after-failure #25

New issue

Closed

opened 2026-03-12 23:49:47 +00:00 by Mark · 2 comments

Mark commented

2026-03-12 23:49:47 +00:00

Owner

Observed during lease-expiry recovery harness.

Current behavior

Crash path emits intent.failed and settlement worker marks settlement lock completed.
A later successful retry execution emits result.committed, but settlement lock returns already_settled and blocks reward settlement.

Problem

Current settlement lock semantics treat failed settlement as terminal, which conflicts with retry/recovery flows where later success should settle.

Proposed

Introduce settlement attempt epochs/versioning keyed by intent + attempt id, OR
Reopen settlement lock when claim lease is reassigned for a new active attempt, OR
Encode terminality by intent lifecycle state and disallow retries explicitly.

Acceptance

Crash -> retry -> success path yields exactly one final settlement for winning attempt
No duplicate settlement effects across attempts
Behavior documented in protocol/recovery spec

Observed during lease-expiry recovery harness. ## Current behavior - Crash path emits `intent.failed` and settlement worker marks settlement lock completed. - A later successful retry execution emits `result.committed`, but settlement lock returns `already_settled` and blocks reward settlement. ## Problem Current settlement lock semantics treat failed settlement as terminal, which conflicts with retry/recovery flows where later success should settle. ## Proposed - Introduce settlement attempt epochs/versioning keyed by intent + attempt id, OR - Reopen settlement lock when claim lease is reassigned for a new active attempt, OR - Encode terminality by intent lifecycle state and disallow retries explicitly. ## Acceptance - Crash -> retry -> success path yields exactly one final settlement for winning attempt - No duplicate settlement effects across attempts - Behavior documented in protocol/recovery spec

Mark referenced this issue

2026-03-12 23:49:47 +00:00

[Federation] Multi-node mesh with tailnet #19

Mark referenced this issue

2026-03-12 23:58:07 +00:00

[Federation] Multi-node mesh with tailnet #19

Mark commented

2026-03-12 23:58:07 +00:00

Author

Owner

Implemented fix in working tree:

Settlement locks are now attempt-scoped (intent_id, attempt) using settlement_locks_v2.
Claim lease tracks attempt epochs and increments on reassignment after failure/expiry.
Worker emits attempt on claim/result payloads.
Settlement worker uses attempt-aware locking and attempt-aware claim-lease completion.

Verification:

Unit coverage added (test_retry_after_failure_can_settle_new_attempt)
Lease recovery harness now shows expected behavior:
- attempt 1 failed settlement succeeds
- attempt 2 retry success settlement also succeeds
- no duplicate settlement within an attempt

This resolves the retry-after-failure settlement lock inflection.

Implemented fix in working tree: - Settlement locks are now attempt-scoped (`intent_id`, `attempt`) using `settlement_locks_v2`. - Claim lease tracks attempt epochs and increments on reassignment after failure/expiry. - Worker emits `attempt` on claim/result payloads. - Settlement worker uses attempt-aware locking and attempt-aware claim-lease completion. Verification: - Unit coverage added (`test_retry_after_failure_can_settle_new_attempt`) - Lease recovery harness now shows expected behavior: - attempt 1 failed settlement succeeds - attempt 2 retry success settlement also succeeds - no duplicate settlement within an attempt This resolves the retry-after-failure settlement lock inflection.

Mark closed this issue

2026-03-12 23:58:07 +00:00

Mark referenced this issue

2026-03-14 18:59:03 +00:00

[Phase 0] Foundation - MVP Completion #27

Mark referenced this issue

2026-03-14 19:12:24 +00:00

[Reliability] Dead-letter replay tooling and retention policy #23

Mark commented

2026-03-14 19:12:38 +00:00

Author

Owner

Phase Dependency Mapping

Linked to Phase Tracker: #31 (Phase 4: Optimization)

Prerequisites:

#11 (SQLite ledger) ✅ - settlement tracking
#19 (Multi-node mesh) - distributed settlement

Type: Reliability/Accounting
Priority: P2 (financial correctness)

Purpose: Prevent double-spend in settlement

Acceptance Criteria:

Lock state machine for in-flight settlements
Retry with idempotency keys
Failure recovery without double-pay
Audit trail for all settlement attempts

Critical for: Economic integrity

## Phase Dependency Mapping **Linked to Phase Tracker:** #31 (Phase 4: Optimization) **Prerequisites:** - #11 (SQLite ledger) ✅ - settlement tracking - #19 (Multi-node mesh) - distributed settlement **Type:** Reliability/Accounting **Priority:** P2 (financial correctness) **Purpose:** Prevent double-spend in settlement **Acceptance Criteria:** - Lock state machine for in-flight settlements - Retry with idempotency keys - Failure recovery without double-pay - Audit trail for all settlement attempts **Critical for:** Economic integrity

Mark referenced this issue

2026-03-14 19:12:48 +00:00

[Meta] Create project milestones and organize roadmap #26

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/mesh-seed#25

No description provided.

Rows
Columns