Mark/mesh-seed

Fork 0

[Reliability] Dead-letter replay tooling and retention policy #23

New issue

Closed

opened 2026-03-12 17:24:43 +00:00 by Mark · 3 comments

Mark commented

2026-03-12 17:24:43 +00:00

Owner

Inflection from v1 backplane implementation.

Current state

substrate/backplane.py now dead-letters messages after retry threshold in consumer-group flow.

Gap

No operator tooling yet for:

listing dead-lettered messages by channel/time
replaying dead-letter envelopes safely with idempotency controls
retention/TTL policy for mesh.deadletter

Proposed

Add scripts/deadletter_inspect.py and scripts/deadletter_replay.py
Add replay guardrails (max batch, dry-run, idempotency prefix)
Add retention policy docs + periodic cleanup job

Acceptance

Dead-letter queue can be inspected and replayed from CLI
Replay does not duplicate already-settled intents
Policy documented in specs/v1-protocol-and-gates.md

Inflection from v1 backplane implementation. ## Current state `substrate/backplane.py` now dead-letters messages after retry threshold in consumer-group flow. ## Gap No operator tooling yet for: - listing dead-lettered messages by channel/time - replaying dead-letter envelopes safely with idempotency controls - retention/TTL policy for `mesh.deadletter` ## Proposed - Add `scripts/deadletter_inspect.py` and `scripts/deadletter_replay.py` - Add replay guardrails (max batch, dry-run, idempotency prefix) - Add retention policy docs + periodic cleanup job ## Acceptance - Dead-letter queue can be inspected and replayed from CLI - Replay does not duplicate already-settled intents - Policy documented in `specs/v1-protocol-and-gates.md`

Mark commented

2026-03-12 18:00:22 +00:00

Author

Owner

Status update (implemented in current working tree):

Added dead-letter operator CLI: scripts/deadletter_cli.py
- inspect subcommand for queue visibility
- replay subcommand with dry-run + idempotency prefix
Backplane retry -> dead-letter behavior implemented in substrate/backplane.py

Remaining: retention/cleanup policy automation and optional split into separate inspect/replay scripts if preferred.

Status update (implemented in current working tree): - Added dead-letter operator CLI: `scripts/deadletter_cli.py` - `inspect` subcommand for queue visibility - `replay` subcommand with dry-run + idempotency prefix - Backplane retry -> dead-letter behavior implemented in `substrate/backplane.py` Remaining: retention/cleanup policy automation and optional split into separate inspect/replay scripts if preferred.

Mark commented

2026-03-13 01:27:22 +00:00

Author

Owner

Closing update:

Implemented and verified in working tree:

dead-letter inspect/replay CLI in scripts/deadletter_cli.py
replay guardrails including settled-intent protection by default
retention cleanup command (cleanup --maxlen)
policy/docs updated in specs/v1-protocol-and-gates.md and USAGE.md

Acceptance criteria are satisfied; closing issue.

Closing update: Implemented and verified in working tree: - dead-letter inspect/replay CLI in `scripts/deadletter_cli.py` - replay guardrails including settled-intent protection by default - retention cleanup command (`cleanup --maxlen`) - policy/docs updated in `specs/v1-protocol-and-gates.md` and `USAGE.md` Acceptance criteria are satisfied; closing issue.

Mark closed this issue

2026-03-13 01:27:28 +00:00

Mark referenced this issue

2026-03-14 18:59:03 +00:00

[Phase 0] Foundation - MVP Completion #27

Mark commented

2026-03-14 19:12:24 +00:00

Author

Owner

Phase Dependency Mapping

Linked to Phase Tracker: #31 (Phase 4: Optimization)

Prerequisites:

#11 (SQLite ledger) ✅ - for replay state
#8 (Integration test) - test replay mechanism
#19 (Multi-node mesh) - distributed replay

Purpose: Reliability engineering
Priority: P2 (production readiness)

Acceptance Criteria:

Failed intents queue to dead-letter
Replay with exponential backoff
Retention policy (30 days default)
Admin tooling for inspection/replay
Metrics on failure rates

Related: #24 (stream hygiene), #25 (settlement lock)

## Phase Dependency Mapping **Linked to Phase Tracker:** #31 (Phase 4: Optimization) **Prerequisites:** - #11 (SQLite ledger) ✅ - for replay state - #8 (Integration test) - test replay mechanism - #19 (Multi-node mesh) - distributed replay **Purpose:** Reliability engineering **Priority:** P2 (production readiness) **Acceptance Criteria:** - Failed intents queue to dead-letter - Replay with exponential backoff - Retention policy (30 days default) - Admin tooling for inspection/replay - Metrics on failure rates **Related:** #24 (stream hygiene), #25 (settlement lock)

Mark referenced this issue

2026-03-14 19:12:36 +00:00

[Reliability] Add stream hygiene controls for test and replay runs #24

Mark referenced this issue

2026-03-14 19:12:48 +00:00

[Meta] Create project milestones and organize roadmap #26

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/mesh-seed#23

No description provided.

Rows
Columns