feat(slo): define SLO specs for all mesh-seed services #242

Merged

Mark merged 2 commits from slo-specs-150 into master

2026-07-22 16:54:06 +00:00

Mark commented

2026-07-22 15:28:43 +00:00

Owner

Define Availability (99.9%), Latency P95 (<2s), and Success Rate (>95%)
SLO targets for 9 services in mesh-seed.

See slo/specs/ for per-service YAML definitions with SLI sources,
measurement windows, burn-rate config, and rationale.

Closes #150

Define Availability (99.9%), Latency P95 (<2s), and Success Rate (>95%) SLO targets for 9 services in mesh-seed. See slo/specs/ for per-service YAML definitions with SLI sources, measurement windows, burn-rate config, and rationale. Closes #150

Mark added 1 commit

2026-07-22 15:28:43 +00:00

feat(slo): define SLO specs for all mesh-seed services (#150 )

CI / Lint and Format (pull_request) Failing after 28s

Details

CI / Type Check (pull_request) Successful in 40s

Details

CI / Security Check (pull_request) Failing after 49s

Details

CI / Code Quality Checks (pull_request) Successful in 12s

Details

CI / Unit Tests (pull_request) Failing after 32s

Details

CI / Full Test Suite (pull_request) Has been skipped

Details

CI / Integration Tests (pull_request) Successful in 33s

Details

CI / Smoke Tests (pull_request) Failing after 31s

Details

f97562ff60

Define Availability (99.9%), Latency P95 (<2s), and Success Rate (>95%)
SLO targets for 9 services in mesh-seed:

Deployed services (k8s):
  - gateway: API gateway (port 8080, 2 replicas)
  - mesh-node: Core runtime (port 9100, 2 replicas)
  - mesh-worker: Ephemeral worker pool (HPA 1-10)
  - mesh-operator: K8s controller (1 replica)

Infrastructure:
  - redis: Backplane/cache (StatefulSet, single replica)
  - prometheus: Metrics collection (TSDB, 30d retention)
  - grafana: Dashboards and visualization

Substrate components:
  - backplane: Redis pub/sub message bus
  - ledger: SQLite economic state database

Each spec includes:
  - SLI source (Prometheus metric name, probe, recording rules)
  - Measurement window (30d rolling for availability;
    5m+1h multi-window for latency burn-rate)
  - Rationale comments explaining each target
  - Dependencies and their SLI impact
  - Instrumentation status for un-instrumented metrics

Closes #150

Mark added 1 commit

2026-07-22 16:46:28 +00:00

feat(slo): implement compliance tracking, burn-rate alerts, and Grafana dashboard (#150 )

CI / Type Check (pull_request) Successful in 35s

Details

CI / Lint and Format (pull_request) Failing after 1m33s

Details

CI / Security Check (pull_request) Failing after 48s

Details

CI / Code Quality Checks (pull_request) Successful in 35s

Details

CI / Integration Tests (pull_request) Successful in 29s

Details

CI / Smoke Tests (pull_request) Failing after 1m2s

Details

CI / Unit Tests (pull_request) Failing after 3m16s

Details

CI / Full Test Suite (pull_request) Has been skipped

Details

fe7bed0f67

- Add 27 Prometheus recording rules (monitoring/prometheus/slo_rules.yml)
  computing per-service SLI ratios for all 9 services across availability,
  latency P95, and success rate SLOs
- Add 26 burn-rate alerting rules (monitoring/prometheus/slo_alerts.yml)
  with multi-window detection: critical at 14.4x/5m, warning at 6x/30m
- Update prometheus.yml to load SLO recording and alerting rules
- Add 14-panel Grafana SLO Compliance dashboard with:
  - Burn-rate timeseries for gateway, mesh-node, worker, redis
  - P95 latency panels with threshold overlays
  - SLO compliance overview and violation counts
  - Error budget usage table
  - Active alert timeline
- Add scripts/slo_compliance_report.py CLI tool for querying
  Prometheus and reporting compliance, error budget, and burn rate
- Add validation script and update slo/README.md with monitoring docs

Closes #150