Mark/mesh-seed

Fork 0

SLO instrumentation gap: Redis P95 latency metric not instrumented #246

New issue

Closed

opened 2026-07-22 18:09:58 +00:00 by Mark · 1 comment

Mark commented

2026-07-22 18:09:58 +00:00

Owner

Finding

The Redis SLO spec (slo/specs/redis.yaml) defines a P95 latency SLO of <2s using the metric redis_command_duration_seconds. However, this metric is marked as un_instrumented: true — the Prometheus Redis exporter does not currently capture per-command latency distributions. The SLO currently uses the gateway request latency as a proxy metric.

SLI spec location: slo/specs/redis.yaml, latency_p95 section
Metric: redis_command_duration_seconds (histogram, percentile: 0.95)
Current proxy: mesh_http_request_duration_seconds{job="mesh-gateway"}
Reproduction steps:

Deploy the mesh-seed stack with Prometheus monitoring
Query Prometheus for redis_command_duration_seconds — metric does not exist
The slo_recording_rule for redis latency will resolve against the proxy metric, not Redis-native latency

Expected: A Prometheus Redis exporter (e.g., oliver006/redis_exporter) sidecar on the Redis StatefulSet that exposes redis_command_duration_seconds_bucket histogram, enabling real Redis-native latency measurement.

Actual: Redis latency SLO is measured via gateway HTTP request latency as proxy, which does not reflect actual Redis command performance.

Priority: Medium — Redis is tier-0 infrastructure (every service depends on it), but the proxy metric provides a reasonable approximation until instrumented.

Kanban card: t_067f4545 (SLO findings from mesh-seed SLO spec review)

### Finding The Redis SLO spec (`slo/specs/redis.yaml`) defines a P95 latency SLO of <2s using the metric `redis_command_duration_seconds`. However, this metric is marked as `un_instrumented: true` — the Prometheus Redis exporter does not currently capture per-command latency distributions. The SLO currently uses the gateway request latency as a proxy metric. **SLI spec location:** slo/specs/redis.yaml, latency_p95 section **Metric:** redis_command_duration_seconds (histogram, percentile: 0.95) **Current proxy:** mesh_http_request_duration_seconds{job="mesh-gateway"} **Reproduction steps:** 1. Deploy the mesh-seed stack with Prometheus monitoring 2. Query Prometheus for `redis_command_duration_seconds` — metric does not exist 3. The slo_recording_rule for redis latency will resolve against the proxy metric, not Redis-native latency **Expected:** A Prometheus Redis exporter (e.g., oliver006/redis_exporter) sidecar on the Redis StatefulSet that exposes `redis_command_duration_seconds_bucket` histogram, enabling real Redis-native latency measurement. **Actual:** Redis latency SLO is measured via gateway HTTP request latency as proxy, which does not reflect actual Redis command performance. **Priority:** Medium — Redis is tier-0 infrastructure (every service depends on it), but the proxy metric provides a reasonable approximation until instrumented. --- *Kanban card: t_067f4545 (SLO findings from mesh-seed SLO spec review)*

Mark added the

enhancement

label

2026-07-22 18:10:04 +00:00

Mark added the

blocked:preflight

label

2026-07-31 04:26:41 +00:00

Mark removed the

blocked:preflight

label

2026-08-08 14:27:22 +00:00

Mark commented

2026-08-12 10:34:17 +00:00

Author

Owner

Reopened by kilo-gsd-runner (kanban t_2f5c2666): the 2026-08-12 03:29 kilo-gsd run closed this issue after "All phases passed", but NO commit referencing #246 landed on origin/master (HEAD == cfaf330, #241 work) and the deliverable was not implemented — slo/specs/redis.yaml still has un_instrumented: true with the mesh_http_request_duration_seconds proxy; no redis_exporter sidecar or redis_command_duration_seconds_bucket histogram exists anywhere in the repo. The Verify gate also ran the wrong plan (mesh-seed-241 fallback, no #246 plan in .kilo/plans), so its PASS exercised unrelated migration checks. Closing was premature; issue is open again for a real fix.

Reopened by kilo-gsd-runner (kanban t_2f5c2666): the 2026-08-12 03:29 kilo-gsd run closed this issue after "All phases passed", but NO commit referencing #246 landed on origin/master (HEAD == cfaf330, #241 work) and the deliverable was not implemented — slo/specs/redis.yaml still has un_instrumented: true with the mesh_http_request_duration_seconds proxy; no redis_exporter sidecar or redis_command_duration_seconds_bucket histogram exists anywhere in the repo. The Verify gate also ran the wrong plan (mesh-seed-241 fallback, no #246 plan in .kilo/plans), so its PASS exercised unrelated migration checks. Closing was premature; issue is open again for a real fix.

Mark

2026-08-12 15:30:52 +00:00

closed this issue
removed the
enhancement
label

Mark referenced this issue from a commit

2026-08-13 08:46:25 +00:00

feat(#246): instrument Redis P95 latency with sidecar exporter

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/mesh-seed#246

No description provided.

Rows
Columns