GPU-003: GPU Monitoring

Scope: medium Risk: low Spec: aligned Type: feature
Summary: Add real-time GPU utilization monitoring to provide operational visibility into per-device GPU metrics (memory, compute, temperature) across the mesh.
Suggestion: Extend the GPU manager (or add a new GPU monitor module) to periodically query real nvidia-smi data instead of using synthetic values in apply_load(), add a CLI command (e.g. mesh gpu status) for live per-GPU health, and expose an API endpoint for dashboard integration. The existing ledger gpu_metrics table and resource_monitor.py's nvidia-smi collection are building blocks that need bridging into a cohesive operator-facing monitoring capability.
Rationale: Scope is medium: requires changes across GPU manager (real vs. synthetic metrics), likely a new CLI module, tests, and possibly a monitoring thread — but builds on existing infrastructure (gpu_metrics table, resource_monitor GPU collection, GPU device tracking) so no new subsystems. Risk is low: purely additive observability — no changes to allocation/routing logic or intent flow. Spec is aligned: the roadmap lists GPU Cluster (month 6) with monitoring as prerequisite, GPU docs all list utilization monitoring as future work, and the existing GPU-001/002 series completes detection+routing leaving monitoring as the natural next step. Type is feature: adds new operational capability, not a bug fix or refactor.

Action: Review each blocking factor above. Remove blocked:needs-decision label to route back to triage.

## ⏸ Needs Decision **Blocking factors:** - **Didn't meet auto-approve criteria** — scope=medium, risk=low, spec=aligned --- **Scope:** medium **Risk:** low **Spec:** aligned **Type:** feature **Summary:** Add real-time GPU utilization monitoring to provide operational visibility into per-device GPU metrics (memory, compute, temperature) across the mesh. **Suggestion:** Extend the GPU manager (or add a new GPU monitor module) to periodically query real nvidia-smi data instead of using synthetic values in apply_load(), add a CLI command (e.g. `mesh gpu status`) for live per-GPU health, and expose an API endpoint for dashboard integration. The existing ledger gpu_metrics table and resource_monitor.py's nvidia-smi collection are building blocks that need bridging into a cohesive operator-facing monitoring capability. **Rationale:** Scope is medium: requires changes across GPU manager (real vs. synthetic metrics), likely a new CLI module, tests, and possibly a monitoring thread — but builds on existing infrastructure (gpu_metrics table, resource_monitor GPU collection, GPU device tracking) so no new subsystems. Risk is low: purely additive observability — no changes to allocation/routing logic or intent flow. Spec is aligned: the roadmap lists GPU Cluster (month 6) with monitoring as prerequisite, GPU docs all list utilization monitoring as future work, and the existing GPU-001/002 series completes detection+routing leaving monitoring as the natural next step. Type is feature: adds new operational capability, not a bug fix or refactor. **Action:** Review each blocking factor above. Remove `blocked:needs-decision` label to route back to triage.

Mark added

triage-approved

in-progress

and removed

blocked:needs-decision

labels

2026-05-28 07:18:57 +00:00

Mark removed the

triage-approved

in-progress

labels

2026-05-29 14:52:41 +00:00

Mark added the

triage-approved

label

2026-06-07 04:24:48 +00:00

Mark removed the

triage-approved

label

2026-06-15 15:30:52 +00:00

Mark added the

blocked:needs-decision

label

2026-06-20 20:43:50 +00:00

Mark commented

2026-06-20 20:43:50 +00:00

Author

Owner

Triage: decompose failed

Triage pipeline could not create a sub-issue for auto-decomposition (API error or repo permissions). Marking this blocked:needs-decision to prevent re-evaluation. Remove the label to retry.

## Triage: decompose failed Triage pipeline could not create a sub-issue for auto-decomposition (API error or repo permissions). Marking this `blocked:needs-decision` to prevent re-evaluation. Remove the label to retry.

Mark added the

blocked:preflight

label

2026-07-31 04:26:44 +00:00

Mark commented

2026-08-11 04:32:35 +00:00

Author

Owner

Approved — removing blocked:needs-decision. Triage failed to auto-approve, but this is low/medium risk, aligned.

Mark added

in-progress

and removed

blocked:needs-decision

blocked:preflight

labels

2026-08-11 04:33:16 +00:00

Mark removed the

in-progress

label

2026-08-12 05:09:12 +00:00

Mark referenced this issue from a commit

2026-08-12 06:03:40 +00:00

feat(#197): add GPU monitoring with saturation, thermal, starvation, orphan signals

Mark closed this issue

2026-08-12 07:28:01 +00:00

Mark commented

2026-08-12 07:28:38 +00:00

Author

Owner

Closed by fixup verification

The GPU monitoring deliverable for this issue has landed and is verified on origin/master.

Commit: f98427f feat(#197): add GPU monitoring with saturation, thermal, starvation, orphan signals (10 files, +1083)
Verified on remote: git ls-remote origin master contains f98427f as ancestor
Lint gate: PASS (ruff check)
Verify gate: PASS 6/6 (pytest test_gpu_monitor / test_gpu_manager / test_multi_gpu_routing, ruff on touched files, import check)
Independent re-run on current tree: 56 passed in 0.15s

kilo-gsd Review-phase controller stalled after launch (known stall pattern on this repo, cf. mesh-seed#78/#199), so the runner never executed its normal close path. Closing here with the above evidence.

## Closed by fixup verification The GPU monitoring deliverable for this issue has landed and is verified on `origin/master`. - Commit: `f98427f feat(#197): add GPU monitoring with saturation, thermal, starvation, orphan signals` (10 files, +1083) - Verified on remote: `git ls-remote origin master` contains f98427f as ancestor - Lint gate: PASS (ruff check) - Verify gate: PASS 6/6 (pytest test_gpu_monitor / test_gpu_manager / test_multi_gpu_routing, ruff on touched files, import check) - Independent re-run on current tree: 56 passed in 0.15s kilo-gsd Review-phase controller stalled after launch (known stall pattern on this repo, cf. mesh-seed#78/#199), so the runner never executed its normal close path. Closing here with the above evidence.

Mark referenced this issue

2026-08-12 08:07:28 +00:00

GPU-001: GPU Detection and Allocation #195

Mark referenced this issue

2026-09-17 05:54:22 +00:00

Phase 3 — Queue-depth auto-scaler + cloud worker capability metadata #299

Rows
Columns

GPU-003: GPU Monitoring #197

⏸ Needs Decision

Triage: decompose failed

Closed by fixup verification