Mark/mesh-seed
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[Phase 6.10] Observability (Metrics/Logs/Traces) #79

New issue
Closed
opened 2026-04-02 20:55:30 +00:00 by Mark · 2 comments
Mark commented 2026-04-02 20:55:30 +00:00
Owner
Copy link

Overview

Metrics, logging, and tracing for mesh observability.

Problem Statement

Current mesh has limited observability:

  • No metrics (latency, throughput, error rates)
  • Basic logging (print statements)
  • No tracing across multi-hop intents
  • No alerting on failures
  • Can't debug issues in production

Proposed Solution

6.10.1: Metrics Collection

class MeshMetrics:
    """Collect and expose mesh metrics."""
    
    # Counters
    intents_received: Counter
    intents_completed: Counter
    intents_failed: Counter
    tokens_used: Counter
    
    # Histograms
    intent_latency: Histogram
    llm_latency: Histogram
    
    # Gauges
    active_workers: Gauge
    queue_depth: Gauge

6.10.2: Metrics Endpoints

endpoints:
  GET /metrics:
    format: Prometheus text format
    returns: Mesh metrics for scraping
    
  GET /v1/metrics/summary:
    returns: JSON metrics summary

6.10.3: Structured Logging

# config/logging.yaml
logging:
  format: json
  level: INFO
  output: stdout
  fields:
    - timestamp
    - level
    - logger
    - message
    - intent_id
    - trace_id
    - duration_ms

# Example log:
{
    "timestamp": "2026-04-02T10:30:00Z",
    "level": "INFO",
    "logger": "intent_router",
    "message": "Intent routed to capability",
    "intent_id": "abc123",
    "capability": "writer_v1",
    "duration_ms": 15
}

6.10.4: Distributed Tracing

class TraceContext:
    """Trace context for multi-hop intents."""
    
    trace_id: str  # Shared across all hops
    span_id: str  # Unique per hop
    parent_span_id: str  # Previous hop
    
def start_trace(intent_id: str) -> TraceContext:
    """Start new trace for intent"""
    
def child_span(parent: TraceContext) -> TraceContext:
    """Create child span for next hop"""

6.10.5: Metrics Dashboard

# Key metrics to expose:
mesh_intents_total{status="success|failure"}
mesh_intent_duration_seconds{intent_type}
mesh_llm_tokens_total{model, type="input|output"}
mesh_llm_cost_usd_total{model}
mesh_active_intents
mesh_queue_depth

# Optional: Grafana dashboard JSON

Deliverables

  • runtime/metrics.py - Metrics collection
  • Prometheus metrics endpoint (/metrics)
  • Structured JSON logging
  • Trace context for multi-hop
  • Log correlation via intent_id
  • Metrics summary endpoint
  • Unit tests

Dependencies

  • None (standalone)

Estimated Duration

1 week

Success Criteria

  • Metrics exposed in Prometheus format
  • Structured logs in JSON format
  • Intent IDs in all logs
  • Latency tracked per intent
  • Token/cost metrics available

Related

  • Parent: #69 (Phase 6 Meta)
## Overview Metrics, logging, and tracing for mesh observability. ## Problem Statement Current mesh has limited observability: - No metrics (latency, throughput, error rates) - Basic logging (print statements) - No tracing across multi-hop intents - No alerting on failures - Can't debug issues in production ## Proposed Solution ### 6.10.1: Metrics Collection ```python class MeshMetrics: """Collect and expose mesh metrics.""" # Counters intents_received: Counter intents_completed: Counter intents_failed: Counter tokens_used: Counter # Histograms intent_latency: Histogram llm_latency: Histogram # Gauges active_workers: Gauge queue_depth: Gauge ``` ### 6.10.2: Metrics Endpoints ```yaml endpoints: GET /metrics: format: Prometheus text format returns: Mesh metrics for scraping GET /v1/metrics/summary: returns: JSON metrics summary ``` ### 6.10.3: Structured Logging ```python # config/logging.yaml logging: format: json level: INFO output: stdout fields: - timestamp - level - logger - message - intent_id - trace_id - duration_ms # Example log: { "timestamp": "2026-04-02T10:30:00Z", "level": "INFO", "logger": "intent_router", "message": "Intent routed to capability", "intent_id": "abc123", "capability": "writer_v1", "duration_ms": 15 } ``` ### 6.10.4: Distributed Tracing ```python class TraceContext: """Trace context for multi-hop intents.""" trace_id: str # Shared across all hops span_id: str # Unique per hop parent_span_id: str # Previous hop def start_trace(intent_id: str) -> TraceContext: """Start new trace for intent""" def child_span(parent: TraceContext) -> TraceContext: """Create child span for next hop""" ``` ### 6.10.5: Metrics Dashboard ```yaml # Key metrics to expose: mesh_intents_total{status="success|failure"} mesh_intent_duration_seconds{intent_type} mesh_llm_tokens_total{model, type="input|output"} mesh_llm_cost_usd_total{model} mesh_active_intents mesh_queue_depth # Optional: Grafana dashboard JSON ``` ## Deliverables - [ ] `runtime/metrics.py` - Metrics collection - [ ] Prometheus metrics endpoint (`/metrics`) - [ ] Structured JSON logging - [ ] Trace context for multi-hop - [ ] Log correlation via intent_id - [ ] Metrics summary endpoint - [ ] Unit tests ## Dependencies - None (standalone) ## Estimated Duration 1 week ## Success Criteria - [ ] Metrics exposed in Prometheus format - [ ] Structured logs in JSON format - [ ] Intent IDs in all logs - [ ] Latency tracked per intent - [ ] Token/cost metrics available ## Related - Parent: #69 (Phase 6 Meta)
Mark commented 2026-07-15 11:57:04 +00:00
Author
Owner
Copy link

Reopened — kilo-gsd closed #79 without implementing deliverables

The kilo-gsd runner hit a pre-dispatch gate failure (dirty worktree from concurrent issue #75) and the Bugfix agent committed unrelated work to clear the gate instead of stashing it. No observability (metrics/logging/tracing) code was implemented. The forced commits have been reverted; the worktree is clean. Re-dispatching with clean state.

## Reopened — kilo-gsd closed #79 without implementing deliverables The kilo-gsd runner hit a pre-dispatch gate failure (dirty worktree from concurrent issue #75) and the Bugfix agent committed unrelated work to clear the gate instead of stashing it. No observability (metrics/logging/tracing) code was implemented. The forced commits have been reverted; the worktree is clean. Re-dispatching with clean state.
Mark commented 2026-07-16 05:24:29 +00:00
Author
Owner
Copy link

Fixed in commit 010465f — Phase 6.10 Observability (Metrics/Logs/Traces) fully implemented.

Delivered:

  • runtime/tracing.py: TraceContext, W3C traceparent propagation
  • runtime/structured_logging.py: Structured JSON logging with MDC
  • runtime/prometheus_metrics.py: Prometheus metrics (latency, tokens, cost, queue depth)
  • runtime/http_entrypoint.py: POST /metrics + POST /v1/metrics/summary endpoints
  • substrate/tracing.py: Re-export shim for backplane compatibility

Tests: 45/45 passing. Closing.

Fixed in commit `010465f` — Phase 6.10 Observability (Metrics/Logs/Traces) fully implemented. Delivered: - `runtime/tracing.py`: TraceContext, W3C traceparent propagation - `runtime/structured_logging.py`: Structured JSON logging with MDC - `runtime/prometheus_metrics.py`: Prometheus metrics (latency, tokens, cost, queue depth) - `runtime/http_entrypoint.py`: POST /metrics + POST /v1/metrics/summary endpoints - `substrate/tracing.py`: Re-export shim for backplane compatibility Tests: 45/45 passing. Closing.
Mark closed this issue 2026-07-16 05:24:36 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
mesh-seed#142-gate-verify-pass-07b1f89
Labels
Clear labels   
blocked:children

   
blocked:commit-ready-failed

   
blocked:depends-231

Depends on #231 (Discord Bot Foundation) landing first. Do not pick until #231 is closed.

  
blocked:depends-236

Depends on #236 (Phase 1) landing first. Do not pick until #236 is closed.

  
blocked:depends-237

Depends on #237 (Phase 2) landing first. Do not pick until #237 is closed.

  
blocked:depends-238

Depends on #238 (Phase 3) landing first. Do not pick until #238 is closed.

  
blocked:depends-239

Depends on #239 (Phase 4) landing first. Do not pick until #239 is closed.

  
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:gate-lint

   
blocked:gate-research

   
blocked:needs-attention

Janitor escalated: pipeline terminated without producing expected changes. Human review required.

  
blocked:needs-decision

   
blocked:no-changes

   
blocked:no-procedure

   
blocked:preflight

Preflight check failed

  
blocked:verify-failed

   
bug

Something is broken

  
enhancement

New feature or request

  
feature-request

   
high-priority

Critical path items

  
in-progress

Issue is currently being worked on by a quickfix runner

  
lint-cleanup

Repo lint baseline is dirty

  
needs-decompose

   
needs-triage

   
ops-retry

   
ops-task

Operations task — install, configure, deploy, API calls

  
phase-5

Phase 5 Self-Building Infrastructure

  
quickfix

Pipe to opencode for direct fix

  
security/medium

Medium severity security issue

  
self-building

Autonomous code generation and self-improvement

  
triage-approved

Approved by triage — queued for fixer pipeline

No labels
blocked:children
blocked:commit-ready-failed
blocked:depends-231
blocked:depends-236
blocked:depends-237
blocked:depends-238
blocked:depends-239
blocked:gate-fetch
blocked:gate-fetch
blocked:gate-lint
blocked:gate-research
blocked:needs-attention
blocked:needs-decision
blocked:no-changes
blocked:no-procedure
blocked:preflight
blocked:verify-failed
bug
enhancement
feature-request
high-priority
in-progress
lint-cleanup
needs-decompose
needs-triage
ops-retry
ops-task
phase-5
quickfix
security/medium
self-building
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/mesh-seed#79
No description provided.