Mark/mesh-seed
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Phase 3 — Queue-depth auto-scaler + cloud worker capability metadata #299

New issue
Closed
opened 2026-09-17 05:54:22 +00:00 by Mark · 4 comments
Mark commented 2026-09-17 05:54:22 +00:00
Owner
Copy link

Source

  • Vikunja task #197 (Phase 3 follow-up)
  • Parent: Mark/mesh-seed#56 (Phase 3 umbrella, closed 2026-09-15)
  • Sibling Phase 3 slices: Mark/mesh-seed#298 (trust tiers in routing, in-progress), PR #297 (GPU capability class, merged)
  • Reference templates already in tree: templates/workers/compositions/cloud_burst.yaml (declares auto_scale: true, scale_trigger: queue_depth) — no implementation yet

Goal

Make the substrate react to queue depth by emitting deterministic scale decisions, and prepare a cloud worker template that advertises the capability/GPU/trust metadata the router needs to consider cloud workers as first-class candidates. Keep the first path single-node and test-friendly; do not require Tailnet federation or a live cloud account for unit verification.

Scope

1. runtime/auto_scaler.py — queue-depth reactive decisions

The template cloud_burst.yaml already advertises auto_scale: true + scale_trigger: queue_depth, but no implementation exists. Build a deterministic, single-node-safe module that:

  • Observation: reads queue depth from a pluggable source — primary path uses the existing runtime/host_scheduler.HostMetrics.queue_depth and a simple in-process counter; a QueueDepthProbe protocol/ABC allows stubbing for tests.
  • Decision rules with explicit thresholds and hysteresis:
    • scale_up when queue_depth ≥ SCALE_UP_THRESHOLD (default 8) for ≥ SCALE_UP_HOLD_S (default 30s)
    • scale_down when queue_depth ≤ SCALE_DOWN_THRESHOLD (default 1) AND active worker count > 1, held for SCALE_DOWN_HOLD_S (default 60s)
    • no_op when between thresholds, when metrics are missing/invalid, or when held by hysteresis
    • Idle/empty queue is no_op; never scale-down the last worker
  • Apply step: a ScaleAction enum + decide(metrics, state) -> ScaleDecision pure function so tests cover all four cases (empty, rising, sustained, falling) plus invalid/missing metrics without touching the network.
  • Safe no-op: when no CloudSpawner is wired, apply(decision) must log and return a structured {"applied": false, "reason": "spawner_unconfigured"} rather than raising — the substrate stays runnable without federation.
  • Pluggable spawner: define a WorkerSpawner ABC (async spawn(template_id, capability_manifest) -> WorkerHandle); the real cloud path is out of scope here, but the contract must exist so the kilo-gsd slice can wire CloudSpawner later without changing the decision logic.

2. Cloud worker template — templates/workers/compositions/cloud_burst.yaml

The template already exists but lacks the capability/GPU/trust metadata the router requires (added in the sibling Phase 3 slices):

  • Add a capabilities block advertising the canonical capability classes (muscle, nerve already declared; add gpu_class if GPU, trust_tier per substrate/trust_tiers.py).
  • Add metadata block with: provider (hetzner/aws/local), region, instance_type, gpu_class (canonical name from runtime/gpu_class.py), trust_tier (HIGH/MEDIUM/LOW), scale_strategy (cloud_burst|reserved|spot), cost_per_hour_usd.
  • Add autoscale block that references the new module: decision_module: runtime.auto_scaler, scale_up_threshold, scale_down_threshold, scale_up_hold_s, scale_down_hold_s.
  • Keep auto_scale: true + scale_trigger: queue_depth for backward compatibility with anything that already reads those keys.

3. Tests (tests/unit/)

  • test_auto_scaler.py — at minimum:
    • empty queue → no_op (no scale-down of last worker)
    • rising queue past scale_up_threshold within hold window → scale_up
    • sustained high queue → repeated scale_up (or held no_op after the first decision, per hysteresis policy)
    • falling queue past scale_down_threshold with workers > 1 → scale_down
    • invalid metrics (None, negative, NaN) → no_op with reason: invalid_metrics
    • missing metrics source → no_op with reason: probe_unconfigured
    • decision is pure (same input → same output, no I/O)
    • apply(decision) with no spawner wired returns structured no-op
  • test_cloud_burst_template.py — validate the YAML loads, advertises required capability fields, references the auto-scaler module, and round-trips through yaml.safe_load.
  • Update any existing scheduler/template tests if assumptions change.

4. Docs (docs/reference/)

  • New docs/reference/AUTO_SCALER.md documenting: thresholds + hysteresis, decision flow, the spawner ABC, single-node boundary (no federation, no live cloud calls), and how to wire a CloudSpawner later without changing the decision logic.
  • Cross-link from the existing routing docs (runtime/intent_router.py is a natural anchor) so the trust-tier + GPU-class slices know the auto-scaler is queue-driven, not capability-driven.

Acceptance criteria

  • runtime/auto_scaler.py exposes decide(metrics, state) -> ScaleDecision and apply(decision, spawner=None); both pure/testable on the single-node path.
  • templates/workers/compositions/cloud_burst.yaml advertises capability + GPU + trust metadata and references the auto-scaler config.
  • tests/unit/test_auto_scaler.py covers empty / rising / sustained / falling queues + invalid/missing metrics.
  • tests/unit/test_cloud_burst_template.py validates the template structure.
  • docs/reference/AUTO_SCALER.md documents thresholds, hysteresis, the spawner ABC, and the single-node boundary.
  • The substrate runs without a wired spawner (no federation, no live cloud account required).

Out of scope

  • Live cloud provisioning (Hetzner/AWS API calls).
  • Tailnet federation / cross-mesh scaling.
  • Replacing the existing runtime/host_scheduler.py selection logic.
  • A CloudSpawner implementation — only the ABC + a NullSpawner no-op so the auto-scaler can run end-to-end without one.
  • Modifying substrate/trust_tiers.py policy tables (the cloud_burst template just consumes them).

Reference (already in tree)

  • runtime/host_scheduler.py — HostMetrics carries queue_depth, active_tasks, completed_tasks_1m.
  • runtime/gpu_class.py (added by PR #297) — canonical GPU class registry + class_satisfies.
  • substrate/trust_tiers.py — TrustTier (LOW/MEDIUM/HIGH) + TrustTierManager.resolve_tier.
  • templates/workers/compositions/cloud_burst.yaml — the template this issue extends.
  • tests/unit/test_host_scheduler.py — patterns for fake HostMetrics.
## Source - Vikunja task #197 (Phase 3 follow-up) - Parent: Mark/mesh-seed#56 (Phase 3 umbrella, closed 2026-09-15) - Sibling Phase 3 slices: Mark/mesh-seed#298 (trust tiers in routing, in-progress), PR #297 (GPU capability class, merged) - Reference templates already in tree: `templates/workers/compositions/cloud_burst.yaml` (declares `auto_scale: true`, `scale_trigger: queue_depth`) — no implementation yet ## Goal Make the substrate react to queue depth by emitting deterministic scale decisions, and prepare a cloud worker template that advertises the capability/GPU/trust metadata the router needs to consider cloud workers as first-class candidates. Keep the first path single-node and test-friendly; do not require Tailnet federation or a live cloud account for unit verification. ## Scope ### 1. `runtime/auto_scaler.py` — queue-depth reactive decisions The template `cloud_burst.yaml` already advertises `auto_scale: true` + `scale_trigger: queue_depth`, but no implementation exists. Build a deterministic, single-node-safe module that: - **Observation**: reads queue depth from a pluggable source — primary path uses the existing `runtime/host_scheduler.HostMetrics.queue_depth` and a simple in-process counter; a `QueueDepthProbe` protocol/ABC allows stubbing for tests. - **Decision rules** with explicit thresholds and hysteresis: - `scale_up` when queue_depth ≥ `SCALE_UP_THRESHOLD` (default 8) for ≥ `SCALE_UP_HOLD_S` (default 30s) - `scale_down` when queue_depth ≤ `SCALE_DOWN_THRESHOLD` (default 1) AND active worker count > 1, held for `SCALE_DOWN_HOLD_S` (default 60s) - `no_op` when between thresholds, when metrics are missing/invalid, or when held by hysteresis - Idle/empty queue is `no_op`; never scale-down the last worker - **Apply step**: a `ScaleAction` enum + `decide(metrics, state) -> ScaleDecision` pure function so tests cover all four cases (empty, rising, sustained, falling) plus invalid/missing metrics without touching the network. - **Safe no-op**: when no `CloudSpawner` is wired, `apply(decision)` must log and return a structured `{"applied": false, "reason": "spawner_unconfigured"}` rather than raising — the substrate stays runnable without federation. - **Pluggable spawner**: define a `WorkerSpawner` ABC (`async spawn(template_id, capability_manifest) -> WorkerHandle`); the real cloud path is out of scope here, but the contract must exist so the kilo-gsd slice can wire `CloudSpawner` later without changing the decision logic. ### 2. Cloud worker template — `templates/workers/compositions/cloud_burst.yaml` The template already exists but lacks the capability/GPU/trust metadata the router requires (added in the sibling Phase 3 slices): - Add a `capabilities` block advertising the canonical capability classes (`muscle`, `nerve` already declared; add `gpu_class` if GPU, `trust_tier` per `substrate/trust_tiers.py`). - Add `metadata` block with: `provider` (hetzner/aws/local), `region`, `instance_type`, `gpu_class` (canonical name from `runtime/gpu_class.py`), `trust_tier` (HIGH/MEDIUM/LOW), `scale_strategy` (cloud_burst|reserved|spot), `cost_per_hour_usd`. - Add `autoscale` block that references the new module: `decision_module: runtime.auto_scaler`, `scale_up_threshold`, `scale_down_threshold`, `scale_up_hold_s`, `scale_down_hold_s`. - Keep `auto_scale: true` + `scale_trigger: queue_depth` for backward compatibility with anything that already reads those keys. ### 3. Tests (`tests/unit/`) - `test_auto_scaler.py` — at minimum: - empty queue → `no_op` (no scale-down of last worker) - rising queue past `scale_up_threshold` within hold window → `scale_up` - sustained high queue → repeated `scale_up` (or held `no_op` after the first decision, per hysteresis policy) - falling queue past `scale_down_threshold` with workers > 1 → `scale_down` - invalid metrics (None, negative, NaN) → `no_op` with `reason: invalid_metrics` - missing metrics source → `no_op` with `reason: probe_unconfigured` - decision is pure (same input → same output, no I/O) - `apply(decision)` with no spawner wired returns structured no-op - `test_cloud_burst_template.py` — validate the YAML loads, advertises required capability fields, references the auto-scaler module, and round-trips through `yaml.safe_load`. - Update any existing scheduler/template tests if assumptions change. ### 4. Docs (`docs/reference/`) - New `docs/reference/AUTO_SCALER.md` documenting: thresholds + hysteresis, decision flow, the spawner ABC, single-node boundary (no federation, no live cloud calls), and how to wire a `CloudSpawner` later without changing the decision logic. - Cross-link from the existing routing docs (`runtime/intent_router.py` is a natural anchor) so the trust-tier + GPU-class slices know the auto-scaler is queue-driven, not capability-driven. ## Acceptance criteria - [ ] `runtime/auto_scaler.py` exposes `decide(metrics, state) -> ScaleDecision` and `apply(decision, spawner=None)`; both pure/testable on the single-node path. - [ ] `templates/workers/compositions/cloud_burst.yaml` advertises capability + GPU + trust metadata and references the auto-scaler config. - [ ] `tests/unit/test_auto_scaler.py` covers empty / rising / sustained / falling queues + invalid/missing metrics. - [ ] `tests/unit/test_cloud_burst_template.py` validates the template structure. - [ ] `docs/reference/AUTO_SCALER.md` documents thresholds, hysteresis, the spawner ABC, and the single-node boundary. - [ ] The substrate runs without a wired spawner (no federation, no live cloud account required). ## Out of scope - Live cloud provisioning (Hetzner/AWS API calls). - Tailnet federation / cross-mesh scaling. - Replacing the existing `runtime/host_scheduler.py` selection logic. - A `CloudSpawner` implementation — only the ABC + a `NullSpawner` no-op so the auto-scaler can run end-to-end without one. - Modifying `substrate/trust_tiers.py` policy tables (the cloud_burst template just consumes them). ## Reference (already in tree) - `runtime/host_scheduler.py` — `HostMetrics` carries `queue_depth`, `active_tasks`, `completed_tasks_1m`. - `runtime/gpu_class.py` (added by PR #297) — canonical GPU class registry + `class_satisfies`. - `substrate/trust_tiers.py` — `TrustTier` (LOW/MEDIUM/HIGH) + `TrustTierManager.resolve_tier`. - `templates/workers/compositions/cloud_burst.yaml` — the template this issue extends. - `tests/unit/test_host_scheduler.py` — patterns for fake `HostMetrics`. --- <!-- forgejo-id: Mark/mesh-seed:queue-depth-autoscaler-phase3 -->
Mark commented 2026-09-17 07:00:55 +00:00
Author
Owner
Copy link

Re-running the kilo-gsd pipeline for #299. Previous run landed the wrong fix (sibling slice #298 changes) because the Verify gate fell back to a different plan than the one written for #299 (.kilo/plans/1789624624336-trust-tier-routing-phase3.md). Re-spawned kilo-gsd.py with explicit --plan /home/b/Projects/mesh-seed/.kilo/plans/1789624630493-queue-depth-autoscaler-phase3.md (the plan for #299). Tmux session: kilo-gsd-mesh-seed-299-v2, log: /home/b/.hermes/logs/kilo-monitor/mesh-seed-299-v2.log.

Re-running the kilo-gsd pipeline for #299. Previous run landed the wrong fix (sibling slice #298 changes) because the Verify gate fell back to a different plan than the one written for #299 (`.kilo/plans/1789624624336-trust-tier-routing-phase3.md`). Re-spawned `kilo-gsd.py` with explicit `--plan /home/b/Projects/mesh-seed/.kilo/plans/1789624630493-queue-depth-autoscaler-phase3.md` (the plan for #299). Tmux session: `kilo-gsd-mesh-seed-299-v2`, log: `/home/b/.hermes/logs/kilo-monitor/mesh-seed-299-v2.log`.
Mark commented 2026-09-18 13:49:47 +00:00
Author
Owner
Copy link

v3 respawn — explicit plan path + ruff-format remediation (2026-09-18)

Verifier (t_a05877d2) confirmed v2's fix did NOT land on origin/master. v2 completed Phases 1-7 with all 7 verify commands passing, but Phase 8/9 Commit failed at 00:28:43 on the pre-commit ruff-format hook (Would reformat: runtime/auto_scaler.py).

v3 changes:

  • Explicit --plan /home/b/Projects/mesh-seed/.kilo/plans/1789628725743-queue-depth-autoscaler-phase3.md (the v2 corrected plan from 2026-09-17 00:13 — written after v1 was found to be wrong; supersedes the v1 plan file)
  • recovery-clean branch recreated at origin/master HEAD 81998de (no leftover #298 commits)
  • Working tree clean (multi-tenant WIP stashed)
  • kilo-gsd-monitor.sh patched (additive, opt-in via KILO_PLAN_FILE env var) to forward --plan to kilo-gsd.py — this fixes the "Verify falls back to wrong sibling plan" failure mode from the original v1 attempt
  • The v2 plan already includes the ruff-format pre-commit remediation (running ruff format after Implement and before Commit)

Substrate state (verifiable now):

  • monitor tmux: kilo-monitor-mesh-seed-299 (NOT kilo-gsd-... — that's the worker's session name, set by the runner itself)
  • monitor log: /home/b/.hermes/logs/kilo-monitor/mesh-seed-299.log
  • kilo-gsd.py running with: reasoning=kilo/minimax/minimax-m3 coding=kilo/minimax/minimax-m3 (cli) (CLI args, env-var fallback avoided)
  • Forgejo #299 labels: feature-request + triage-approved (no in-progress yet — claim happens in Phase 1)

Sibling slices unchanged: PR #297 (GPU class, merged), #298 (trust tiers in routing, landed via PR #300).

## v3 respawn — explicit plan path + ruff-format remediation (2026-09-18) Verifier (`t_a05877d2`) confirmed v2's fix did NOT land on origin/master. v2 completed Phases 1-7 with all 7 verify commands passing, but Phase 8/9 Commit failed at 00:28:43 on the pre-commit `ruff-format` hook (`Would reformat: runtime/auto_scaler.py`). **v3 changes:** - Explicit `--plan /home/b/Projects/mesh-seed/.kilo/plans/1789628725743-queue-depth-autoscaler-phase3.md` (the v2 corrected plan from 2026-09-17 00:13 — written after v1 was found to be wrong; supersedes the v1 plan file) - `recovery-clean` branch recreated at origin/master HEAD `81998de` (no leftover #298 commits) - Working tree clean (multi-tenant WIP stashed) - `kilo-gsd-monitor.sh` patched (additive, opt-in via `KILO_PLAN_FILE` env var) to forward `--plan` to kilo-gsd.py — this fixes the "Verify falls back to wrong sibling plan" failure mode from the original v1 attempt - The v2 plan already includes the ruff-format pre-commit remediation (running `ruff format` after Implement and before Commit) **Substrate state (verifiable now):** - monitor tmux: `kilo-monitor-mesh-seed-299` (NOT `kilo-gsd-...` — that's the worker's session name, set by the runner itself) - monitor log: `/home/b/.hermes/logs/kilo-monitor/mesh-seed-299.log` - kilo-gsd.py running with: `reasoning=kilo/minimax/minimax-m3 coding=kilo/minimax/minimax-m3 (cli)` (CLI args, env-var fallback avoided) - Forgejo #299 labels: `feature-request` + `triage-approved` (no `in-progress` yet — claim happens in Phase 1) Sibling slices unchanged: PR #297 (GPU class, merged), #298 (trust tiers in routing, landed via PR #300).
Mark commented 2026-09-18 15:07:38 +00:00
Author
Owner
Copy link

Re-dispatch attempt v4 — 2026-09-18 08:07 PT — triage-runner

Substrate state at exit:

  • Implementation COMPLETE on local branch fix/mesh-seed-299-autoscaler (3 commits, ahead of origin/master HEAD 81998de by 3 commits)
    • 05189ce feat(#299): queue-depth auto-scaler + cloud_burst capability metadata (423 lines runtime/auto_scaler.py + 22 lines cloud_burst.yaml additions + 353 lines test_auto_scaler.py + 93 lines test_cloud_burst_template.py)
    • dba0b32 Mark unit tests for test_auto_scaler and test_cloud_burst_template (pytestmark = pytest.mark.unit)
    • 1550f02 docs(#299): AUTO_SCALER reference, registry plumbing, router cross-link (AUTO_SCALER.md new + AUTO_SCALING.md pointer + intent_router.py docstring + template_registry.py sections)
  • Verify gates ALL PASS:
    • ruff format --check: 5 files already formatted
    • ruff check: All checks passed
    • pytest tests/unit/test_auto_scaler.py tests/unit/test_cloud_burst_template.py: 53 passed in 0.13s
    • pytest tests/unit/: 1896 passed, 6 skipped (only pre-existing embedding_service / embedding_ledger test errors are excluded — they fail due to huggingface-hub version mismatch, unrelated)
    • TemplateRegistry().validate_template('cloud_burst', 'composition') == []
    • _extract_config propagates capabilities + metadata + autoscale blocks
    • Pre-commit hooks: ruff, ruff-format, bandit, codespell, pyupgrade, datetime, AGENTS.md, python-version, critical-imports — all Passed

PUSH BLOCKED — Forgejo substrate outage:

git push origin fix/mesh-seed-299-autoscaler fails with send-pack: unexpected disconnect while reading sideband packet + HTTP/2 200 / content-length: 0. The server accepts the git-receive-pack POST and claims HTTP 200 but returns an empty response — git cannot parse the result packet.

Curl-level probe of the same instance:

  • GET https://forgejo.tail018ac4.ts.net/Mark/mesh-seed → HTTP 500 (HTML repo page degraded)
  • GET https://forgejo.tail018ac4.ts.net/Mark/mesh-seed/issues/299 → HTTP 500 (issue page degraded)
  • GET https://forgejo.tail018ac4.ts.net/api/v1/repos/Mark/mesh-seed/issues/299 → HTTP 200 (JSON API works)
  • GET https://forgejo.tail018ac4.ts.net/api/v1/repos/Mark/mesh-seed/issues/299/comments → HTTP 200 (comments API works)

The HTML page + git receive-pack paths are both degraded. This is the SAME wire-in trap shape that affected #298 (runner-push-failed at 23:33 on 2026-09-17, recovered via cherry-pick PR #300 the next morning). Verifier should:

  1. Probe Forgejo repo/issue HTML pages for HTTP 500 -> if recovered, retry push (git push origin fix/mesh-seed-299-autoscaler from /home/b/Projects/mesh-seed).
  2. If still 500, escalate to operator (server-side git receive-pack is broken).
  3. When push lands, open PR to merge fix/mesh-seed-299-autoscaler into master, then close #299.

Acceptance criteria from the issue — all addressed:

  • runtime/auto_scaler.py exposes decide(metrics, state) -> ScaleDecision and apply(decision, spawner=None); both pure/testable on the single-node path.
  • templates/workers/compositions/cloud_burst.yaml advertises capability + GPU + trust metadata and references the auto-scaler config.
  • tests/unit/test_auto_scaler.py covers empty / rising / sustained / falling queues + invalid/missing metrics.
  • tests/unit/test_cloud_burst_template.py validates the template structure.
  • docs/reference/AUTO_SCALER.md documents thresholds, hysteresis, the spawner ABC, and the single-node boundary.
  • The substrate runs without a wired spawner (no federation, no live cloud account required).

Skipped: AGENTS.md entry (file is write-protected in this environment; the AUTO_SCALER.md section 1 cross-link + intent_router.py docstring note cover the discoverability intent).

## Re-dispatch attempt v4 — 2026-09-18 08:07 PT — triage-runner **Substrate state at exit:** - **Implementation COMPLETE on local branch `fix/mesh-seed-299-autoscaler`** (3 commits, ahead of origin/master HEAD 81998de by 3 commits) - `05189ce feat(#299): queue-depth auto-scaler + cloud_burst capability metadata` (423 lines runtime/auto_scaler.py + 22 lines cloud_burst.yaml additions + 353 lines test_auto_scaler.py + 93 lines test_cloud_burst_template.py) - `dba0b32 Mark unit tests for test_auto_scaler and test_cloud_burst_template` (pytestmark = pytest.mark.unit) - `1550f02 docs(#299): AUTO_SCALER reference, registry plumbing, router cross-link` (AUTO_SCALER.md new + AUTO_SCALING.md pointer + intent_router.py docstring + template_registry.py sections) - **Verify gates ALL PASS:** - `ruff format --check`: 5 files already formatted - `ruff check`: All checks passed - `pytest tests/unit/test_auto_scaler.py tests/unit/test_cloud_burst_template.py`: **53 passed in 0.13s** - `pytest tests/unit/`: 1896 passed, 6 skipped (only pre-existing embedding_service / embedding_ledger test errors are excluded — they fail due to huggingface-hub version mismatch, unrelated) - `TemplateRegistry().validate_template('cloud_burst', 'composition') == []` - `_extract_config` propagates capabilities + metadata + autoscale blocks - Pre-commit hooks: ruff, ruff-format, bandit, codespell, pyupgrade, datetime, AGENTS.md, python-version, critical-imports — all Passed **PUSH BLOCKED — Forgejo substrate outage:** `git push origin fix/mesh-seed-299-autoscaler` fails with `send-pack: unexpected disconnect while reading sideband packet` + `HTTP/2 200 / content-length: 0`. The server accepts the git-receive-pack POST and claims HTTP 200 but returns an empty response — git cannot parse the result packet. Curl-level probe of the same instance: - `GET https://forgejo.tail018ac4.ts.net/Mark/mesh-seed` → **HTTP 500** (HTML repo page degraded) - `GET https://forgejo.tail018ac4.ts.net/Mark/mesh-seed/issues/299` → **HTTP 500** (issue page degraded) - `GET https://forgejo.tail018ac4.ts.net/api/v1/repos/Mark/mesh-seed/issues/299` → **HTTP 200** (JSON API works) - `GET https://forgejo.tail018ac4.ts.net/api/v1/repos/Mark/mesh-seed/issues/299/comments` → **HTTP 200** (comments API works) The HTML page + git receive-pack paths are both degraded. This is the SAME wire-in trap shape that affected #298 (runner-push-failed at 23:33 on 2026-09-17, recovered via cherry-pick PR #300 the next morning). Verifier should: 1. Probe Forgejo repo/issue HTML pages for HTTP 500 -> if recovered, retry push (`git push origin fix/mesh-seed-299-autoscaler` from `/home/b/Projects/mesh-seed`). 2. If still 500, escalate to operator (server-side git receive-pack is broken). 3. When push lands, open PR to merge `fix/mesh-seed-299-autoscaler` into master, then close #299. Acceptance criteria from the issue — all addressed: - [x] `runtime/auto_scaler.py` exposes `decide(metrics, state) -> ScaleDecision` and `apply(decision, spawner=None)`; both pure/testable on the single-node path. - [x] `templates/workers/compositions/cloud_burst.yaml` advertises capability + GPU + trust metadata and references the auto-scaler config. - [x] `tests/unit/test_auto_scaler.py` covers empty / rising / sustained / falling queues + invalid/missing metrics. - [x] `tests/unit/test_cloud_burst_template.py` validates the template structure. - [x] `docs/reference/AUTO_SCALER.md` documents thresholds, hysteresis, the spawner ABC, and the single-node boundary. - [x] The substrate runs without a wired spawner (no federation, no live cloud account required). Skipped: AGENTS.md entry (file is write-protected in this environment; the AUTO_SCALER.md section 1 cross-link + intent_router.py docstring note cover the discoverability intent).
Mark commented 2026-09-18 23:11:50 +00:00
Author
Owner
Copy link

Verifier confirmation — fix landed on remote (2026-09-18 16:15 PT)

PR opened: #301

Branch fix/mesh-seed-299-autoscaler is now on origin at commit 0f150af (4 commits ahead of master HEAD 81998de):

  • 05189ce feat(#299): queue-depth auto-scaler + cloud_burst capability metadata (4 files, +897)
  • dba0b32 Mark unit tests for test_auto_scaler and test_cloud_burst_template
  • 1550f02 docs(#299): AUTO_SCALER reference, registry plumbing, router cross-link
  • 0f150af style: ruff format runtime/structured_logging.py (pre-existing lint debt)

Verification (verifier lane, Execution lens):

  • pytest tests/unit/test_auto_scaler.py tests/unit/test_cloud_burst_template.py — 53 passed in 0.19s
  • ruff format --check on all changed files — PASS
  • ruff check on all changed files — All checks passed
  • pre-commit run on the structured_logging commit — all 11 hooks PASS (the v2/v3 phantom AUTO_SCALER.md auto-fix conflict has resolved itself now that the file is in the working tree)
  • Forgejo substrate probe: HTML 200 (149968 bytes), API 200, git push returns cleanly with sideband packets — recovered from the 09:31 PT outage.

Acceptance criteria (all satisfied on remote):

  • runtime/auto_scaler.py exposes decide(metrics, state) -> ScaleDecision and apply(decision, spawner=None) — pure/testable on single-node path
  • templates/workers/compositions/cloud_burst.yaml advertises capability + GPU + trust + autoscale blocks
  • tests/unit/test_auto_scaler.py covers empty / rising / sustained / falling queues + invalid/missing metrics + structured no-op
  • tests/unit/test_cloud_burst_template.py validates template structure + yaml.safe_load round-trip
  • docs/reference/AUTO_SCALER.md documents thresholds, hysteresis, WorkerSpawner ABC, single-node boundary
  • Substrate runs without wired spawner (no federation, no live cloud account)

Recommendation: merge PR #301 to close #299. The single-comment "AGENTS.md entry" omission from the v3 handoff remains unaddressed (file is write-protected); discoverability is achieved via docs/reference/AUTO_SCALER.md §1 + the intent_router.py docstring note. Not a blocker.

## Verifier confirmation — fix landed on remote (2026-09-18 16:15 PT) **PR opened:** https://forgejo.tail018ac4.ts.net/Mark/mesh-seed/pulls/301 Branch `fix/mesh-seed-299-autoscaler` is now on `origin` at commit `0f150af` (4 commits ahead of `master` HEAD `81998de`): - `05189ce` feat(#299): queue-depth auto-scaler + cloud_burst capability metadata (4 files, +897) - `dba0b32` Mark unit tests for test_auto_scaler and test_cloud_burst_template - `1550f02` docs(#299): AUTO_SCALER reference, registry plumbing, router cross-link - `0f150af` style: ruff format runtime/structured_logging.py (pre-existing lint debt) **Verification (verifier lane, Execution lens):** - `pytest tests/unit/test_auto_scaler.py tests/unit/test_cloud_burst_template.py` — **53 passed in 0.19s** - `ruff format --check` on all changed files — PASS - `ruff check` on all changed files — All checks passed - `pre-commit run` on the structured_logging commit — all 11 hooks PASS (the v2/v3 phantom AUTO_SCALER.md auto-fix conflict has resolved itself now that the file is in the working tree) - Forgejo substrate probe: HTML 200 (149968 bytes), API 200, `git push` returns cleanly with sideband packets — recovered from the 09:31 PT outage. **Acceptance criteria** (all satisfied on remote): - [x] `runtime/auto_scaler.py` exposes `decide(metrics, state) -> ScaleDecision` and `apply(decision, spawner=None)` — pure/testable on single-node path - [x] `templates/workers/compositions/cloud_burst.yaml` advertises capability + GPU + trust + autoscale blocks - [x] `tests/unit/test_auto_scaler.py` covers empty / rising / sustained / falling queues + invalid/missing metrics + structured no-op - [x] `tests/unit/test_cloud_burst_template.py` validates template structure + yaml.safe_load round-trip - [x] `docs/reference/AUTO_SCALER.md` documents thresholds, hysteresis, WorkerSpawner ABC, single-node boundary - [x] Substrate runs without wired spawner (no federation, no live cloud account) **Recommendation:** merge PR #301 to close #299. The single-comment "AGENTS.md entry" omission from the v3 handoff remains unaddressed (file is write-protected); discoverability is achieved via `docs/reference/AUTO_SCALER.md` §1 + the `intent_router.py` docstring note. Not a blocker.
Mark 2026-09-18 23:12:21 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
mesh-seed#142-gate-verify-pass-07b1f89
Labels
Clear labels   
blocked:children

   
blocked:commit-ready-failed

   
blocked:depends-231

Depends on #231 (Discord Bot Foundation) landing first. Do not pick until #231 is closed.

  
blocked:depends-236

Depends on #236 (Phase 1) landing first. Do not pick until #236 is closed.

  
blocked:depends-237

Depends on #237 (Phase 2) landing first. Do not pick until #237 is closed.

  
blocked:depends-238

Depends on #238 (Phase 3) landing first. Do not pick until #238 is closed.

  
blocked:depends-239

Depends on #239 (Phase 4) landing first. Do not pick until #239 is closed.

  
blocked:gate-fetch

   
blocked:gate-fetch

   
blocked:gate-lint

   
blocked:gate-research

   
blocked:needs-attention

Janitor escalated: pipeline terminated without producing expected changes. Human review required.

  
blocked:needs-decision

   
blocked:no-changes

   
blocked:no-procedure

   
blocked:preflight

Preflight check failed

  
blocked:verify-failed

   
bug

Something is broken

  
enhancement

New feature or request

  
feature-request

   
high-priority

Critical path items

  
in-progress

Issue is currently being worked on by a quickfix runner

  
lint-cleanup

Repo lint baseline is dirty

  
needs-decompose

   
needs-triage

   
ops-retry

   
ops-task

Operations task — install, configure, deploy, API calls

  
phase-5

Phase 5 Self-Building Infrastructure

  
quickfix

Pipe to opencode for direct fix

  
security/medium

Medium severity security issue

  
self-building

Autonomous code generation and self-improvement

  
triage-approved

Approved by triage — queued for fixer pipeline

No labels
blocked:children
blocked:commit-ready-failed
blocked:depends-231
blocked:depends-236
blocked:depends-237
blocked:depends-238
blocked:depends-239
blocked:gate-fetch
blocked:gate-fetch
blocked:gate-lint
blocked:gate-research
blocked:needs-attention
blocked:needs-decision
blocked:no-changes
blocked:no-procedure
blocked:preflight
blocked:verify-failed
bug
enhancement
feature-request
high-priority
in-progress
lint-cleanup
needs-decompose
needs-triage
ops-retry
ops-task
phase-5
quickfix
security/medium
self-building
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/mesh-seed#299
No description provided.