No supervisor keeps per-repo Forgejo runner containers alive — only 3/8 live, rest silently stale #81

New issue

Closed

opened 2026-08-14 09:31:04 +00:00 by Mark · 0 comments

Mark commented

2026-08-14 09:31:04 +00:00

Owner

Summary

No systemd unit, cron, or other supervisor keeps the per-repo Forgejo Actions runner containers alive on blackhole. They are manual start.sh/podman run invocations, so they silently go stale. Currently only 3 of 8 registered runner dirs have a live container. This is the root cause behind runner outages (e.g. sandboxed-sh) and lets CI rot silently.

Evidence (probe 2026-08-14)

runner dir	.runner registered	live container
nodedesk	yes	RUNNING
kinapp (kimi_agent_kin_app_sync)	yes	RUNNING
transcriber	yes	RUNNING
kimi (kimi_agent_openscad_parametric_gui)	yes	no container
mcheroes	yes	no container
kaha	yes	no container
kab10	yes	no container
sandboxed-sh	yes (Aug 12)	no container
generic localhost-runner	yes	no container

No supervisor: no systemd unit, no cron entry — all runner daemons are manual start.sh runs.

Steps to reproduce

On blackhole: podman ps shows only nodedesk/kinapp/transcriber runner containers.
ps aux | grep forgejo-runner → daemons for the other dirs absent.
Host reboot or container crash → runner silently disappears; CI runs stack in waiting until an operator notices.

Expected

A supervisor (systemd unit per runner, or a cron watchdog) restarts any runner container that dies, so CI stays online without manual intervention.

Actual

Runners are started by hand; a crashed/stopped container stays down until an operator notices.

Action needed

Add supervision for ~/forgejo-runner-*/ data dirs (systemd user units or a watchdog script) and verify a killed container comes back within a minute.

Filed by reporter from kanban card t_86cbfbe7 (probe of closed issue #78).

## Summary No systemd unit, cron, or other supervisor keeps the per-repo Forgejo Actions runner containers alive on blackhole. They are manual `start.sh`/`podman run` invocations, so they silently go stale. Currently only **3 of 8** registered runner dirs have a live container. This is the root cause behind runner outages (e.g. sandboxed-sh) and lets CI rot silently. ## Evidence (probe 2026-08-14) | runner dir | .runner registered | live container | |---|---|---| | nodedesk | yes | RUNNING | | kinapp (kimi_agent_kin_app_sync) | yes | RUNNING | | transcriber | yes | RUNNING | | kimi (kimi_agent_openscad_parametric_gui) | yes | no container | | mcheroes | yes | no container | | kaha | yes | no container | | kab10 | yes | no container | | sandboxed-sh | yes (Aug 12) | no container | | generic localhost-runner | yes | no container | No supervisor: no systemd unit, no cron entry — all runner daemons are manual start.sh runs. ## Steps to reproduce 1. On blackhole: `podman ps` shows only nodedesk/kinapp/transcriber runner containers. 2. `ps aux | grep forgejo-runner` → daemons for the other dirs absent. 3. Host reboot or container crash → runner silently disappears; CI runs stack in `waiting` until an operator notices. ## Expected A supervisor (systemd unit per runner, or a cron watchdog) restarts any runner container that dies, so CI stays online without manual intervention. ## Actual Runners are started by hand; a crashed/stopped container stays down until an operator notices. ## Action needed Add supervision for `~/forgejo-runner-*/` data dirs (systemd user units or a watchdog script) and verify a killed container comes back within a minute. Filed by reporter from kanban card t_86cbfbe7 (probe of closed issue #78).