[verify-E2E] Restart org-subscriber daemon on unstable and confirm commit `1f4e66c` lands a fixit card from Mark/mesh-seed#285 #47

New issue

Closed

opened 2026-09-22 06:27:56 +00:00 by Mark · 1 comment

Mark commented

2026-09-22 06:27:56 +00:00

Owner

Context

workers/forgejo-kanban-ingress-wrapper.sh was broken at the Python parse step. Recent commits reverted a working argv-as-list shape back to a stdin-heredoc with \\n line-continuations inside a Python f-string expression. Symptoms:

Python 3.10+ rejects the f-string with SyntaxError: f-string expression part cannot include a backslash at line 102 of the wrapper ({title.replace("'", "'\\''")}).
Even if it compiled, the literal \\n was being passed over ssh stdin as two chars (\\ + n) instead of newlines — bash would receive a malformed command with literal backslashes between tokens.

Net effect: every issues.labeled webhook event for a triage-approved label has silently failed at the wrapper since the reversion. This is why ~30 sweep cards, Mark/mesh-seed#285, and 15 cybertactics/veilwalkers issues all routed through the daemon → wrapper path without producing fixit cards.

What was fixed

Commit 1f4e66c fix(worker): restore argv-as-list ssh invocation in forgejo-kanban-ingress-wrapper.sh on Mark/org_mode main:

Replaces the stdin-heredoc shell-out with argv-as-list (matches the shape that 4424657 shipped before the revert, merged with dcaaf91's --assignee triage-runner and 164bb5b's blackhole.local ssh target).
Verified the embedded Python compiles rc=0 on this host's Python 3.11.15.
Verified via stub-ssh that the produced argv reaches hermes cleanly with title, body, and idempotency-key intact (no shell quoting surface).

Why this issue exists (the verification gap)

The org-subscriber daemon runs on a separate host (unstable.tail018ac4.ts.net), NOT on bblhd's host. The webhook delivery chain is:

Forgejo hook (per-repo) -> unstable daemon endpoint (:9090/webhooks/forgejo) -> daemon dispatches forgejo-kanban-ingress-wrapper.sh -> wrapper SSHes blackhole.local -> blackhole hermes kanban create on the fixit board

From bblhd's host I can verify the wrapper file is correct (done) but I cannot trace whether the daemon on unstable is actually running, nor invoke the daemon to confirm end-to-end card creation. The kernel signal was diagnostic-only.

What needs to happen on unstable to close the loop

Confirm unstable.tail018ac4.ts.net has the org-subscriber daemon running (check systemctl / ps -ef | grep -E 'org-mode|forgejo-kanban').
Confirm the daemon process was started AFTER commit 1f4e66c so it reads the updated wrapper script (file is at /home/b/Projects/org_mode/workers/forgejo-kanban-ingress-wrapper.sh on unstable; the daemon invokes it by name with no startup cache).
- If the daemon was started before 22:40 UTC 2026-09-21 (the commit timestamp), restart it.
Trigger a test event: remove-then-re-add the triage-approved label on Mark/mesh-seed#285 (currently has bug, triage-approved). The daemon should fire forgejo-kanban-ingress-wrapper.sh → argv-as-list ssh invocation → hermes kanban create --idempotency-key forgejo:<url> --assignee triage-runner → fixit board gets one new card with title [followup] mesh-seed#204: complete multi-tenant wiring — tests, runtime, docker,... and a body containing the --body block from the wrapper.
Verify on blackhole: hermes kanban --board fixit list shows a new card from that webhook event.

Acceptance

After step 4, Mark/mesh-seed#285 has a fixit card against it (idempotency-key forgejo:<the URL of #285>).

If the daemon is not running on unstable, the cards will not appear even with the wrapper fix in place — that's a separate restart task and out of scope for this quickfix.

Commit: Mark/org_mode@1f4e66c on main
Prior commits for context: 4424657 (claimed rewrite), 164bb5b (blackhole.local SSH target), dcaaf91 (--assignee triage-runner)
15 cybertactics + 7 veilwalkers issues currently have triage-approved but no fixit card — they will all flow through once the daemon restarts and sees this fix.

## Context `workers/forgejo-kanban-ingress-wrapper.sh` was broken at the Python parse step. Recent commits reverted a working argv-as-list shape back to a stdin-heredoc with `\\n` line-continuations inside a Python f-string expression. Symptoms: 1. Python 3.10+ rejects the f-string with `SyntaxError: f-string expression part cannot include a backslash` at line 102 of the wrapper (`{title.replace("'", "'\\''")}`). 2. Even if it compiled, the literal `\\n` was being passed over ssh stdin as two chars (`\\` + `n`) instead of newlines — bash would receive a malformed command with literal backslashes between tokens. Net effect: every `issues.labeled` webhook event for a `triage-approved` label has silently failed at the wrapper since the reversion. This is why ~30 sweep cards, Mark/mesh-seed#285, and 15 cybertactics/veilwalkers issues all routed through the daemon → wrapper path without producing fixit cards. ## What was fixed Commit `1f4e66c fix(worker): restore argv-as-list ssh invocation in forgejo-kanban-ingress-wrapper.sh` on `Mark/org_mode main`: - Replaces the stdin-heredoc shell-out with argv-as-list (matches the shape that `4424657` shipped before the revert, merged with `dcaaf91`'s `--assignee triage-runner` and `164bb5b`'s `blackhole.local` ssh target). - Verified the embedded Python compiles `rc=0` on this host's Python 3.11.15. - Verified via stub-ssh that the produced argv reaches `hermes` cleanly with title, body, and idempotency-key intact (no shell quoting surface). ## Why this issue exists (the verification gap) The org-subscriber daemon runs on a separate host (`unstable.tail018ac4.ts.net`), NOT on bblhd's host. The webhook delivery chain is: ``` Forgejo hook (per-repo) -> unstable daemon endpoint (:9090/webhooks/forgejo) -> daemon dispatches forgejo-kanban-ingress-wrapper.sh -> wrapper SSHes blackhole.local -> blackhole hermes kanban create on the fixit board ``` From bblhd's host I can verify the wrapper file is correct (done) but I cannot trace whether the daemon on `unstable` is actually running, nor invoke the daemon to confirm end-to-end card creation. The kernel signal was diagnostic-only. ## What needs to happen on unstable to close the loop 1. Confirm `unstable.tail018ac4.ts.net` has the org-subscriber daemon running (check systemctl / `ps -ef | grep -E 'org-mode|forgejo-kanban'`). 2. Confirm the daemon process was started AFTER commit `1f4e66c` so it reads the updated wrapper script (file is at `/home/b/Projects/org_mode/workers/forgejo-kanban-ingress-wrapper.sh` on unstable; the daemon invokes it by name with no startup cache). - If the daemon was started before 22:40 UTC 2026-09-21 (the commit timestamp), restart it. 3. Trigger a test event: remove-then-re-add the `triage-approved` label on `Mark/mesh-seed#285` (currently has `bug, triage-approved`). The daemon should fire `forgejo-kanban-ingress-wrapper.sh` → argv-as-list ssh invocation → `hermes kanban create --idempotency-key forgejo:<url> --assignee triage-runner` → fixit board gets one new card with title `[followup] mesh-seed#204: complete multi-tenant wiring — tests, runtime, docker,...` and a body containing the `--body` block from the wrapper. 4. Verify on blackhole: `hermes kanban --board fixit list` shows a new card from that webhook event. ## Acceptance After step 4, `Mark/mesh-seed#285` has a fixit card against it (idempotency-key `forgejo:<the URL of #285>`). If the daemon is not running on unstable, the cards will not appear even with the wrapper fix in place — that's a separate restart task and out of scope for this quickfix. ## Related - Commit: `Mark/org_mode@1f4e66c` on `main` - Prior commits for context: `4424657` (claimed rewrite), `164bb5b` (blackhole.local SSH target), `dcaaf91` (--assignee triage-runner) - 15 cybertactics + 7 veilwalkers issues currently have `triage-approved` but no fixit card — they will all flow through once the daemon restarts and sees this fix.

Mark added the

triage-approved

quickfix

bug

labels

2026-09-22 06:27:56 +00:00

Mark commented

2026-09-22 15:37:57 +00:00

Author

Owner

Status: RESOLVED 2026-09-22

Two fixes landed on Mark/org_mode main:

1f4e66c — restored argv-as-list ssh invocation in forgejo-kanban-ingress-wrapper.sh (replaced the stdin-heredoc shell-out with argv-as-list ssh + hermes CLI args).
d19ea52 — added shlex.quote() for body and title in the wrapper so the remote bash doesn't mangle multiline bodies or titles with parens/backticks.

E2E verification (2026-09-22 from bblhd's host)

Restarted the daemon on unstable.tail018ac4.ts.net via systemctl --user restart org-subscriber.service (was running 19 days with stale wrapper).
Created Mark/veilwalkers#10 test issue with body containing parens + backticks. Webhook delivered, action chain dispatched wrapper, wrapper exit=0, kanban card created (t_d1e21adc).

Substrate fix also required

48 Mark/* webhooks were pointing at http://unstable.tail018ac4.ts.net:9090/... (Tailscale MagicDNS hostname). Forgejo's webhook delivery could not resolve that hostname. All bulk-PATCHed to http://100.123.237.4:9090/... (bare Tailscale IP) so webhooks actually arrive at the daemon. Without this fix, even the wrapper-patched daemon would have received no events for those repos.

Stashes preserved

Pre-existing local patches on unstable (stash-1.patch 3957 bytes, stash-2.patch 25311 bytes) were dropped from git stash during the reset and preserved at /tmp/orgmode-stash-snapshot/ on unstable.tail018ac4.ts.net for operator review. These are operator-level local commits (pre-existing worker edits + m007-deploy patches), not the wrapper fix.

Final state

Mark/veilwalkers#10 → t_d1e21adc (ready, assignee=triage-runner) on fixit.
triage-runner: done=73, ready=11, running=1 — pipeline is healthy.
Mark/org_mode main is at 1f4e66c + d19ea52 ahead of bf0d318.
48 webhook URLs across Mark/* repos patched to bare Tailscale IP.

## Status: RESOLVED 2026-09-22 Two fixes landed on `Mark/org_mode` main: 1. **`1f4e66c`** — restored argv-as-list ssh invocation in `forgejo-kanban-ingress-wrapper.sh` (replaced the stdin-heredoc shell-out with argv-as-list ssh + hermes CLI args). 2. **`d19ea52`** — added `shlex.quote()` for body and title in the wrapper so the remote bash doesn't mangle multiline bodies or titles with parens/backticks. ## E2E verification (2026-09-22 from bblhd's host) - Restarted the daemon on `unstable.tail018ac4.ts.net` via `systemctl --user restart org-subscriber.service` (was running 19 days with stale wrapper). - Created `Mark/veilwalkers#10` test issue with body containing parens + backticks. Webhook delivered, action chain dispatched wrapper, wrapper exit=0, kanban card created (`t_d1e21adc`). ## Substrate fix also required 48 Mark/* webhooks were pointing at `http://unstable.tail018ac4.ts.net:9090/...` (Tailscale MagicDNS hostname). Forgejo's webhook delivery could not resolve that hostname. All bulk-PATCHed to `http://100.123.237.4:9090/...` (bare Tailscale IP) so webhooks actually arrive at the daemon. Without this fix, even the wrapper-patched daemon would have received no events for those repos. ## Stashes preserved Pre-existing local patches on `unstable` (`stash-1.patch` 3957 bytes, `stash-2.patch` 25311 bytes) were dropped from git stash during the reset and preserved at `/tmp/orgmode-stash-snapshot/` on `unstable.tail018ac4.ts.net` for operator review. These are operator-level local commits (pre-existing worker edits + m007-deploy patches), not the wrapper fix. ## Final state - `Mark/veilwalkers#10` → `t_d1e21adc (ready, assignee=triage-runner)` on fixit. - `triage-runner: done=73, ready=11, running=1` — pipeline is healthy. - `Mark/org_mode` main is at `1f4e66c + d19ea52` ahead of `bf0d318`. - 48 webhook URLs across Mark/* repos patched to bare Tailscale IP.

Mark closed this issue

2026-09-22 15:37:57 +00:00