#38 - org-subscriber daemon drops all Mark/org_mode webhooks with 401 — hook 102 has no secret while daemon requires HMAC - Mark/org_mode - cyguin git: Adlerian Hierarchical Self-Actualization Syllogism

Mark commented

2026-08-21 23:01:22 +00:00

Owner

Summary

Every Mark/org_mode event sent to the org-subscriber daemon is rejected with 401 Unauthorized. The daemon webhook endpoint requires an HMAC secret, but Forgejo hook id 102 on Mark/org_mode has no secret configured. Zero Mark/org_mode events reach the daemon — issue close/label events (e.g. the org_mode#29 close on 2026-08-21) are silently dropped.

Pre-existing (known pitfall, verified 2026-08-07) — NOT a regression from the #29 fix, but it breaks the event substrate the Module O swarm bridge (org_mode#29) is supposed to run on.

Reproduction

~/.hermes/scripts/forgejo-api.sh GET /repos/Mark/org_mode/hooks → hook id 102, active, url http://unstable.tail018ac4.ts.net:9090/webhooks/forgejo, config keys = [content_type, url] only — no secret.
grep -n -B2 -A3 secret /home/b/Projects/org_mode/config/daemon.unstable.yaml → secret: "2173fe56e5a7a10a6d6e090c454693720bf1c3e3e62eac031054a85d5ba92df6" set for path /webhooks/forgejo.
Trigger any org_mode issue event (e.g. close issue, apply triage-approved label).
ssh unstable journalctl --user -u org-subscriber -n 300 --no-pager → POST /webhooks/forgejo ... 401 Unauthorized immediately after the event. Verified live 2026-08-21 22:54–22:58: continuous 200/401 pairs, only osint-hub (secret-bearing) hooks return 200.

Actual behavior

~42 x 401 per 40 min on the daemon for org_mode events; zero org_mode events reach the daemon. The org_mode#29 close (22:50:23Z) and the triage-approved re-apply fired webhooks that were all 401-dropped (journal 401 batches 22:50:43–22:52:48 = retries).

Expected behavior

org_mode events reach the daemon: daemon returns 200 and the event dispatches (kanban card created / run recorded).

Fix candidates (agent-reachable)

Forgejo-side: set secret on hook 102 (PATCH persists, GET won't echo it — verify functionally).
Daemon-side: clear secret in daemon.unstable.yaml (needs explicit permission — out of reporter scope).

Kanban origin: card t_99b7d377 (post-deploy probe of Mark/org_mode#29).

## Summary Every Mark/org_mode event sent to the org-subscriber daemon is rejected with 401 Unauthorized. The daemon webhook endpoint requires an HMAC secret, but Forgejo hook id 102 on Mark/org_mode has no secret configured. Zero Mark/org_mode events reach the daemon — issue close/label events (e.g. the org_mode#29 close on 2026-08-21) are silently dropped. **Pre-existing** (known pitfall, verified 2026-08-07) — NOT a regression from the #29 fix, but it breaks the event substrate the Module O swarm bridge (org_mode#29) is supposed to run on. ## Reproduction 1. `~/.hermes/scripts/forgejo-api.sh GET /repos/Mark/org_mode/hooks` → hook id 102, active, url `http://unstable.tail018ac4.ts.net:9090/webhooks/forgejo`, config keys = [content_type, url] only — **no secret**. 2. `grep -n -B2 -A3 secret /home/b/Projects/org_mode/config/daemon.unstable.yaml` → `secret: "2173fe56e5a7a10a6d6e090c454693720bf1c3e3e62eac031054a85d5ba92df6"` set for path `/webhooks/forgejo`. 3. Trigger any org_mode issue event (e.g. close issue, apply triage-approved label). 4. `ssh unstable journalctl --user -u org-subscriber -n 300 --no-pager` → `POST /webhooks/forgejo ... 401 Unauthorized` immediately after the event. Verified live 2026-08-21 22:54–22:58: continuous 200/401 pairs, only osint-hub (secret-bearing) hooks return 200. ## Actual behavior - ~42 x 401 per 40 min on the daemon for org_mode events; zero org_mode events reach the daemon. The org_mode#29 close (22:50:23Z) and the triage-approved re-apply fired webhooks that were all 401-dropped (journal 401 batches 22:50:43–22:52:48 = retries). ## Expected behavior - org_mode events reach the daemon: daemon returns 200 and the event dispatches (kanban card created / run recorded). ## Fix candidates (agent-reachable) 1. Forgejo-side: set secret on hook 102 (PATCH persists, GET won't echo it — verify functionally). 2. Daemon-side: clear `secret` in daemon.unstable.yaml (needs explicit permission — out of reporter scope). Kanban origin: card t_99b7d377 (post-deploy probe of Mark/org_mode#29).

Mark added the

triage-approved

bug

infra

labels

2026-08-21 23:01:22 +00:00

Mark commented

2026-08-27 03:36:29 +00:00

Author

Owner

#38 fix verification: webhook secret sync 2026-08-27Z

Mark commented

2026-08-27 03:38:15 +00:00

Author

Owner

webhook delivery test 03:38Z

Mark commented

2026-08-27 03:39:49 +00:00

Author

Owner

post-fix live trigger

Mark commented

2026-08-27 03:52:43 +00:00

Author

Owner

post-dns-fix live trigger

Mark commented

2026-08-27 03:54:10 +00:00

Author

Owner

Resolution (2026-08-27 ~03:52Z)

Closed-loop breaker was a two-layer bug, not just the missing secret:

Secret mismatch (described in #38 body). Forgejo hook 102 had no config.secret; daemon daemon.unstable.yaml had secret: 2173fe56...ba92df6 for /webhooks/forgejo. PATCH /repos/Mark/org_mode/hooks/102 set the matching secret.
DNS resolution failure (not described in #38 body, but visible in hook_task delivery log). Forgejo container resolv.conf points at 1.1.1.1 / 8.8.8.8 (public DNS), so unstable.tail018ac4.ts.net (Tailscale MagicDNS) did not resolve. Every hook_task delivery for hook 102 from 2026-08-26 ~05:09Z onward returned dial tcp: lookup unstable.tail018ac4.ts.net on 1.1.1.1:53: no such host — before any HMAC check ran. So the symptom (#38 said "every Mark/org_mode event is rejected with 401 Unauthorized") was actually a TCP-layer failure masquerading as an auth failure: Forgejo never even reached the daemon to be 401-rejected.

Fix applied

PATCH /repos/Mark/org_mode/hooks/102 with:

config.secret = 2173fe56e5a7a10a6d6e090c454693720bf1c3e3e62eac031054a85d5ba92df6
config.url = http://100.123.237.4:9090/webhooks/forgejo (bypasses MagicDNS by using the tailnet IPv4 directly)
full event list preserved (16 events: push, issues, issue_, pull_request_)
active = true

Verification

Live trigger (issue_comment #52841 on Mark/org_mode#38) at 03:52:43Z:

journalctl --user -u org-subscriber:
- server.webhooks.router: Webhook received: source=forgejo event=issue_comment.created actor=Mark repo=Mark/org_mode
- INFO: 100.123.237.4:45848 - "POST /webhooks/forgejo HTTP/1.1" 200 OK
hook_task row 13685: is_succeed=1, status=200, body={"status":"ok","event_type":"issue_comment.created"...}
Daemon action_chain ran feature-triage dispatchers (exit=0).

Side findings (not blocking this issue)

Forgejo container DNS config (resolv.conf = 1.1.1.1, 8.8.8.8) is unsuitable for any internal tailnet webhook target. If a future hook needs DNS, either set the URL to a literal tailnet IP (as done here), or change the container's DNS to 100.100.100.100 (Tailscale MagicDNS resolver) — but that needs a container restart and a daemon-reload for resolv.conf to take effect.
forgejo_tailscale conmon (PID 2018666) is running podman start -a against a non-existent container. Worth a separate cleanup ticket; not a closed-loop blocker.

Closes the closed-loop breaker for org_mode: every labeled event on Mark/org_mode now reaches the daemon, the daemon action_chain runs, downstream PM/kanban/E/N/O modules can rely on the event substrate.

## Resolution (2026-08-27 ~03:52Z) Closed-loop breaker was a **two-layer bug**, not just the missing secret: 1. **Secret mismatch (described in #38 body).** Forgejo hook 102 had no `config.secret`; daemon `daemon.unstable.yaml` had `secret: 2173fe56...ba92df6` for `/webhooks/forgejo`. PATCH `/repos/Mark/org_mode/hooks/102` set the matching secret. 2. **DNS resolution failure (not described in #38 body, but visible in `hook_task` delivery log).** Forgejo container `resolv.conf` points at `1.1.1.1` / `8.8.8.8` (public DNS), so `unstable.tail018ac4.ts.net` (Tailscale MagicDNS) did not resolve. Every hook_task delivery for hook 102 from 2026-08-26 ~05:09Z onward returned `dial tcp: lookup unstable.tail018ac4.ts.net on 1.1.1.1:53: no such host` — *before* any HMAC check ran. So the symptom (#38 said "every Mark/org_mode event is rejected with 401 Unauthorized") was actually a TCP-layer failure masquerading as an auth failure: Forgejo never even reached the daemon to be 401-rejected. ### Fix applied PATCH `/repos/Mark/org_mode/hooks/102` with: - `config.secret = 2173fe56e5a7a10a6d6e090c454693720bf1c3e3e62eac031054a85d5ba92df6` - `config.url = http://100.123.237.4:9090/webhooks/forgejo` (bypasses MagicDNS by using the tailnet IPv4 directly) - full event list preserved (16 events: push, issues, issue_*, pull_request_*) - active = true ### Verification Live trigger (issue_comment #52841 on Mark/org_mode#38) at 03:52:43Z: - `journalctl --user -u org-subscriber`: - `server.webhooks.router: Webhook received: source=forgejo event=issue_comment.created actor=Mark repo=Mark/org_mode` - `INFO: 100.123.237.4:45848 - "POST /webhooks/forgejo HTTP/1.1" 200 OK` - `hook_task` row 13685: `is_succeed=1, status=200, body={"status":"ok","event_type":"issue_comment.created"...}` - Daemon action_chain ran feature-triage dispatchers (exit=0). ### Side findings (not blocking this issue) - Forgejo container DNS config (`resolv.conf` = `1.1.1.1`, `8.8.8.8`) is unsuitable for any internal tailnet webhook target. If a future hook needs DNS, either set the URL to a literal tailnet IP (as done here), or change the container's DNS to `100.100.100.100` (Tailscale MagicDNS resolver) — but that needs a container restart and a daemon-reload for resolv.conf to take effect. - `forgejo_tailscale` conmon (PID 2018666) is running `podman start -a` against a non-existent container. Worth a separate cleanup ticket; not a closed-loop blocker. Closes the closed-loop breaker for org_mode: every labeled event on Mark/org_mode now reaches the daemon, the daemon action_chain runs, downstream PM/kanban/E/N/O modules can rely on the event substrate.

Mark closed this issue

2026-08-27 03:54:14 +00:00

Rows
Columns

org-subscriber daemon drops all Mark/org_mode webhooks with 401 — hook 102 has no secret while daemon requires HMAC #38

Summary

Reproduction

Actual behavior

Expected behavior

Fix candidates (agent-reachable)

Resolution (2026-08-27 ~03:52Z)

Fix applied

Verification

Side findings (not blocking this issue)