Mark/nodedesk
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

CI: post-merge CI runs stuck in waiting; e2e never green on main #55

New issue
Closed
opened 2026-08-07 15:39:38 +00:00 by Mark · 6 comments
Mark commented 2026-08-07 15:39:38 +00:00
Owner
Copy link

Findings from QA verification (kanban t_b85d7159)

PR #45 (ci/quality-gates-matrix) was merged to main on 2026-08-07T14:45:26Z. The merge commit is 18a1c168.

What works

  • Lint, Type check, and Unit tests (Node 20 + 22) all passed on the PR branch before merge.
  • The matrix quality-gate pipeline structure is correct and functional.

What is broken

1. Post-merge CI runs stuck in waiting

Two CI runs were triggered by the merge commit (18a1c168):

  • Run #5428 (ci.yml) — status: waiting
  • Run #5429 (ci.yml) — status: waiting

The runner has not picked up these jobs. They have been waiting since the merge at 14:45 UTC.

2. e2e tests have never passed on main

Every e2e workflow run on the main branch has failed across the entire history of the repo (runs #2 through #19). The quality jobs (lint/typecheck/unit) consistently pass, but e2e always fails. This predates PR #45.

Reproduction steps

  1. Check Forgejo Actions for Mark/nodedesk — runs #5428 and #5429 are in waiting status.
  2. Browse the full run history — every e2e run on main has status=failure.

Expected behavior

  • Post-merge CI runs should be picked up by the runner within minutes.
  • e2e tests should pass (or be explicitly quarantined/skipped with a tracking issue).

Actual behavior

  • Runner is not picking up post-merge jobs.
  • e2e tests fail on every push to main.

Context

  • The runner was previously diagnosed as having host-networking issues with service containers (referenced in PR #45 commit messages as P1001).
  • PR #45 quarantined worklog-timer.spec.ts as a timing-dependent flaky test.
  • The runner infrastructure may need attention (registration token scope, label matching, or container networking).

Reported by QA verification for kanban card t_b85d7159.

## Findings from QA verification (kanban t_b85d7159) PR #45 (ci/quality-gates-matrix) was merged to main on 2026-08-07T14:45:26Z. The merge commit is 18a1c168. ### What works - Lint, Type check, and Unit tests (Node 20 + 22) all passed on the PR branch before merge. - The matrix quality-gate pipeline structure is correct and functional. ### What is broken **1. Post-merge CI runs stuck in waiting** Two CI runs were triggered by the merge commit (18a1c168): - Run #5428 (ci.yml) — status: waiting - Run #5429 (ci.yml) — status: waiting The runner has not picked up these jobs. They have been waiting since the merge at 14:45 UTC. **2. e2e tests have never passed on main** Every e2e workflow run on the main branch has failed across the entire history of the repo (runs #2 through #19). The quality jobs (lint/typecheck/unit) consistently pass, but e2e always fails. This predates PR #45. ### Reproduction steps 1. Check Forgejo Actions for Mark/nodedesk — runs #5428 and #5429 are in waiting status. 2. Browse the full run history — every e2e run on main has status=failure. ### Expected behavior - Post-merge CI runs should be picked up by the runner within minutes. - e2e tests should pass (or be explicitly quarantined/skipped with a tracking issue). ### Actual behavior - Runner is not picking up post-merge jobs. - e2e tests fail on every push to main. ### Context - The runner was previously diagnosed as having host-networking issues with service containers (referenced in PR #45 commit messages as P1001). - PR #45 quarantined worklog-timer.spec.ts as a timing-dependent flaky test. - The runner infrastructure may need attention (registration token scope, label matching, or container networking). --- Reported by QA verification for kanban card t_b85d7159.
Mark commented 2026-08-11 05:18:28 +00:00
Author
Owner
Copy link

Bughunt update (2026-08-10, t_61ae8e84): two code-side contributors to the red-CI state are now fixed on main.

  1. Fabricated action SHAs in the legacy .gitea/workflows/ci.yml copy — setup-node@8f152de4… and upload-artifact@5d5fc9b0… do not exist upstream (git fetch: upload-pack: not our ref; upload-artifact has no v3.10.0 tag). That workflow is the legacy alias path the runner also discovers, so any run that picked it up failed at the action-resolution step. Replaced with the verified pins already used by .forgejo/workflows/ci.yml (commit d618eb0).
  2. Dummy STRIPE_* env in the workflows breaks env.test.ts isStripeConfigured returns false when stripe vars are missing — the removal is in PR #59 (ci/verify-green-workflows), not yet merged.

Runner pick-up (post-merge runs stuck in waiting) remains an infrastructure question outside repo code — worth confirming runner registration scope/labels on the Actions runner. e2e on main: see the separate e2e workflow; the quarantined specs are tracked in #53/#54.

Bughunt update (2026-08-10, t_61ae8e84): two code-side contributors to the red-CI state are now fixed on main. 1. Fabricated action SHAs in the legacy `.gitea/workflows/ci.yml` copy — setup-node@8f152de4… and upload-artifact@5d5fc9b0… do not exist upstream (git fetch: `upload-pack: not our ref`; upload-artifact has no v3.10.0 tag). That workflow is the legacy alias path the runner also discovers, so any run that picked it up failed at the action-resolution step. Replaced with the verified pins already used by `.forgejo/workflows/ci.yml` (commit d618eb0). 2. Dummy STRIPE_* env in the workflows breaks env.test.ts `isStripeConfigured returns false when stripe vars are missing` — the removal is in PR #59 (ci/verify-green-workflows), not yet merged. Runner pick-up (post-merge runs stuck in `waiting`) remains an infrastructure question outside repo code — worth confirming runner registration scope/labels on the Actions runner. e2e on main: see the separate e2e workflow; the quarantined specs are tracked in #53/#54.
Mark removed the
bug
 label 2026-08-13 00:12:44 +00:00
Mark closed this issue 2026-08-14 04:15:28 +00:00
Mark reopened this issue 2026-08-14 06:31:07 +00:00
Mark commented 2026-08-14 06:31:07 +00:00
Author
Owner
Copy link

Reopened by QA reporter (kanban t_539d5e5a) — false-positive close

Closed 2026-08-14T04:15:28Z, but the acceptance criterion ("e2e green on main") was not met at close time. Reopening with the root cause.

Evidence

  • Run 5664 (e2e.yml, head 9805e368 = the push containing ALL #55 fixes, incl. the localhost DATABASE_URL restore in 00bf954) FAILED at 293s — identical duration to the pre-fix run 5638 (293s, head 3ff5784).
  • The workflow comment + docs/ci/audit.md §3.8 called this push "the expected-green verification run" — it was not.
  • Meanwhile ci.yml was green on the same head (run 5663, success) and on the #59 merge (run 5637, success). Only e2e.yml is red.

Root cause: host port 5432 conflict under host-networking runner

  • The Actions runner on this host runs job containers with container.network: host (see ~/.forgejo-runner-*/data/config.yaml), and e2e.yml declares a postgres:16 service with ports: ["5432:5432"] and DATABASE_URL: postgresql://test:***@localhost:5432/test.
  • Under host networking the service container must bind the host's port 5432. Any local postgres container already holding 5432 makes the service start fail (or makes the job talk to the wrong instance), and the job dies at ~293s.
  • During runs 5638/5664 the host's pg-nodedesk container (up ~9 days, exited 05:33Z) held 5432. Right now devforge_db_1 (0.0.0.0:5432) holds it — a fresh e2e run (task 9494, triggered ~2026-08-13 23:2x) is failing the same way.
  • This is the exact hazard documented in docs/ci/audit.md §3.10 ("local docker compose up -d db + a CI run on the same host").

Reproduction steps

  1. Ensure a local postgres container holds host port 5432 (e.g. docker compose up -d db from Mark/devforge → devforge_db_1, or pg-nodedesk).
  2. Push any commit to main (or trigger workflow_dispatch on .forgejo/workflows/e2e.yml).
  3. Open Actions → e2e.yml run.

Actual vs expected

  • Actual: e2e job fails after ~293s; the workflow's postgres service cannot reliably bind host port 5432 when a local postgres already holds it. prisma db push / seed / Playwright all run against the wrong (or missing) test DB.
  • Expected: e2e run green on main (as claimed when #55 was closed), regardless of what local postgres containers happen to be running on the runner host.

Suggested fix directions (for the fixer / operator to evaluate)

  • Stop/avoid local postgres containers on the runner host during CI, OR
  • Change the e2e service to a non-conflicting port mapping (5433:5432) with DATABASE_URL updated to match, OR
  • Reconsider container.network: host for the e2e job so the service gets its own network namespace.

Re-verify e2e green on main before closing again.

## Reopened by QA reporter (kanban t_539d5e5a) — false-positive close Closed 2026-08-14T04:15:28Z, but the acceptance criterion ("e2e green on main") was **not met at close time**. Reopening with the root cause. ### Evidence - Run **5664** (e2e.yml, head `9805e368` = the push containing ALL #55 fixes, incl. the localhost DATABASE_URL restore in `00bf954`) **FAILED at 293s** — identical duration to the pre-fix run **5638** (293s, head `3ff5784`). - The workflow comment + `docs/ci/audit.md §3.8` called this push "the expected-green verification run" — it was not. - Meanwhile ci.yml was green on the same head (run 5663, success) and on the #59 merge (run 5637, success). Only e2e.yml is red. ### Root cause: host port 5432 conflict under host-networking runner - The Actions runner on this host runs job containers with `container.network: host` (see `~/.forgejo-runner-*/data/config.yaml`), and e2e.yml declares a `postgres:16` service with `ports: ["5432:5432"]` and `DATABASE_URL: postgresql://test:***@localhost:5432/test`. - Under host networking the service container must bind the **host's** port 5432. Any local postgres container already holding 5432 makes the service start fail (or makes the job talk to the wrong instance), and the job dies at ~293s. - During runs 5638/5664 the host's `pg-nodedesk` container (up ~9 days, exited 05:33Z) held 5432. Right now `devforge_db_1` (0.0.0.0:5432) holds it — a fresh e2e run (task 9494, triggered ~2026-08-13 23:2x) is failing the same way. - This is the exact hazard documented in `docs/ci/audit.md §3.10` ("local docker compose up -d db + a CI run on the same host"). ### Reproduction steps 1. Ensure a local postgres container holds host port 5432 (e.g. `docker compose up -d db` from Mark/devforge → `devforge_db_1`, or `pg-nodedesk`). 2. Push any commit to `main` (or trigger `workflow_dispatch` on `.forgejo/workflows/e2e.yml`). 3. Open Actions → e2e.yml run. ### Actual vs expected - **Actual:** e2e job fails after ~293s; the workflow's `postgres` service cannot reliably bind host port 5432 when a local postgres already holds it. `prisma db push` / seed / Playwright all run against the wrong (or missing) test DB. - **Expected:** e2e run green on main (as claimed when #55 was closed), regardless of what local postgres containers happen to be running on the runner host. ### Suggested fix directions (for the fixer / operator to evaluate) - Stop/avoid local postgres containers on the runner host during CI, OR - Change the e2e service to a non-conflicting port mapping (`5433:5432`) with `DATABASE_URL` updated to match, OR - Reconsider `container.network: host` for the e2e job so the service gets its own network namespace. Re-verify e2e green on main **before** closing again.
Mark commented 2026-08-14 08:43:02 +00:00
Author
Owner
Copy link

Verbatim failing-step log (kanban t_722c7942 spot-check)

Corroborating evidence for the 2026-08-14 reopen above — same run, step-level log pulled from the Actions task store (action_run_job id 14316, action_task id 9485, action_task_step id 46061).

Run: id 5664, index 431 — workflow e2e.yml, event push, ref refs/heads/main
Commit SHA: 9805e368e210e2d8a8c0e86415758659348a50f8
Status: failure (started 2026-08-14T04:30:09Z, stopped 2026-08-14T04:35:02Z, duration 293s)
Failing job/step: job e2e → step "Push database schema" — subsequent steps (Seed database, Build application, Run Playwright tests) skipped.
Run URL: https://forgejo.tail018ac4.ts.net/Mark/nodedesk/actions/runs/431

Log excerpt (verbatim from action_task_step log):

2026-08-14T04:33:06.5289255Z Prisma schema loaded from prisma/schema.prisma
2026-08-14T04:33:06.5317226Z Datasource "db": PostgreSQL database "test", schema "public" at "localhost:5432"
2026-08-14T04:33:06.5953118Z 
2026-08-14T04:33:06.5956843Z Error: P1000: Authentication failed against database server at `localhost`, the provided database credentials for `test` are not valid.
2026-08-14T04:33:06.5957122Z 
2026-08-14T04:33:06.5957272Z Please make sure to provide valid database credentials for the database server at `localhost`.

Consistent with the root cause already documented here: under container.network: host, the postgres:16 service must bind host port 5432; a local postgres container holding 5432 makes the job talk to the wrong instance → P1000 auth failure at ~293s.

Source card: kanban t_722c7942 (spot-check CI run on default branch HEAD).

## Verbatim failing-step log (kanban t_722c7942 spot-check) Corroborating evidence for the 2026-08-14 reopen above — same run, step-level log pulled from the Actions task store (`action_run_job` id 14316, `action_task` id 9485, `action_task_step` id 46061). **Run:** id `5664`, index `431` — workflow `e2e.yml`, event push, ref `refs/heads/main` **Commit SHA:** `9805e368e210e2d8a8c0e86415758659348a50f8` **Status:** failure (started 2026-08-14T04:30:09Z, stopped 2026-08-14T04:35:02Z, duration 293s) **Failing job/step:** job `e2e` → step **"Push database schema"** — subsequent steps (Seed database, Build application, Run Playwright tests) skipped. **Run URL:** https://forgejo.tail018ac4.ts.net/Mark/nodedesk/actions/runs/431 Log excerpt (verbatim from action_task_step log): ``` 2026-08-14T04:33:06.5289255Z Prisma schema loaded from prisma/schema.prisma 2026-08-14T04:33:06.5317226Z Datasource "db": PostgreSQL database "test", schema "public" at "localhost:5432" 2026-08-14T04:33:06.5953118Z 2026-08-14T04:33:06.5956843Z Error: P1000: Authentication failed against database server at `localhost`, the provided database credentials for `test` are not valid. 2026-08-14T04:33:06.5957122Z 2026-08-14T04:33:06.5957272Z Please make sure to provide valid database credentials for the database server at `localhost`. ``` Consistent with the root cause already documented here: under `container.network: host`, the `postgres:16` service must bind host port 5432; a local postgres container holding 5432 makes the job talk to the wrong instance → P1000 auth failure at ~293s. Source card: kanban `t_722c7942` (spot-check CI run on default branch HEAD).
Mark commented 2026-08-14 14:36:26 +00:00
Author
Owner
Copy link

Runner root cause confirmed (kanban t_d025ed96, 2026-08-14):

  • Runs #5428/#5429 sat waiting because NO online runner with the linux label existed from 2026-07-31 13:26Z (zsus Tailscale logged out, zsus-instance-runner last_online) until 2026-08-11 04:19Z (localhost-nodedesk-runner registered on blackhole). Not label mismatch, not token scope, not P1001.
  • They were never picked up (started=0, no action_task rows) and Forgejo's stale-waiting cleanup cancelled them 2026-08-08 19:06:02Z. Terminal.
  • Fix is live: runner id 24 (repo-scoped Mark/nodedesk, labels ubuntu-latest/docker/linux/x64) picks up every push within seconds; last_online now; zero waiting runs. The P1001 port-5432 conflict is a separate e2e failure mode (runs 5638/5664) tracked in this issue.
Runner root cause confirmed (kanban t_d025ed96, 2026-08-14): - Runs #5428/#5429 sat waiting because NO online runner with the `linux` label existed from 2026-07-31 13:26Z (zsus Tailscale logged out, zsus-instance-runner last_online) until 2026-08-11 04:19Z (localhost-nodedesk-runner registered on blackhole). Not label mismatch, not token scope, not P1001. - They were never picked up (started=0, no action_task rows) and Forgejo's stale-waiting cleanup cancelled them 2026-08-08 19:06:02Z. Terminal. - Fix is live: runner id 24 (repo-scoped Mark/nodedesk, labels ubuntu-latest/docker/linux/x64) picks up every push within seconds; last_online now; zero waiting runs. The P1001 port-5432 conflict is a separate e2e failure mode (runs 5638/5664) tracked in this issue.
Mark commented 2026-08-14 15:23:53 +00:00
Author
Owner
Copy link

Reopened by kilo-gsd-runner: kilo-gsd run 2026-08-14 08:17 closed this issue WITHOUT applying any fix — Commit phase reported "No changes to push" (stale 2026-08-13 plan reused), origin/main still at 9805e36, e2e.yml still binds 5432 with DATABASE_URL localhost:5432. Root cause (t_2966c6ac): runner runs containers with container.network=host; host port 5432 owned by devforge_db_1, so the postgres service must use PGPORT 5433. Applying the documented fix now (PGPORT 5433 + DATABASE_URL localhost:5433) and opening a PR.

Reopened by kilo-gsd-runner: kilo-gsd run 2026-08-14 08:17 closed this issue WITHOUT applying any fix — Commit phase reported "No changes to push" (stale 2026-08-13 plan reused), origin/main still at 9805e36, e2e.yml still binds 5432 with DATABASE_URL localhost:5432. Root cause (t_2966c6ac): runner runs containers with container.network=host; host port 5432 owned by devforge_db_1, so the postgres service must use PGPORT 5433. Applying the documented fix now (PGPORT 5433 + DATABASE_URL localhost:5433) and opening a PR.
Mark commented 2026-08-14 17:43:37 +00:00
Author
Owner
Copy link

Fixed and verified — main-branch CI is now green (2026-08-14).

Root cause (kanban t_2966c6ac): two independent e2e failure modes on the Forgejo runner (container.network=host, single-slot runner).

Fix 1 — PGPORT collision (P1000): e2e.yml postgres service bound host port 5432, already owned by devforge_db_1 on the runner host; prisma db push hit the wrong server (P1000, run #5664). Merged via PR #62 (commit 7d834ea): service now runs on PGPORT 5433, ports 5433:5432, health-cmd pg_isready -p 5433, DATABASE_URL localhost:5433.

Fix 2 — Playwright browser mismatch: container image mcr.microsoft.com/playwright:v1.47.2-jammy bakes in browsers for 1.47.2, but npm ci installs @playwright/test 1.60.0 (caret range) — every test failed at launch with "Executable doesnt exist at /ms-playwright/chromium_headless_shell-1223". Merged via PR #63 (commit be45040): image bumped to playwright:v1.60.0-jammy.

Verification (all green):

  • PR #62 CI: run #432, 7/7 jobs success.
  • workflow_dispatch on fix branch (image fix): run #435 e2e success.
  • Post-merge main: run #436 ci success, run #437 ci success, run #438 e2e success (run IDs per Forgejo actions; API run ids 5729/5730/5731).
  • Local repro with CI-equivalent config: prisma db push OK, seed OK, Playwright 28 passed / 1 skipped.

Remaining quarantined tests (worklog-timer #53, attachments #51) are explicitly tracked — not part of this issue.

**Fixed and verified — main-branch CI is now green (2026-08-14).** Root cause (kanban t_2966c6ac): two independent e2e failure modes on the Forgejo runner (container.network=host, single-slot runner). **Fix 1 — PGPORT collision (P1000):** e2e.yml postgres service bound host port 5432, already owned by devforge_db_1 on the runner host; `prisma db push` hit the wrong server (P1000, run #5664). Merged via PR #62 (commit 7d834ea): service now runs on **PGPORT 5433**, ports 5433:5432, health-cmd `pg_isready -p 5433`, DATABASE_URL localhost:5433. **Fix 2 — Playwright browser mismatch:** container image mcr.microsoft.com/playwright:v1.47.2-jammy bakes in browsers for 1.47.2, but npm ci installs @playwright/test 1.60.0 (caret range) — every test failed at launch with "Executable doesnt exist at /ms-playwright/chromium_headless_shell-1223". Merged via PR #63 (commit be45040): image bumped to **playwright:v1.60.0-jammy**. **Verification (all green):** - PR #62 CI: run #432, 7/7 jobs success. - workflow_dispatch on fix branch (image fix): run #435 e2e **success**. - Post-merge main: run #436 ci success, run #437 ci success, run #438 e2e **success** (run IDs per Forgejo actions; API run ids 5729/5730/5731). - Local repro with CI-equivalent config: prisma db push OK, seed OK, Playwright 28 passed / 1 skipped. Remaining quarantined tests (worklog-timer #53, attachments #51) are explicitly tracked — not part of this issue.
Mark closed this issue 2026-08-14 17:43:47 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
blocked:gate-fetch

   
blocked:gate-implement

   
blocked:needs-attention

   
blocked:needs-decision

   
blocked:needs-human

   
blocked:needs-local-checkout

   
blocked:no-changes

   
bug

Bug report

  
feature-request

   
in-progress

   
needs-decompose

Ready to be decomposed into sub-issues

  
needs-triage

   
ops-task

   
quickfix

   
security/high

High severity security issue

  
security/low

Low severity security issue

  
security/medium

Medium severity security issue

  
triage-approved
No labels
blocked:gate-fetch
blocked:gate-implement
blocked:needs-attention
blocked:needs-decision
blocked:needs-human
blocked:needs-local-checkout
blocked:no-changes
bug
feature-request
in-progress
needs-decompose
needs-triage
ops-task
quickfix
security/high
security/low
security/medium
triage-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Mark/nodedesk#55
No description provided.