Regression test test_build_output_directories_not_tracked fails whenever .next build output exists (over-strict assertion, flaky) #27

New issue

Closed

opened 2026-07-31 08:00:25 +00:00 by Mark · 1 comment

Mark commented

2026-07-31 08:00:25 +00:00

Owner

Summary

tests/test_security_scan_config.py::test_build_output_directories_not_tracked fails spuriously whenever .next/ build output exists in the working tree. The fix for #21 landed correctly (.next/ untracked + gitignored, .semgrepignore excludes it, scanner prunes it), but the regression test added with the fix asserts something stricter than the actual security requirement and breaks under completely normal conditions (any local next build or pipeline build).

Reproduction steps

Check out AuWoMe/forgeflow on main (verified on commit f473d0a, tree state 2026-07-31).

Run a production build so build output exists:

npm run build   # next.config output: 'standalone' regenerates .next/ with 1700+ JS artifacts

Run the regression test file:

python3 -m pytest tests/test_security_scan_config.py

Observe: test_build_output_directories_not_tracked FAILS (the other two tests pass).

Current failure output (verified by probe and re-verified at filing time — 1702 JS artifacts under .next/):

>       assert not js_artifacts, (
            f"{dirname}/ must not contain generated JS artifacts, found: {js_artifacts!r}"
        )
E       AssertionError: .next/ must not contain generated JS artifacts, found: [PosixPath('.next/prerender-manifest.js'), ...]
tests/test_security_scan_config.py:43: AssertionError

Actual vs expected behavior

Actual: The test asserts if .next/ exists, it must contain NO JS artifacts (assert not js_artifacts at tests/test_security_scan_config.py:43). Any local next build or pipeline build regenerates .next/ (this is what .gitignore and .semgrepignore are for), so the test fails in normal operation. It passed the 2026-07-30 23:59 kilo-gsd verify gate only because .next happened to be absent at that moment — latent flaky test.
Expected: The security requirement from #21 is that .next/ build output is (a) not tracked by git and (b) excluded from Semgrep scanning — not that it never exists on disk. The test already verifies both requirements in its lower half (git ls-files .next is empty, git check-ignore .next/generated.js returns 0). The "no artifacts on disk" assertion should be relaxed (e.g. dropped, or scoped to tracked files only) so the test passes whenever build output legitimately exists.

Impact

Any developer or CI pipeline that builds before running pytest gets a spurious failure.
Gate runs become order-dependent: pass only when .next was never generated in that checkout.
Verified current state: 1 failed / 2 passed in tests/test_security_scan_config.py; tests/test_pre_commit_loop_detector.py 5/5 passed. No src/ code involved.

Reference

Originating kanban card: t_1c6c1e47 (post-deploy probe for AuWoMe/forgeflow#21)
Fix refs: #21; regression test introduced with the security-scan hardening (f94c948, 2026-07-07/08 remediation commits)

## Summary `tests/test_security_scan_config.py::test_build_output_directories_not_tracked` **fails spuriously whenever `.next/` build output exists in the working tree**. The fix for #21 landed correctly (`.next/` untracked + gitignored, `.semgrepignore` excludes it, scanner prunes it), but the regression test added with the fix asserts something stricter than the actual security requirement and breaks under completely normal conditions (any local `next build` or pipeline build). ## Reproduction steps 1. Check out `AuWoMe/forgeflow` on `main` (verified on commit `f473d0a`, tree state 2026-07-31). 2. Run a production build so build output exists: ```bash npm run build # next.config output: 'standalone' regenerates .next/ with 1700+ JS artifacts ``` 3. Run the regression test file: ```bash python3 -m pytest tests/test_security_scan_config.py ``` 4. Observe: `test_build_output_directories_not_tracked` FAILS (the other two tests pass). Current failure output (verified by probe and re-verified at filing time — 1702 JS artifacts under `.next/`): ``` > assert not js_artifacts, ( f"{dirname}/ must not contain generated JS artifacts, found: {js_artifacts!r}" ) E AssertionError: .next/ must not contain generated JS artifacts, found: [PosixPath('.next/prerender-manifest.js'), ...] tests/test_security_scan_config.py:43: AssertionError ``` ## Actual vs expected behavior - **Actual:** The test asserts `if .next/ exists, it must contain NO JS artifacts` (`assert not js_artifacts` at `tests/test_security_scan_config.py:43`). Any local `next build` or pipeline build regenerates `.next/` (this is what `.gitignore` and `.semgrepignore` are for), so the test fails in normal operation. It passed the 2026-07-30 23:59 kilo-gsd verify gate only because `.next` happened to be absent at that moment — latent flaky test. - **Expected:** The security requirement from #21 is that `.next/` build output is (a) **not tracked by git** and (b) **excluded from Semgrep scanning** — not that it never exists on disk. The test already verifies both requirements in its lower half (`git ls-files .next` is empty, `git check-ignore .next/generated.js` returns 0). The "no artifacts on disk" assertion should be relaxed (e.g. dropped, or scoped to tracked files only) so the test passes whenever build output legitimately exists. ## Impact - Any developer or CI pipeline that builds before running `pytest` gets a spurious failure. - Gate runs become order-dependent: pass only when `.next` was never generated in that checkout. - Verified current state: `1 failed / 2 passed` in `tests/test_security_scan_config.py`; `tests/test_pre_commit_loop_detector.py` 5/5 passed. No `src/` code involved. ## Reference - Originating kanban card: `t_1c6c1e47` (post-deploy probe for AuWoMe/forgeflow#21) - Fix refs: #21; regression test introduced with the security-scan hardening (f94c948, 2026-07-07/08 remediation commits)

Mark added the

bug

label

2026-07-31 08:00:25 +00:00

Mark referenced this issue from a commit

2026-08-11 09:52:41 +00:00

fix(hooks,tests): loop-detector hardening + relax over-strict .next test (#26, #27)

Mark commented

2026-08-11 09:53:46 +00:00

Author

Owner

Fixed on main by commit 07599d0. Removed the over-strict js_artifacts assertion from test_build_output_directories_not_tracked — build output may legitimately exist in the working tree after next build; the actual security requirement is that those files are never tracked and are gitignored so Semgrep's :include .gitignore excludes them. The test now verifies git ls-files is empty for .next/ and out/ and git check-ignore rejects generated files. Verified with .next/ present (1700+ artifacts): 9/9 pytest pass (previously 1 failed).

Fixed on main by commit 07599d0. Removed the over-strict `js_artifacts` assertion from `test_build_output_directories_not_tracked` — build output may legitimately exist in the working tree after `next build`; the actual security requirement is that those files are never tracked and are gitignored so Semgrep's `:include .gitignore` excludes them. The test now verifies `git ls-files` is empty for .next/ and out/ and `git check-ignore` rejects generated files. Verified with .next/ present (1700+ artifacts): 9/9 pytest pass (previously 1 failed).

Mark closed this issue

2026-08-11 09:53:46 +00:00