Fix: E2E test suite runtime exceeds 60s target (regression from #67-#71) #73

New issue

Closed

opened 2026-06-23 20:14:13 +00:00 by Mark · 1 comment

Mark commented

2026-06-23 20:14:13 +00:00

Owner

Problem

The full test suite now takes ~78 seconds. The <60s target was specified in the acceptance criteria of #68 (closed). Source: the new e2e tests in tests/test_pipeline_e2e.py and tests/test_concurrency_contract.py add several full-pipeline runs that dominate the runtime.

Measured times

Before #67 (pre-walk baseline): 0.48s for the full suite (21 tests)
After #67 (pull compensation toggle): 25.69s for 50 tests
After #68 (machine profile constraints): 78.99s for 55 tests
After #71 (failure semantics): 78.51s for 81 tests

Body guidance from #68 (original)

Total runtime target: < 60 seconds for the full e2e suite. If it
exceeds that, the slowest test gets a pytest.mark.slow and CI
runs it in a separate job.

What's needed

Identify the slowest tests via pytest --durations=20.
Mark them with @pytest.mark.slow per body guidance.
Update CI workflow to run pytest -m "not slow" as the default PR check, and a separate job for pytest -m slow.
Optionally: reduce fixture sizes for the slowest tests (the pull-comp on-vs-off test currently runs the full pipeline twice with a non-trivial SVG; a smaller fixture would cut its cost).
Update docs/concurrency.md / PR description to reflect the split.

Acceptance

pytest -m "not slow" completes in < 60 seconds on the current dev machine.
pytest -m slow runs in a separate CI job.
No regression in test coverage — every test still runs, just in a separate job.

Decomposed from easystitch#68

## Problem The full test suite now takes ~78 seconds. The `<60s` target was specified in the acceptance criteria of #68 (closed). Source: the new e2e tests in `tests/test_pipeline_e2e.py` and `tests/test_concurrency_contract.py` add several full-pipeline runs that dominate the runtime. ## Measured times - Before #67 (pre-walk baseline): `0.48s` for the full suite (21 tests) - After #67 (pull compensation toggle): `25.69s` for 50 tests - After #68 (machine profile constraints): `78.99s` for 55 tests - After #71 (failure semantics): `78.51s` for 81 tests ## Body guidance from #68 (original) > Total runtime target: < 60 seconds for the full e2e suite. If it > exceeds that, the slowest test gets a `pytest.mark.slow` and CI > runs it in a separate job. ## What's needed - Identify the slowest tests via `pytest --durations=20`. - Mark them with `@pytest.mark.slow` per body guidance. - Update CI workflow to run `pytest -m "not slow"` as the default PR check, and a separate job for `pytest -m slow`. - Optionally: reduce fixture sizes for the slowest tests (the pull-comp on-vs-off test currently runs the full pipeline twice with a non-trivial SVG; a smaller fixture would cut its cost). - Update `docs/concurrency.md` / PR description to reflect the split. ## Acceptance - `pytest -m "not slow"` completes in < 60 seconds on the current dev machine. - `pytest -m slow` runs in a separate CI job. - No regression in test coverage — every test still runs, just in a separate job. _Decomposed from [easystitch#68](https://forgejo.tail018ac4.ts.net/Mark/easystitch/issues/68)_

Mark added the

bug

triage-approved

needs-decompose

labels

2026-06-23 20:14:13 +00:00

Mark commented

2026-06-23 20:16:00 +00:00

Author

Owner

Auto-triage: needs decompose

Marked triage-approved + needs-decompose. decompose-executor will pick this up and break it into implementable sub-issues.

## Auto-triage: needs decompose Marked `triage-approved` + `needs-decompose`. decompose-executor will pick this up and break it into implementable sub-issues.

Mark

2026-06-25 14:07:38 +00:00