[fixit] Add extraneous-geometry metric to render_compare.py (catches #72/#77 phantom pillars + hallucinated holes) #84

Merged

Mark merged 1 commit from fixit/render-compare-extraneous-metric into main 2026-10-01 17:05:31 +00:00

Conversation 1 Commits 1 Files changed 6 +461 -4

Mark commented 2026-10-01 14:24:57 +00:00

Owner

Copy link

What

Add the symmetric coverage pair to tests/render_compare.py:

• extraneous_fraction — fraction of mesh-surface sample points whose nearest input point is farther than tolerance mm. Symmetric counterpart to coverage_fraction.
• symmetric_chamfer_mm — mean of input→mesh and mesh→input mean nearest-neighbour distances. Single-scalar over- + under-coverage gate.
• New CLI flags --max-extraneous-fraction and --max-symmetric-chamfer-mm.
• Per-spec keys extraneous_fraction_max / symmetric_chamfer_mm_max in expected_metrics() for spec-driven gates.

Why

The Verify gate was one-sided. It scored under-reconstruction (missing geometry) but not over-reconstruction (phantom geometry). At origin/main e8053cb the L-bracket reconstruction hallucinated phantom pillars + an unintended circular hole while passing coverage 0.8176, normals 0.999, faces=38 — all green. The methodology_assembly reconstruction hallucinated a cylindrical bore + semicircular archway while passing coverage 0.8276, normals 0.998, faces=56. Bug class: coverage-blind verification.

Acceptance

Per the originating card body:

1. tests/render_compare.py outputs extraneous and symmetric_chamfer in metrics.json with --max-extraneous-fraction / --max-symmetric-chamfer-mm CLI flags.
2. Existing test_l_bracket_render_compare_passes and test_plate_with_hole_render_compare_passes FAIL on current origin/main when run with --max-extraneous-fraction 0.05 — the metric CATCHES the bug. Demo run output:
   • strategy_l_bracket: extraneous_fraction 0.080 > 0.05 threshold, step faces 16 < 20 spec min
   • strategy_plate_with_hole: extraneous_fraction 0.130 > 0.05 threshold
3. New tests/test_topology_no_extraneous_geometry.py walks every samples/specs/*_spec.py and asserts the new metrics for opted-in specs.
4. CI on fixit/render-compare-extraneous-metric.
5. This PR. Linked from comments on #72 and #77 with the failure output as evidence.

Out of scope

Replacing the existing coverage metric. This PR is additive — it adds the symmetric counterpart. The two together form the symmetric coverage pair.

Test status (current origin/main @ 09be6be)

• tests/render_compare.py::test_render_compare_step_roundtrip PASS
• tests/render_compare.py::test_render_compare_coverage_metric PASS
• tests/render_compare.py::test_render_compare_smoke PASS
• tests/test_topology_no_extraneous_geometry.py[strategy_l_bracket] FAIL (expected — proves metric works)
• tests/test_topology_no_extraneous_geometry.py[strategy_plate_with_hole] FAIL (expected — proves metric works)
• tests/test_topology_no_extraneous_geometry.py[*] SKIP (7 specs without extraneous_fraction_max key — opt-in)

The failing tests are the deliverable, not a bug — they demonstrate the new metric catches phantom geometry that the existing coverage gate misses.

Refs

• Mark/pc2cad#72 (L-bracket phantom pillars + circular hole)
• Mark/pc2cad#77 (methodology_assembly hallucinated top hole)
• Mark/pc2cad#80 (OPPOSES-join BFS over-connection)
• Mark/pc2cad#73 (torus fitter visual verification)

Files

• tests/render_compare.py — ExtraneousMetrics + ChamferMetrics dataclasses, extraneous_metric() + symmetric_chamfer_metric() functions, CLI flags, metrics.json keys, spec-driven gates.
• tests/test_reconstruction.py — added --max-extraneous-fraction + --max-symmetric-chamfer-mm to test_l_bracket_render_compare_passes and test_plate_with_hole_render_compare_passes.
• tests/test_topology_no_extraneous_geometry.py — new parametric regression test walking every spec.
• samples/specs/strategy_l_bracket_spec.py — added extraneous_fraction_max: 0.05 + symmetric_chamfer_mm_max: 1.5.
• samples/specs/strategy_plate_with_hole_spec.py — added extraneous_fraction_max: 0.05 + symmetric_chamfer_mm_max: 1.0.
• pointcloud-to-cad-guide.md — new "Verify gate semantics" section.

## What Add the symmetric coverage pair to `tests/render_compare.py`: - `extraneous_fraction` — fraction of mesh-surface sample points whose nearest input point is farther than `tolerance` mm. Symmetric counterpart to `coverage_fraction`. - `symmetric_chamfer_mm` — mean of input→mesh and mesh→input mean nearest-neighbour distances. Single-scalar over- + under-coverage gate. - New CLI flags `--max-extraneous-fraction` and `--max-symmetric-chamfer-mm`. - Per-spec keys `extraneous_fraction_max` / `symmetric_chamfer_mm_max` in `expected_metrics()` for spec-driven gates. ## Why The Verify gate was one-sided. It scored under-reconstruction (missing geometry) but not over-reconstruction (phantom geometry). At `origin/main` e8053cb the L-bracket reconstruction hallucinated phantom pillars + an unintended circular hole while passing `coverage 0.8176`, `normals 0.999`, `faces=38` — all green. The methodology_assembly reconstruction hallucinated a cylindrical bore + semicircular archway while passing `coverage 0.8276`, `normals 0.998`, `faces=56`. Bug class: coverage-blind verification. ## Acceptance Per the originating card body: 1. `tests/render_compare.py` outputs `extraneous` and `symmetric_chamfer` in `metrics.json` with `--max-extraneous-fraction` / `--max-symmetric-chamfer-mm` CLI flags. 2. Existing `test_l_bracket_render_compare_passes` and `test_plate_with_hole_render_compare_passes` FAIL on current `origin/main` when run with `--max-extraneous-fraction 0.05` — the metric CATCHES the bug. Demo run output: - strategy_l_bracket: `extraneous_fraction 0.080` > 0.05 threshold, `step faces 16` < 20 spec min - strategy_plate_with_hole: `extraneous_fraction 0.130` > 0.05 threshold 3. New `tests/test_topology_no_extraneous_geometry.py` walks every `samples/specs/*_spec.py` and asserts the new metrics for opted-in specs. 4. CI on `fixit/render-compare-extraneous-metric`. 5. This PR. Linked from comments on #72 and #77 with the failure output as evidence. ## Out of scope Replacing the existing coverage metric. This PR is additive — it adds the symmetric counterpart. The two together form the symmetric coverage pair. ## Test status (current origin/main @ 09be6be) - `tests/render_compare.py::test_render_compare_step_roundtrip` PASS - `tests/render_compare.py::test_render_compare_coverage_metric` PASS - `tests/render_compare.py::test_render_compare_smoke` PASS - `tests/test_topology_no_extraneous_geometry.py[strategy_l_bracket]` FAIL (expected — proves metric works) - `tests/test_topology_no_extraneous_geometry.py[strategy_plate_with_hole]` FAIL (expected — proves metric works) - `tests/test_topology_no_extraneous_geometry.py[*]` SKIP (7 specs without `extraneous_fraction_max` key — opt-in) The failing tests are the deliverable, not a bug — they demonstrate the new metric catches phantom geometry that the existing coverage gate misses. ## Refs - Mark/pc2cad#72 (L-bracket phantom pillars + circular hole) - Mark/pc2cad#77 (methodology_assembly hallucinated top hole) - Mark/pc2cad#80 (OPPOSES-join BFS over-connection) - Mark/pc2cad#73 (torus fitter visual verification) ## Files - `tests/render_compare.py` — `ExtraneousMetrics` + `ChamferMetrics` dataclasses, `extraneous_metric()` + `symmetric_chamfer_metric()` functions, CLI flags, metrics.json keys, spec-driven gates. - `tests/test_reconstruction.py` — added `--max-extraneous-fraction` + `--max-symmetric-chamfer-mm` to `test_l_bracket_render_compare_passes` and `test_plate_with_hole_render_compare_passes`. - `tests/test_topology_no_extraneous_geometry.py` — new parametric regression test walking every spec. - `samples/specs/strategy_l_bracket_spec.py` — added `extraneous_fraction_max: 0.05` + `symmetric_chamfer_mm_max: 1.5`. - `samples/specs/strategy_plate_with_hole_spec.py` — added `extraneous_fraction_max: 0.05` + `symmetric_chamfer_mm_max: 1.0`. - `pointcloud-to-cad-guide.md` — new "Verify gate semantics" section.

Mark added 1 commit 2026-10-01 14:24:58 +00:00

test(render_compare): add extraneous-fraction + symmetric-Chamfer metrics ... 05b52d5906

The Verify gate in tests/render_compare.py was one-sided: it scored
under-reconstruction (input points the mesh fails to cover) but not
over-reconstruction (mesh surface that doesn't correspond to any
input point). At origin/main e8053cb the L-bracket and
methodology_assembly reconstructions hallucinated phantom pillars +
extra holes while passing coverage 0.82, normals 0.998, faces>=20 —
all green. The bug class is the 'coverage-blind verification'
anti-pattern.

Add the symmetric coverage pair:

- extraneous_fraction: sample the mesh surface uniformly, report the
  fraction whose nearest input point lies > tolerance mm away.
  Symmetric counterpart to coverage_fraction.
- symmetric_chamfer_mm: mean of input->mesh and mesh->input mean
  nearest-neighbour distances. Single-scalar over- + under-coverage
  gate.
- New CLI flags --max-extraneous-fraction and --max-symmetric-chamfer-mm
  in render_compare.py; per-spec keys extraneous_fraction_max and
  symmetric_chamfer_mm_max for spec-driven gates.

Wire the new metrics into the existing tests/test_reconstruction.py
l_bracket and plate_with_hole Verify gates. Add the canonical
regression test tests/test_topology_no_extraneous_geometry.py that
parametrically walks every samples/specs/*_spec.py and asserts the
new metrics. Update pointcloud-to-cad-guide.md with a 'Verify gate
semantics' section explaining the symmetric coverage pair.

The new tests FAIL on current origin/main for the two opted-in specs
(strategy_l_bracket, strategy_plate_with_hole) — this is the
acceptance criterion #2 from the originating card: 'fail on current
main when run with --max-extraneous-fraction 0.05 (proves the metric
catches the bug)'. The metric WORKS; a follow-up compose fix for
Mark/pc2cad#72 is the next card.

Refs: Mark/pc2cad#72 (L-bracket phantom pillars), #77
(methodology_assembly hallucinated top hole), #80 (OPPOSES-join BFS
over-connection), #73 (torus fitter visual verification).

Mark referenced this pull request 2026-10-01 14:25:51 +00:00

bug(compose): L-bracket topology broken — output is two disconnected slabs, not a joined L (synth_bracket coverage 0.229) #72

Mark referenced this pull request 2026-10-01 14:25:51 +00:00

bug(compose): methodology_assembly hallucinated top hole + arched cutout (coverage 0.847) #77

Mark merged commit d9ac95242f into main 2026-10-01 17:05:31 +00:00

Mark deleted branch fixit/render-compare-extraneous-metric 2026-10-01 17:05:31 +00:00

Mark referenced this pull request from a commit 2026-10-01 17:05:33 +00:00

Merge pull request '[fixit] Add extraneous-geometry metric to render_compare.py (catches #72/#77 phantom pillars + hallucinated holes)' (#84) from fixit/render-compare-extraneous-metric into main

Mark commented 2026-10-01 17:06:06 +00:00

Author

Owner

Copy link

Merged via fixup lane (kanban t_pr_Mark_pc2cad_84_merge).

• Merge commit: d9ac95242f99f95520b72396a3c0ebf456e571fe on main
• Merged at: 2026-10-01T17:05:31Z
• Method: default merge (no squash)
• Branch fixit/render-compare-extraneous-metric retained (delete_branch_after_merge requested but Forgejo returned 405 on the closure path; head branch still listed)

Referenced issues are intentionally left open — PR body uses Refs: (not Closes:), so they will not auto-close on merge. Per the PR description:

The failing tests are the deliverable, not a bug — they demonstrate the new metric catches phantom geometry that the existing coverage gate misses.

The two failing tests on this branch (strategy_l_bracket extraneous 0.080 > 0.05, strategy_plate_with_hole extraneous 0.130 > 0.05) are the canonical evidence for #72 (L-bracket phantom pillars + circular hole) and #77 (methodology_assembly hallucinated top hole). Those issues should be closed once reconstruction quality is fixed to bring extraneous_fraction ≤ 0.05, not on PR merge.

#73 (torus fitter) and #80 (OPPOSES-join BFS over-connection) are tangentially referenced — no specific fix is delivered by this PR.

CC @Mark

**Merged via fixup lane (kanban `t_pr_Mark_pc2cad_84_merge`).** - Merge commit: `d9ac95242f99f95520b72396a3c0ebf456e571fe` on `main` - Merged at: 2026-10-01T17:05:31Z - Method: default `merge` (no squash) - Branch `fixit/render-compare-extraneous-metric` retained (delete_branch_after_merge requested but Forgejo returned 405 on the closure path; head branch still listed) **Referenced issues are intentionally left open** — PR body uses `Refs:` (not `Closes:`), so they will not auto-close on merge. Per the PR description: > The failing tests are the deliverable, not a bug — they demonstrate the new metric catches phantom geometry that the existing coverage gate misses. The two failing tests on this branch (`strategy_l_bracket` extraneous 0.080 > 0.05, `strategy_plate_with_hole` extraneous 0.130 > 0.05) are the canonical evidence for #72 (L-bracket phantom pillars + circular hole) and #77 (methodology_assembly hallucinated top hole). Those issues should be closed once reconstruction quality is fixed to bring extraneous_fraction ≤ 0.05, not on PR merge. #73 (torus fitter) and #80 (OPPOSES-join BFS over-connection) are tangentially referenced — no specific fix is delivered by this PR. CC @Mark

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

blocked:needs-decision

  

blocked:preflight

Held by kilo-gsd preflight (lint dirty etc.)

  

bug

Bug report

  

in-progress

Pipeline is working on this issue

  

in-progress

Claimed by an active kilo-gsd pipeline run

  

lint-cleanup

Hold the repo until pre-flight lint is clean

  

needs-decompose

  

quickfix

  

triage-approved

  

wontfix

Will not be worked on

No labels

blocked:needs-decision

blocked:preflight

bug

in-progress

in-progress

lint-cleanup

needs-decompose

quickfix

triage-approved

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Mark/pc2cad!84

No description provided.