Digitize endpoint loses semantic content on real photos (photo-to-embroidery not viable) #101

New issue

Closed

opened 2026-07-29 15:58:50 +00:00 by Mark · 0 comments

Mark commented

2026-07-29 15:58:50 +00:00

Owner

Problem

Feeding a real photograph through POST /v1/digitize produces a geometric/color segmentation that does not preserve recognizable content from the source.

Repro

curl -s -X POST http://localhost:8080/v1/digitize \
  -F "image=@/path/to/photo.jpg;type=image/jpeg" \
  -F 'settings={"max_colors":8,"min_stitch_length_mm":1.5}' \
  -o digitize.json

POST the resulting stitchObjects through /v1/export to produce a PES file.

Observed

Using an 800x600 photo of a dimly-lit floor (a person's foot on lower-left, small orange crumb in middle):

Output: 33 distinct gray/brown thread colors even when max_colors=4 is requested. The output boundary regions form an abstract Voronoi-like decomposition of the photo's value gradient, not a stitched rendition of the source scene.
168,533 stitches across 66 stitchObjects; final PES 1.1 MB.
Visual render of the PES output: no foot, no crumb, no recognizable objects. Only irregular gray segmented regions.

Pre-flattening the input to 4 quantized colors via PIL (Image.quantize(colors=4)) before upload improves bounds (5 distinct colors in output instead of 33) but still produces abstract segmentation, no recognizable scene.

The same pipeline works correctly on already-stylized graphic inputs (e.g. a PNG of a red square, blue hexagon, green triangle, yellow circle on white) — all four shapes are emitted as correct dense tatami fills with the right colors.

Validation already done

head -c 8 /tmp/out.pes -> #PES0001 (correct Brother PES magic)
pyembroidery.read() round-trip -> 21,776 stitches read back on the geometric test, matches HTTP X-Stitch-Count header
File is a valid PES (any conforming viewer would accept it)
Color count and dimensions match across the geometric test

So the PES file itself is fine. The bug is purely digitization quality on photographic inputs — the pipeline has no concept of semantic regions, only of color-region adjacency.

Root cause hypothesis

The current pipeline (per element_classifier + physical_optimizer in this repo) takes already-quantized inputs and treats each color cluster as a region. On a photograph that already has thousands of distinct pixel values, the region detector over-segments into fragments that track tiny color shifts in the photo rather than coherent shapes. The existing max_colors cap is honored by PhysicalOptimizer (verified by tests/test_settings_threading.py::test_physical_optimizer_honors_settings_max_colors) but appears not to be applied to the upstream region detection that establishes the 66-30 split before the optimizer runs.

Proposed fix

Add pre-digitize color quantization (k-means or median-cut) that enforces max_colors BEFORE the region detector classifies colors. This would:

Honor the documented max_colors setting at the right layer
Merge 33 close gray shades into 4-8 distinct shapes
Group photo into coherent semantic regions before per-region stitching

Workaround for users

Pre-flatten with PIL before upload:

from PIL import Image
img = Image.open("photo.jpg").convert("RGB")
img.quantize(colors=4).save("flat.jpg", "JPEG")

## Problem Feeding a real photograph through `POST /v1/digitize` produces a geometric/color segmentation that does not preserve recognizable content from the source. ## Repro ```bash curl -s -X POST http://localhost:8080/v1/digitize \ -F "image=@/path/to/photo.jpg;type=image/jpeg" \ -F 'settings={"max_colors":8,"min_stitch_length_mm":1.5}' \ -o digitize.json ``` POST the resulting `stitchObjects` through `/v1/export` to produce a PES file. ## Observed Using an 800x600 photo of a dimly-lit floor (a person's foot on lower-left, small orange crumb in middle): - Output: 33 distinct gray/brown thread colors even when `max_colors=4` is requested. The output boundary regions form an abstract Voronoi-like decomposition of the photo's value gradient, not a stitched rendition of the source scene. - 168,533 stitches across 66 stitchObjects; final PES 1.1 MB. - Visual render of the PES output: no foot, no crumb, no recognizable objects. Only irregular gray segmented regions. Pre-flattening the input to 4 quantized colors via PIL (`Image.quantize(colors=4)`) before upload improves bounds (5 distinct colors in output instead of 33) but still produces abstract segmentation, no recognizable scene. The same pipeline works correctly on **already-stylized graphic inputs** (e.g. a PNG of a red square, blue hexagon, green triangle, yellow circle on white) — all four shapes are emitted as correct dense tatami fills with the right colors. ## Validation already done - `head -c 8 /tmp/out.pes` -> `#PES0001` (correct Brother PES magic) - `pyembroidery.read()` round-trip -> 21,776 stitches read back on the geometric test, matches HTTP `X-Stitch-Count` header - File is a valid PES (any conforming viewer would accept it) - Color count and dimensions match across the geometric test So the PES file itself is fine. The bug is purely **digitization quality on photographic inputs** — the pipeline has no concept of semantic regions, only of color-region adjacency. ## Root cause hypothesis The current pipeline (per `element_classifier` + `physical_optimizer` in this repo) takes already-quantized inputs and treats each color cluster as a region. On a photograph that already has thousands of distinct pixel values, the region detector over-segments into fragments that track tiny color shifts in the photo rather than coherent shapes. The existing `max_colors` cap is honored by `PhysicalOptimizer` (verified by `tests/test_settings_threading.py::test_physical_optimizer_honors_settings_max_colors`) but appears not to be applied to the upstream region detection that establishes the 66-30 split before the optimizer runs. ## Proposed fix Add pre-digitize color quantization (k-means or median-cut) that enforces `max_colors` BEFORE the region detector classifies colors. This would: - Honor the documented `max_colors` setting at the right layer - Merge 33 close gray shades into 4-8 distinct shapes - Group photo into coherent semantic regions before per-region stitching ## Workaround for users Pre-flatten with PIL before upload: ```python from PIL import Image img = Image.open("photo.jpg").convert("RGB") img.quantize(colors=4).save("flat.jpg", "JPEG") ```