Skip to content

feat(image-studio): background removal on the CPU - #457

Merged
lightningpixel merged 3 commits into
devfrom
feat/background-removal
Nov 16, 2026
Merged

lightningpixel merged 3 commits into
devfrom
feat/background-removal

Conversation

@lightningpixel

Copy link
Copy Markdown
Owner

Closes #421

What it does

Image Studio gets a Remove background tool. The cut-out is added to the history and shown on the checkerboard; the original stays one click away in the history strip. A Mask toggle shows the matte, and an edge threshold hardens soft edges. Re-running always starts from the original, not from a previous cut-out.

It needs no extension and no GPU:

  • Model: BiRefNet-lite (MIT, ONNX fp32, 224 MB) from onnx-community/BiRefNet_lite-ONNX, pinned to a revision and checked by size and SHA-256 after download.
  • Runtime: onnxruntime 1.23.2 on the CPU, pip-installed into the backend's own environment on first use, not added to requirements.txt (a requirements change forces every existing user through a full Python reinstall). 1.23.2 is the last release with Intel Mac wheels.
  • First use shows a one-time setup card (~249 MB) with two-step progress (runtime, then model) and Retry on error.

Backend

An image_ops registry modelled on mesh_ops, under /image:

  • GET /image/ops
  • GET /image/op/{op}/status → { ready, missing, download_bytes, preparing }
  • POST /image/op/{op}/prepare → SSE progress (same format as the LLM downloads); idempotent, a second call joins the running one
  • POST /image/op/{op} { path, params } → { path, mask_path }; 409 not_ready until prepared; inference runs off the event loop

Outputs go to <workspace>/Images/ and never overwrite. onnxruntime is never imported at module load.

Frontend

  • ImageOpsToolbar + setup card, client in imageOpsApi.ts, pure helpers in imageOps.ts
  • EditPanel's param fields moved to a shared ParamFields component (no visual change)
  • consumeSse accepts a request init so the setup stream is a POST
  • The before/after slider was removed: the canvas shows the selected image only

Testing

  • npm test passes (Python 512, JS 352 + 237, 0 failures); lint clean; tsc unchanged from dev
  • 52 new backend tests run without network or onnxruntime (fake session); the real-model test skips when onnxruntime is missing
  • Real end-to-end in a clean Python 3.11 venv: install + download in ~35 s, 8–16 s per image on CPU, clean cut-outs on cluttered photos
  • Tested in the app by the maintainer
  • Not verified: the on-demand install on macOS / Linux, behind a proxy, and from the packaged app's Python

Image Studio gets a "Remove background" tool. The cut-out lands in the
history with the original as its source, so the before/after slider
compares them; a Mask toggle shows the matte and an edge threshold
hardens soft edges.

It needs no extension and no GPU: BiRefNet-lite (MIT, ONNX, pinned
revision, checked by size and SHA-256) runs with onnxruntime on the CPU.
Both are fetched on first use behind a one-time setup card with
progress (~249 MB): onnxruntime 1.23.2 is pip-installed into the
backend's own environment on demand rather than added to
requirements.txt, which would force every user through a full Python
reinstall; 1.23.2 is the last release with Intel Mac wheels.

Backend: an image_ops registry modelled on mesh_ops, under /image:
GET /image/ops, GET /image/op/{op}/status, POST /image/op/{op}/prepare
(SSE, LLM download format) and POST /image/op/{op} (409 until
prepared). Outputs go to <workspace>/Images/ and never overwrite.

EditPanel's param fields move to a shared ParamFields component, used
by both panels.

Closes #421
A GET could start the one-time install from any page (an image tag is
enough); the renderer already uses POST.
…e/after slider

The canvas shows the cut-out itself; the original stays one click away
in the history strip.
@lightningpixel
lightningpixel merged commit 14c0c1a into dev Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant