UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run)

[cailmdaley]

UberSeg on #933's tile detection: on data as on sims, the SEG_VIGNET stamps come from the tile SExtractor run's own segmentation map. Contains #922 (masked pixels), which this builds on.

UberSeg on a blended galaxy (tile 202.301, NUMBER 15152, MAG_AUTO 22.7). Left: the tile VIGNET, with the brighter neighbour's footprint masked (orange). Middle: SEG_VIGNET, the SExtractor segmentation relabelled to DR6 NUMBERs. Right: the UberSeg weight. Every pixel nearer the neighbour's footprint than to the galaxy's own (outlined) gets zero weight, 33% of the stamp here, and the galaxy keeps full weight.

Why every label goes through one map. The join drops SExtractor detection 23409, and 23409 is also an unrelated DR6 NUMBER on the same tile; all 8 dropped labels on 202.301 collide like this. A partial relabel (middle) leaves those pixels reading as that other object. The full relabel (right) marks them −1, an anonymous neighbour.

Problem

UberSeg (uberseg_weight, seg_has_neighbour, Ngmix._check_central_seg_label) needs a segmentation stamp per object, on VIGNET's grid. It compares the stamp's labels with the row's NUMBER. #925 writes these stamps as the tile catalogue's SEG_VIGNET column in two ways: from the DR6 segmentation map in the catalogue converter, and from SExtractor's SEGMENTATION check image. #933 deletes the converter. Data now runs ShapePipe's own tile SExtractor, and match_catalogue joins it to DR6, so each row takes DR6's NUMBER and unpaired detections leave the catalogue. The SExtractor path now serves data and sims alike. On data, though, the check image is labelled with SExtractor's numbers, while the rows carry DR6's.

The two numberings overlap almost completely. On tile 202.301, 35,971 of the 36,014 final NUMBERs are also SExtractor NUMBERs on the tile, and all 8 dropped detections have SExtractor labels that are also final NUMBERs of other rows. A stamp that keeps any old label can therefore show a neighbour carrying the object's own number. UberSeg would then give the neighbour's pixels to the object.

Change

• Order. sextractor_runner cuts SEG_VIGNET (add_seg_vignet) right after SExtractor and before match_catalogue. The join relabels a SEG_VIGNET column it finds.
• Centring. SEG_VIGNET is cut where SExtractor cuts VIGNET. SExtractor 2.25.0 (src/analyse.c, ix=(int)(obj->mx+0.49999)) truncates the 0-based barycentre plus 0.49999, and the SIF's Debian 2.25.0+ds-3 package doesn't patch that file. vignet_centre mirrors the rule on X_IMAGE_DBL - 1 and Y_IMAGE_DBL - 1. It differs from round-to-nearest at exact halves with an odd floor and for fractional parts in (0.5, 0.50001). The float32 X_IMAGE can't resolve those cases, so SExtractor also writes the double positions, which are dropped once the stamps are cut.
• Label design: one full relabel. match_catalogue maps every stamp label through a single lookup table. A paired row's SExtractor NUMBER becomes its DR6 NUMBER. A dropped row's label, and any label not in the catalogue, becomes UNMATCHED_LABEL = −1. Sky stays 0. The pairing is one-to-one, so this keeps the property UberSeg relies on: the row's own footprint is exactly the set of pixels labelled with its NUMBER. This works because the consumers only ask own-versus-other. The dropped footprint stays a neighbour (its light is still in the image) under a label no row can have.
  This holds only if DR6 NUMBERs are unique integers that the int32 NUMBER and SEG_VIGNET columns hold exactly. The join now refuses a catalogue with a repeated, non-positive, non-integer or above-2³¹−1 NUMBER; for example, 4294967295 would narrow to −1, the dropped label. That is the only code change to match_catalogue. Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933 already had the relabel; this PR tests it against collisions and guards its premise.
  A partial relabel, which maps only the paired labels and leaves the rest, collides exactly as described above. The tests include it as a negative control.
• Memory. Under blend_handling: uberseg, tile_detect asks for 3,000 MB more (DETECT_SEG_MEM_MB), i.e. 7,000 MB. On 202.301, peak RSS is 1.53 GiB under noisefill and 2.93 GiB under uberseg. add_seg_vignet reads the 400 MB check image and adds an int32 column the size of VIGNET, which the join holds twice. Scaled to Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933's worst tile (1.84 GiB), uberseg needs about 3.5 GiB. Runtime is 170 s against 80 s, well inside 20 min.
• Merge resolutions. The converter, its runner, tests and config_tile_Uc.ini stay deleted. pipeline/config.py takes Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933's stricter ${VAR:-default} expansion and keeps UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925's getexpandedboolean. tile.smk keeps Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933's single tile_detect, and detect_env adds blend_env("tile_detect") to SP_MATCH_CATALOGUE. astra.yaml keeps catalogue_neighbour_marking deleted and drops the converter from the blend and defect texts. Some masked-pixels tests in test_ngmix_defect_fill.py built tile VIGNETs with the converter. They now mark stamps as SExtractor does (cut_stamps) on the same DR6 patch. UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925's make_cat exclusion test moves to the SExtractor tests.
• Params pins, regenerated with --update-params-pin (SIF plus host snakemake 9.23.1). params_pin_noisefill.json is byte-identical to Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933's params_pin.json, so blend_handling: noisefill plans exactly Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933's chain. The uberseg default moves tile_detect and tile_ngmix only.

Verification

• Label tests (tests/module/test_sextractor_seg_vignet.py) go through sextractor_runner with SExtractor's outputs stubbed. They use a crowd in which each paired row's DR6 NUMBER is another object's SExtractor NUMBER, one of them the dropped detection's. Each own footprint carries the final NUMBER and nothing else does. The dropped footprint is −1. uberseg_weight and seg_has_neighbour give the same answers as on the SExtractor-labelled stamps. Without a join (sims), the stamps are the check image's. Each of these injected bugs fails the tests: the join running before add_seg_vignet, relabelling only paired labels, dropping the NUMBER guard or its type and range checks, and centring with np.rint.
• Centring tests check vignet_centre against hand-written centres: exact halves (31.5 → 30, 32.5 → 31), fractions just below and above 0.50001, and a negative 0-based position. A scene test cuts its expected stamps by plain slicing, not by cut_stamps, at those hand-written centres. It includes objects where np.rint of the double positions is wrong, and one where SExtractor's rule on the float32 positions is wrong.
• Boundary tests for the NUMBER guard: repeats, 0, −3, 5.5, 5.0, 2³¹ and 2³²−1 are refused, and 2³¹−1 is accepted.
• One real tile, DR6 202.301, staged from /project/def-mjhudson/unions-wl/ into /scratch/cdaley/uberseg933/tile/. The real sextractor_runner ran with the committed config_tile_Sx.ini, SP_SEG_VIGNET=True and SP_MATCH_CATALOGUE set, and post-processing off (allocation 23087393). Results:
  • Of 202.301's 36,022 detections, none falls where SExtractor's rule and np.rint disagree, and none has an exact half (checked by a separate SExtractor run that writes X/Y_IMAGE_DBL, allocation 23094126). The numbers below were measured before the centring fix, so they hold unchanged.
  • The join paired 36,014 rows, dropped 8 and left 51 DR6-only objects.
  • The central pixel carries the row's final NUMBER in 36,013 of 36,014 stamps. The exception is a 24.2 mag object whose barycentre lies on sky (centre label 0); its own label is in the stamp.
  • No stamp has a pixel outside the row's own SExtractor footprint carrying its NUMBER.
  • Alignment: the check script finds VIGNET's own grid from the pixels (VIGNET = image − background) for 35,857 objects; 45 of these lie off the rounded X/Y_IMAGE centre (half-pixel ties). On that grid, SEG_VIGNET equals the relabelled check image for all 35,857. For the other 157 objects, VIGNET holds a constant value on some of the object's own pixels, so the grid can't be found from the pixels. For those, SEG_VIGNET equals the relabelled cut at the rounded centre.
  • VIGNET's −1e30 marks agree with SEG_VIGNET's foreign labels to 99.2% both ways.
  • A neighbour appears in 70.1% of stamps, and UberSeg zero-weights 26.8% of stamp pixels on average (dilation 0 or 1). These match UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925's DR6 converter numbers (70%, 27%).
• Full suite (Nibi allocation 23136112, at a45b70e, develop e3ef252 merged in): 973 passed, 4 skipped; tests/workflow: 22 passed.

Merge notes

• Retargeting UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925 onto Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933 brings feat/masked-pixels's 18 commits into the diff, as they are already in UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925.

• git merge-tree against current heads. develop is 615e73f (Compress the merged campaign HDF5 with lzf #937, Read the tile WCS header log lock-free in make_post_process #938 and make_cat: write the final catalogue once per save stage, node-local #939 merged).

  branch	result	the same conflicts on
  develop	clean	–
  Centre the PSF-star box on a KDE peak of the FWHM preselection #934 fix/stable-fwhm-mode	clean	–
  Wire the UNIONS external healsparse masks into the workflow (MASK_<flag value>_<name> columns) #886 feat/wire-external-masks	params_pin.json, workflow/config.yaml	Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933
  Exposure-level HealSparse maps: instrument defects and exposure count #887 feat/instrument-defect-map	params_pin.json, workflow/README.md, workflow/config.yaml	Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933
  Retire in-code catalogue versioning and the pre-Snakemake run layers #880 feat/retire-versioning	docs/source/pipeline_canfar.md	Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933
  Drop the unused tile weight vignets (campaign boundary) #932 chore/drop-unused-weight-vignets	tests/unit/test_config_parse.py, params_pin.json	Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933
  Pixel scale from the WCS: ngmix prior and PSF star preselection #927 fix/pixel-scale-from-wcs	ngmix_package/__init__.py, ngmix.py, ngmix_runner.py, test_ngmix.py, test_ngmix_uberseg.py	UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925 (masked-pixels)

  The port adds no new conflict.

• Campaign boundary: the uberseg default changes tile_detect and tile_ngmix prologues relative to Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933. A campaign that ran on Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue #933 resumes unchanged only with blend_handling: noisefill.

Out of scope

• A full UberSeg campaign through ngmix, and the noisefill-vs-UberSeg comparison that decides the default (A/B validation of off-by-default shape-measurement options #813).
• match_catalogue still holds the catalogue twice. Streaming the join would cut tile_detect's peak memory under both blend handlings.
• The tile run had post-processing off, which adds per-epoch HDUs and touches no LDAC_OBJECTS column, so it doesn't measure the post-processing's memory or time with the larger catalogue.

Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code

[cailmdaley]

Yikes, is there any way to use uberseg without actually doubling the disk and memory?

[cailmdaley]

Any way to use UberSeg without doubling disk and memory? Yes. It doesn't double anything today, and two small changes bring its overhead close to zero. Measured on the smk-ab925 A/B (same 25 tiles, code 4570a907).

What it costs now (measured)

• Final products: nothing. make_cat drops SEG_VIGNET. final_cat_smk-ab925-us.hdf5 and -nf.hdf5 are both exactly 953,220,227 B.
• Scratch: SEG_VIGNET is int32 51×51, the same bytes as VIGNET: ~323 MiB per tile (calculated as 32.6K rows × 2601 × 4), living in the tile sexcat until the tile is cleaned. For 330 tiles that's ≤104 GiB, ~3.6% of p3a's 2.8 TiB peak.
• tile_shape memory: +4.13 GB per tile (≈ +10%; median MaxRSS 40.1 vs 35.9 GB), +245 MiB per ngmix chunk (1.51 vs 1.24 GB).

Why memory grows 16×, not 1×

TileCat.get_data (ngmix.py ~l.556) opens the whole tile catalogue through FITSCatalogue, which has memmap=False by default (file_io.py l.441). With 16 chunks, each chunk process loads the full VIGNET and SEG_VIGNET columns. So both stamp columns are held 16 times per tile. This hits noisefill too: VIGNET alone is ~323 MiB × 16 ≈ 5 GB of duplicated stamps per tile_shape job.

Fixes, cheapest first

1. Load only the chunk's rows, using memmap=True or a row-range read. This cuts the uberseg overhead to ~1/16, and it removes ~15/16 of the VIGNET memory that noisefill already pays. It applies to every campaign whatever blend_handling says, and it's the bigger win for tile_shape's memory request.
2. Store less in the seg stamp. uberseg_weight, seg_has_neighbour and _check_central_seg_label only ever compare the stamp against 0 and the object's own NUMBER, so a neighbour's identity is never used.
   • A uint8 3-state stamp (0 sky, 1 own, 2 other) is 1/4 the size.
   • A bitpacked own-footprint mask is ~10 MiB per tile, with "other" taken from VIGNET's −1e30 markers. That pixel alignment is not checked.
3. Not worth it: cutting seg stamps from the seg check image at shape time. That means a ~400 MB per-tile image to stage, and it brings back the second file and the grid-alignment problem that UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) #925's single-file design removed.

Under #933

On data, #925's SExtractor-mode path is the live one. config_tile_Sx.ini already writes CHECKIMAGE = BACKGROUND, SEGMENTATION, so it adds no new file, only a transient copy inside tile_detect.

Unverified: why the per-chunk delta (245 MiB) sits below the column size (323 MiB). The likely cause is the Snakemake benchmark's sampling.

Claude Opus 5.5 on behalf of Cail, from the nibi Surveyor's measurements