Detect tiles with SExtractor on data too; take NUMBER from the UNIONS catalogue

[cailmdaley]

Closes #936. Refs #924.

Data and image sims now detect tiles the same way: ShapePipe runs SExtractor (config_tile_Sx.ini) on the tile, and on data each detection takes its NUMBER from the UNIONS DR6 catalogue (Stephen Gwyn's MegaPipe run) by a positional join. Stephen's catalogue defines which objects exist, their IDs, and the photometry for photo-z. ShapePipe's own run defines everything the shape measurement reads: windowed centroids, FLAGS_WIN, the VIGNET stamps with their neighbour marking, sizes and fluxes. Shape, photometry and photo-z catalogues still share one TILE_UNIQUE_ID, as #897 intended.

Why. Since #897, data took its sample from DR6 through a converter (read_ext_sexcat), while sims ran SExtractor. That had three problems:

• DR6 has no windowed centroids, so the converter put the isophotal barycentre into XWIN_*. That is 0.16 px from the windowed centroid at the median and 1.6 px at p99.
• The converter rebuilt SExtractor's neighbour marking from the DR6 segmentation map.
• It lacked nine SExtractor columns (Questions and uncertainties about Stephen Gwyn's official DR6 SExtractor run #924).

Running SExtractor removes all three and puts data and sims on one code path, so sim calibration describes the data.

How.

• tile_get_catalogue fetches only CFIS.<tile>.r.cat.
• In sextractor_runner, match_catalogue runs after SExtractor and before the multi-epoch post-processing. It pairs each detection with its mutual nearest DR6 object within 1 px (MATCH_RADIUS), and the detection takes that object's NUMBER. Unpaired detections leave the catalogue, and a SEG_VIGNET column, if present, is relabelled to match.
• Sims export an empty MATCH_CATALOGUE, so their command is unchanged.
• Guard. The run stops when the unpaired detections or the unpaired DR6 objects exceed both 2% of their side (MATCH_MIN_FRACTION = 0.98) and 20 objects (MATCH_TOLERATED_UNPAIRED). This catches a tile image from another release (DR5 pixels pair at about 95%) and a drifted detection configuration. It tolerates deblending differences near large objects and a few unpaired children on near-empty edge tiles.
• Single-image SExtractor. Without a detection image, sextractor_runner now calls SExtractor with one image and one weight. It used to call it in dual-image mode with the same image twice, which changed MAG_AUTO by more than 0.01 for 3.7% of objects and FLUX_RADIUS for 20%. On the 40 CCDs of exposure 2254826, the PSF-star selection is identical: the same 2,202 stars, and the same FWHM mode on every CCD.
• MEMORY_PIXSTACK is 3,000,000 in default_tile.sex. At 300,000, SExtractor truncated the pixel stack at very large objects on four of the eight test tiles (13 times around Alioth), and dropped detections behind them.
• Removed: read_ext_sexcat and its tests, config_tile_Uc.ini, the segmentation-map fetch, the runner link in tile.smk, merge_final_cat --tile-detection with SEXTRACTOR_ONLY_COLUMNS, and the decision detection.catalogue_neighbour_marking.
• Record. astra.yaml rewrites detection.tile_detection and preparation.object_position_columns, and pins the match keys and MEMORY_PIXSTACK.

Validation on eight DR6 tiles, using the real sextractor_runner and the delivered exposure headers. The tiles are three typical high-latitude ones, one at b = 18°, the A2199 cluster, Alioth's halo, and two survey-edge tiles at 96% zero weight.

• Pairs agree to 4e-5 px at the median and 7e-5 px at p99.
• At least 99.7% of detections and 99.3% of DR6 objects pair.
• DR6 objects without a detection are 0–0.7% per tile, concentrated around very large objects. At most 79 per tile fall in 20 < MAG_AUTO < 24.5. They keep their photometry but have no shape.
• On 186.307, every detection pairs, and MAG_AUTO matches DR6 to 0.01 mag for all but 0.03% of objects.
• Explained (Questions and uncertainties about Stephen Gwyn's official DR6 SExtractor run #924). On tiles with many blends, MAG_AUTO differs from DR6 by more than 0.01 mag for 2–11% of objects. SExtractor's deblender gives contested pixels to children using unseeded rand() (refine.c gatherup()), and one stream runs through the image. After the first object whose draw count differs from MegaPipe's build, deblended objects get different pixels. Positions match; deblended photometry differs by a few percent per object. No configuration setting changes this. Data and sims run the same binary, so this doesn't affect calibration.

tile	case	detections paired	DR6 objects paired	DR6-only (20–24.5)	wall	peak RSS
186.307	typical	1.0000	0.9989	45 (4)	119 s	1.75 GiB
381.259	A2199	0.9996	0.9928	288 (26)	157 s	1.68 GiB
216.292	Alioth halo	0.9972	0.9926	200 (79)	160 s	1.84 GiB
066.168	survey edge	1.0000	0.9944	5 (1)	24 s	0.29 GiB

Cost. The whole step takes 23–160 s and peaks at 1.84 GiB. SExtractor in the image is built without threads, so tile_detect asks for 1 core, 4 GB and 20 min, with one retry that doubles memory and runtime. On Nibi it bills 1 core-equivalent instead of 8. That is about 1% of tile_ngmix's CPU per tile.

Tests.

• Join: the radius bound, mutual pairing, the guard on each side (including the small-tile tolerance and a DR5-like 95% case), relabelling, and row alignment.
• The SExtractor command line in single and dual mode.
• Malformed ${ config values.
• That data's tile_detect waits for the catalogue fetch and sims export an empty path.

Each test was seen to fail under an injected bug. tests/unit, tests/module and tests/workflow: 828 passed, 1 skipped. astra validate passes. An Opus review and two GPT reviews found no blocking defects, and their fixes are folded in.

Campaign boundary. completeness.py changes SCRIPT_HASH, which is in the params of 13 rules. Resuming any campaign on develop after the merge, sims included, therefore reruns every finished unit, and tile and exposure SExtractor measurements change with single-image mode. Merge between campaigns. The tile mirror a campaign reads must be DR6, or every data tile stops at the guard.

Rebasing #925. In sextractor_runner, add_seg_vignet must run before match_catalogue, or the relabelling misses the stamps.

Validation scripts and tables: candide:/n17data/cdaley/scratch/xwin-assoc/tiles/ (table_pix3e6.md, where.png).

Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code

[cailmdaley]

is the underlying problem that the DR6 catalog doesn't include a sextractor quantity that we need it to? i need a little more exposition to understand. what are the barycenter vs windowed centroid? why is each one used where it is? does shapepipe need to use this other quantity? reimplementing a part of sextractor seems like not the right solution. we could ask the new columns be included in the DR6 catalog eventually, but what's the first-principles fix here?

[cailmdaley]

You're right. The first-principles fix is to run ShapePipe's own tile SExtractor on data, exactly as sims do, and take DR6's IDs by a positional join. I'm rewriting this PR to do that, and dropping the port.

Barycentre vs windowed centroid. The barycentre (X_IMAGE) is the flux-weighted mean over the isophotal footprint, which here means the pixels above 1σ. Every footprint pixel counts equally, so noise at the edge and asymmetric footprints pull it around. Asymmetric footprints come from faint wings or a deblender cut against a neighbour. The windowed centroid (XWIN_IMAGE) iterates a Gaussian-weighted mean whose width is set by the half-light radius, so it converges on the light peak and is close to PSF-fit precision. On tile 186.307 they differ by 0.16 px at the median and 1.6 px at p99 in the shear range (20 < mag < 24.5).

Where ShapePipe uses it. The position sets the final RA/Dec and mask queries, epoch membership, PSF interpolation and the multi-epoch stamp centres. It is also the origin of the ngmix Jacobian, where the centroid prior (σ = 1 px) is centred. At 0.16 px, only the prior is plausibly affected. My expectation is that using the barycentre would move shear very little; I haven't measured it. The real problem is that sims use SExtractor's XWIN, so data and sims would be running different centroid definitions, and sim calibration would not exactly describe the data, mostly in blends.

DR6 lacks the column, but we don't need it from DR6. #929 showed that default_tile.sex on the DR6 image reproduces the DR6 catalogue object for object. I've now tested this on 186.307:

• Membership. Joining each SExtractor row to its mutual nearest DR6 object within 1 px gives 41,201 one-to-one pairs with a p99 offset of 7e-5 px, so every SExtractor object inherits its DR6 NUMBER. SExtractor's ASSOC mode is worse: it silently drops 2 objects at the image edge.
• What we lose. 45 DR6 objects have no SExtractor row (0.11%). They are faint children that MegaPipe deblends out of 12 bright parents; 4 are in the shear range. They would have photometry and no shape.
• Cost. SExtractor takes 34 s and 0.5 GB on 1 thread. The converter plus this port takes 47 s and 2.6 GB.
• The port is not exact. Against native XWIN it matches at the median (3e-5 px), but 1.4% of objects are more than 0.1 px off, and its flags miss most of them. So it would leave sims and data on slightly different centroids anyway.
• Requirement. The tile image must be the DR6 image. On the DR5 pixels, only 96% of objects join.

What the rewrite does. Data runs tile_detect as sextractor_runner in both modes. A short post-step joins each object to DR6 and renumbers NUMBER, and the SEG_VIGNET labels for UberSeg get mapped the same way. The converter, the windowed-centroid port, the DR6 segmentation-map fetch and the DR6-only column handling (#924) all go, and native FLAGS_WIN and the full SExtractor column set come back.

Asking for the new XWIN columns in a future DR6 release would also work, but it wouldn't help us: running SExtractor ourselves is cheap and keeps data and sims on one code path.

Scripts and outputs: candide:/n17data/cdaley/scratch/xwin-assoc/.

Claude Opus 5.5 on behalf of Cail

[cailmdaley]

The rewrite is in, and the description is updated. One correction to my comment above: default_tile.sex reproduces DR6's detections and positions exactly, but sextractor_runner calls SExtractor in dual-image mode, which shifts MAG_AUTO and FLUX_RADIUS against DR6 for about 20% of objects (≥0.01 mag for 3.7%). Data and sims stay consistent with each other. The fix to a single-image call is #936.

Claude Opus 5.5 on behalf of Cail