make_cat: write the final catalogue once per save stage, node-local

[cailmdaley]

Problem

tile_make_cat spends 10–20 min per tile rewriting its output FITS file. FITSCatalogue.add_col rebuilds the table HDU from every existing column plus the new one, writeto(overwrite=True)s the whole file, then closes and reopens it. make_cat adds ~243 of the final ~271 columns this way: TILE_ID/TILE_UNIQUE_ID, the ngmix block and the per-epoch PSF slot block (SaveCatalogue.process), and the MASK_* columns. So the catalogue gets written roughly 243 times.

Measured on nibi:

• py-spy on a live tile_make_cat worker: ~94% of samples in astropy writeto under SaveCatalogue.process's add_col loop.
• add_col costs 2.05 s per column on NFS /scratch and 0.06 s node-local.
• A single astropy write of the 33K × 271 table takes 19 s on NFS and 0.4 s node-local.

Change

1. FITSCatalogue.add_cols(columns) appends a dict of columns with one HDU rebuild and one write. add_col now just calls add_cols with one item. The per-column FITS type, repeat count, TDIM and string-width logic moved into _make_fits_col without changes, so both paths build the same columns.
2. make_cat writes once per save stage: once for TILE_ID + TILE_UNIQUE_ID, once per SaveCatalogue.process call (ngmix, PSF slots), and once for all MASK_* columns. That's four writes per tile instead of ~243.
3. Node-local build. make_cat_runner takes an optional WORK_DIR. The catalogue is built there, then moved to the run's output directory once it's complete. Any work file left by an earlier attempt is removed first, because save_as_fits appends to an existing file. config_tile_Mc.ini sets WORK_DIR = $SP_LOCAL/make_cat, the per-tile node-local store that tile_local() already exports for tile_make_cat. That rule's EXIT trap already removes the store. tile.smk is unchanged: the published path, the completeness check and the final_cat copy all still read $SP_RUN/output/run_sp_tile_Mc/.... No params changed, so finished tiles are not rerun.

Verification

• tests/module/test_file_io_add_cols.py: one add_cols call produces a file byte-identical to successive add_col calls (each of which writes and reopens the file). Also checks headers, column names/formats/TDIM and data HDU by HDU. Covers a plain catalogue and the SExtractor layout (table between other HDUs), with int16/int32/int64/float32/bool/string/vector/2-D columns. Also checks that a non-array column raises before anything is written, and that new_cat/new_cat_inst with hdu_no/ext_name leaves the source untouched, carries the surrounding HDUs over, and matches the add_col chain.

• tests/module/test_make_cat.py::test_make_cat_runner_work_dir_publishes_same_catalogue runs all three save stages (ngmix, per-epoch PSF slots, one MASK_EXT_PATHS band). The catalogue published through WORK_DIR is byte-identical to one built in place, WORK_DIR is left empty, and a stale work file does not leak in. If the stale-file removal is disabled, this test fails.

• Targeted module tests (test_file_io_add_cols, test_make_cat, test_make_cat_mask_ext, test_mask_query) and the config unit tests pass (72 module tests, 108 unit tests) in the shapepipe-develop-240b37e4 container.

• Real data (Nibi job 23055211): the g11 tile 181.308 catalogue (34,946 rows × 271 columns) was rebuilt from its first 28 columns, adding the other 243 once through develop's per-column add_col and once through add_cols.

  where	develop add_col × 243	add_cols
  node-local disk	60.3 s	0.34 s
  NFS /scratch	859.8 s	7.4 s

  Both outputs are byte-identical (md5 9a43e3be…, 69,382,080 B), with equal headers and column names. The script's all data equal: False line comes from np.array_equal without equal_nan on NaN-bearing columns; the identical md5 settles equality. With the node-local working file (WORK_DIR), make_cat's catalogue writes drop from ~14 min on NFS to under a second, plus one copy.

• Full suite (Nibi job 23055353, pytest tests -m "not slow" in the develop SIF): 831 passed, 4 skipped, 2 deselected.

Out of scope

• make_post_process (tile_detect) adds a single column (N_EPOCH), so batching doesn't apply there. Its cost is one rewrite of the LDAC sexcat, on top of the per-epoch save_as_fits HDU appends before it.
• _get_fits_col_type maps np.bool_ to D. Wire the UNIONS external healsparse masks into the workflow (MASK_<flag value>_<name> columns) #886 changes it to L. add_cols goes through the same function, so whichever mapping lands applies to both paths.
• TDIM for columns with two or more trailing dimensions is written in numpy order rather than FITS order, so a (n, 2, 3) column reads back as (n, 3, 2). This behaviour already exists, and add_cols keeps it so output stays identical. No current final_cat column is multi-dimensional; the g11 catalogue carries no TDIM cards.

Merge notes

git merge-tree against #886 (feat/wire-external-masks), #925 (feat/uberseg-seg-vignet) and #933 (fix/dr6-windowed-positions) is clean. Against #887 (feat/instrument-defect-map) the only conflict is in workflow/README.md, which #887 already has with develop itself; this branch doesn't touch that file. On develop, cfis_image_sims/config_tile_Mc.ini is a symlink to the cfis one, so it picks up WORK_DIR too. #886 replaces that symlink with a regular file, so after it merges, image-sims builds in place until the WORK_DIR line is added to its copy too (a one-line follow-up; output is identical either way).

Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code