Read the tile WCS header log lock-free in make_post_process

[cailmdaley]

Problem

tile_detect's post-processing (make_post_process in sextractor_script.py) builds the per-epoch EXP_NAME/CCD_N HDUs and N_EPOCH by looking up the tile's header log (log_exp_headers<tile>.sqlite, written by merge_headers) once per exposure × CCD:

for idx_j in range(n_hdu):            # ~40 CCDs
    if exp not in f_wcs: ...           # one sqlite transaction
    w = f_wcs[exp][idx_j]["WCS"]       # another, plus unpickling all ~40 WCS of the exposure

That is ~80 transactions per exposure, and each takes and releases a POSIX lock on the file. On NFS /scratch those locks are slow:

• On a Nibi login node, 440 keyed reads with sqlite ?mode=ro (locking) took 18.4 s, 80–250 ms per query on a busy store. With ?immutable=1 (no locking) the same reads took 0.25 s. Some freshly written stores took 27–30 s per locked query, or never returned.
• In the smk-p3a campaign, post-processing took a median 283 s, p90 704 s and max 1306 s per tile. Four tiles hit the time limit twice, at 30 min and then at 1 h. py-spy showed the main thread in sqlitedict.select ← __getitem__/__contains__ ← make_post_process, and the SqliteDict worker thread in D state on rpc_wait_bit_killable.

Reading each exposure once would still take one lock per exposure, which costs ~30 s on a bad file, so the read has to avoid locks entirely. The loop also re-unpickles every exposure's WCS list once per CCD. That cost is separate from locking and is not small: see the numbers below.

Change

• New shapepipe.pipeline.sqlite_store:
  • ImmutableSqliteDict is a read-only Mapping over a SqliteDict file, opened with file:<path>?immutable=1. Its keys, rowid order and values (decoded with sqlitedict's own decode) match what SqliteDict returns for stores written with SqliteDict's default encoding, which is how every ShapePipe store is written.
  • read_sqlitedict(path) loads a whole store into a dict.
  • A missing file raises FileNotFoundError. SqliteDict would silently create an empty store instead.
• make_post_process loads the header log once with read_sqlitedict, then indexes the dict. The "exposure missing from header file" check now runs once per exposure, outside the CCD loop. Output is unchanged.

immutable=1 is only correct when no process writes the file during the read. sqlite neither sees concurrent changes nor replays a hot journal. This holds here because merge_headers writes and closes the header log in an earlier rule. As a guard against a writer that is still running or was killed mid-transaction, ImmutableSqliteDict refuses to open a store with a -journal or -wal file next to it and raises RuntimeError. Without the guard, a reviewer reproduced immutable=1 returning 49 of 50 rows with uncommitted values, where SqliteDict rolled the journal back.

Verification

• New tests/module/test_sqlite_store.py:
  • On a header log written by merge_headers from split_exp-style WCS .npy files, read_sqlitedict returns the same keys, in the same order (TILE_ID first), with values whose pickles are byte-identical to SqliteDict's.
  • It also checks the Mapping semantics against SqliteDict: lookup, in, len, KeyError, and rowid order after an overwrite.
  • A missing file raises and is not created.
  • A read leaves no journal file.
  • A -journal or -wal file next to the store makes the read raise. One case uses a real SqliteDict writer killed mid-transaction: the read raises, and SqliteDict then rolls back to the committed row.
  • A path containing a space, ?, # and % reads correctly through the file: URI.
• tests/module/test_sextractor_post_process.py gains a test that an exposure missing from the header log still raises KeyError. The existing tests pass.
• Real store from a Nibi run, log_exp_headers-202-301.sqlite (9.7 MB, 6 exposures × 40 CCDs, NFS /scratch, login node):

  • read_sqlitedict matches SqliteDict(flag="r").items() exactly: same key order, byte-identical pickles.

  • Timings:

    read	time
    develop's access pattern through SqliteDict (482 queries)	60.6 s (126 ms/query)
    same access pattern, lock-free	24.6 s (repeated unpickling alone)
    read_sqlitedict	2.2 s cold, 0.7 s warm

• End to end on the real tile 202.301 (36,022 objects; Nibi job 23055352, compute node, store on NFS /scratch): develop's make_post_process took 78.6 s and this branch's 7.1 s. The output is identical across all 9 HDUs (PRIMARY, LDAC_IMHEAD, LDAC_OBJECTS, EPOCH_0–EPOCH_5), and EPOCH_*/N_EPOCH equal the campaign's own sexcat. Full suite on the branch: 835 passed, 6 skipped.

Scope

This PR only touches make_post_process, where the time is going. Other code reads SqliteDict stores in the same per-key way, and each could switch to ImmutableSqliteDict as a drop-in in a follow-up. None of these are converted here, to keep the diff away from files that open PRs are reworking:

• ngmix.Vignet (6–7 vignet stores read per object) and ngmix_runner's pre-check, which scans every key of the PSF and galaxy vignet stores.
• make_cat.save_psf_data, which does one read of galaxy_psf_cat[str(id)] per object (~35K per tile). make_cat.py is touched by Wire the UNIONS external healsparse masks into the workflow (MASK_<flag value>_<name> columns) #886/Exposure-level HealSparse maps: instrument defects and exposure count #887.
• The f_wcs reads in vignetmaker, psfex_interp and mccd_interpolation_script.

git merge-tree against #933, #925 and #886 is clean. Against #887 it conflicts only in workflow/README.md, and #887 conflicts there with develop as well. On #933, make_post_process reads the header log exactly as on develop, so this change still applies after #933 lands.

Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code