[dev] Stream HDF5 files with datalad-fuse's remfile backend, without a custom cache

[CodyCBakerPhD]

Stacked on #1957. This is a try-out branch: it uses the remfile-dev branch of CodyCBakerPhD/datalad-fuse, which is datalad/datalad-fuse#131 (pluggable backends, remfile) plus an init_annex option.

Changes

• Adapter: get_adapter() creates datalad-fuse's RemoteFilesystemAdapter with:
  • backends="remfile,fsspec": remfile for NWB and other HDF5-based files; fsspec for other files, and for HDF5-based files that remfile fails to open.
  • init_annex=False: git-annex is never initialized in a clone where it is not.
• Removed:
  • The coalescing fsspec cache (_coalescing_block_cache(), _read_in_small_blocks()).
  • annex_initialized(), replaced by datalad-fuse's annex_initialized attribute.
• Garbage collection pause (_GCPausedFile): now only for files read with fsspec. remfile fetches data in the reading thread, so no finalizer can run in another thread while h5py holds its lock. With fsspec that can still deadlock even with a patched h5py: in fsspec's I/O thread, hdmf 6.2.0's HDMFIO.__del__ calls is_open(), which is bool(h5py.File) and waits for h5py's lock.
• Dependencies: remfile is added to the datalad extra.
• CI: the datalad-dev job installs the remfile-dev branch. The datalad job keeps the released datalad-fuse, where streaming is skipped.
• Docs: the validate docs describe the requirements.

Why remfile

Reading the structure of a 16 MB NWB file in test_annexed_readable_reads_little:

Backend	Downloaded
remfile	0.6 MB
fsspec, default cache	64 MB

Validating the 53 NWB files of 000006 (-J 8, stock h5py 3.16):

Path	Time	Downloaded
remfile	33–61 s	340 MiB
fsspec with the coalescing cache and garbage collection paused (#1957)	16–18 s	179 MiB

Both gave the same validation results, and nothing was downloaded into the clone. So remfile trades some bandwidth for needing no custom cache, no garbage collection pause and no h5py patch for HDF5 files.

Tests

Locally, the datalad-fuse, validate and cmd_validate tests pass:

Setup	Result
remfile-dev + remfile	102 passed
remfile-dev without remfile	100 passed, 2 skipped
released datalad-fuse 0.5.1	86 passed, 16 skipped

test_annexed_readable_reads_little now requires remfile and asserts that less than a tenth of the file is downloaded.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.12%. Comparing base (e0aac05) to head (5f9d5f9).

Additional details and impacted files

@@                       Coverage Diff                       @@
##           claude/datalad-fuse-remfile    #1964      +/-   ##
===============================================================
- Coverage                        91.14%   91.12%   -0.02%     
===============================================================
  Files                               94       94              
  Lines                            14866    14765     -101     
===============================================================
- Hits                             13550    13455      -95     
+ Misses                            1316     1310       -6     

Flag	Coverage Δ
unittests	91.12% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.