[dev] Use datalad-fuse's remfile backend and all other dev branches

[CodyCBakerPhD]

Stacked on #1956. This tries the development version of datalad-fuse from datalad/datalad-fuse#131, which adds pluggable backends, including remfile for HDF5-based files.

Changes

• Adapter choice: get_adapter() uses datalad_fuse.adapter.RemoteFilesystemAdapter (dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131) when it is available, and falls back to the released datalad_fuse.fsspec.FsspecAdapter (0.5.1) otherwise.
  • With dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131 and remfile installed, NWB and other HDF5 files are read through remfile; other files still go through fsspec.
  • The backends can be configured with DataLad's datalad.fusefs.backends setting.
• S3 export fallback: get_annexed_readable() also tries the S3 exporttree URLs from dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131 when git-annex lists no URL, for older datasets that lack versioned S3 URLs.
• Docs: a note in the validate docs.
• Tests: test_annexed_readable_remfile checks that an NWB file is opened through datalad-fuse's RemfileWrapper and read with h5py over the local HTTP range server. It is skipped unless both dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131 and remfile are installed.
• CI: a new datalad-dev job installs git-annex, remfile and datalad-fuse from refs/pull/131/head. The existing datalad job keeps testing the PyPI release.

Locally, the datalad-fuse and validate stream tests pass with both datalad-fuse 0.5.1 (where the remfile test is skipped) and #131 plus remfile.

Once #131 is released, the CI install can switch to that release and the datalad extra's lower bound can be raised.

Follow-up branch: dev datalad-fuse and h5py instead of the workarounds

Branch claude/use-dev-datalad-fuse-h5py (4484f3a) builds on this PR. It has no PR of its own yet. It moves the low-level streaming workarounds out of dandi-cli and into the libraries where they belong, using development branches of those libraries.

Removed from dandi/support/datalad_fuse.py	Replaced by
_coalescing_block_cache() and _read_in_small_blocks()	datalad-fuse's fsspec_options (block_size, cache_type, cache_options) and its "coalescing-blockcache": CodyCBakerPhD/datalad-fuse#1, on top of datalad/datalad-fuse#131
annex_initialized()	datalad-fuse's init_annex=False and annex_initialized (same PR)
_GCPausedFile (garbage collection paused while streamed files are open)	h5py no longer waits for its global lock when deallocating while another thread holds it: CodyCBakerPhD/h5py#1

The fsspec counterpart is CodyCBakerPhD/filesystem_spec#1. It makes fsspec's own "blockcache" fetch each run of missing blocks with one request, so datalad-fuse could drop its own cache. That branch is not yet installed or used by the follow-up branch.

CI on the follow-up branch: the datalad and datalad-dev jobs install the datalad-fuse and h5py forks from git; datalad-dev also installs remfile. Building h5py from source needs the HDF5 headers, which the jobs install first (apt-get install libhdf5-dev, HDF5_DIR=/usr/lib/x86_64-linux-gnu/hdf5/serial).

Results so far (local only):

• The datalad-fuse, validate and cmd_validate tests pass: 102 passed, with and without remfile.
• Streaming the 53 NWB files of 000006 with the garbage-collection workaround removed did not deadlock:
  • -J 1: 87 s.
  • -J 8: 30 s, with the same results.
  • Nothing was downloaded into the clone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.59459% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.15%. Comparing base (a65f6ee) to head (bb56977).

Files with missing lines	Patch %	Lines
dandi/support/datalad_fuse.py	87.50%	2 Missing ⚠️

Additional details and impacted files

@@                             Coverage Diff                             @@
##           claude/validate-stream-via-datalad-fuse    #1957      +/-   ##
===========================================================================
+ Coverage                                    91.13%   91.15%   +0.01%     
===========================================================================
  Files                                           94       94              
  Lines                                        14806    14837      +31     
===========================================================================
+ Hits                                         13494    13525      +31     
  Misses                                        1312     1312              

Flag	Coverage Δ
unittests	91.15% <94.59%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.