Repository navigation
Replace the ctypes statx call in the filesystem row provider #1494
Description
Problem
The filesystem provider derives an object's version as ino:birthtimeNs:mtimeNs:size so that it equals the JS SDK's version string. Python's os.stat does not expose birth time on Linux, so the module declares the Linux statx struct by hand through ctypes and calls libc.statx directly. _put then stamps a random sub-millisecond mtime on each file so that two writes inside one coarse tick differ. The module itself says this provider exists for "the test and single-host filesystem" case. A hand-declared kernel struct is a maintenance and portability cost (field layout, _AT_FDCWD, the STATX_* mask, and glibc symbol availability) for a provider that is not meant for production storage.
Evidence
All references are to main at 47084e83a, src/adcp/reporting/ledger/row_storage_object.py unless stated.
-
Lines 414-465:
_statx_birth_time_nsdefinesTimespecandStatxasctypes.Structuresubclasses (fieldsstx_maskthroughstx_btime, padded with("_rest", ctypes.c_uint8 * 128)) and callslibc.statx(_AT_FDCWD, os.fsencode(path), 0, _STATX_BASIC_STATS_AND_BTIME, ctypes.byref(buffer)). -
Lines 417-419: "only the test and single-host filesystem provider needs it, and a Python build without
_ctypesmust still import the ledger." -
Lines 467-485:
_birth_time_nsusesstatxon Linux and falls back tost_birthtime_ns,st_birthtime, or0. -
Lines 488-490:
def _filesystem_version(path: str) -> str: stat = os.stat(path) return f"{stat.st_ino}:{_birth_time_ns(path, stat)}:{stat.st_mtime_ns}:{stat.st_size}"
-
Lines 547-550, in
_put: "stamp a random sub-millisecond mtime so an identical re-creation inside one coarse tick still differs",stamp = (int(time.time() * 1000) * 1_000_000) + secrets.randbelow(1000) * 1000. -
Lines 605-622: the new
statmethod (from feat(reporting)!: align row storage with AdCP stored names and the period date rule #1485) depends on the same version string. -
JS SDK,
adcp-clientmainat7c7f730f4,src/lib/reporting/ledger/row-storage-object.ts:290-291:`${stat.ino}:${stat.birthtimeNs}:${stat.mtimeNs}:${stat.size}`. -
The interop test (test(reporting): prove JS and Python chunks verify in one database and bucket #1470) reads filesystem objects written by the other SDK by their recorded version, which is what ties the two strings together.
Suggested fix
Pick one of the following:
- Write the version as a UUID into a sidecar file (or into an extended attribute where available) at
_puttime and read it back inget,stat, anddelete. Both SDKs then read a stored value instead of deriving one from inode metadata, and the random mtime stamp goes away. This needs the same change in the JS provider. - Keep
os.statfields only (ino:mtimeNs:size, with the random stamp for uniqueness) and drop cross-SDK parity for the filesystem provider: regenerate the interop fixtures and let the interop test assert parity on the database rows and the chunk bytes, not on the filesystem version string.
Either removes the ctypes code. The second is smaller; the first keeps the interop test as it is.
Introduced in #1467.
Activity