Skip to content

Replace the ctypes statx call in the filesystem row provider #1494

Open

Description

@KonstantinMirin

Problem

The filesystem provider derives an object's version as ino:birthtimeNs:mtimeNs:size so that it equals the JS SDK's version string. Python's os.stat does not expose birth time on Linux, so the module declares the Linux statx struct by hand through ctypes and calls libc.statx directly. _put then stamps a random sub-millisecond mtime on each file so that two writes inside one coarse tick differ. The module itself says this provider exists for "the test and single-host filesystem" case. A hand-declared kernel struct is a maintenance and portability cost (field layout, _AT_FDCWD, the STATX_* mask, and glibc symbol availability) for a provider that is not meant for production storage.

Evidence

All references are to main at 47084e83a, src/adcp/reporting/ledger/row_storage_object.py unless stated.

  • Lines 414-465: _statx_birth_time_ns defines Timespec and Statx as ctypes.Structure subclasses (fields stx_mask through stx_btime, padded with ("_rest", ctypes.c_uint8 * 128)) and calls libc.statx(_AT_FDCWD, os.fsencode(path), 0, _STATX_BASIC_STATS_AND_BTIME, ctypes.byref(buffer)).

  • Lines 417-419: "only the test and single-host filesystem provider needs it, and a Python build without _ctypes must still import the ledger."

  • Lines 467-485: _birth_time_ns uses statx on Linux and falls back to st_birthtime_ns, st_birthtime, or 0.

  • Lines 488-490:

    def _filesystem_version(path: str) -> str:
        stat = os.stat(path)
        return f"{stat.st_ino}:{_birth_time_ns(path, stat)}:{stat.st_mtime_ns}:{stat.st_size}"
  • Lines 547-550, in _put: "stamp a random sub-millisecond mtime so an identical re-creation inside one coarse tick still differs", stamp = (int(time.time() * 1000) * 1_000_000) + secrets.randbelow(1000) * 1000.

  • Lines 605-622: the new stat method (from feat(reporting)!: align row storage with AdCP stored names and the period date rule #1485) depends on the same version string.

  • JS SDK, adcp-client main at 7c7f730f4, src/lib/reporting/ledger/row-storage-object.ts:290-291: `${stat.ino}:${stat.birthtimeNs}:${stat.mtimeNs}:${stat.size}`.

  • The interop test (test(reporting): prove JS and Python chunks verify in one database and bucket #1470) reads filesystem objects written by the other SDK by their recorded version, which is what ties the two strings together.

Suggested fix

Pick one of the following:

  • Write the version as a UUID into a sidecar file (or into an extended attribute where available) at _put time and read it back in get, stat, and delete. Both SDKs then read a stored value instead of deriving one from inode metadata, and the random mtime stamp goes away. This needs the same change in the JS provider.
  • Keep os.stat fields only (ino:mtimeNs:size, with the random stamp for uniqueness) and drop cross-SDK parity for the filesystem provider: regenerate the interop fixtures and let the interop test assert parity on the database rows and the chunk bytes, not on the filesystem version string.

Either removes the ctypes code. The second is smaller; the first keeps the interop test as it is.

Introduced in #1467.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpythonPull requests that update python code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions