Conversation
Reverse-engineered and implemented against a live standalone ESXi 8.0.3 host (no vCenter), closing most of the gap versus the proprietary VDDK: - Direct ESXi (no vCenter) connectivity: nfc_service() previously hardcoded the NfcService moref as "nfcService" (vCenter's name), which fails on bare ESXi (moref is "ha-nfc-service" there). Now resolved dynamically via RetrieveInternalContent, same as VDDK itself does. Also fixes connect_authd() for tickets that omit `host` (implicit on a direct-ESXi ticket). - VixDiskLib_GetInfo: capacity and physical geometry come free from the OPEN_FILE reply (offsets already in the wire frame). biosGeo, adapterType, and uuid are fetched via DDB_GET, matching real VDDK's behavior and cost exactly. - DDB_GET (VMDK descriptor lookups): generic key/value NFC message, values are ASCII text on the wire (not binary), matching how a VMDK descriptor's DDB section is stored. - VixDiskLib_QueryAllocatedBlocks: allocated-block bitmap query. Verified against a live disk to exactly match native VDDK's output, including two non-obvious wire details: a field-order swap that's invisible in a zero-offset capture, and 4-byte bitmap padding that only shows up for small chunk counts. - Changed Block Tracking: turned out to need no NFC work at all -- VirtualMachine.QueryChangedDiskAreas is public VIM API. Added thin wrappers (enable_change_tracking / disk_change_id / query_changed_disk_areas) and documented real-world characteristics (extent granularity, wildcard changeId semantics) from live testing. - Investigated NFC_DELTA_DISK: found it's an optional VMFS-only VDDK client optimization (per `strings` on libvixDiskLib.so), not a correctness requirement -- reading, writing, and querying allocated blocks on an actual snapshot delta file already work with the existing NFC_DISK-only implementation. Documented a real gotcha found along the way: querying allocated blocks on the same still-open handle a write just went through can see stale data. Adds unit tests (bitmap decode/merge, DDB_GET wire format, CBT dataclass conversion, validation errors) and integration tests (GetInfo, QueryAllocatedBlocks, CBT full cycle, delta-disk read/write/ query) validated against a live ESXi 8.0.3 lab. Full protocol details and the reverse-engineering process are in docs/nfc_auth.md, docs/nfc_open.md, docs/nfc_read.md, docs/cbt.md, and docs/reverse_engineering_procedure.md.
This was referenced Sep 19, 2026
Author
|
Splitting this into 4 focused, independently-reviewable PRs instead:
Each was verified to build, pass its own tests, and (where relevant) validate live against the test lab standalone against current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reverse-engineered and implemented against a live standalone ESXi 8.0.3 host (no vCenter), closing most of the gap versus the proprietary VDDK:
nfc_service()previously hardcoded theNfcServicemoref as"nfcService"(vCenter's name), which fails outright on bare ESXi (moref is"ha-nfc-service"there) withvmodl.fault.ManagedObjectNotFound. Now resolved dynamically viaRetrieveInternalContent, the same way real VDDK does it. Also fixesconnect_authd()for tickets that omithost(implicit on a direct-ESXi ticket — the authd endpoint is the host you're already logged into).VixDiskLib_GetInfo— capacity and physical geometry come free from theOPEN_FILEreply (already-received wire bytes, no extra round trip).biosGeo/adapterType/uuidare fetched viaDDB_GET, matching real VDDK's behavior and cost exactly.DDB_GET— generic VMDK descriptor key/value lookup over NFC. Values are ASCII text on the wire (not binary), matching how a VMDK descriptor's DDB section is actually stored.VixDiskLib_QueryAllocatedBlocks— allocated-block bitmap query. Verified against a live disk to exactly match native VDDK's own output, including two non-obvious wire details found the hard way: a field-order swap invisible in astartSector=0capture, and 4-byte bitmap padding that only shows up for small chunk counts.VirtualMachine.QueryChangedDiskAreasis public VIM API. Added thin wrappers (enable_change_tracking/disk_change_id/query_changed_disk_areas) and documented real-world characteristics (extent granularity vs. fragmentation, wildcardchangeIdsemantics) from live testing.NFC_DELTA_DISKinvestigated — found it's an optional VMFS-only VDDK client optimization (perstringsonlibvixDiskLib.so), not a correctness requirement. Reading, writing, and querying allocated blocks on an actual snapshot delta file already work with the existingNFC_DISK-only implementation. Along the way, found and documented a real gotcha: querying allocated blocks on the same still-open handle a write just went through can see stale (pre-write) data — reproduced identically on native VDDK, so it's real server/VMFS behavior, not a client bug.Test plan
DDB_GETwire format, CBT dataclass conversion,query_allocated_blocksvalidation errors — no lab needed (pytest tests/unit)GetInfo,QueryAllocatedBlocks(including on a delta disk), CBT full cycle (enable → snapshot → write → snapshot → query), wildcardchangeId— validated against a live standalone ESXi 8.0.3 labpytest tests/unit tests/integration(excluding the native-VDDK cross-check tests, which needtox -e integration's isolated subprocess env due to an unrelated OpenSSL ABI conflict between VDDK's bundled OpenSSL and system OpenSSL when both load in the same process — documented indocs/ssl_hook.md)Full protocol details and the reverse-engineering process (including dead ends and the exact captures that resolved each ambiguity) are in
docs/nfc_auth.md,docs/nfc_open.md,docs/nfc_read.md,docs/cbt.md, anddocs/reverse_engineering_procedure.md.🤖 Generated with Claude Code