Conversation
Reverse-engineered and implemented against a live standalone ESXi 8.0.3 host (no vCenter), closing most of the gap versus the proprietary VDDK: - Direct ESXi (no vCenter) connectivity: nfc_service() previously hardcoded the NfcService moref as "nfcService" (vCenter's name), which fails on bare ESXi (moref is "ha-nfc-service" there). Now resolved dynamically via RetrieveInternalContent, same as VDDK itself does. Also fixes connect_authd() for tickets that omit `host` (implicit on a direct-ESXi ticket). - VixDiskLib_GetInfo: capacity and physical geometry come free from the OPEN_FILE reply (offsets already in the wire frame). biosGeo, adapterType, and uuid are fetched via DDB_GET, matching real VDDK's behavior and cost exactly. - DDB_GET (VMDK descriptor lookups): generic key/value NFC message, values are ASCII text on the wire (not binary), matching how a VMDK descriptor's DDB section is stored. - VixDiskLib_QueryAllocatedBlocks: allocated-block bitmap query. Verified against a live disk to exactly match native VDDK's output, including two non-obvious wire details: a field-order swap that's invisible in a zero-offset capture, and 4-byte bitmap padding that only shows up for small chunk counts. - Changed Block Tracking: turned out to need no NFC work at all -- VirtualMachine.QueryChangedDiskAreas is public VIM API. Added thin wrappers (enable_change_tracking / disk_change_id / query_changed_disk_areas) and documented real-world characteristics (extent granularity, wildcard changeId semantics) from live testing. - Investigated NFC_DELTA_DISK: found it's an optional VMFS-only VDDK client optimization (per `strings` on libvixDiskLib.so), not a correctness requirement -- reading, writing, and querying allocated blocks on an actual snapshot delta file already work with the existing NFC_DISK-only implementation. Documented a real gotcha found along the way: querying allocated blocks on the same still-open handle a write just went through can see stale data. Adds unit tests (bitmap decode/merge, DDB_GET wire format, CBT dataclass conversion, validation errors) and integration tests (GetInfo, QueryAllocatedBlocks, CBT full cycle, delta-disk read/write/ query) validated against a live ESXi 8.0.3 lab. Full protocol details and the reverse-engineering process are in docs/nfc_auth.md, docs/nfc_open.md, docs/nfc_read.md, docs/cbt.md, and docs/reverse_engineering_procedure.md.
Captured native VDDK's ZLIB and SkipZ IO compression with the same SSL-hook technique used for FastLZ: same opcode/offset-36 framing, just different type values (1 zlib, 3 SkipZ). ZLIB's extra is a plain zlib stream, decodable with Python's stdlib directly. SkipZ turned out to not be general compression at all -- it just omits runs of zero bytes, keeping everything else raw, which only became visible after switching from an all-non-zero test pattern (which correctly fell back to uncompressed, same as it would for FastLZ/zlib) to one with real zero-filled runs. Implemented as new _skipz_compress/_skipz_decompress helpers alongside the existing FastLZ path in NfcDisk.write/readinto, wired through VixDiskLibHandle's existing compression-flag mapping. Validated against the live ESXi lab for all three algorithms, with SkipZ's byte-level (non-sector-aligned) run encoding specifically cross-checked by writing with openvixdisklib and reading back with native VDDK.
doccaz
marked this pull request as ready for review
September 19, 2026 23:55
4 tasks
doccaz
marked this pull request as draft
September 20, 2026 14:25
doccaz
marked this pull request as ready for review
September 20, 2026 14:54
We'll use the same typing.Protocol approach as cloudbase#4, minimizing merge conflicts.
We'll extend the integration tests to cover the newly introduced compression algorithms: zlib and skipz. In case of skipz, we'll generate a pattern that contains large enough zero blocks to trigger skipz compression.
Member
|
Looks great, thanks! I've added some integration tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Captured native VDDK's ZLIB and SkipZ IO compression with the same SSL-hook technique used for FastLZ (
docs/ssl_hook.md): same opcode/offset-36 framing, just different compression-type values (1 zlib, 3 SkipZ; FastLZ stays 2).zlibdirectly, no custom framing at all.Implemented as
_skipz_compress/_skipz_decompressinopenvixdisklib/nfc_open.py(no external dependency, unlike FastLZ'spyfastlz) alongside the existing FastLZ path inNfcDisk.write/readinto, wired throughVixDiskLibHandle's existing compression-flag mapping.skip_decompression(the existing FastLZ extension) now works for all three algorithms for free, since it only depended on the shared offset-36 extra-length field.Full wire format for all three algorithms:
docs/nfc_read.md. New "Step 17" indocs/reverse_engineering_procedure.mdwith the capture methodology and dead ends (the all-non-zero-pattern false start for SkipZ).Test plan
openvixdisklibagainst the live ESXi labopenvixdiskliband reading back with native VDDK on the same fragment — confirms the server accepts arbitrary run boundaries, not just the sector-aligned ones the captures happened to usepytest tests/unit tests/integration, excluding VDDK-cross-check/perf)Note on PR sequencing
Stacked on top of the unmerged direct-ESXi/GetInfo/QueryAllocatedBlocks/CBT work (PRs #3–#6) — this branch's history includes their combined commit, so the diff shown here will shrink to just the one new commit (
0e22ea0) once those merge. That commit only touchesnfc_open.py's compression constants/write/readintoandopenvixdisklib.py's compression-flag mapping — no overlap with the other PRs' code.