Add Dataset::to_json for creation-schema round-trip - #180
Open
thiago-carneiro wants to merge 3 commits into
Open
thiago-carneiro wants to merge 3 commits into
thiago-carneiro wants to merge 3 commits into
Conversation
Serialize the logical schema (variable names, data types, dimensions, chunk grids, compressors, long names, coordinates and stored user attributes) into the creation JSON consumed by from_json(). The output is validated against the dataset creation schema before being returned. Key translation decisions: - dataType comes from the zarr dtype metadata (V2 numpy-style strings reverse-mapped, V3 MDIO names, struct fields translated per field), not the top-level spec dtype, which is the byte view for structs. - statsV1/unitsV1/attributes are read from the stored array metadata (V2 .zattrs, V3 zarr.json) because the in-memory UserAttributes flatten unitsV1 objects, a form the creation schema rejects. - Dimension coordinates use the object form so the create path registers the dimension; other variables use the string form. - Datasets with header variables are rejected loudly: the creation schema has no representation for them, so the round-trip would silently drop them (creation-side support is a prerequisite). Tests: round-trip (V2+V3) with sampling and full creation-JSON equality, explicit schema validation, loud header-variable rejection; struct-array and header-variable round-trip preservation are prerequisite-gated (GTEST_SKIP).
Relocate the creation-JSON serialization helpers next to their parsing inverses (to_zarr_dtype, blosc resolution helpers) in dataset_factory.h, inside mdio::internal. Pure relocation: no signature, body, or call-site changes; dataset.h already includes dataset_factory.h.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #178
Summary
Adds
Dataset::to_json()(V2 and V3): serializes an open dataset back to thedataset-creation JSON, validated against the creation schema, so the result
feeds
Dataset::from_json()unchanged. This enables schema-faithful copieswithout hand-rolling variable specs: one call to get the creation JSON, one
call to recreate the store (see #178).
The output carries the logical schema: variable names, data types (scalar and
struct arrays, mapped from the V2 numpy dtypes / V3 data_type descriptors),
dimensions (dimension coordinates in object form so the create path registers
them), chunk grids, compressors (Blosc in numcodecs and codec-pipeline forms),
long names, coordinates, and the stored user attributes
(
statsV1/unitsV1/attributes) — read from the kvstore because thein-memory
UserAttributesflattenunitsV1objects, a form the creationschema rejects.
Datasets with header variables are rejected loudly: the creation schema has no
representation for them, so serializing would silently drop them. Creation-side
header-variable support is a prerequisite for lifting that restriction.
Changes
mdio/dataset.h: publicDataset::to_json(IncludeDefaults)(+65 lines).mdio/dataset_factory.h(new): the serialization helpers, inmdio::internalnext to the existing creation helpers (dtype mapping,compressor translation, struct fields, dimensions).
structured arrays round-trip correctly.
Testing
toJsonRoundTrip(V2/V3),toJsonOutputPassesSchemaValidation(V2/V3),toJsonRejectsHeaderVariablesLoudly.toJsonHeaderVariableRoundTripPreservationis skipped pending creation-sideheader-variable support (the rejection path above is its precondition).
mdio-python 1.2.3.
Notes
Independently mergeable (cut from
main; touches different regions than#177's fix). Downstream consumers already migrated onto this API.