[MikeCarlo]

Summary

Changes custom_extensions[].data from a JSON-encoded string to a structured object with vendor-defined keys, implementing Proposal A from discussion #481.

This is an intentional hard break in the unreleased 0.3.0.dev0 spec. There is no deprecation period, since the project is still in early incubation. There is no 0.2.0 release: documents that declare version: 0.2.0.dev0 must declare 0.3.0.dev0. The field is also renamed to nonportable_extensions. The old name is not accepted.

Motivation

Today data is an opaque JSON string:

custom_extensions:
  - vendor_name: SNOWFLAKE
    data: '{"warehouse": "ANALYTICS_WH", "some_unmapped_flag": true}'

Problems with this:

• Values are not first-class YAML/JSON, so JSON Schema can't see them.
• Nested structure is hidden behind JSON.parse.
• Every consumer has to special-case-parse each vendor's string.

Converters increasingly use extensions to carry unmapped or partially mapped properties, so this friction keeps growing.

Change

data is now an object, and custom_extensions has been renamed nonportable_extensions to enforce the ideal that "Core semantics MUST NOT depend on extension contents":

nonportable_extensions:
  - vendor_name: SNOWFLAKE
    data:
      warehouse: ANALYTICS_WH
      some_unmapped_flag: true

• No reserved keys. Keys are entirely vendor-defined. The core schema does not describe the structure under data. Vendor schemas live outside core-spec.
• Any JSON value is allowed as a value: strings, numbers, booleans, null, arrays, nested objects.
• data is still required. {} is valid.
• A raw string is still possible by putting it under any key the vendor chooses, e.g. data: { text: '{"…"}' }. A bare string is no longer valid.
• Portability. Nothing defined in the core specification relies on any property in the extension area. Behavior defined only through extensions is non-portable and must not be presented as portable Ossie behavior. Producers should use core constructs wherever they can express the meaning.

YAML quoting

data must be JSON after YAML loads it. validation/validate.py rejects values that are not JSON. Quote scalars that YAML would otherwise read as a boolean, a number, or a date. Worked examples: extensions/nonportable_extensions.md.

Unquoted dates, timestamps, and the non-finite forms .inf and .nan are rejected. The same characters in quotes are strings and pass:

# rejected
nonportable_extensions:
  - vendor_name: DB_VENDOR
    data:
      released: 2024-01-01
      stamped: 2024-01-01T00:00:00Z
      ratio: .inf
      missing: .nan

# accepted
nonportable_extensions:
  - vendor_name: DB_VENDOR
    data:
      released: "2024-01-01"
      stamped: "2024-01-01T00:00:00Z"
      ratio: ".inf"
      missing: "NaN"

yes, NO, true, false, and 010 also pass when quoted ("yes", "NO", "true", "false", "010"). Left unquoted they still pass validation, but as a boolean or an integer (and 010 as the integer 8), not as the text that was written. The validator cannot catch that. Unquoted 1e3 stays the string 1e3 under this project's PyYAML, but other loaders may read 1000, so quote "1e3" when the characters matter.

Files changed

File	Change
core-spec/ossie-schema.json	Field renamed to nonportable_extensions. $defs/NonportableExtension.data is "type": "object" with no keys defined under data. Document version is 0.3.0.dev0.
core-spec/spec.md	Non-portable Extensions: portability boundaries, JSON representability, migration from custom_extensions.
core-spec/spec.yaml	data: string becomes data: object.
extensions/nonportable_extensions.md	Field guide with quoted and unquoted data examples.
examples/tpcds_semantic_model.yaml	Extensions converted to native maps under the new field name.
python/src/ossie/models.py	OssieNonportableExtension.data is dict[str, Any].
docs/index.md	FAQ and glossary wording updated.
validation/validate.py, python/tests/*, validation/tests/test_validate.py	Object data, rejection of dates and non-finite floats, and the residual retyping cases.

Validation

Local checks after the rebase onto 0.3.0.dev0:

Command	Result
uv run validation/test_validate.py	✅ 59 tests
uv run --with pytest --with pyyaml --with jsonschema --with sqlglot -m pytest validation/tests/	✅ 613 passed
uv run --with pytest --with jsonschema --with ./python -m pytest python/tests/	✅ 278 passed
uv run scripts/generate-spec-types.py --check	✅ in sync
uv run validation/validate.py on TPC-DS, flights, and renamed-fields examples	✅ passed

GitHub required workflows are still waiting on fork-workflow approval, so CI is not green yet.

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue
• Breaking changes to the spec are clearly called out in the summary

Ontology

Ontology files were updated for the field rename (ontology/mapping.json, ontology/ontology.md).

Converters (Out of scope)

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate
• CONTRIBUTING.md is updated if the contribution process changed (no process change)

Examples

• examples/ are added or updated for any new spec constructs or converter support

Tests

• All existing tests pass (pytest / CI green) - core tests (local pytest passed; GitHub CI is waiting on fork-workflow approval)
• New functionality is covered by tests

Compliance

• No third-party dependencies are added without PMC/IPMC approval

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2

Open (2)

• The table uses || prefixes, which typically renders as an extra empty column in standard Markdown… · New
• model_config = ConfigDict(frozen=True) no longer effectively makes OssieCustomExtension… · New

What changed in this PR

Updates the Ossie spec and Python model to represent custom_extensions[].data as a structured object (vendor-defined map) instead of a JSON-encoded string, aligning validation and examples with the new shape.

Changes:

• Update JSON Schema / YAML spec to require custom_extensions[].data as an object
• Update Python model (OssieCustomExtension) and tests to accept objects and reject non-objects
• Rewrite documentation and examples to use native YAML mappings for extension data

File	Description
core-spec/​ossie-schema.json	Changes CustomExtension.data schema type to object and updates description
core-spec/​spec.yaml	Updates the spec type annotations from string → object for custom_extensions[].data
core-spec/​spec.md	Updates the Custom Extensions documentation and examples to use YAML objects
docs/​index.md	Updates FAQ/glossary wording to match object-based extension data
examples/​tpcds_semantic_model.yaml	Migrates example extension data from JSON strings to native YAML objects
python/​src/​ossie/​models.py	Updates OssieCustomExtension.data from str → dict[str, Any]
python/​tests/​test_validation.py	Updates/extends validation tests for object-based data
python/​tests/​test_serialization.py	Adds round-trip tests for nested custom extension data
python/​tests/​test_models.py	Updates fixtures/examples using custom_extensions.data
validation/​tests/​test_validate.py	Adds schema-level validation tests for object vs non-object extension data

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ekut]

Ran the converter suites against this branch merged onto current main (2491533), since the converter workflows on this PR are still waiting for approval. Spec, schema, example and the Python package agree with each other, and the core checks listed in the description pass. What breaks on merge is the converters, which still treat data as a string:

• 7 of the 14 Python converter suites fail (172 tests); all 14 are green on main. In six of them the converter writes data with json.dumps(...) and the schema or the model now rejects it; Sigma builds OssieCustomExtension(data=json.dumps(...)) on the in-repo package, so Sigma imports raise wherever they write an extension. The seventh, the ontology converter, fails the other way: its own CustomExtension has data: str, so it can no longer load examples/tpcds_semantic_model.yaml as updated here.
• A green suite doesn't mean the converter handles the new form. Given its own vendor's extension with data as an object, Cube, Omni, Microsoft, Databricks and Sigma raise TypeError from json.loads, ThoughtSpot raises "expected a JSON string", and Honeydew, NVIDIA and OrionBelt catch the error and drop the extension silently. For OrionBelt, round-tripping its TPC-DS OBML fixture with object data drops all 36 comment entries (the string path keeps them) and renames the column customer_full_name to its SQL expression, with no warning. GoodData already accepts both forms (_get_gooddata_extension).
• converters/README.md, section "Custom Extensions", still tells converter authors that data is a JSON string to parse. ThoughtSpot's README.md and docs/vendor-payload.md say it is never a nested object.

How would you like to sequence this: the converter changes in this PR, or readers first accept both forms the way GoodData does and then the schema flips? The order also interacts with #523: once the converters move to apache/ossie-converters, their CI no longer runs here, and these failures show up at the first submodule bump there instead. Open #378 adds new json.dumps writes for DBT, so it needs the same change whichever lands first.

Per-converter results

Converter	Suite, main → this PR	Writes data as	Reading object data
cube	864 passed → 124 failed	JSON string	TypeError
databricks (python)	103 passed → 103 passed	JSON string	TypeError
dbt	176 passed → 176 passed	—	—
gooddata	94 passed → 94 passed	JSON string	works
honeydew	151 passed → 151 passed	JSON string	dropped silently
microsoft	410 passed → 1 failed	JSON string	TypeError
nvidia	89 passed → 2 failed	JSON string	dropped silently
omni	105 passed → 105 passed	JSON string	TypeError
ontology	170 passed → 1 failed	own model, data: str	fails to load
orionbelt	269 passed → 7 failed	JSON string	dropped silently
sigma	103 passed → 34 failed	JSON string, rejected by the model	TypeError
snowflake	186 passed → 186 passed	—	drops all extensions
thoughtspot	977 passed → 3 failed	JSON string	ConversionError
wisdom	37 passed → 37 passed	—	reports, doesn't parse

Repro: cd converters/<name> && uv sync && uv run pytest (needs uv >= 0.9). Every failure is data rejected as a string by the schema or the model, except ontology, which rejects the object. The Java converters (Databricks Java, Polaris, Salesforce) were not run; Polaris models data as String.

[MikeCarlo]

@ekut We discussed in the last two OSSIE meetings that because we are currently in incubation that breaking changes occur now. Thus, the move to the new Converter Repo makes the most sense to me. Enforce the new standard in the test in the new converter repo.

Tagging @jbonofre about the communication in our meeting notes about breaking changes occurring now.

Here is how I would see the next steps moving:

1. Update the spec with this breaking change, bump the spec with a major version
2. Move the converters to the new repo
3. Validate and update converters to adhere to the new spec as part of testing validations on the new repo

This enables a clean cut between the two bodies of work required.

This PR is in response to this discussion:
#481

[christianeu-db]

+1 to JB's comment in #481 - please bring this to a [DISCUSS] on the dev list so the community can weigh in before merging.

The opaque-string friction within converters is real, so I'm sympathetic to wanting to lower the per-converter cost. My main concern is that this could weaken portability in practice. Today, vendor extensions live in opaque strings. Implementations can ignore those strings and still operate on the portable parts of the model, assuming more semantics continue to land in the portable core.

Making vendor extensions structured and easily parseable could create an incentive to put more semantics there rather than in the portable core. Instead of a vendor's state being one opaque value, it can now be N individually addressable values. Over time, these could become a de facto extension mechanism that dominates the file format. We're seeing a bit of this with expressions already: we have a portable set, but around 1/3 of converters write vendor-specific dialects that other implementations may not understand (test for simple metrics in #576).

Vendors also tend to have features that are almost the same but not quite. With opaque strings, these differences stay isolated. Once those keys become structured, even under vendor-specific sections, implementations may start depending on subtly different representations of the same concepts. Over time, that could make interoperability harder rather than easier.

I also wonder whether having named vendors in the core spec implies more standardization of these extensions than actually exists. The spec says vendor names can be arbitrary strings, and there's no enforcement of the structure or semantics of their contents. I think it's reasonable for the core spec to define an extension mechanism, but I'd prefer we avoid giving the impression that vendor-specific contents are part of the portable semantic contract. #146 is adding support for annotations in the ontology layer, but that's explicitly not intended as a vendor-specific escape hatch.

More broadly, I'd like us to preserve the incentive for vendors to contribute common semantics to the portable core rather than build out parallel vendor-specific representations. Otherwise, we risk Ossie becoming a collection of vendor-specific schemas rather than a common semantic interchange format.

Separately, it's probably worth a PR to clarify how vendor extensions are documented. For example, we could use obviously fictional vendors in the core spec and move real vendor examples into a separate vendors/ folder, where vendors or individuals can register their extensions without requiring a core-spec change. I can take a pass at that.

None of this is meant to dismiss the ergonomics problem - the per-consumer parsing pain is real. If the main goal is to reduce that burden, could we address some of it through shared parsing/accessor utilities for custom_extensions in ossie-converters, without promoting the payload into the core schema? Happy to collaborate on that!

[christianeu-db]

Following #522 (review), let's run this through a discussion. This does seem to make vendor extensions internals appear as top-level extensions in Ossie

[flyrain]

Every changes in a github discussion like #481 will be sent to dev mailing list. Given that #481 exists, do we need another thread?

[christianeu-db]

Every changes in a github discussion like #481 will be sent to dev mailing list. Given that #481 exists, do we need another thread?

@flyrain there is a discuss thread here. Let's use that to align

[jbonofre]

As we have already a thread (before the GH discussion), better to use it and route contributors to the dev mailing list.

[kayemkim]

One gap between the two readers the spec now names. OssieDocument.from_ossie_yaml accepts a date or timestamp under data, and model_dump_json writes it back as a string, so the document validate.py fails with date is not a JSON value comes out of the SDK as one that passes, with the value retyped on the way. The loader already walks data in _invalid_below (models.py:82) to refuse .inf and .nan there, and the reason it gives, that JSON cannot express the value, covers a date too. The docstring leaves a date under ai_context out of scope on purpose, and data is walked with the same free-form flag, but data now has a spec sentence that ai_context does not, so a date/datetime branch in that walk (and bytes from !!binary, which dumps as "hi") would make the two readers agree on a document while model construction stays lenient as the docstring intends. Keys below the top level are the smaller half of the same gap: dict[str, Any] rejects {1: x} directly under data but accepts it one level down and dumps the key as "1".

The rest holds. A date, a timestamp, .inf, .nan, a non-string key, !!binary, a reused anchor, the old field name and a string data each produce one [Schema] line at the right path at model, dataset, field and metric level; yes, 010 and 1e3 pass as the section says; the core examples, the ontology documents and flights.mapping.yaml validate unchanged. The tree merges clean onto main and onto #697 and #703, and no custom_extensions is left outside the migration text.

[christianeu-db]

@MikeCarlo thanks for incorporating the feedback from the mailing list! This resolves all my core semantics concerns.

I've left a few comments on the spec text, mostly around trimming wording. The biggest opportunity is removing migration details and references to the old custom_extensions name/shape. Given we're still pre-release, I think it's fine for the spec to describe only the new structure. As we get closer to merge, we can file issues in ossie-converters to track any necessary migration work.

I'll defer to @kayemkim for the initial signoff on validation.

I'll also update the mailing list with the current state: JB's three proposed changes have been incorporated, and I recommend moving forward with lazy consensus.

[kayemkim]

The validation side is good from me. Merged onto current main with the README hunk below resolved, the validation and Python suites pass, each block in the new field guide loads and validates the way its heading says, and the cases from my last review still give one [Schema] line each at the right path.

The SDK loader difference from that review is unchanged. from_ossie_yaml takes the guide's released and stamped lines and writes them back as strings, while validate.py rejects both. It doesn't need to hold this PR, and I can send it as a follow-up after the merge if you'd rather not widen this one.

The ! in the new title fails the PR title check: the pattern in pr-title-check.yml allows no ! before the colon, and feat(core-spec): … with the same text passes. The branch also conflicts with main since #491 landed after your rebase. It is one hunk in extensions/README.md, where both sides add rows to the file table, and keeping main's dialects/ row above your two resolves it.

[christianeu-db]

@MikeCarlo this looks good to merge. I updated the title to remove the "!" so the title check passes. The only remaining issue I see is the merge conflict. Once that's resolved, let's target merging by EOD Tuesday Pacific time to give consensus a bit more time to settle.

[christianeu-db]

@MikeCarlo - let's keep this mutable this now. Addressing mutability can be covered in a followup PR

Since data became a MappingProxyType, a document with an extension can't be copied or pickled, and json.dumps(ext.data) fails:

copy.deepcopy(doc): TypeError: cannot pickle 'mappingproxy' object
doc.model_copy(deep=True): TypeError: cannot pickle 'mappingproxy' object
pickle.dumps(doc): TypeError: cannot pickle 'mappingproxy' object
json.dumps(ext.data): TypeError: Object of type mappingproxy is not JSON serializable

With the TPC-DS example, all four worked before this commit. Other frozen models hold plain lists (datasets, primary_key), so could we drop this commit and keep data a dict?

[christianeu-db]

lgtm - thanks for going through a few iterations on this @MikeCarlo