fix(types): generate the open asset unions, and derive the vocabulary they route by

[KonstantinMirin]

fix(types): generate the open asset unions, and derive the vocabulary they route by

Closes #1486.

The defect

Format.assets routed 14 of the 21 asset_type values the pinned
schema declares, and RepeatableAssetGroup.assets routed 12 of its 21.

The missing values did not fail. They matched the _unknown Tag and validated
as UnknownFormatAsset / UnknownGroupAsset, so a consumer received an untyped
object with the asset's fields in __pydantic_extra__ where the schema promises
a typed one — while the generated class for every missing value sat in
src/adcp/types/domains/core/format.py, unused.

missing from Format.assets : ['card', 'daast_tracker', 'display_tag',
                              'pixel_tracker', 'published_post',
                              'vast_tracker', 'zip']
missing from group slots   : those, plus ['brief', 'catalog']

This is the unpaid half of #742. That issue correctly identified that a closed
generated union breaks consumers when the spec grows additively, and #744 fixed
it by replacing the generated union with a hand-written open one carrying a
fallback arm. Openness became permanent and completeness became manual.
_forward_compat's own docstring used pixel_tracker as its EXAMPLE of a
"novel asset_type"; by then the example was an instance of the gap.

Why adding arms would not have fixed it

There were four hand-maintained lists in aliases.py, not two: the two
unions, and two frozensets of 14 and 12 asset_type values. The frozensets
decide first — the callable Discriminator reads one and returns _unknown
BEFORE pydantic reaches the arms:

return asset_type if asset_type in _KNOWN_INDIVIDUAL_ASSET_TYPES else "_unknown"

So an annotation can show 21 arms while every document for the other seven
still downgrades. Arm counting is not evidence here, and that is demonstrable
rather than asserted: drop one value from the generated vocabulary on this
branch and leave all 21 Tag arms in place, and the two instruments disagree —

                              arm count      parse level
Format.assets                 21 arm(s)      20/21, 'pixel_tracker' -> UnknownFormatAsset
RepeatableAssetGroup.assets   21 arm(s)      21/21  (perturbation was scoped to one slot)

The acceptance evidence below is at the parse level for that reason.

What changed

The union is emitted where its arms are declared. _OPEN_ASSET_UNIONS in
scripts/post_generate_fixes.py writes, per slot, into
domains/core/format.py:

• the fallback class (extra='allow', asset_type: str),
• the asset_type vocabulary, as a frozenset of the literals the arms carry,
• the callable discriminator that reads it,
• the tagged union, one Tag per arm plus _unknown,

and then repoints the carrying field at the alias.

The generator knows each arm's asset_type literal because it emitted that
literal, so nothing is declared and nothing can go stale: a schema that adds
an asset type adds its class AND its tag in the same run.

src/adcp/types/_forward_compat.py's asset patch is deleted, not rewired —
with the generated annotation correct there is nothing left to patch — and
aliases.py re-exports the four names instead of defining them. That module no
longer imports aliases at all; the asset patch was its only reason to.

                              before            after
_forward_compat asset patch   2 field rewrites  absent
unions in aliases.py          2 hand-written    0 (re-export)
vocabularies in aliases.py    2 frozensets      0 (generated)

The two slots keep their genuine asymmetry: each union is derived from its own
base class, because a group slot carries no nested repeatable_group. The group
container is derived too — by its item_type discriminator rather than by name,
for the reason restore_format_asset_numbered_aliases already derives it: the
generator renumbers that class (Assets94 once, Assets29 now).

_unknown is still an arm, so completeness costs none of the openness #744
bought: a value the schema does not declare still reaches the fallback.

Acceptance evidence: parse level, with a control

Every declared asset_type validated through a real document on both slots, and
the class it lands in. The control row carries an asset_type the schema does
not declare and must land in the FALLBACK — a probe whose rows all refuse would
print a clean-looking table while grading nothing.

== Format.assets: 21 declared          before      after
   card                                UnknownFormatAsset   Assets25
   daast_tracker                       UnknownFormatAsset   Assets28
   display_tag                         UnknownFormatAsset   Assets23
   pixel_tracker                       UnknownFormatAsset   Assets26
   published_post                      UnknownFormatAsset   Assets24
   vast_tracker                        UnknownFormatAsset   Assets27
   zip                                 UnknownFormatAsset   Assets16
   (the other 14)                      own class            own class
   (control) totally_new…              UnknownFormatAsset   UnknownFormatAsset
   -> 14/21 own class                  -> 21/21 own class

== RepeatableAssetGroup.assets: 21 declared
   those 7, plus brief and catalog      UnknownGroupAsset   own class
   (control) totally_new…               UnknownGroupAsset   UnknownGroupAsset
   -> 12/21 own class                  -> 21/21 own class

The vocabulary the probe iterates comes from
schemas/cache/<bundle>/core/format.json, not from generated code. Both arms of
properties.assets.items.oneOf are selected by their item_type const rather
than by position, and read at their own depth — the individual arm carries its
asset oneOf directly, the group arm one level further down under
properties.assets.items — because a fixed-depth or positional reader gets one
right and the other empty.

The guards, and what breaks them

Six derived refusals in the generator, each fire-probed on a mutated copy of the
generated module (never on the schema: mutating the authority side reddens any
comparison against that authority and proves only that the plumbing is
connected):

perturbation	refusal
an arm the derivation cannot see, while the field still carries it	Format.assets and FormatAssetUnion disagree … Carried but not an arm of the union: ['Assets26']
the field carries a class the base does not own	… Carried but not an arm of the union: ['Renders']
two arms tagged the same	asset_type 'image' is declared by both Assets and Assets26; a discriminated union cannot tag two arms the same
the carrying field renamed	Format.assets is not declared, so FormatAssetUnion has nothing to open
the carrying field stops being a list	Format.assets is not a list annotation (…); FormatAssetUnion assumes it is
no class declares item_type='repeatable_group'	… the asset unions have no group arm and no group container

The membership check is the one that keeps the derivation from passing
vacuously: the field's current element union must resolve to exactly the arm
set the alias was built from. A schema refresh that adds an asset type moves
both sides together and passes; anything else fails here rather than leaving the
field closed.

Tests

Four tests in tests/test_forward_compat_assets.py asserted that
pixel_tracker parses as UnknownFormatAsset. The obligation (a novel value
reaches the fallback) is right; the value was wrong — they agreed with the stale
frozenset rather than with the schema, and the gap's own symptom is what made
them pass. A fifth restated the stale 14-value list as its own assertion. The
novel value is now derived from the generated classes, so a schema refresh
cannot turn the test's novel value into a declared one without the test
noticing.

TestDeclaredAssetTypesAreNotDegraded is new: parametrized over the generated
classes on both slots (21 + 21), asserting at the parse level that each declared
type reaches its own class with no untyped extras. Dropping one entry from the
generated vocabulary fails exactly one case, naming it:

FAILED …::test_each_declared_individual_type_reaches_its_class[pixel_tracker-Assets26]
E  AssertionError: assert <class '…format.UnknownFormatAsset'> is <class '…format.Assets26'>

One further case compares that population against the two arms of the pinned
schema. Without it, a generator that stopped emitting a class would shrink the
parametrization and stay green — grading fewer rows and saying nothing about the
one that vanished.

Public surface

adcp.types.UnknownFormatAsset and UnknownGroupAsset now resolve to the
classes in domains.core.format rather than the ones in aliases. The public
name tracks the class the unions actually route to, exactly as before, so
isinstance against the public name is unaffected.
tests/fixtures/public_api_snapshot.json records the repoint, regenerated with
scripts/regenerate_public_api_snapshot.py.

Both unions stay syntactic type aliases — the generator writes
FormatAssetUnion = Annotated[…] as source into the generated module — so a
static checker reads them directly. That makes the adopter-facing typing better,
not merely unregressed. aliases.py built the unions out of
_format_asset_class("image"), a runtime call returning type, so pyright saw a
variable rather than a type alias; an adopter writing list[FormatAssetUnion]
gets four errors on main today:

error: Variable not allowed in type expression (reportInvalidTypeForm)   x4

Measured on the same fixture, same environment, --no-incremental on both sides:

                                   before (main)   after
pyright  list[FormatAssetUnion]    4 errors        0 errors
mypy --strict                      Success         Success

Out of scope

The public <Type>FormatAsset / <Type>FormatGroupAsset aliases still name 14
and 12 of the 21 slot classes — there is no CardFormatAsset, ZipFormatAsset,
DisplayTagFormatAsset, PixelTrackerFormatAsset, VastTrackerFormatAsset,
DaastTrackerFormatAsset or PublishedPostFormatAsset. Routing no longer
depends on that list, so the gap is now only a naming one, and adding public
names is an API decision rather than a bug fix. It should probably be generated
from the same derivation rather than extended by hand, or it drifts the way the
vocabulary did. Happy to follow up separately.

[aao-secretariat]

Ladon verdict: Approve

Approve — clean fix for #1486, no blocking or medium findings.

This PR derives FormatAssetUnion/GroupFormatAssetUnion and their routed vocabulary from the generated base slot classes, fixing the case where 7/21 individual and 9/21 group asset_types silently downgraded to UnknownFormatAsset.

What I checked:

• Forward-compat preserved: the _unknown fallback arm is retained, so genuine novel discriminator values still parse. No fallback arm removed, no discriminator key changed on the wire — not a high finding under the forward-compat rule.
• Not a breaking change: no public name added/removed; newly-routed typed classes add no required fields beyond the base, so no previously-valid payload now fails to deserialize. Snapshot repoint is the only public-API movement. fix: prefix is appropriate — semver signal is correct.
• Import layering: the numbered Assets94 import sits under TYPE_CHECKING in aliases.py, which the layering allowlist permits.
• Fail-closed derivation: empty scan, duplicate asset_type, and reserved-tag collisions each raise ImportError.
• Vocabulary-drift and corrected pixel_tracker→derived-novel tests are thorough.

Decision rationale: no critical/high/medium findings. gated_paths is false, so row 2 does not apply despite review_decision: REVIEW_REQUIRED. high_risk is true only via a (modified) entry on aliases.py, but with zero medium-or-higher findings rows 4/5 do not fire. No no-auto-approve team match. Falls through to row 9 → approve.

[KonstantinMirin]

Moved to draft — please don't spend review time on this shape yet.

This derives the arms and the discriminator vocabularies at the consumer, in aliases.py, which leaves _forward_compat still rewriting the generated assets annotation at import. It makes the hand-maintained list derived rather than literal; it does not remove the need for a list. On reflection that is a fix layered on the mechanism that caused the drift, rather than a fix to it.

The version I should have opened does it in codegen: emit each slot's open union, its _unknown fallback arm and its discriminator from one derivation, so the generated annotation is already correct and the import-time patch for assets disappears. In our fork that takes _forward_compat.py from 563 lines to 45 and leaves zero hand-written unions in aliases.py.

I'm porting that onto current main and will update this PR in place rather than opening a competing one. The defect statement, the measurement and the reproduction in the body above all still stand — only the mechanism changes.

[KonstantinMirin]

Replaced with the generator-side fix and ready for review. The defect, the schema citation and the measurement are unchanged; the mechanism is different, and it is the one I should have opened first.

What changed from the shape you may have glanced at earlier:

• Codegen now emits, per asset slot, the fallback class, the asset_type vocabulary, the callable discriminator that reads it and the tagged union — as source into domains/core/format.py, beside the classes they route to. The generator knows each arm's asset_type literal because it emitted that literal, so a schema that adds an asset type adds its class and its tag in the same run.
• _forward_compat's asset patch is deleted rather than rewired, and aliases.py re-exports the four names instead of defining them. The previous revision left both in place and only made the lists derived, which fixed the bookkeeping on the mechanism rather than the mechanism.

Evidence, with the control reported separately because an instrument that refuses every row prints a clean "nothing downgraded" while grading nothing:

                              before 47084e83a     after
Format.assets                 14/21 own class      21/21
RepeatableAssetGroup.assets   12/21 own class      21/21
rows refused                   0                    0
rows downgraded               16                    0
(control) undeclared type     fallback             fallback

Arm counting would not have caught this and is worth stating explicitly: the callable Discriminator reads its vocabulary before pydantic reaches the arms, so dropping one vocabulary entry with all 21 arms intact leaves an arm-level check green and takes the parse-level one red on exactly that value. Both directions are fire-probed by perturbing the generated module, never the schema.

One unexpected result: this improves adopter typing rather than merely preserving it. On main, FormatAssetUnion's arms resolve through _format_asset_class("image"), a runtime call returning type, so list[FormatAssetUnion] in an adopter module reports Variable not allowed in type expression four times. Because the generator writes the union as literal source naming real classes, that goes to zero.

Gates are matched on both sides, same environment and flags: identical FAILED sets, the pass delta is exactly the 43 tests added, and both sides reconcile against their own --collect-only count. The nine pre-existing failures at main are untouched and none touches assets; I've noted them rather than classified them away.

[aao-secretariat]

Ladon is not auto-reviewing this PR because it modifies the review system itself (LADON.md). A human reviewer should review and merge this PR; Ladon resumes on subsequent PRs once these changes land on main.