Public access to hidden attributes (_job_*, _prov): no reader exists #1562

Platform-managed columns — _job_start_time, _job_duration, _job_version, _prov, _singleton — are stored per row and have no public reader. to_arrays('_job_start_time') and proj('_prov') both raise DataJointError: Attribute '...' not found., because Heading.attributes (heading.py:258-267) excludes hidden names and every accessor derives from it.

reference/specs/job-metadata.md documented to_arrays('_job_start_time', ...) as the access path. That call has never worked; the docs are corrected in datajoint/datajoint-docs#288 to describe the SQL workaround until this lands. Supersedes datajoint-docs#1553.

Deferred out of 2.3.4 deliberately — the mechanism touches query machinery and should not be designed under release pressure.

What works today, so the gap is scoped correctly

Measured on MySQL 8.0, not assumed:

So the missing piece is reading values through a supported surface. Restriction already works in its string form, and a correctly decoded read is reachable today in about six lines — which means the design can be judged on ergonomics and safety rather than on feasibility.

Proposed direction: QueryExpression.hidden

A property returning a copy whose heading also carries the hidden attributes:

Analysis.hidden.to_dicts()               # includes _job_* and a decoded _prov
Analysis.hidden.proj('_job_version')
Target.insert(Source.hidden)             # INSERT ... SELECT carries them

Returning a copy is what makes it safe: describe(), alter(), diagram and every un-opted query keep the heading they have now.

The invariant

.hidden changes what is selected and returned. It never changes what is matched on. Join and restriction keys must keep coming from the visible set unconditionally. Three findings make that non-negotiable:

• expression.py:398,430 build the USING list from heading.names. Hidden namesakes there reproduce exactly the NATURAL JOIN defect the USING clause was introduced to prevent (specs/job-metadata.md, "Excluding Hidden Attributes from Binary Operators").
• Hidden attributes never get a lineage row (lineage.py:366-369 skips any column starting with _), so assert_join_compatibility (condition.py:264-289) would raise "lineage missing on one side" for every join of two metadata-bearing tables.
• condition.py:441 selects restriction keys the same way, and that one fails silently.

Where the work is

• heading.py — an opt-in flag plus with_hidden(); an unconditionally-filtered name list for matching; select() (:641) must carry hidden through, join() (:698) must not. _attributes already holds everything with correct json/uuid/codec/dtype flags (:499-503), so nothing new is loaded.
• expression.py — the hidden property (copy idiom at :1507-1516); proj's validator at :572-576.
• table.py:877-886 — the INSERT ... SELECT branch currently appends _prov by name. Generalizing it is tempting but needs thought: copying _singleton, or _job_* from one table to another, is not meaningful. The _prov-specific carry may be correct because it is specific.

Two defects found on this path

• expression.py:578-583 is a dead validator. next(a for a in mentions if not self.heading.names) tests whether the heading is empty, not whether a is in it. It never fires, for any name.
• proj(x='_prov') silently drops the column — _prov matches rename_pattern, bypassing the positional check, and Heading.select's loop never reaches it. No error, no column. Same family as Mapping restriction naming a hidden attribute silently drops the predicate #1561.

The alternative worth taking seriously

Keep hidden attributes fully hidden and add a narrow documented reader instead — a function taking an expression and attribute names, doing the SELECT and decode_attribute itself. It serves the known consumer (provenance-export, which needs _job_version and _prov per row for audit packets) with decoded values and restriction composition, and touches no query machinery.

The case for it is not weak: attributes' filter currently does five distinct jobs — join-key selection, restriction-key selection, output shaping, insert validation, and DDL round-tripping through describe() ↔ prepare_declare ↔ alter. Splitting them is where silent breakage comes from. The case against is that it is a second way to query, outside the algebra.

Decide between them before implementing.

What must not change

.hidden would expose _singleton, so primary_key becomes ['_singleton'] in the opted-in view. Correct, but keys() and restriction run off the primary key — verify on a singleton table.

Acceptance

• Reading returns a decoded _prov dict on both MySQL (json) and PostgreSQL (jsonb); backend-consistent decoding is the likeliest thing to break.
• Default to_dicts() byte-identical to today.
• A * B with _job_* on both sides joins on the visible key only, no lineage error.
• describe() and alter() round-trip unchanged, including a singleton.
• test_entry_provenance.py's raw-SQL _raw_prov helper is replaced by the new API — that is the real acceptance test.
• Also closes the declare.py:965 error message, which tells users to "use proj()" — currently false.