Mapping restriction naming a hidden attribute silently drops the predicate #1561

A mapping restriction naming an attribute that exists on the table but is hidden has its predicate dropped, and every row is returned.

Nothing about JSON is involved. Earlier revisions of this issue demonstrated it with _prov, whose JSON type made it look like a JSON-path problem; it is not.

Repro — no JSON

@schema
class Comp(dj.Computed):          # with config.jobs.add_job_metadata on
    definition = """
    -> Src
    ---
    val : int32
    """

Two populated rows. _job_version is a varchar(64):

The first should match nothing. The emitted SQL carries no predicate at all. Same on both backends, so the defect is backend-agnostic, and it applies to _job_start_time, _job_duration and _prov equally.

What is not broken

Worth stating, because the earlier framing of this issue implied otherwise: JSON path restriction works correctly and portably on a visible attribute.

Doc & {"data.system": "PyRat"}     # `data : json`

returns 1 of 2 rows on MySQL and on PostgreSQL alike. translate_attribute (condition.py:59-61) hands the path to adapter.json_path_expr, which emits json_value() on one and jsonb_extract_path_text() on the other. That machinery is fine and is not what this issue is about.

The reason a JSON-typed hidden attribute looks worse is only that the portable spelling runs through the mapping form, which is the form being dropped — so for _prov the workaround has to be backend-specific SQL. That is a consequence of this bug, not a separate JSON defect.

Cause

condition.py:409:

common_attributes = set(c.split(".", 1)[0] for c in condition).intersection(query_expression.heading.names)
if not common_attributes:
    return not negate  # no matching attributes -> evaluates to True

heading.names is visible-only, so a hidden name lands in the same bucket as an attribute the table does not have.

Suggested fix

Match against the names present in _attributes rather than names, so a hidden attribute resolves instead of being discarded.

The two cases are different facts and are distinguishable:

• scan_id on Session — absent from _attributes entirely. Ignoring it is deliberate and must stay: Session & key has to work when key carries attributes from a more detailed downstream table, and make() depends on that.
• _job_version on a Computed table — present in _attributes, filtered only out of names. The column exists; the caller named a real thing.

So the key-passing idiom is untouched. A dict reaching & can only acquire a hidden name by hand: every key DataJoint produces — keys(), to_dicts(), fetch1(), key_source — is built from the visible heading, so no generated query changes.

Nothing hidden becomes readable by this. Restriction on these columns already works through the string form; #1562 is the separate question of reading values back.

One obstacle in the same change: condition.py:347 does query_expression.heading[key_match["attr"]].uuid, and Heading.__getitem__ resolves through attributes, so it raises KeyError on a hidden name. It needs the _attributes fallback Heading.as_sql already uses (heading.py:398-404).

Noticed in passing, not filed

Doc & {"data": {"system": "PyRat", "n": 7}} — restricting by a whole JSON object rather than a path — raises QuerySyntaxError on both backends. prep_value only special-cases a dict value when the key is a JSON path (condition.py:343-344), so a whole-object comparison falls through and emits invalid SQL. It errors rather than answering wrongly, so it is a poor message rather than a correctness problem. Mentioned only so the next person who hits it knows it is understood.

Documentation

reference/specs/boundary-provenance.md, how-to/record-data-origin.md, the data-entry tutorial and reference/specs/job-metadata.md advertised the mapping form on _prov. Corrected in datajoint/datajoint-docs#288.