Repository navigation
feat(core-spec): add Layer 3 wide table query interface - #653
Conversation
Add core-spec/wide_table_semantics.md, the Layer 3 specification of the layered query interface. A Layer 3 query names dimensions, metrics, and one filter. The engine lowers the query to a Layer 2 query that evaluates each metric with AGG(). This version covers the shared-dimension case: it joins only along many-to-one relationships, and it raises an error for ambiguous or unsupported cases. Update the pointers to the Layer 3 specification: - spec_layers.md: link the new file in place of apache#246, and name metrics, a filter, and datasets in the Layer 3 summary. - relational_semantics.md: link the new file in place of apache#246, and link Layers 1 and 3 from the conformance section. Co-authored-by: Isaac <no-reply@databricks.com>
855afbe to
6868df5
Compare
ekut
left a comment
There was a problem hiding this comment.
The numbers in the worked examples check out. I ran Sections 1.3 and 6.1 and Appendices B.1, B.2, B.4 and B.5 through an independent engine on ClickHouse 26.9 (dactopus-ossie-clickhouse). Every query there that the text answers with rows gets the same rows, empty-input values included.
The text also settles four questions left open in the #246 review:
- Two unconnected metrics without dimensions give one row (B.4).
- Dimensions of two datasets that meet only through a fact raise
E_NO_ANCHOR, and a metric of the fact makes the query valid (B.3). - A direct relationship next to a longer path raises
E_AMBIGUOUS_PATH(Section 4). - A metric of a coarser dataset by a dimension of a finer one raises
E_NO_PATH(Section 4, B.3).
| CROSS JOIN (SELECT AGG(c.c2) AS c2 FROM customers c) b2; | ||
| ``` | ||
|
|
||
| On the data in Appendix B.1, the value is 650 / 4 = 162.5. |
There was a problem hiding this comment.
An engine that joins customers to the orders rows gets 650 / 3 = 216.67 here, because it counts only the customers with orders (dactopus-ossie-clickhouse does today). Could the example rule that out? For example: "Not 650 / 3: c2 counts C4, which has no orders."
There was a problem hiding this comment.
That's a good point. I'll add a line clarifying that each measure is computed independently, so rows contributing to one measure aren't dropped due to joins with tables used by other measures. This avoids the row loss that can occur when pre-joining tables to compute multiple aggregates in a single query.
|
|
||
| Conformance is semantic: each query must produce the specified rows or error. Bindings define the representation of queries, results, and errors, and interoperability between their implementations. | ||
|
|
||
| An engine conforms if it satisfies Sections 3, 4, 5, and 8 through every binding it offers. It need not generate SQL text, but SHOULD expose its lowering, for example through `EXPLAIN`. |
There was a problem hiding this comment.
Can a conforming engine answer, as an extension, a query that this version rejects? For example:
- a filter on the many side, as an existence test:
{"dimensions": ["customers.region"], "filter": "orders.amount > 100"}gives EAST and WEST, where B.5 expectsE_UNSUPPORTED_FILTER_PATH(Section 5.6); - a filter over two datasets of one join tree, such as
orders.status = 'paid' AND customers.region = 'EAST', where Section 3.4 expectsE_CROSS_DATASET_FILTER; - a model-scoped
SUM(orders.amount) / COUNT(*), where Section 5.2 expectsE_AMBIGUOUS_MEASURE_GRAIN.
dactopus-ossie-clickhouse answers all three today. Each of these errors is a MUST, so as written such an engine does not conform. Section 3.5 has the engine reject a part or property "that it does not support", which leaves room for an engine to support more than this version defines. Is that the intended route: the extension applies only when the query carries a property the engine defines, and the Section 8 error applies otherwise? A.1 lists existence filters as a future extension, so an engine that answers the first case by default could disagree with a later version. If this is the intent, one sentence in Section 9 would say it.
There was a problem hiding this comment.
Yes, that is the intended route. This version is deliberately conservative. It accepts only the cases whose answers we are confident about, and it rejects the rest so that a later version can still define them.
Engines are welcome to do more than this version defines. By default, though, an engine should still raise the Section 8 error, for the reason you give: a later version may define these cases with a different answer. A gate keeps the default behavior aligned with each version, and the engine can keep its own behavior behind its property. An engine that does not know the property rejects the query with E_INVALID_QUERY and does not ignore it (Section 3.5), so a query that depends on an extension fails visibly on other engines.
I expect all three of your cases to become conformant later, either in Layer 3 itself or through a model extension and new rules:
- Existence filters are listed under one-to-many paths in A.1.
- Filters over several datasets are listed under multiple filters and cross-dataset expressions in A.1.
SUM(orders.amount) / COUNT(*)can already be written as a dataset-scoped metric onorders. A later version could add a rule that gives a model-scopedCOUNT(*)a home dataset. Until then, a metric'scustom_extensionsentry can carry the gate.
I'll add this to Section 9:
An engine MAY answer a query that this version rejects only when the query or the model requests that behavior explicitly, for example through an engine-defined query property or a
custom_extensionsentry. Otherwise, the engine MUST raise the error that Section 8 specifies.
- Section 5.2: state that each branch aggregates the rows of its own home dataset, so revenue_per_customer is 650 / 4. A plan that computes both components over a join from orders to customers (e.g. inner, left) drops C4 and gives 650 / 3. - Section 9: an engine MAY answer a query that this version rejects only when the query or the model requests that behavior explicitly; otherwise it MUST raise the Section 8 error. Co-authored-by: Isaac <no-reply@databricks.com>
| | 10 | `E_UNAGGREGATED_FINER_GRAIN_REFERENCE` | A model-scoped metric has a field reference outside every aggregate. | Section 5.2 | | ||
| | 11 | `E_DEFERRED_WINDOWED_METRIC` | A model-scoped metric contains a window function. Deferred. | Section 5.2 | | ||
| | 12 | `E_AMBIGUOUS_MEASURE_GRAIN` | An aggregate in a model-scoped metric has arguments that read several datasets (deferred), has a `FILTER` that reads a dataset other than the one its arguments read, or has arguments that read no dataset. | Section 5.2 | | ||
| | 13 | `E_NO_PATH` | No metric of the query reaches the dataset of a dimension. | Section 5.3 | |
There was a problem hiding this comment.
This row is vacuously true for a query without metrics: "No metric of the query reaches the dataset of a dimension" holds when the query has no metrics at all. Because the first matching row wins, every dimension only query would raise E_NO_PATH, and rows 15 and 16 could never fire for one. That contradicts B.3 (E_NO_ANCHOR) and B.5 (E_UNSUPPORTED_FILTER_PATH).
| | 13 | `E_NO_PATH` | No metric of the query reaches the dataset of a dimension. | Section 5.3 | | |
| | 13 | `E_NO_PATH` | The query has metrics, and none of them reaches the dataset of a dimension. | Section 5.3 | |
There was a problem hiding this comment.
Good catch, thanks. Applied. I also made the dimension explicit in rows 13 and 14 and in Section 5.3 ("there is a dimension whose dataset none of them reaches"), because the old wording was also vacuously true for a query without dimensions.
|
|
||
| Only many-to-one steps are supported. Layer 2's `AGG()` also handles joins in the reverse direction (Layer 2 Section 8.3), but those paths are outside this version's shared-dimension scope. | ||
|
|
||
| The lowering uses `LEFT JOIN` to preserve rows without matches. Engines trust declared cardinality; results are undefined when the data violates it. Self-relationships do not participate in paths. |
There was a problem hiding this comment.
| The lowering uses `LEFT JOIN` to preserve rows without matches. Engines trust declared cardinality; results are undefined when the data violates it. Self-relationships do not participate in paths. | |
| The lowering uses `LEFT JOIN` to preserve rows without matches. Declared cardinality determines which paths exist; it does not change the result. When the data violates it, the result is still that of the Layer 2 lowering, in which a home row counts once in each group it matches. Self-relationships do not participate in paths. |
There was a problem hiding this comment.
Agreed. Layer 3 is defined by its Layer 2 lowering, and AGG() does not depend on declared cardinality (Layer 2 Section 8.3). Applied, with "after the filter" added. Section 4 now defines declared cardinality, and the sentences that assumed it holds now say so (Sections 3.2 and 3.4, the step in Section 4, and S1). A.1 lists a RELY-style declaration that lets engines trust it as a future Layer 2 extension
|
|
||
| ### B.3 Path cases | ||
|
|
||
| - **Shared dimensions:** `store_sales` and `store_returns` both relate to `item` and `date_dim`. Querying their respective metrics by `item.brand` and `date_dim.year` succeeds with two branches. |
There was a problem hiding this comment.
I'm a bit confuse here 😄
Bullet 2 says store_sales also relates to store, and the "No anchor" bullet depends on that store_sales -> store relationship ("Adding store_sales.total_sales makes the query valid"). In that model, though, store_sales has two paths to date_dim, so bullet 1 query by date_dim.year would raise E_AMBIGUOUS_PATH instead of succeeding.
Maybe bullet 2 could start with "If store_sales also relates to store`..." and bullet 4 could say it uses that model.
There was a problem hiding this comment.
Right, the bullets mixed two models. B.3 now states one model up front. Only the closing-date relationship from store to date_dim is conditional, because it creates the second path. The example in Section 4 says the same.
- Sections 5.3 and 8: rows 13 and 14 and the Section 5.3 bullets name "a dimension whose dataset ..." explicitly, and row 13 applies only to queries with metrics. A query without metrics or without dimensions meets neither row, so E_NO_ANCHOR and E_UNSUPPORTED_FILTER_PATH can fire for a dimension-only query. - Section 4: define declared cardinality as the many-to-one direction of a relationship, which the data need not satisfy. It affects only which paths exist. When the data violates it, the result is still that of the Layer 2 lowering. Sections 3.2, 3.4, 4 (step) and 6 (S1) state which claims depend on the data satisfying it, and Section 3.4 says that the filter is evaluated on each row the joins produce. - Appendix B.3: state one model up front; only the relationship from store to date_dim through its closing date is conditional. The Section 4 example says the same. Co-authored-by: Isaac <no-reply@databricks.com>
- Sections 5.2 and 8: rows 10 to 12 now apply to the model-scoped metrics that the query names. Read literally, they applied to every model-scoped metric of the model, so one metric that this version defers, such as SUM(order_lines.quantity * products.price), made every Layer 3 query against that model fail. Rows 13 and 14 already apply only to the metrics of the query. - Appendix A.1: list trusted cardinality, a model declaration in the manner of SQL RELY constraints that lets an engine assume declared cardinality, as a future extension. It changes measure evaluation, so it belongs in Layer 2, which already lists RELY hints as a future extension. In this version, the result is that of the Layer 2 lowering whether or not the data satisfies the declared cardinality. Co-authored-by: Isaac <no-reply@databricks.com>
kayemkim
left a comment
There was a problem hiding this comment.
The Section 5.2 and Appendix B numbers hold when the illustrative lowering is run on DuckDB over the B.1 and B.2 tables: 650/4 against 650/3 for the single-join plan, the three-branch result, the two NORTH rows from the plan that joins b_visits on b_orders.region alone, and the five B.5 cases. I also walked the B.3 to B.5 queries through the Section 8 order with the new row 13 wording; each reaches the row the text names and nothing fires earlier. One cross-PR note: #691 bumps the version header in relational_semantics.md but does not know this file, so line 22 here needs the same bump in whichever of the two lands second.
A model's metric whose aggregates each read one dataset is split into components, one per aggregate, as Layer 3 (apache/ossie#653, section 5.2) requires. Components with the same home dataset form a branch with the question's dimensions and row filters; branches are merged by the existing stitching, a missing key gives a component its empty-input value, and the metric's expression is evaluated over the merged values. customer_lifetime_value now divides total sales by all customers, not by those with a sale (52624.71 where 0.13.0 gave 57919.73), and store_productivity is answered. A dimension or row filter that the dataset of an aggregate cannot reach is refused with a message naming it, the metric and the dataset; so are a window function or a column outside every aggregate in such a metric, and a filter that writes aggregates of different datasets. An aggregate that reads several datasets is planned as before. design.md cites #653 and the relational spec in place of the closed proposal #246. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… hints A model's metric with an aggregate over several datasets (or a FILTER on another dataset) next to an aggregate over a dataset the first does not read is refused in every question and reported by validate: one join would count the second's dataset over the first's rows, and section 5.2 of apache/ossie#653 rejects the metric. An aggregate's FILTER no longer gives its home, so COUNT(*) FILTER (WHERE other.x) names no dataset and is refused in a metric that would be split or mixed. A written aggregate filter follows the same rules, and an aggregate that names no dataset no longer lifts the check. The hint in a reachability error appears only when a dataset metric can express the count: COUNT(DISTINCT key) of the counted dataset, through one visible relationship from the suggesting dataset. Tests: a NULL dimension value from the dimension's own dataset against a missing branch row, every key of both branches in the TPC-DS comparisons, the mixed and FILTER cases, written filters, both hint cases and the plan of a filter on a split metric. design.md records the decision and drops the open item. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A model's metric whose aggregates each read one dataset is split into components, one per aggregate, as Layer 3 (apache/ossie#653, section 5.2) requires. Components with the same home dataset form a branch with the question's dimensions and row filters; branches are merged by the existing stitching, a missing key gives a component its empty-input value, and the metric's expression is evaluated over the merged values. customer_lifetime_value now divides total sales by all customers, not by those with a sale (52624.71 where 0.13.0 gave 57919.73), and store_productivity is answered. A dimension or row filter that the dataset of an aggregate cannot reach is refused with a message naming it, the metric and the dataset; so are a window function or a column outside every aggregate in such a metric, and a filter that writes aggregates of different datasets. An aggregate that reads several datasets is planned as before. design.md cites #653 and the relational spec in place of the closed proposal #246. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… hints A model's metric with an aggregate over several datasets (or a FILTER on another dataset) next to an aggregate over a dataset the first does not read is refused in every question and reported by validate: one join would count the second's dataset over the first's rows, and section 5.2 of apache/ossie#653 rejects the metric. An aggregate's FILTER no longer gives its home, so COUNT(*) FILTER (WHERE other.x) names no dataset and is refused in a metric that would be split or mixed. A written aggregate filter follows the same rules, and an aggregate that names no dataset no longer lifts the check. The hint in a reachability error appears only when a dataset metric can express the count: COUNT(DISTINCT key) of the counted dataset, through one visible relationship from the suggesting dataset. Tests: a NULL dimension value from the dimension's own dataset against a missing branch row, every key of both branches in the TPC-DS comparisons, the mixed and FILTER cases, written filters, both hint cases and the plan of a filter on a split metric. design.md records the decision and drops the open item. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
|
||
| ```json | ||
| { | ||
| "dimensions": ["customers.region"], |
There was a problem hiding this comment.
Earlier I see that layer 3 is not a SQL relation.
Wouldn't it be possible to express this example still as a SQL query?
something like below
select agg(orders.total_revenue), agg(returns.total_returns) from metrics group by customers.regioni.e. the select the metrics, and add the dimensions in group by.
Isn't that still a reasonable to query, at layer 3 as well? The benefit of this would be all tools that can work with SQL can use it. They can do Layer 2 as well, however they would need to know the joins.
|
|
||
| ### 3.4 Filter | ||
|
|
||
| `filter` is one Layer 1 boolean expression, evaluated before aggregation. |
There was a problem hiding this comment.
say the metric is average price, and we would like to ignore the outliers (say remove the first 1 percentile and the last 1 percentile), can we express it at this layer? i.e. can filter include metrics?
An example that along this line would be useful.
|
|
||
| - A **step** crosses a relationship from `from` to `to`. If the data satisfies the declared cardinality, a step matches at most one `to` row for each `from` row. | ||
| - A **path** from dataset X to dataset Y is a sequence of steps from X to Y with no repeated dataset. The path from a dataset to itself is empty. Y is **reachable** from X if such a path exists. | ||
| - The lowering joins Y to X along the path from X to Y. If two or more paths lead from X to Y, the engine MUST raise [`E_AMBIGUOUS_PATH`](#8-errors), even if one path is shorter. For example, if `store` relates to `date_dim` through its closing date, the sale date and the closing date give two paths from `store_sales` to `date_dim` (Appendix B.3). |
There was a problem hiding this comment.
I believe such ambiguous paths are valid from a model perspective, otherwise model validation should have warned when the model was authored.
Consider the case where there are two paths from A to D, and the query selects a metric from A and a field in D.
How would we query this in layer 3? Would there be a way to indicate preferred join path?
| { | ||
| "dimensions": ["customers.region"], | ||
| "metrics": [ | ||
| {"metric": "orders.total_amount", "alias": "order_amount"}, |
There was a problem hiding this comment.
How would we query things like total_amount in the last 7 days?
How would we query total_amount in the last 7 days, for each of the last 3 days? (i.e for each each, look back 7 days, compute aggregate and emit a row. There would be at least 3 rows in the output)
A metric of a dataset grouped by a field of a dataset whose rows reference it, such as orders.order_count by order_lines.product_id, was refused: one SELECT over the many side would count each order once per line. In a survey of 92 store-owner questions over the commerce model, the refusal hit 18 of them, and agents answered with up to 25 queries, one per value. The planner now answers it as relational_semantics.md section 8.1 defines it: a spine of the question's distinct (dimensions, key) pairs, planned as a question over the many side with its joins, FINAL and filters, joined to the metric's own rows and grouped by the dimensions, so each row of the one side counts once per group. The branch is stitched with the others, and the spine is merged with no values, so a group without a row of the one side keeps its empty-input value. Metrics of components keep the reachability rule of apache/ossie#653 section 5.3, and the linking dataset must be one the question names. Compliance cases b04, b18 and b20 pass. tests/test_bridge.py compares answers with DuckDB running the section 9 reference strategy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
This PR adds core-spec/wide_table_semantics.md, specifying Layer 3, the Wide Table Query Interface, of the layered query interface in spec_layers.md. A Layer 3 query names dimensions, metrics, and an optional filter. The engine chooses the joins, lowers the query to Layer 2 (#354), and returns the result.
This builds directly on #246 by willpugh and the expressions working group working group. #246 established the core guarantees for querying an Ossie model: preserving fact rows, avoiding double counting, combining facts across shared dimensions, and rejecting queries that cannot be evaluated safely.
This PR preserves those guarantees for a smaller initial subset, while building on the Layer 2 query semantics and existing core spec.
What carries forward from #246
Query shape (§5.1.1). Queries specify dimensions, metrics (#246's Measures), and a pre-aggregation filter (Where). Results group by all dimensions, with one row for grand totals.
Correctness guarantees (§6.1). The five user-visible guarantees carry forward as S1–S5 in this spec: fact preservation, no fan-out double counting, preservation of multi-fact groups, no unsafe re-aggregation, and no silently wrong answers. S6 adds that filtering cannot introduce new result groups.
Join and stitching semantics (§6.6, §6.8). LEFT JOIN preserves unmatched fact rows in NULL groups. Facts aggregate independently and combine on shared dimensions using NULL-safe equality. Grand totals combine through a CROSS JOIN of one-row results. Missing groups use SQL empty-input aggregate values (§6.11).
Path resolution (§6.9). Ambiguous paths raise E_AMBIGUOUS_PATH; the engine never silently chooses a path. The initial spec retains this rule for many-to-one traversal.
Filtering and errors (§6.2, Appendix A). Filters apply before aggregation to every metric. The spec retains the relevant error vocabulary and the principle of returning an error rather than an incorrect result.
Key differences from #246
Measure evaluation belongs to Layer 2. #246 defines its own fan-out and chasm-trap rewrites, including pre-aggregation (§6.7), bridge de-duplication (§6.8.1), and stitching (§6.8.2). This spec instead evaluates measures through Layer 2's AGG() (#354). Layer 3 chooses paths, joins, filter placement, and how independently aggregated results combine. It introduces no separate measure-evaluation rules.
Model definitions come from the core spec. #246 §4 defines datasets, fields, relationships, metrics, grain, namespaces, and cardinality inference. This spec reuses the existing core model, including dataset-scoped and model-scoped metrics (#591). Relationship direction establishes many-to-one cardinality without requiring primary or unique key declarations.
A smaller initial query surface. This version supports many-to-one traversal through unambiguous paths, with dimensions and filters reachable from every metric. It defers one-to-many and many-to-many traversal, including bridges; HAVING, sorting and limits; scalar queries; window functions; and more complex filtering. Consumers needing richer query behavior can use Layer 2 directly, selecting their own joins and filters while retaining measure-aware aggregation.
Intent
The goal is a minimal, interoperable Layer 3 baseline for star schemas and multiple fact tables sharing dimensions.
Unsupported or ambiguous cases raise errors rather than relying on heuristics. Appendix A identifies possible extensions, including per-metric overrides, explicit relationship selection, one-to-many traversal, and richer query operations.
These can be standardized incrementally. Their defaults must preserve the results of queries accepted by this version, following #246's additive-extension principle.
NB
The query interface and binding are intentionally non-normative, except that Layer 3 must not present itself as a SQL relation exposing measure columns or AGG(), since its result domain can depend on the requested metrics.
Related Issues
Relates to #452, #354. Derived from #246 (by wpugh & earlier working group discussions)
Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
Validation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructsTests
pytest/ CI green)Compliance
NOTICEand/orLICENSEhave been updated per ASF policyAI disclosure
Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. I have reviewed and verified every change.