[kayemkim]

Summary

A dialect registered in extensions/dialects.md also has to be listed in validate.py, in SQLGLOT_DIALECTS with the reader that parses it or in NON_SQL_DIALECTS, or the validator warns about a token the registry says is fine. The same goes for the SDK enums, which #617 kept as the well-known values: OssieDialect should carry what dialects.md registers and OssieVendor what vendors.md does. Nothing checked either, and the vendor side already disagreed. OssieVendor had SEMANTIDO, which #207 added to the enum only, and lacked HONEYDEW, POWER_BI and THOUGHTSPOT, which the spec table had before #617 moved it.

Four commits:

• validation: the union of the two sets equals the dialects.md table, the sets are disjoint, only ANSI_SQL and OSSIE_SQL_2026 use sqlglot's default reader, and every reader resolves through sqlglot.Dialect.get_or_raise, so a misspelled one fails here rather than as a ValueError on the first document in that dialect. With this in place the validator's unsupported-dialect warning only ever names a token that is not in the registry, and the comment on that warning now says so. The assertion messages name the files to edit.
• python: add the three vendors to OssieVendor and a SEMANTIDO row to vendors.md.
• python tests: OssieDialect and OssieVendor against the two tables.
• ci: extensions/** added to the path triggers of validation-ci and python-ci, so a change to a table alone runs both suites.

This makes registering a dialect a three-file change: the table, validate.py and OssieDialect. That is the point of the check, since #558, #598 and #607 each missed one of the places that existed then. If the registry should stay free of the validator, the direction that can go is registry to code; the other one, that everything the code names is registered, holds either way.

The first revision pinned the tables to the schema enum. With #617 the enum is gone and the registry carries the check, as said on the list. The Holistics half left with the converters (#523).

Verification

• validation/tests/: 274 passed, three new (main: 271). python/tests/: 268 passed, two new (main: 266). generate-spec-types.py --check clean.
• Mutations tried in place, each failing only the assertion meant for it, with the set difference naming the token: a row added to dialects.md only, HOLISTICS_AQL dropped from NON_SQL_DIALECTS, DUCKDB added to SQLGLOT_DIALECTS only, MDX in both sets, bigqeury, ANSI_SQL given a real reader, a row dropped from vendors.md, HONEYDEW dropped from the enum, a padded table row (still read). On main's enum and table the vendor pin reports SEMANTIDO on one side and HONEYDEW, POWER_BI and THOUGHTSPOT on the other.

Related Issues

Discussion on dev@: https://lists.apache.org/thread/0tql9g8myg255hhonhbzo906rdpgstcf. Follows #617.

Checklist

Validation

• New validation cases are covered by tests

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval

[jbonofre]

The docstring says SQLGLOT_DIALECT mirrors DIALECT_MAP and NOT_SQL mirrors the skip list, but nothing checks that. Each side is compared only to the schema enum, so the two can disagree and every test still passes. For example, changing "DATABRICKS": "databricks" to "spark" in sqlrefs.py passes all three tests here.
Could we compare them directly?

from importlib.util import module_from_spec, spec_from_file_location

import pytest


def _validate_module():
    # validate.py exits at import time without jsonschema.
    pytest.importorskip("jsonschema")
    path = snapshots.REPO_ROOT / "validation" / "validate.py"
    spec = spec_from_file_location("ossie_validate", path)
    assert spec is not None and spec.loader is not None
    module = module_from_spec(spec)
    spec.loader.exec_module(module)
    return module


def test_the_tables_match_validate_py() -> None:
    validate = _validate_module()

    assert sqlrefs.NOT_SQL == validate.SKIP_SQL_VALIDATION
    assert sqlrefs.SQLGLOT_DIALECT == {
        d: reader
        for d, reader in validate.DIALECT_MAP.items()
        if d not in validate.SKIP_SQL_VALIDATION
    }

Also, converter-holistics-ci.yml doesn't run on changes under validation/**, so a change to DIALECT_MAP alone would not run this test.

That said, as the converters will move, it's not a big deal 😄

[kayemkim]

Compared directly now in ecc467b, with validate.py loaded the way its own tests do.

[jbonofre]

This checks whether each reader is None, but not that a non-None reader is a real sqlglot dialect.
A typo like "bigqeury" passes. Then sqlglot.parse_one raises ValueError: Unknown dialect 'bigqeury', which validate_sql_expression doesn't catch, because it only catches ParseError, TokenError and RecursionError.

Could we add a separate test? Keeping it separate means the enum check still runs when sqlglot isn't installed:

def test_every_sqlglot_reader_exists() -> None:
    sqlglot = pytest.importorskip("sqlglot")

    for reader in _VALIDATE.DIALECT_MAP.values():
        if reader is not None:
            sqlglot.Dialect.get_or_raise(reader)

[kayemkim]

Added as a separate test in ecc467b, behind importorskip so the enum check keeps running without sqlglot.

[jbonofre]

The Holistics side has the same gap: a misspelled reader on SQLGLOT_DIALECT passes all three tests. sqlrefs.columns catches only ParseError, TokenError and RecursionError, so the ValueError isn't turned into UnparseableSQL.

The conversion then crashes instead of reporting an issue. sqlglot is a runtime dependency here, so a plain import works:

import sqlglot


def test_every_reader_is_a_sqlglot_dialect() -> None:
    for reader in sqlrefs.SQLGLOT_DIALECT.values():
        if reader is not None:
            sqlglot.Dialect.get_or_raise(reader)

[kayemkim]

Added in ecc467b with a plain import, since sqlglot is a runtime dependency here.

[kayemkim]

Pushed ecc467b with the three. The Holistics test now compares NOT_SQL and SQLGLOT_DIALECT with SKIP_SQL_VALIDATION and DIALECT_MAP directly, loading validate.py the way its own tests do, and both sides check every reader with Dialect.get_or_raise. The validate.py one is a separate test behind importorskip("sqlglot") so the enum check still runs without it. I tried the cases you named on this branch: "spark" for DATABRICKS, bigqeury on either side and a dropped DAX skip entry each fail now, and both suites pass.

On the validation/** trigger, agreed it can wait for the move. I'll leave the notes on #558 and #607 and open the CONTRIBUTING.md paragraph as its own PR.

[christianeu-db]

FYI: @kayemkim I put together a PR here / linked thread, which removes the validation of enums / puts the dialect table more as a community extension table rather than a core part of the spec.

Could be nice to coordinate these approaches (e.g. I think it makes sense to have a table-derived set of constants but perhaps outside the core spec

[kayemkim]

Thanks for the pointer. I read #617 and ran this branch on top of it: the two tests that read the schema enum fail with KeyError, the table cross-check and the reader tests still pass. If the enum goes, I would point those two at the table in extensions/dialects.md instead, so the registry carries the check. I put the rest on the list thread and on #617. Whichever lands first, I will do the rebase.

[jbonofre]

The converters moved to apache/ossie-converters repository, so converters/ no longer exists on main.

Could you rebase from main and drop this file?

[kayemkim]

Rebased on main after #523 and dropped the Holistics file. The same two tables now live in ossie-converters, and I will pin them there once that repository carries the schema. What stays is the validate.py side: DIALECT_MAP and the skip list against the enum, plus the check that every reader names a sqlglot dialect. 249 passed locally.

Once #617 is in, the enum comparison moves to the table in extensions/dialects.md and the two renamed sets, as said on the list.

[kayemkim]

Rebased on main with #617 in. The schema enum is gone, so the two tests now compare SQLGLOT_DIALECTS and NON_SQL_DIALECTS with the table in extensions/dialects.md, next to the default-reader and reader-exists checks. The vendor pin I mentioned on #617 is here too: OssieVendor gains HONEYDEW, POWER_BI and THOUGHTSPOT, vendors.md gains SEMANTIDO, and both SDK enums are compared with their tables. A registry-only change would not have run either suite, so extensions/** is added to the validation-ci and python-ci triggers. Registering a dialect is now a three-file change, table plus validate.py plus OssieDialect, which is what the thread asked for; if you would rather keep the registry free of the validator, the registry-to-code direction is the one to drop. 274 and 268 passed locally, 33/33 across the matrix.

[christianeu-db]

thanks for this - it will make registering new tokens safer.

@jbonofre please take a look & remove the change request flag if you agree

[christianeu-db]

(non-blocking) it might be easier for people adding / modifying the dialects/extensions if we printed the diffs - could be a followup pr

[kayemkim]

Done in #670: the message now names the file each missing token belongs in.

[kayemkim]

@jbonofre no rush on this one. When you get back to it: with #598 approved and #558 and #607 waiting on a rebase onto #617, landing this before them would let their CI check that each registers the dialect in all three places.

[jbonofre]

Thanks @kayemkim! It looks good to me now 😄

Both of my earlier points are addressed: the sqlglot reader check is a separate test now, and the converters/ file is gone after the rebase. Great one!