[fupelaqu]

Summary

Seven Elasticsearch mapping types now have a declared SQL type, a user who may read an index but not its mapping gets a typed schema, and index patterns whose indices disagree on a field type it the way DuckDB would (SCHEMA-7).

On main, scaled_float, half_float, unsigned_long, date_nanos, ip, constant_keyword and flattened read as ANY in DESCRIBE, JDBC, Arrow and set operations, and constant_keyword answers NULL. A user with only the index read privilege (GET <index> 403, _field_caps 200) has no schema at all, and every SELECT without a LIMIT, UNION ALL or scroll of that user fails with "Failed to get ES version". A pattern field mapped keyword in one index and long in another is untyped.

What changes

• Mapping types. One reader, SQLTypes.apply, gains the new arms:

  mapping type	declared	reported	value per row
  scaled_float	DOUBLE	DOUBLE	Double, the _source value
  half_float	REAL	REAL	Float
  unsigned_long	BIGINT	BIGINT	Long; above 2^63-1 a BigInteger, never wrapped
  date_nanos	TIMESTAMP	TIMESTAMP	ZonedDateTime UTC, nanoseconds kept
  ip	KEYWORD	VARCHAR	String
  constant_keyword	KEYWORD	VARCHAR	String, read from docvalue_fields (the value is not in _source)
  flattened	ANY	ANY	as read (unchanged)

  Elasticsearch-side choices (proven HAVING key kinds, sort unmapped_type, docvalue_fields) read the column's real mapping type, never the SQL type, so typing a column does not change what is sent for it.

• constant_keyword answers its value. A statement reading such a column (or *) adds it to docvalue_fields in both bridges, on search, scroll, PIT pages, _msearch legs and the window base; the hit reader takes it from the hit's fields.

• A read-only user is typed from _field_caps. One query-only seam (IndicesApi.loadQuerySchema) fetches the mapping as today and falls back to GET <source>/_field_caps on a 403 or a pattern with no member, inside the same single-flight cache compute: the mapping is asked for at most once per source per cache TTL, never once per statement. DDL never sees a _field_caps schema (new TableType.FieldCapabilities).

• A read-only user's reads without a LIMIT answer. The scroll strategy and the slicing read the server version through the same once-per-client function the date dialect already uses: the cluster's version, else, when GET / is denied, the major the client module is built for (ES 6 and 7 modules page with search_after; ES 8 and 9 open a PIT, whose _settings lookup is denied once per index set per TTL and leaves one slice).

• Index patterns. Members are sorted by index name. A field the members disagree on takes the type of DuckDB's union lattice, as set operations do (keyword + long or keyword + date: VARCHAR; DATE + TIMESTAMP: TIMESTAMP), when every member's value reads into it; otherwise it stays ANY, as today (a refused pair, a nested result, two date formats, a member with a non-native mapping type).

• DDL round trip. ALTER TABLE … ADD COLUMN keeps every field's mapping type and parameters. A retype that changes the mapping type reindexes (ip to VARCHAR maps text; date_nanos to DATE) or is refused when it cannot be done (constant_keyword to VARCHAR). SHOW CREATE TABLE of a table holding these types re-parses and recreates the same mapping types. A type written in a column's options (OPTIONS (type = 'wildcard')) is kept, shown by DESCRIBE / SHOW CREATE, and re-sent by ALTER (main re-sent the SQL type's own Elasticsearch type, which Elasticsearch refused).

• date_nanos arithmetic. dn ± n (days), per document, keeps nanoseconds in search and transform scripts.

• Docs: documentation/client/gateway.md (mapping types, index patterns, a user who may not read the mapping), documentation/sql/known_limitations.md, documentation/sql/dql_statements.md.

Release notes

• Newly typed: the seven mapping types above in DESCRIBE, JDBC, Arrow, set-operation slots, GROUP BY keys (a half_float key is a Float), WHERE literal rules, MIN / MAX, and CTAS / materialized-view columns. constant_keyword answers its value (was NULL). Pattern fields the indices disagree on take the lattice type.
• Fixed: a user with only the index read privilege is typed, gets the privileged user's answers and the same type refusals, and its SELECT, UNION ALL and scroll without a LIMIT answer (main: "Failed to get ES version").
• Newly refused by type, with the rule and reason a native column of the same SQL type already gets (DuckDB refuses them too). Main's answers were run on ES 8: a Painless string concatenation, a string compare that misses 1.50, values of another type than the column's label, or a Painless error:
  • arithmetic on an ip or constant_keyword column (main: ip + 1 = '10.0.0.11');
  • arithmetic on a pattern field the lattice types VARCHAR, by the KEYWORD rules;
  • a CASE, COALESCE, NULLIF, GREATEST or LEAST mixing one of these columns with an arm of no common type ("Cannot mix values of type … - an explicit cast is required");
  • a comparison between one of these columns and another type family (scaled_float = keyword, ip > long, half_float > date), and date_nanos * 2, date_nanos / 2.
• Also newly refused:
  • HAVING hf = 0.1 on a half_float key: 0.1 has no exact half_float value;
  • a several-value cell of one of these types, read from the mapping, fails as it does for any typed column; declaring the column ARRAY<X> (ALTER TABLE … ALTER COLUMN … SET DATA TYPE ARRAY<X>) answers it as a list and keeps the mapping type.
• Newly failing with an Elasticsearch error: ORDER BY over an index list or pattern where one index lacks the column, when the column is a type written in DDL that reads back as its own SQL type (DOUBLE + scaled_float, TIMESTAMP + date_nanos, REAL + half_float, KEYWORD + ip or constant_keyword), or a pattern column that merges to ANY: Elasticsearch answers No mapping found for [<field>] in order to sort on (main sent an unmapped_type). A declared type over a different Elasticsearch type (KEYWORD + wildcard, INT + long) keeps main's unmapped_type.
• Newly accepted: a pattern column that merges to ANY because one index declares a DDL type beside a native field is untyped, so type rules no longer apply to it (w + 1 over a wildcard + keyword pattern is sent; main refused it).
• Read-only user residuals: _field_caps carries no _meta (a declared DATE or TIME reads TIMESTAMP, ARRAY declarations are invisible), no date format (dates are read ISO-first), reports constant_keyword as keyword (NULL as on main, now typed VARCHAR) and lists runtime fields as keyword columns (* shows them, NULL). Literals are forwarded as written. If the first GET / of a client fails for any reason, the build major stands for that client's life.
• Binary change: ColumnTypeChanged gains a second parameter list (fromMappingType, toMappingType), so its constructor, apply and copy signatures change: binary-incompatible for a caller that constructs it. Extensions, arrow and JDBC do not construct it. TableType gains a case object (additive).

Acceptance evidence

• Types (RUN on ES 8 and ES 6.8 REST): every type of the table over projection, *, WHERE literals, GROUP BY keys, HAVING key filters, MIN / MAX / SUM, ORDER BY over a pattern, arithmetic, CAST and conditionals (MappingTypeReadingSpec and its runners).
• Read-only user (RUN on ES 8, security on): JavaClientReadOnlyUserSpec — typed from _field_caps, the same answers and refusals as the privileged client except the residuals above, at most one mapping request per source per TTL, SELECT * without a LIMIT, pattern and UNION ALL answer with rows = _count and one GET / per client.
• Patterns (RESOLVE matrix + RUN): MergedSchemaSpec (every pair, member permutations, lists, formats); SELECT k FROM s7_p* over keyword "007" + long 7 answers VARCHAR "007", "7" on ES 8 and ES 6.8.
• DDL (RUN on ES 8): GET _mapping after ADD COLUMN differs by the new field only; the retypes above; SHOW CREATE TABLE re-parses and recreates the mapping types.
• Gate: a census on ES 8 against main of every type × consumer: answered → refused only for the items listed above; main's answers for the conditional and comparison cells were run on main and classified.
• Consumer shapes: JavaClientSchemaConsumerSpec reads describe and the rows of a typed index and a pattern as arrow, JDBC and extensions do (plain, scroll, UNION ALL leg).

Recorded, not fixed

• NestedElements.mappedAsNested still reads the SQL type's Elasticsearch type, not the column's mapping type (no new type maps to nested, so nothing is widened).
• SHOW CREATE TABLE prints the index settings Elasticsearch generated (uuid, creation_date, provided_name, version.created), so running its text as written fails with "unknown setting" (as on main): CREATE TABLE does not strip them, ALTER and CTAS do.
• The parser rejects dn + 1 as a function argument (CASE, COALESCE, GREATEST, CAST, DATE_TRUNC) and CASE … END IS NOT NULL in WHERE ("end of input expected"), as on main.
• A pattern field mapped keyword in one index and text in another is decided by the first text member in index-name order (documented).
• date_nanos: a shift of a function's result, per-group results and ingest processors stay in milliseconds (documented).
• COALESCE(scaled_float | half_float | unsigned_long, date) is now refused; main's answer was only rendered, not run.

Test plan

• sql/test 2292/0; core/test 1591/0; every bridge (es6, es7, es8, es9 and the shared one) green; es6 Jest, es6 REST, es7, es8 client unit specs green; + compile (2.12 and 2.13)
• ES 8.18.3: mapping types, sort and client specs 72/0 on the final code; read-only user, mapping types and consumer specs 36/0; the 13 specs of the touched area (date arithmetic and functions, grouping, scroll and sliced-scroll completeness, set operations, temporal literals, schema reading) 309/0
• ES 6.8.23 REST: mapping types, client and schema-reading specs 64/0 (6 skipped as ES-7+-only); the final sort cells 12/0
• CI lint line on the committed bytes
• Cost table, main against this branch (interleaved, warm JIT): plain SELECT, pattern schema merge, SELECT * with constant_keyword, schema load via _field_caps, GET / counts — running; posted here before merge
• CI: ES 7.17 and ES 9 legs

🤖 Generated with Claude Code

[fupelaqu]

Cost table: main (663e6df) against this branch (aaec7d3)

Measured on one machine, main and branch interleaved (main, branch, main, branch), 2 rounds each. Unit timings: JDK 11, warm-up iterations discarded, a full GC before each run, inputs built outside the timed loop, 7 timed runs per round. Elasticsearch timings: 8.18.3 with security on, a fresh container per leg, JDK 17. Medians are pooled over the 2 rounds; Δ = branch median against main median.

Summary

• Parse + resolve + render of a plain SELECT (no new type, 500 to 5,000 columns): within ±5%, one cell +16%. Resolve + render alone: +9% to +66% at 2,000 and 5,000 columns (7 to 14 µs more per statement), -22% at 500 columns (noisy); the branch's new doc-value column scan, timed alone, takes 26 to 43 µs.
• Date reads: 5 to 15% faster.
• Pattern schema merge: +28% to +41% (2,000 fields × 200 indices: 121 → 164 ms); +116% in the smallest disagreeing shape, where 6 of the branch's 14 runs contained GC pauses of 230 to 650 ms. It runs once per pattern per schema-cache lifetime; building the 200 index schemas it merges takes about 4 s on both sides in the same run (timed once, not JIT-warm).
• SELECT * over 20,000 documents (10,000 rows read): +36% with a constant_keyword column, which main does not return at all and the branch now reads; +5% without one.
• A user with only the index read privilege: requests as intended (below). On main, that user's un-LIMITed statements fail; on the branch they answer.
• Not attributed yet: the privileged user's statement wall time on Elasticsearch is +31% to +52% with identical requests. Those cells ran each statement shape at most 18 times in a fresh JVM, so they are not a warm-JIT measurement. A re-run with a 200-statement warm-up per shape is running; its result will follow in this PR.

Plain SELECT, resolve and render (unit)

measurement	parameters	unit	main median [min–max]	branch median [min–max]	Δ
resolve + render	cols=2000, stmt=named	us	55.8 [48.9–60.4]	62.2 [52.7–76.8]	+11%
resolve + render	cols=2000, stmt=star	us	14.9 [12.3–17.2]	21.7 [18.7–31.0]	+46%
resolve + render	cols=5000, stmt=named	us	90.7 [74.1–127]	98.7 [94.1–116]	+9%
resolve + render	cols=5000, stmt=star	us	21.0 [17.6–27.5]	34.8 [33.1–37.4]	+66%
resolve + render	cols=500, stmt=named	us	99.7 [49.1–158]	77.8 [51.8–179]	-22%
resolve + render	cols=500, stmt=star	us	91.8 [56.0–363]	69.4 [33.4–242]	-24%
doc-value column scan (branch only)	cols=2000, stmt=named	ns	–	27,191 [21,407–61,256]	branch only
doc-value column scan (branch only)	cols=2000, stmt=star	ns	–	33,686 [20,160–67,099]	branch only
doc-value column scan (branch only)	cols=5000, stmt=named	ns	–	39,905 [33,040–44,731]	branch only
doc-value column scan (branch only)	cols=5000, stmt=star	ns	–	28,767 [26,454–35,075]	branch only
doc-value column scan (branch only)	cols=500, stmt=named	ns	–	26,247 [19,338–1,225,920]	branch only
doc-value column scan (branch only)	cols=500, stmt=star	ns	–	42,557 [20,492–155,338]	branch only
parse + resolve + render	cols=2000, stmt=named	us	932 [872–1,016]	961 [917–1,065]	+3%
parse + resolve + render	cols=2000, stmt=star	us	162 [134–243]	157 [143–202]	-3%
parse + resolve + render	cols=5000, stmt=named	us	984 [939–1,138]	1,013 [963–1,038]	+3%
parse + resolve + render	cols=5000, stmt=star	us	136 [131–155]	157 [145–168]	+16%
parse + resolve + render	cols=500, stmt=named	us	1,307 [1,040–1,571]	1,244 [1,048–1,519]	-5%
parse + resolve + render	cols=500, stmt=star	us	336 [243–542]	325 [203–606]	-3%

Pattern schema merge (unit; disagree1 = one field mapped long in half the indices and keyword in the other half)

measurement	parameters	unit	main median [min–max]	branch median [min–max]	Δ
pattern schema merge	fields=1000, members=100, variant=agree	ms	52.6 [45.9–103]	71.4 [66.0–86.9]	+36%
pattern schema merge	fields=1000, members=100, variant=disagree1	ms	56.9 [52.0–113]	72.6 [63.0–83.3]	+28%
pattern schema merge	fields=2000, members=200, variant=agree	ms	121 [105–161]	164 [155–210]	+36%
pattern schema merge	fields=2000, members=200, variant=disagree1	ms	119 [115–128]	162 [150–177]	+36%
pattern schema merge	fields=500, members=30, variant=agree	ms	48.1 [35.4–83.5]	67.6 [49.8–1,143]	+41%
pattern schema merge	fields=500, members=30, variant=disagree1	ms	41.4 [34.1–49.3]	89.6 [32.9–794]	+116%

Date reads (unit)

measurement	parameters	unit	main median [min–max]	branch median [min–max]	Δ
hit rows, field-caps schema (branch only)	hits=20000, version=8	us	–	0.772 [0.688–1.21]	branch only
hit rows, mapped dates	hits=20000, version=8.18.3	us	0.875 [0.759–0.981]	0.827 [0.787–0.937]	-5%
hit rows, untyped	hits=20000, version=8	us	2.57 [2.26–2.90]	2.27 [1.98–2.50]	-12%
one value, date read	values=20000	ns	121 [90.1–341]	103 [83.0–267]	-15%
one value, untyped read	values=20000	ns	1,926 [1,759–2,203]	1,721 [1,541–2,658]	-11%

Elasticsearch 8: SELECT * and schema loads

measurement	parameters	unit	main median [min–max]	branch median [min–max]	Δ
SELECT * full read	index=ac8_ck, docs=20000, client=admin	ms	75.8 [40.6–117]	103 [76.8–170]	+36%
SELECT * full read	index=ac8_nock, docs=20000, client=admin	ms	76.1 [41.3–155]	80.0 [57.0–138]	+5%
schema load, cold	client=admin, source=ac8_p*, loader=loadPatternSchema	ms	2.33 [1.38–3.65]	3.12 [2.54–4.06]	+34%
schema load, cold	client=admin, source=ac8_p*, loader=loadQuerySchema	ms	–	1.28 [1.02–1.53]	branch only
schema load, cold	client=admin, source=ac8_t, loader=loadQuerySchema	ms	–	1.33 [1.08–1.68]	branch only
schema load, cold	client=admin, source=ac8_t, loader=loadSchema	ms	2.73 [1.60–4.50]	3.50 [2.60–27.7]	+28%
schema load, cold	client=reader, source=ac8_p*, loader=loadQuerySchema	ms	–	2.26 [1.94–2.55]	branch only
schema load, cold	client=reader, source=ac8_t, loader=loadQuerySchema	ms	–	2.60 [2.11–2.88]	branch only
schema load, cold	client=reader, source=ac8_t, loader=loadSchema	ms	1.25 [0.800–26.4]	1.62 [1.24–2.33]	+30%
schema load, pattern with no index	client=reader, source=ac8_nomatch*, loader=loadQuerySchema	ms	–	1.79 [1.44–1.88]	branch only
schema load, warm (cache hit)	client=admin, source=ac8_p*, loader=loadPatternSchema	us	0.881 [0.423–1.72]	1.20 [0.698–2.27]	+36%
schema load, warm (cache hit)	client=admin, source=ac8_p*, loader=loadQuerySchema	us	–	0.295 [0.219–0.341]	branch only
schema load, warm (cache hit)	client=admin, source=ac8_t, loader=loadQuerySchema	us	–	0.344 [0.297–0.469]	branch only
schema load, warm (cache hit)	client=admin, source=ac8_t, loader=loadSchema	us	1.98 [1.11–4.26]	2.19 [1.47–3.00]	+10%
schema load, warm (cache hit)	client=reader, source=ac8_p*, loader=loadQuerySchema	us	–	0.310 [0.218–0.513]	branch only
schema load, warm (cache hit)	client=reader, source=ac8_t, loader=loadQuerySchema	us	–	0.801 [0.508–1.23]	branch only
schema load, warm (cache hit)	client=reader, source=ac8_t, loader=loadSchema	us	860 [511–1,347]	1,038 [700–2,216]	+21%

Elasticsearch 8: statement wall time (not JIT-warm: see the summary)

single = SELECT id, k FROM ac8_t LIMIT 5, pattern = SELECT id FROM ac8_p* LIMIT 5, nomatch = SELECT * FROM ac8_nomatch* LIMIT 5 (no index), unlimited = SELECT id, k FROM ac8_t. cold = schema cache cleared just before. For reader, main has no schema for that user, and its unlimited statement fails ("Failed to get ES version"): main's time there is the time to fail.

measurement	parameters	unit	main median [min–max]	branch median [min–max]	Δ
statement wall time	client=admin, stmt=nomatch, phase=cold	ms	2.60 [1.98–3.34]	5.13 [4.51–7.10]	+98%
statement wall time	client=admin, stmt=nomatch, phase=warm	ms	1.73 [1.27–2.16]	2.56 [2.27–3.06]	+48%
statement wall time	client=admin, stmt=pattern, phase=cold	ms	4.80 [3.85–6.73]	6.73 [6.09–8.02]	+40%
statement wall time	client=admin, stmt=pattern, phase=warm	ms	2.85 [2.34–3.70]	4.23 [3.64–5.55]	+48%
statement wall time	client=admin, stmt=single, phase=cold	ms	5.47 [4.14–7.31]	8.25 [7.08–15.4]	+51%
statement wall time	client=admin, stmt=single, phase=warm	ms	3.48 [2.44–4.22]	5.28 [4.45–8.23]	+52%
statement wall time	client=admin, stmt=unlimited, phase=cold	ms	9.68 [7.52–13.7]	13.1 [11.5–15.6]	+35%
statement wall time	client=admin, stmt=unlimited, phase=warm	ms	6.99 [5.10–12.6]	9.17 [8.21–11.0]	+31%
statement wall time	client=reader, stmt=nomatch, phase=cold	ms	2.29 [1.75–2.61]	3.97 [3.63–4.40]	+74%
statement wall time	client=reader, stmt=nomatch, phase=warm	ms	2.21 [1.73–2.92]	1.83 [1.72–2.01]	-17%
statement wall time	client=reader, stmt=pattern, phase=cold	ms	3.29 [2.50–4.04]	7.30 [5.96–11.8]	+122%
statement wall time	client=reader, stmt=pattern, phase=warm	ms	3.18 [2.42–3.94]	3.25 [2.75–4.26]	+2%
statement wall time	client=reader, stmt=single, phase=cold	ms	3.22 [2.44–4.39]	7.40 [6.40–8.53]	+130%
statement wall time	client=reader, stmt=single, phase=warm	ms	3.11 [2.50–3.73]	3.59 [3.11–6.15]	+16%
statement wall time	client=reader, stmt=unlimited, phase=cold	ms	2.28 [1.79–3.82]	11.2 [9.94–14.5]	+393%
statement wall time	client=reader, stmt=unlimited, phase=warm	ms	2.22 [1.81–5.50]	6.82 [6.18–7.56]	+208%

Requests

Every request count is the same on both sides for the privileged client. For the read-only user:

• on main, every statement re-sends the denied GET <index>, and every un-LIMITed statement sends a GET / (which fails it): 6 GET / after 6 un-LIMITed statements;
• on the branch, the schema comes from one _field_caps per source per cache lifetime, GET / is sent once per client, and the first un-LIMITed statement on an index set adds one _settings and one shard lookup per cache lifetime.

🤖 Generated with Claude Code

[fupelaqu]

Statement wall time with a warm JIT (follow-up to the cost table)

The statement rows of the cost table were not JIT-warm (each shape ran at most 18 times in a fresh JVM). Re-measured: each client first runs every statement shape 200 times, round-robin, with its schema cache warm, as a running application would; then 30 timed round-robin runs per shape. Main and branch interleaved, 2 rounds each (n = 60 per side), Elasticsearch 8.18.3 with security on.

client	statement	main median [p10–p90] ms	branch median [p10–p90] ms	Δ
admin	single	1.53 [1.37–1.80]	1.53 [1.35–1.71]	-0%
admin	pattern	1.47 [1.29–1.64]	1.39 [1.19–1.65]	-6%
admin	nomatch	0.96 [0.83–1.12]	0.96 [0.81–1.13]	-0%
admin	unlimited	3.90 [3.45–4.42]	3.73 [3.29–4.31]	-4%
read-only	single	1.61 [1.43–2.04]	1.30 [1.04–1.71]	-19%
read-only	pattern	1.56 [1.40–1.85]	1.26 [0.96–1.52]	-19%
read-only	nomatch	1.18 [1.04–1.45]	0.85 [0.65–1.04]	-28%
read-only	unlimited	1.30 [1.13–1.53] — fails	3.40 [2.63–4.51] — answers	not comparable

• Privileged client: no difference once the JIT is warm (both rounds agree, identical requests: none beyond the searches). The +31% to +52% of the first table was warm-up on a longer code path.
• Read-only user: faster on the branch. Over the 120 timed statements, main sent 120 denied GET <index> requests (the schema is asked for on every statement) and 30 GET / (one per un-LIMITed statement, each failing it with "Failed to get ES version"); the branch sent none.
• Statements: single = SELECT id, k FROM ac8_t LIMIT 5, pattern = SELECT id FROM ac8_p* LIMIT 5, nomatch = SELECT * FROM ac8_nomatch* LIMIT 5 (no index), unlimited = SELECT id, k FROM ac8_t.

🤖 Generated with Claude Code