Skip to content

Add LTTB downsampling as a built-in Table Model table function - #18725

Open
DaZuiZui wants to merge 1 commit into
apache:masterfrom
DaZuiZui:feat/lttb-table-function
Open

DaZuiZui wants to merge 1 commit into
apache:masterfrom
DaZuiZui:feat/lttb-table-function

Conversation

@DaZuiZui

Copy link
Copy Markdown
Contributor

Description

Closes #18532

Adds LTTB (Largest-Triangle-Three-Buckets) as a built-in Table Model table function for visualization-oriented downsampling, following the conventions of the existing M4 table function.

Function definition

-- Target-count mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time', N => 500);

-- Count-window mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time', SIZE => 100, SLIDE => 100);

-- Time-window mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time',
                   SIZE => 1m, SLIDE => 1m, ORIGIN => 2026-01-01T00:00:00.000+00:00);
  • DATA (set semantics) / TIMECOL / SIZE / SLIDE / ORIGIN reuse the M4 argument semantics and window construction (half-open [start, start+SIZE) windows, ORIGIN alignment, count windows for integer SIZE).
  • N selects target-count mode: the whole partition is buffered and standard LTTB reduces each participant column to exactly N points (all points when M <= N), always preserving the first and last eligible points. Buckets follow [floor(i*(M-2)/(N-2))+1, floor((i+1)*(M-2)/(N-2))+1); the final bucket uses the last point as vertex C. Output uses window_index = 0.
  • SIZE selects window/bucket mode: each window is one LTTB bucket. Per participant column the anchor is chained across windows (the first eligible point seeds it), vertex C is the average point of the next window that carries data for the column (or the window's own last eligible point when there is none), and empty buckets leave the anchor untouched.
  • Participant columns are inferred automatically (INT32/INT64/FLOAT/DOUBLE; other types are rejected with the shared "not allowed columns" message). NULL/non-finite values are ignored per column; columns with no eligible points emit NULL. Area ties resolve to the earliest timestamp (stable input order). Integral columns keep exact long values in the output.
  • Output schema follows M4: window_start, window_end | window_index, partition columns, then <col>_time, <col> per participant column, aligned by output position and padded with NULL.

Validation

Rejected at analysis time: neither/both of N and SIZE; N < 3; N given as a duration; N with SLIDE/ORIGIN; SLIDE/ORIGIN without SIZE; ORIGIN in count-window mode; SLIDE mode differing from SIZE; descending ORDER BY; non-TIMESTAMP TIMECOL; unsupported participant types. All new messages are i18n constants in both en and zh CommonMessages.

Execution and distribution

  • TableDistributedPlanGenerator.canSplitTableFunctionProcessor now excludes LTTB (like FFT), so the whole partition is gathered and ordered (MergeSort over exchanges) before the function runs — fragment-local LTTB samples are not mergeable.
  • Target-count mode buffers primitives per column with a hard cap (MAX_BUFFERED_POINTS), mirroring the row caps used by LOWPASS/XCORR; window mode keeps bounded state (current/next windows plus per-column anchor).

Alternatives considered

  • Fully independent windows in SIZE mode (no chained anchor) would simplify overlapping windows but diverges from the issue's "previously selected point / next-bucket average" definition; chained anchors were chosen.
  • Reusing M4TableFunction's ValueOperator was rejected because LTTB requires numeric values for triangle areas.

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths (LTTBTableFunctionTest, TableFunctionTest#testLTTB* — verified under both en and zh locales).
  • added integration tests (IoTDBLTTBTableFunctionIT, passes with -PTableSimpleIT).

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.commons.udf.builtin.relational.tvf.LTTBTableFunction (new)
  • org.apache.iotdb.commons.queryengine.plan.relational.function.TableBuiltinTableFunction
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.StatementAnalyzer (tryAppendLTTBModeArgument)
  • org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlanner
  • org.apache.iotdb.db.queryengine.plan.relational.planner.distribute.TableDistributedPlanGenerator
  • org.apache.iotdb.commons.i18n.CommonMessages (en/zh)

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature request] Add LTTB downsampling as a built-in table function for the Table Model

1 participant