Skip to content

[Performance] Reduce aligned flush duplicate timestamp overhead - #18724

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex/aligned-flush-null-check
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex/aligned-flush-null-check

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Reduce duplicate timestamp flush overhead

  • Cache the unsorted value index once per row instead of resolving it repeatedly.
  • Reuse the null check already performed while selecting the last non-null value for a duplicate timestamp.
  • Replace per-column Pair<Long, Integer> state with primitive arrays, avoiding boxing in the hot loop.
  • Apply the same logic to both the normal aligned flush path and the deleted-measurement mapping path.

The duplicate timestamp merge behavior is unchanged: each value column keeps the last non-null value at a timestamp.

Tests

  • Added correctness coverage for duplicate timestamps with the full measurement set.

  • Added correctness coverage after deleting a measurement, exercising column remapping.

  • Added an opt-in manual performance UT. It stays skipped during normal test runs and can be enabled with:

    mvn test -pl iotdb-core/datanode -Dtest=AlignedFlushDuplicateTimeCheckPerformanceTest -Diotdb.aligned.flush.duplicate-check.perf.enabled=true

Benchmark workload: 16,384 rows, 8 aligned columns, four rows per duplicated timestamp, 128 scans per round, five rounds.

Implementation CPU per scan Allocated per scan
Legacy 1.099 ms 3,846,576 bytes
Optimized 0.366 ms 128 bytes

The optimized loop used about 33.33% of the legacy CPU time in this targeted benchmark.

Verification:

mvn -nsu test -pl iotdb-core/datanode '-Dtest=MemTableFlushTaskTest,AlignedFlushDuplicateTimeCheckPerformanceTest'

13 tests ran: 12 passed and the opt-in performance UT was skipped as expected.


This PR has:

  • been self-reviewed.
  • added comments explaining the intent where the optimized duplicate merge is non-obvious.
  • added or updated unit tests to cover the changed code paths.

Key changed/added classes in this PR
  • AlignedWritableMemChunk
  • MemTableFlushTaskTest
  • AlignedFlushDuplicateTimeCheckPerformanceTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants