Skip to content

feat(global-index): support checkpoints for Lumina index builds - #382

Open
lszskye wants to merge 4 commits into
apache:mainfrom
lszskye:lumina_checkpoint
Open

lszskye wants to merge 4 commits into
apache:mainfrom
lszskye:lumina_checkpoint

Conversation

@lszskye

@lszskye lszskye commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Add checkpoint support for Lumina global index builds.

This change:

  • introduces GlobalIndexCheckpointFileManager as a plugin-facing checkpoint storage interface;

  • lets GlobalIndexFileManager optionally manage checkpoint scanning and I/O;

  • adds task-scoped checkpoint paths in the following form:
    [index_type]-global-index-[field_name]-[range_from]-[range_to]-[task_id]-[id].index.ckpt

  • discovers the latest checkpoint using its numeric ID and continues allocating IDs from it;

  • adds GlobalIndexer::SupportsCheckpoint() so checkpoint storage is created only for index implementations that support it;

  • adds an optional task_id to GlobalIndexWriteTask::WriteIndex;

  • adapts Paimon checkpoint storage to Lumina's checkpoint manager;

  • restores Lumina builds from the latest matching checkpoint;

  • deletes matching checkpoints after a successful build;

  • when a build using an existing checkpoint fails, deletes the checkpoint and retries once from scratch;

  • isolates checkpoints by index type, field, row range, and task ID.

Index implementations that do not support checkpoints keep their existing behavior.

Tests

src/paimon/core/global_index/global_index_file_manager_test.cpp
src/paimon/core/utils/file_store_path_factory_test.cpp
src/paimon/global_index/lumina/lumina_global_index_test.cpp
test/inte/global_index_test.cpp

API and Format

  • adds GlobalIndexer::SupportsCheckpoint(), which returns false by default;
  • adds GlobalIndexCheckpointFileManager;
  • adds an optional task_id argument to GlobalIndexWriteTask::WriteIndex.

@lszskye
lszskye requested a review from lxy-9602 September 23, 2026 02:33
Comment thread include/paimon/global_index/global_index_write_task.h
Comment thread src/paimon/core/global_index/global_index_file_manager.h
Comment thread src/paimon/core/global_index/global_index_file_manager.h Outdated
Comment thread src/paimon/core/global_index/global_index_file_manager.h Outdated
Comment thread src/paimon/core/global_index/global_index_write_task.cpp
Comment thread src/paimon/global_index/lumina/lumina_checkpoint_manager.cpp
Comment thread src/paimon/global_index/lumina/lumina_global_index.cpp
Comment thread src/paimon/global_index/lumina/lumina_global_index.cpp
Comment thread src/paimon/global_index/lumina/lumina_global_index.cpp
lxy-9602
lxy-9602 previously approved these changes Sep 23, 2026

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants