Skip to content

Preserve a ref's retention policy when its snapshot moves - #3770

Closed
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:preserve-branch-retention
Closed

1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:preserve-branch-retention

Conversation

@1fanwang

@1fanwang 1fanwang commented Aug 8, 2026 •

Copy link
Copy Markdown

Rationale for this change

Refs can be created with retention settings, but two move paths rebuild the ref without carrying those settings forward: branch writes and current-snapshot moves. The next metadata update then clears max ref age, max snapshot age, and min snapshots to keep.

Fast-forward already preserves the policy. This makes branch writes, set-current-snapshot, and rollback do the same. Refs created without a policy still stay unset.

Are these changes tested?

Targeted unit regression tests fail on current main and pass with this branch. Branch write also passes against the REST catalog.

Raw logs
$ F=pyiceberg/table/update/snapshot.py
$ T=tests/table/test_manage_snapshots.py
$ git checkout origin/main -- "$F"
$ uv run pytest "$T" -k retention -q
..FFF...FFF...                                                           [100%]
E       assert None == 86400000
E       assert None == 99999
6 failed, 8 passed, 20 deselected in 4.99s

$ git checkout HEAD -- "$F"
$ uv run pytest "$T" -q
..................................                                       [100%]
34 passed in 1.20s

$ make test-integration-setup
$ I=tests/integration/test_writes/test_writes.py
$ uv run pytest "$I" -m integration -k retention -q
.                                                                        [100%]
1 passed, 123 deselected in 6.74s
$ make test-integration-cleanup

Are there any user-facing changes?

A ref keeps the retention policy it was created with when its snapshot moves. There is no API change.

AI Disclosure

AI disclosure: GitHub Copilot CLI (GPT-6 Astra) helped rebase the stale PR, resolve the current main conflict, refresh the PR description, and rerun the tests.

Copilot AI lite review requested due to automatic review settings August 8, 2026 04:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a retention-policy regression in the snapshot commit/write path: when writing a new snapshot to an existing branch, the branch’s retention configuration (max_ref_age_ms, max_snapshot_age_ms, min_snapshots_to_keep) is now preserved instead of being dropped by the emitted set-snapshot-ref update.

Changes:

  • Preserve existing branch retention fields when _SnapshotProducer._commit() updates a branch ref during writes.
  • Add unit tests validating retention is preserved across multiple writes and remains unset when not configured.
  • Add an integration test covering the REST catalog scenario where branch retention previously regressed.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pyiceberg/table/update/snapshot.py Carries existing ref retention fields into the SetSnapshotRefUpdate emitted during snapshot commits to a branch.
tests/table/test_manage_snapshots.py Adds unit coverage to ensure branch retention survives repeated writes and remains unset if not configured.
tests/integration/test_writes/test_writes.py Adds an integration regression test for preserving branch retention on repeated writes via REST catalog.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

create_branch() accepts max_ref_age_ms, max_snapshot_age_ms and
min_snapshots_to_keep, but any later operation that moves the ref drops all
three. Both paths that move a ref rebuild it from a SetSnapshotRefUpdate that
omits the retention fields, so the applied ref falls back to the defaults:

  - writing to a branch, via _SnapshotProducer._commit()
  - set_current_snapshot(), and the rollbacks that delegate to it

A branch created with a retention policy therefore loses it on its first write,
which is the point at which the policy starts to matter.

Carry the existing ref's retention fields through on both paths, mirroring
Java's TableMetadata.Builder.setBranchSnapshotInternal, which rebuilds the moved
ref with SnapshotRef.builderFrom(ref, replacementSnapshotId). create_tag() and
create_branch() still set the fields explicitly, so a ref with no policy does
not acquire one.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang force-pushed the preserve-branch-retention branch from 198ed00 to 4f48ea4 Compare August 8, 2026 06:41
@1fanwang 1fanwang changed the title Preserve branch retention policy when writing to a branch Preserve a ref's retention policy when its snapshot moves Aug 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Sep 8, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants