CAS: refuse new nested server_root_id, delete S3 staging on abort, actionable member refusals

[filimonov]

A cas_server_root_id may no longer contain /, for new and existing members alike. Decommission, staging and the GC floor lookup select a member's objects by path prefix, so decommissioning a also removes the namespaces, staging and mountpoint objects of a member a/b. A server whose id contains / refuses to open the disk and must be recreated under a new id.

Changes

• validateServerRootId refuses / with BAD_ARGUMENTS. It runs at disk config read, in Pool::open and in SYSTEM CAS DROP POOL MEMBER / clickhouse-disks cas-drop-member, so a nested id can neither mount nor be decommissioned. The message tells the operator to recreate the server under a new <cas_server_root_id> without /.
• An aborted transaction deletes its S3 staging object, best-effort, as a commit does. Before, aborted staging stayed until the next mount of the same server_root_id.
• The bucket requirements document the AbortIncompleteMultipartUpload lifecycle rule: an upload killed mid-way is invisible to object LIST and is never swept.
• The owner and decommission refusals name the recovery that works:
  • retired owner: configure a new cas_server_root_id; do not clear the tombstone;
  • foreign owner: restore the uuid file or start under a new id; decommission the old id once its slot is terminated or fenced;
  • SYSTEM CAS DROP POOL MEMBER on a hard-killed member: wait until the disk's GC leader fences it, about one lease TTL plus one renew period after it first saw the lease; SYSTEM CAS GC RUN on the leader helps, but two back-to-back runs do not fence.

Risks / notes

• Backward incompatible: a server whose cas_server_root_id contains / (for example from a {shard}/{replica} template) refuses to open the disk with BAD_ARGUMENTS. Recreate it under a new id without /, such as {shard}-{replica}. Data stored under the old id is not visible under the new id; the old id's objects stay in the pool and keep their blobs referenced, and SYSTEM CAS DROP POOL MEMBER refuses the old id, so it cannot reclaim them.
• Decommissioning a still selects the objects of a legacy a/b left in the pool by an earlier build. Such a member can no longer mount.

Cleaning up a legacy id with / (example id a/b)

Before upgrading, on 26.6.4 (preferred):

1. Stop the server whose id is a/b. If it must stay in the cluster, first start a replacement under an id without / and let it replicate.
2. From another member, run SYSTEM CAS DROP POOL MEMBER 'a/b' FROM DISK '<disk>'. 26.6.4 accepts / in the id. A live lease is refused; a hard-killed member must be fenced first, as described above.
3. Rerun until it returns slot_removed = true and empty warnings. The warning namespace(s) are still owned by this member means GC rounds must run first.
4. Upgrade.

Dropping a/b selects keys under a/b/ only, never a/bc or a, but it does select a deeper a/b/c. Never drop a parent a while a/b must survive: both versions select a/b's objects for a.

After upgrading, if the steps above were missed:

1. Delete nothing by hand.
   • cas/ref_catalog is one object for the whole pool, updated by compare-and-swap by writers and GC. Its a/b/... rows cannot be removed by hand.
   • cas/ns/stream/<life>/ and cas/ns/state/<life>/ are keyed by life id, not by server_root_id. Only the catalog maps them to a/b.
   • gc/server-roots/a/b/ (owner, epoch, mount) is the slot that decommission claims. Without owner and mount it refuses with unknown pool member. The manifest sweep reads its floor from mount.
   • staging/a/b/, roots/a/b/ and eligible cas/manifests/a/b/ debris are drained by cas-drop-member itself.
   • blobs/: GC reclaims a blob once no namespace references it.
2. Keep the a/b server stopped. Run the 26.6.4 offline tool against the pool from a config whose CAS disk points at the same pool with <readonly>true</readonly>:
   clickhouse-disks -C config.xml --disk cas cas-drop-member 'a/b'
3. Rerun it after GC rounds until it prints slot_removed=true and no warning= lines. The upgraded servers' GC deletes the removed catalog rows and then the unreferenced blobs.

Not verified: that a 26.6.4 binary reads every object this build writes. No test runs a 26.6.4 cas-drop-member against a pool written by this build.

Testing

• New gtests (fail on the base): an existing member with a nested id refuses to open, without claiming the mount or minting an epoch; decommission of a nested id is refused and leaves its owner object untouched; abort deletes S3 staging; the refusal texts. Guards: system.cas_mounts still names a legacy nested slot whole; abort after storage shutdown does not throw; the decommission staging drain removes what the writer staged under the local and native key prefixes; the next mount sweeps a staging object whose abort delete was lost.
• CAS* unit tests green in release and ASan; test_cas_drop_pool_member passes, including a new case: SYSTEM CAS DROP POOL MEMBER 'node2/nested' is refused with BAD_ARGUMENTS.

Changelog category (leave one):

• Backward Incompatible Change

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CAS: cas_server_root_id may not contain /. A server whose id contains / (for example from a {shard}/{replica} template) refuses to start the disk and must be recreated under a new id without /; data stored under the old id is not visible under the new one. An aborted transaction deletes its S3 staging object. SYSTEM CAS DROP POOL MEMBER and the owner refusals name the state or command that resolves them.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [596fced]

[k-morozov]

Seems there is a corner case here.

As I can see above here we decided support old server_root with '`/'.

We have entry in ref_catalog only after inserting data (precommit stage). So we could have server_rrot_id a/b without entry in ref_catalog and serverRootSubtreeEmpty returns true instead false.

This repoduces the same problem.

[filimonov]

Agreed. / is now refused for every cas_server_root_id, including ids that already own their owner object, so a nested id can no longer mount or be decommissioned and the subtree check does not have to reason about one. An operator recreates such a member under a new id without /. Fixed in 52cb4b5.