Repository navigation
CAS: packed parts - #2579
Draft
k-morozov wants to merge 58 commits into
Draft
CAS: packed parts#2579k-morozov wants to merge 58 commits into
k-morozov wants to merge 58 commits into
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #2159 from Altinity/feature/antalya-26.6/CAS [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 # Conflicts: # .github/workflows/fast_builds.yml # .github/workflows/master.yml # .github/workflows/pull_request.yml # .github/workflows/pull_request_community.yml # ci/jobs/functional_tests.py # ci/jobs/scripts/clickhouse_proc.py # ci/jobs/scripts/clickhouse_service.py # ci/workflows/pull_request.py # ci/workflows/pull_request_community.py # ci/workflows/release_branches.py # docs/concepts/features/configuration/server-config/storing-data.mdx # programs/disks/DisksApp.cpp # programs/disks/ICommand.h # programs/server/Server.cpp # src/Common/FailPoint.cpp # src/Common/ThreadStatus.h # src/Core/ServerSettings.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.h # src/Disks/DiskObjectStorage/DiskObjectStorageTransaction.cpp # src/Disks/DiskObjectStorage/MetadataStorages/IMetadataStorage.h # src/Disks/DiskObjectStorage/MetadataStorages/MetadataStorageFactory.cpp # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h # src/Disks/DiskObjectStorage/ObjectStorages/Local/LocalObjectStorage.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/Disks/DiskObjectStorage/RegisterDiskObjectStorage.cpp # src/Disks/IDiskTransaction.h # src/IO/ReadPipeline.cpp # src/IO/ReadPipeline.h # src/IO/S3/copyS3File.cpp # src/IO/S3/copyS3File.h # src/IO/S3Common.cpp # src/IO/WriteBufferFromS3.cpp # src/IO/tests/gtest_writebuffer_s3.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Interpreters/ServerAsynchronousMetrics.cpp # src/Parsers/ASTSystemQuery.h # src/Parsers/tests/gtest_Parser.cpp # src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp # src/Storages/MergeTree/DataPartStorageOnDiskFull.cpp # src/Storages/MergeTree/DataPartsExchange.cpp # src/Storages/MergeTree/IMergeTreeDataPart.cpp # src/Storages/MergeTree/MergeTask.cpp # src/Storages/MergeTree/MergeTreeData.cpp # tests/integration/test_replicated_database/test.py # tests/queries/0_stateless/02253_empty_part_checksums.sh # tests/queries/0_stateless/02254_projection_broken_part.sh # tests/queries/0_stateless/02255_broken_parts_chain_on_start.sh # tests/queries/0_stateless/02369_lost_part_intersecting_merges.sh # tests/queries/0_stateless/02370_lost_part_intersecting_merges.sh # tests/queries/0_stateless/02444_async_broken_outdated_part_loading.sh # tests/queries/0_stateless/04215_replicated_missing_covered_part_on_start.sh # tests/queries/0_stateless/04327_reader_executor_metrics.sql # tests/queries/0_stateless/04328_reader_executor_kpi_async_metric.sql
Source PR: #2159 - `.github/workflows/*.yml` are generated: kept at the base version; the "Check Workflows" job regenerates them from `ci/workflows/*.py` (`python3 -m praktika yaml`), which now include the CAS lanes. - Stateless tests 02253/02254/02255/02369/02370/02444/04215 stay deleted: `antalya-26.8` converted them to integration tests (7f7a30e). - `ci/defs/altinity_jobs.py`: dropped the duplicated `cas_functional_tests_jobs` (already on base from 5aaf212). Adapted: DiskObjectStorageTransaction - PR's `dispatch` mapped onto base `addOperation` (made eager also for `transactionIsStagingOverlay`), keeps `appliesOperationsEagerly` for in-memory metadata Adapted: RegisterDiskObjectStorage - `use_fake_transaction` selection dropped, fake transactions were removed on base (c6254c3) Adapted: copyS3File - `copy_mode` threaded through base `copyS3FileImpl`; ranged copies use `ObjectStorageCopyMode::Default` Adapted: DataPartStorageOnDiskFull - in-flight reads moved into the base `*Impl` overrides (packed skip-index overlay lives in the base wrapper) Adapted: MergeTask/IMergeTreeDataPart - `use_parent_transaction` passed to base `getProjectionNoInitialize` Adapted: DataPartsExchange - kept base `REPLICATION_PROTOCOL_VERSION_WITH_INVALIDATED_SYSTEM_COLUMNS = 10`, negotiation tops out at `WITH_CA_CONFIRM = 11` Adapted: MergeTreeData - empty covering part already commits its disk transaction on base, PR's CAS-only commit dropped as redundant Adapted: functional_tests.py - RustFS start folded into base SeaweedFS-based `start` Source-PR: #2159 (#2159)
…next commit) --- Original cherry-pick message follows: Merge pull request #2300 from Altinity/feature/antalya-26.6/CAS-improvements CAS improvements # Conflicts: # src/Common/ErrorCodes.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h # src/IO/ReadBufferFromS3.cpp # src/IO/ReadSettings.h # src/IO/S3/tests/TestPocoHTTPServer.h # src/IO/WriteSettings.h
- `ErrorCodes.cpp`: kept antalya-26.8 numbering (1009-1018) and the `CLICKHOUSE_PARSER_MINIMAL_BUILD` guard; added the PR's `CAS_WRITE_UNATTRIBUTED` (1037) and `CAS_DELETE_MARKER` (1038), END = 1038. Dropped the PR's comments describing the antalya-26.6 1009-1011 layout, which does not exist on antalya-26.8. - `S3ObjectStorage`: combined the PR's `client_slot`/`client` reference with antalya-26.8's `client_restricts_server_credentials`, the iterator's `description` with the PR's `attempt_seed`, and antalya-26.8's `UnknownSize`-aware size and `object.etag` args with the PR's `refresh_callback`. - `TestPocoHTTPServer.h`: kept `makeMockServerParams` and added the PR's dedicated thread pool and destructor. - Adapted the PR's `S3::setClickhouseAttemptNumber` calls to `S3::setClickHouseAttemptNumber`, renamed on antalya-26.8 by e8e1e12. Adapted: setClickhouseAttemptNumber -> setClickHouseAttemptNumber (renamed by e8e1e12 on antalya-26.8) Source-PR: #2300 (#2300)
…next commit) --- Original cherry-pick message follows: Merge PR #2307 (cas/configurable-mount-lease-timing) into feature/antalya-26.6/CAS-improvements-cicd-fixes Brings k-morozov's configurable CAS mount lease TTL / renewal period (#2307) so the heartbeat-model work can land on top of it while keeping the author's commits intact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx # Conflicts: # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> # Conflicts: # docs/en/antalya/cas/configuration.md # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp
All changes from #2307 (`mount_lease_ttl_ms` / `mount_renew_period_ms` settings, members, wiring into `PoolConfig`, validation, docs) are already present on the base. The conflicting hunks keep the base side, which has later additions on top of them (`unsafe_remount_no_delay`, the `attempt_timeout_ms` check, `connect_timeout_cap_ms`, and the refined `cas_mount_renew_period_ms` doc text). Source-PR: #2307 (#2307)
…next commit) --- Original cherry-pick message follows: Merge pull request #2349 from Altinity/cas/memworker-split-allocated-source MemoryWorker: use sanitizer info for allocated counter # Conflicts: # src/Common/MemoryWorker.cpp
…next commit) --- Original cherry-pick message follows: Merge pull request #2385 from Altinity/fix/antalya-26.6/lldb-budget-and-sigcont CI: lldb stack budget by build flavor, and keep lldb's JIT breakpoint out of the server # Conflicts: # ci/tests/test_print_stacktraces.py # tests/clickhouse-test
tests/clickhouse-test: applied the PR's lldb budget and JIT-loader flag on top of the antalya-26.8 shape of get_stacktraces_from_lldb (sudo/shutil.which lldb path and keep_output_on_error kept; no _ensure_lldb_installed, which antalya-26.8 does not have). Docstring keeps the 'liveness check' call site that exists on antalya-26.8. ci/tests/test_print_stacktraces.py: kept deleted, as on antalya-26.8, where 3a69545 removed all of ci/tests/. The PR's test changes to that file are therefore not carried over. Source-PR: #2385 (#2385)
…next commit) --- Original cherry-pick message follows: Merge pull request #2292 from Altinity/enable-cas-s3-stress Enable cas s3 stress # Conflicts: # .github/workflows/master.yml # .github/workflows/pull_request.yml # ci/tests/test_stress_job.py
Kept the antalya-26.8 `needs` lists of `finish_workflow` in `master.yml` and `pull_request.yml` and added only the four `stress_test_amd_*_cas_s3_storage` jobs introduced by the source PR. Kept both `ensure_ripgrep_installed` and `get_additional_envs` imports in `ci/tests/test_stress_job.py`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Source-PR: #2292 (#2292)
…next commit) --- Original cherry-pick message follows: Merge pull request #2444 from Altinity/docs/antalya-26.6/metrics-and-events Document Antalya-only profile events and metrics. # Conflicts: # src/Common/ProfileEvents.cpp
…next commit) --- Original cherry-pick message follows: Merge pull request #2511 from Altinity/fix/antalya-26.6/cas-renewal-memory-guard Keep the CAS mount lease renewal alive over the server memory limit # Conflicts: # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp
k-morozov
changed the base branch from
feature/antalya-26.8/auto-grp-pr-2159
to
antalya-26.8
October 16, 2026 10:55
zvonand
added
the
forwardport
This reverts commit 133b29d.
…eature/cas-packed-parts
The port references removed DEB_AMD_* artifacts and generated workflow jobs with stale dependencies on antalya-26.8. Source-PR: #2292 (#2292) Review: https://gist.github.com/k-morozov/0e7a4ee2ce4675fbf22c01e79adfb350
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: