Backport #7858 and #7864 to release-1.22 for rc.2 - #7863
Merged
CharlieTLe merged 2 commits intoSep 25, 2026
Merged
Conversation
…ervations (cortexproject#7858) * Ingester: fix cortex_ingester_ingestion_delay_seconds losing most observations The histogram was registered with NativeHistogramMinResetDuration: 1, an untyped constant that Go implicitly converts to time.Duration(1), i.e. 1 nanosecond, instead of the intended 1 hour used by every other native histogram in this file. When the native histogram's bucket count exceeds NativeHistogramMaxBucketNumber (100), client_golang's limitBuckets() first tries maybeReset(), which fully resets the histogram (wiping both native and classic bucket counts, keeping only the latest observation) if at least NativeHistogramMinResetDuration has elapsed since the last reset. With an effectively-zero duration, that condition is satisfied on virtually every call, so instead of gracefully reducing resolution (bucket width doubling / zero bucket widening), the histogram repeatedly self-resets and silently drops the large majority of observations. In a 100k-sample simulation this loses ~86% of observations, corrupting both _count/_sum and the classic le="600" bucket that operators alert on for ingestion lag. Fix it to 1 * time.Hour, matching every other histogram in this file. Fixes cortexproject#7731 Signed-off-by: ankit090701 <ankitanku090701@gmail.com> * Add CHANGELOG.md entry for cortexproject#7744 Signed-off-by: ankit090701 <ankitanku090701@gmail.com> * Address review feedback from @yeya24 - Drop the CHANGELOG entry: the bug was introduced in cortexproject#7443, which was merged after the latest release (v1.21.1, 2026-06-04) and has never shipped, so there's nothing for users to be informed about fixing. - Fix check-modernize lint failure: use `for range numObservations` instead of `for i := 0; i < numObservations; i++` in the new test, since the loop index was never used. Signed-off-by: ankit090701 <ankitanku090701@gmail.com> * remove test Signed-off-by: SungJin1212 <tjdwls1201@gmail.com> --------- Signed-off-by: ankit090701 <ankitanku090701@gmail.com> Signed-off-by: SungJin1212 <tjdwls1201@gmail.com> Co-authored-by: ankit090701 <ankitanku090701@gmail.com> (cherry picked from commit f172a5a) Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe
requested review from
danielblando
and removed request for
a team
September 25, 2026 19:38
CharlieTLe
changed the title
friedrichg
approved these changes
Sep 25, 2026
Every integration leg is failing at Preload Images: docker pull quay.io/minio/minio:RELEASE.2024-05-28T17-19-04Z Error response from daemon: unauthorized: access to the requested resource is not authorized MinIO has withdrawn its public images from quay.io, after doing the same on Docker Hub (which is why cortexproject#7837 moved us to quay.io). No mirror still serves the multi-arch RELEASE.2024-05-28T17-19-04Z image. Host an unmodified copy of MinIO's official RELEASE.2024-07-04T14-25-45Z image under docker.io/cortexproject/minio. It was copied with `crane copy`, so its index digest, sha256:5db7e40b69f0c3ad5a878521ff5029468e3070ef146c084dc2540e2d492075c4, is identical to the source and it keeps linux/amd64 and linux/arm64. Its binary reports the commit behind MinIO's signed release tag, and its SLSA provenance records a build of github.com/minio/minio from MinIO's own release commit. The image runs as root like the previous one, so the e2e harness works unchanged. The development docker-compose stacks previously pulled the untagged (latest) image; pin them to the same tag, since that is the only tag we host. Signed-off-by: Charlie Le <charlie_le@apple.com> (cherry picked from commit 6fb32b6)
CharlieTLe
force-pushed
the
backport-1.22-rc2
branch
from
September 25, 2026 22:19
9b63494 to
76ef7bf
Compare
CharlieTLe
merged commit Sep 25, 2026
6b86a56
into
cortexproject:release-1.22
74 of 75 checks passed
CharlieTLe
added a commit
that referenced
this pull request
Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports #7858 to
release-1.22forv1.22.0-rc.2.cortex_ingester_ingestion_delay_secondsis new in 1.22 (#7443) and is listed in the 1.22.0 CHANGELOG. In rc.1 it is registered withNativeHistogramMinResetDuration: 1, which is 1ns rather than the intended 1h. Once the native histogram exceeds 100 buckets it resets on nearly every observation, dropping ~86% of observations and corrupting_count,_sumand the classicle="600"bucket used for ingestion-lag alerting (#7731).#7858 dropped its CHANGELOG entry on the grounds that the bug never shipped; backporting it keeps that true for 1.22.0.
One-line change in
pkg/ingester/metrics.go, cherry-picked cleanly with-x.Verification
go build -tags "netgo slicelabels" ./...— cleango test -tags "netgo slicelabels" ./pkg/ingester/...— okAlso cherry-picks #7864 (minio image now hosted at
docker.io/cortexproject/minio). Without it every integration leg fails at Preload Images because MinIO withdrew its images from quay.io, and the rc.2 tag build could not publish images.