Repository navigation
Conversation
14db4ac to
bad9884
Compare
b481005 to
ac73e0f
Compare
3b1d505 to
7dda555
Compare
7e77e41 to
f9645af
Compare
15913cb to
06a2a69
Compare
3b49b2a to
0b3e646
Compare
📜 Change history & discussion (Agora / pg.ddx.io)I have everything needed. 🧵 Related discussion
🔗 Related commits / prior art
📋 Commitfest
🧭 Context for reviewers
Generated by pg-history via the Agora MCP server (pg.ddx.io). |
| SELECT sum(reads) AS stats_bulkreads_before | ||
| FROM pg_stat_io WHERE context = 'bulkread' \gset | ||
| FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset |
There was a problem hiding this comment.
The \gset variables are still named stats_bulkreads_before/stats_bulkreads_after, but this query now deliberately measures the normal context (the comment above was rewritten to say the reads are no longer bulkreads). The names now contradict what they hold. Since these exact lines are already being touched, rename them to reflect the new semantics (e.g. stats_normal_reads_before) for both \gset sites and the comparison on the following line. This also requires the matching update in contrib/amcheck/expected/check_heap.out.
| SELECT sum(reads) AS stats_bulkreads_before | |
| FROM pg_stat_io WHERE context = 'bulkread' \gset | |
| FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset | |
| SELECT sum(reads) AS stats_normal_reads_before | |
| FROM pg_stat_io WHERE context = 'normal' AND object = 'relation' \gset |
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
This silently changes the user-visible semantics of the pg_buffercache.usagecount column. BUF_STATE_GET_USAGECOUNT returned the full clock-sweep count (0..BM_MAX_USAGE_COUNT, historically 0..5), whereas BUF_STATE_GET_COOLSTATE returns only bit 0 (0=COOL, 1=HOT). The column is still declared smallint and documented as "Clock-sweep access count", so users querying usagecount will now silently get only 0/1 with no doc or column-name update. This is a backward-incompatible behavior change that needs the documentation (pgbuffercache.sgml) updated and, arguably, the column renamed to reflect HOT/COOL semantics. (high confidence)
| { | ||
| buffers_used++; | ||
| usagecount_total += BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount_total += BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
usagecount_total now accumulates only 0/1 per buffer instead of 0..BM_MAX_USAGE_COUNT, so the derived usagecount_avg column changes range/meaning without any doc update. The SGML still labels it a clock-sweep average. Update the documentation to match the new HOT/COOL semantics. (high confidence)
| CHECK_FOR_INTERRUPTS(); | ||
|
|
||
| usage_count = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usage_count = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
pg_buffercache_usage_counts() now buckets buffers only into indices 0 and 1 (COOL/HOT); the usage_count output column will never exceed 1. This is fine for array bounds since BM_MAX_USAGE_COUNT is redefined to BUF_COOLSTATE_HOT (=1), but the SQL column name usage_count and its doc ("A possible buffer usage count") are now misleading. Update pgbuffercache.sgml and the expected regression output to reflect the reduced value domain. (high confidence)
| allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0; | ||
| } | ||
| else | ||
| allow_strat = allow_sync = false; | ||
|
|
||
| if (allow_strat) | ||
| { | ||
| /* During a rescan, keep the previous strategy object. */ | ||
| if (scan->rs_strategy == NULL) | ||
| scan->rs_strategy = GetAccessStrategy(BAS_BULKREAD); | ||
| } | ||
| else | ||
| { | ||
| if (scan->rs_strategy != NULL) | ||
| FreeAccessStrategy(scan->rs_strategy); | ||
| scan->rs_strategy = NULL; | ||
| } | ||
| allow_sync = false; |
There was a problem hiding this comment.
The if body is now a single statement after removing the allow_strat assignment, but the braces are retained. PostgreSQL style (and pgindent-adjacent convention) omits braces around single-statement bodies. Since this block was directly modified by the diff, drop the braces to keep the code reading as if it had always been written this way:
if (!RelationUsesLocalBuffers(scan->rs_base.rs_rd) &&
scan->rs_nblocks > NBuffers / 4)
allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0;
else
allow_sync = false;Confidence: moderate.
| allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0; | |
| } | |
| else | |
| allow_strat = allow_sync = false; | |
| if (allow_strat) | |
| { | |
| /* During a rescan, keep the previous strategy object. */ | |
| if (scan->rs_strategy == NULL) | |
| scan->rs_strategy = GetAccessStrategy(BAS_BULKREAD); | |
| } | |
| else | |
| { | |
| if (scan->rs_strategy != NULL) | |
| FreeAccessStrategy(scan->rs_strategy); | |
| scan->rs_strategy = NULL; | |
| } | |
| allow_sync = false; | |
| allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0; | |
| else | |
| allow_sync = false; |
| else | ||
| elevel = DEBUG2; | ||
|
|
||
| /* Set up static variables */ |
There was a problem hiding this comment.
This comment is now dangling. It solely described the removed vac_strategy = bstrategy; assignment; the only other static in this file (anl_context) is set up in do_analyze_rel, not here. Leaving an empty comment block referring to nonexistent code violates the minimal-diff/comment-accuracy discipline. Remove the comment (and the now-doubled blank line).
| read_stream_next_block(ReadStream *stream) | ||
| { | ||
| *strategy = stream->ios[0].op.strategy; | ||
| return read_stream_get_block(stream, NULL); | ||
| } |
There was a problem hiding this comment.
With strategy removed, read_stream_next_block() no longer has any distinguishing behavior: it is now an exported thin wrapper around read_stream_get_block(stream, NULL) and has no callers anywhere in the tree (verified: only the callback functions apw_read_stream_next_block and collect_corrupt_items_read_stream_next_block match by name; no invocation of read_stream_next_block(...) exists). Its sole prior reason to exist over calling the internal helper directly was reporting the strategy, which this patch deletes. Recommend removing the function (and its prototype in read_stream.h) as dead transitional scaffolding, or, if it is intentionally retained as public API, keep it clearly justified. Leaving a caller-less exported wrapper contradicts the minimal-diff / YAGNI discipline. (moderate confidence)
| current = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer); | ||
| if (current >= (uint32) NBuffers) | ||
| { | ||
| wrapped = current % NBuffers; | ||
| if (pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer, | ||
| ¤t, wrapped)) | ||
| StrategyControl->completePasses++; |
There was a problem hiding this comment.
completePasses can be under-counted under concurrency, corrupting bgwriter pacing. StrategySyncStart() relies on nextVictimBuffer staying below 2*NBuffers so its pending-wrap compensation (nextVictimBuffer / NBuffers) is at most 1. With batching, many concurrent backends can each land a fetch_add(batch_size) before any of them reaches the spinlock, so the shared counter can grow to NBuffers + MaxBackends*batch_size. When it exceeds 2*NBuffers, wrapped = current % NBuffers subtracts multiple NBuffers-multiples but completePasses is incremented only once, permanently losing pass counts. This is reachable in the small-NBuffers case (minimum 16, batch capped at NBuffers): 2*NBuffers is easily exceeded by concurrent claims. Consider incrementing completePasses by current / NBuffers instead of a flat +1, and/or bounding growth so the counter cannot exceed 2*NBuffers.
| current = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer); | |
| if (current >= (uint32) NBuffers) | |
| { | |
| wrapped = current % NBuffers; | |
| if (pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer, | |
| ¤t, wrapped)) | |
| StrategyControl->completePasses++; | |
| current = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer); | |
| if (current >= (uint32) NBuffers) | |
| { | |
| wrapped = current % NBuffers; | |
| if (pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer, | |
| ¤t, wrapped)) | |
| StrategyControl->completePasses += current / NBuffers; |
| MyBatchPos = start; | ||
| MyBatchEnd = start + batch_size; |
There was a problem hiding this comment.
A batch can straddle the NBuffers boundary without any wrap being registered on the shared counter. When start < NBuffers but start + batch_size > NBuffers, the wrap branch (start >= NBuffers) is skipped, yet victim = MyBatchPos % NBuffers wraps internally as MyBatchPos crosses NBuffers. The completePasses increment for that crossing is deferred until a later fetch-add happens to return start >= NBuffers. Between those events, StrategySyncStart() sees nextVictimBuffer already advanced past NBuffers and relies solely on its nextVictimBuffer / NBuffers term to compensate. That compensation is only correct while the counter stays below 2*NBuffers (see the related finding). This coupling is fragile and should be documented or handled explicitly, since a mid-batch wrap is now the common case (batch_size defaults to 32).
| #include "storage/subsystems.h" | ||
| #include "port/pg_numa.h" |
There was a problem hiding this comment.
Include is out of group/alphabetical order: port/pg_numa.h is placed after the storage/* block. Per the tree's convention (and pgindent grouping), it belongs with the other port/* headers, immediately after port/atomics.h.
| #include "storage/subsystems.h" | |
| #include "port/pg_numa.h" | |
| #include "storage/subsystems.h" |
0b3e646 to
3ed29c4
Compare
There was a problem hiding this comment.
🔍 OCR found 43 issue(s).
- 25 inline, 18 in summary (inline capped at 25)
📄 src/backend/access/nbtree/nbtree.c (L1732-L1732)
The strategy argument was removed from this ReadBufferExtended call, but the comment above still claims "we want to use a nondefault buffer access strategy." This comment is now stale and misleading -- no strategy is passed anymore. Update the comment to reflect that the buffer access strategy is now handled elsewhere (or drop that sentence). (moderate confidence)
📄 src/backend/commands/analyze.c (L124-L124)
The /* Set up static variables */ comment is now stale: this diff removed the only statement it introduced (vac_strategy = bstrategy;), leaving the comment describing nothing. Remove the dangling comment so the code reads as if it had always been written this way (minimal-diff hygiene). (high confidence)
📄 src/backend/commands/vacuum.c (L178-L179)
This removal makes VACUUM (BUFFER_USAGE_LIMIT ...) and ANALYZE (BUFFER_USAGE_LIMIT ...) a hard error. The grammar (gram.y utility_option_list) still accepts any option name as a generic DefElem, so with this branch gone the option now falls through to the unrecognized "VACUUM"/"ANALYZE" option ereport below. This is a backward-compatibility break of a documented, user-visible SQL option, and it breaks existing dump/restore scripts and tooling that pass BUFFER_USAGE_LIMIT. If the intent is to fully retire the feature, the corresponding user-facing surfaces (SQL docs in doc/src/sgml/ref/{vacuum,analyze}.sgml and psql tab-completion in tab-complete.in.c, which still advertise BUFFER_USAGE_LIMIT) must be updated in the same change so the option isn't offered while silently erroring. Confidence: high.
📄 src/backend/storage/buffer/freelist.c (L161-L163)
completePasses accounting is inconsistent with StrategySyncStart's compensation term and can double-count / go backwards.
StrategySyncStart() still returns *complete_passes = completePasses + nextVictimBuffer / NBuffers. But now:
-
This
else ifincrements completePasses for a batch that crosses the wrap point without wrapping the shared counter. The counter is then >= NBuffers, so a subsequent StrategySyncStart addsnextVictimBuffer / NBuffers(>=1) on top of the increment already applied here -> the same pass is counted twice. -
The
if (start >= NBuffers)branch above wrapscurrent % NBuffersbut increments completePasses by only 1 even whencurrentdrifted to e.g.2*NBuffers; the sync-start compensation term drops from 2 to 0 across that wrap, so the visible pass count decreases.
Because BgBufferSync does passes_delta = strategy_passes - prev_strategy_passes; strategy_delta += passes_delta*NBuffers; Assert(strategy_delta >= 0), a non-monotonic completePasses can trip that assertion and corrupts bgwriter pacing. Confidence: high. The batched wrap accounting needs to be made consistent with StrategySyncStart's nextVictimBuffer / NBuffers term (i.e. don't both pre-count the crossing here and let sync-start count it again).
📄 src/backend/storage/buffer/freelist.c (L137-L142)
This if (start >= NBuffers) wrap path can lose passes when the counter has drifted more than one NBuffers period. current may be >= 2*NBuffers under batching/contention, but wrapped = current % NBuffers collapses it and completePasses is bumped by only 1, while StrategySyncStart's nextVictimBuffer / NBuffers compensation was >1 before the wrap and becomes 0 after. Net effect: the reported completed-pass count can decrease across the wrap, which BgBufferSync's Assert(strategy_delta >= 0) does not tolerate. Confidence: high. Consider incrementing completePasses by current / NBuffers (the number of whole periods being discarded) so the (nextVictimBuffer, completePasses) pair stays monotonic.
📄 src/backend/storage/buffer/freelist.c (L324-L330)
Under force_cool, resetting trycounter = NBuffers on the ref-bit-clear transition (not just on the HOT->COOL demote) means a highly contended, all-HOT pool where PinBuffer keeps re-setting BUF_REFBIT (it does so on every pin, bufmgr.c:3302) can keep this backend clearing ref bits and resetting the counter without ever landing on a reclaimable COOL buffer. The intended "second unproductive full pass -> elog(ERROR, no unpinned buffers)" backstop can then be deferred well beyond the ~3 passes the comment claims. Consider only resetting trycounter on the actual HOT->COOL demotion (real forward progress toward a victim), and treating a bare ref-clear as non-progress so the escalation/backstop stays bounded. Confidence: moderate.
📄 src/backend/storage/buffer/bufmgr.c (L86-L86)
Dead flag: BUF_COOLED is defined here, documented in the SyncOneBuffer header, and set via result |= BUF_COOLED, but no caller ever inspects it. SyncOneBuffer's only cool_if_hot=true caller is BgBufferSync, which reacts only to BUF_WRITTEN and BUF_REUSABLE. Per the minimalism/YAGNI discipline, either wire BUF_COOLED into a consumer (e.g. bgwriter pacing/instrumentation) or drop the flag, its result |= assignment, and the doc line. (high confidence)
📄 src/backend/storage/buffer/bufmgr.c (L2320-L2325)
Comment-drift / misleading placement. This "Admit the newly loaded page COOL" comment sits directly above the unrelated BM_PERMANENT conditional, but the actual COOL admission is the removal of BUF_USAGECOUNT_ONE from the set_bits |= BM_TAG_VALID; line above (a freshly admitted buffer now has coolstate 0 == COOL). As written the comment misleads readers into thinking the BM_PERMANENT block performs the cooling admission. Move the comment to sit above the set_bits |= BM_TAG_VALID; line. (moderate confidence)
📄 src/backend/storage/buffer/bufmgr.c (L2964-L2968)
Same comment-placement issue as the other admission site: the "Admit COOL (probation)" comment is above the BM_PERMANENT conditional rather than the set_bits |= BM_TAG_VALID; line where BUF_USAGECOUNT_ONE was previously set. Relocate it above set_bits |= BM_TAG_VALID; so the comment describes the line that actually performs COOL admission. (moderate confidence)
📄 src/backend/utils/activity/pgstat_io.c (L443-L449)
This comment block is now stale/dangling. The if blocks it described (the B_CHECKPOINTER/B_BG_WRITER/autovac restrictions on the removed bulkread/bulkwrite/vacuum IOContexts) were deleted, leaving the comment sitting directly above return true; describing nothing. Per PG comment discipline (comments must describe what the code does now) and minimal-diff hygiene, remove the orphaned comment so the function reads as if it had always been written this way.
💡 Suggested change
Before:
/*
* Some BackendTypes do not currently perform any IO in certain
* IOContexts, and, while it may not be inherently incorrect for them to
* do so, excluding those rows from the view makes the view easier to use.
*/
return true;
After:
return true;
📄 src/bin/scripts/vacuumdb.c (L355-L355)
This removes the user-visible --buffer-usage-limit option (getopt entry, handler, validation, and help text), but the corresponding documentation is not removed: doc/src/sgml/ref/vacuumdb.sgml still contains a full <varlistentry> for --buffer-usage-limit (around line 135). A user-visible removal must delete the matching doc entry in the same patch, otherwise the docs describe an option the binary no longer accepts. Please remove the SGML block as part of this change. (high confidence)
📄 src/bin/scripts/vacuuming.c (L267-L272)
This removes the --buffer-usage-limit handling from vacuumdb, a user-visible option shipped since PG16. Together with the coordinated removals in vacuumdb.c and vacuuming.h, this deletes an established CLI feature. This is a backward-compatibility break: existing scripts invoking vacuumdb --buffer-usage-limit=... will now fail with an unknown-option error. Such a removal needs extraordinary justification and a design discussion on -hackers; nothing in the diff explains why. Additionally, doc/src/sgml/ref/vacuumdb.sgml still documents --buffer-usage-limit and is not part of this change set, so the docs will describe an option that no longer exists. (high confidence)
📄 src/include/storage/bufmgr.h (L357-L360)
Dead section header and whitespace churn. All prototypes that lived under "/* in freelist.c /" (GetAccessStrategy, FreeAccessStrategy) were removed, so this comment now labels nothing, and the removal left a stray double blank line before "/* inline functions /". This will trip git diff --check / pgindent style expectations. Remove the empty "/ in freelist.c */" section entirely and collapse to a single blank line.
💡 Suggested change
Before:
/* in freelist.c */
/* inline functions */
After:
/* inline functions */
📄 src/include/storage/bufmgr.h (L221-L222)
API/ABI break on widely-used exported entry points (moderate confidence on impact). ReadBufferExtended(), ReadBufferWithoutRelcache(), and the ExtendBufferedRel*() family drop their BufferAccessStrategy parameter, while GetAccessStrategy()/GetAccessStrategyWithSize()/GetAccessStrategyBufferCount()/GetAccessStrategyPinLimit()/FreeAccessStrategy() are removed outright. These are core buffer-manager APIs used pervasively by out-of-tree extensions; every such extension will fail to compile against this header. On HEAD a signature change is allowed, but wholesale removal of the strategy concept with no replacement is a substantial, non-obvious semantic change (ring buffers gone; scan/VACUUM no longer bounded to a small buffer ring). For pgsql-hackers this needs (a) a reference to the design thread / commitfest entry justifying dropping ring buffers, and (b) confirmation it is a single logical change -- as submitted it bundles at least three independent things (remove BufferAccessStrategy, replace usage_count clock sweep with HOT/COOL cooling state, add NUMA-batched clock sweep), which should be split into separately-committable, individually-bisectable patches.
📄 src/include/pgstat.h (L294-L294)
This change reduces IOCONTEXT_NUM_TYPES from 6 to 2, which shrinks the fixed-stats structs PgStat_BktypeIO/PgStat_PendingIO/PgStat_IO (dimensioned [IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES], lines 328-337). PgStat_IO is a fixed stats kind that is serialized to the persisted stats file. The comment above PGSTAT_FILE_FORMAT_ID (line 216-217) explicitly requires bumping the format ID "whenever any of these data structures change", and on startup pgstat_read_statsfile() only discards a stale file when format_id != PGSTAT_FILE_FORMAT_ID. Since PGSTAT_FILE_FORMAT_ID (0x01A5BCBC) is unchanged in this patch, a stats file written by a pre-patch server will be accepted and read with the new, smaller layout, corrupting the fixed-stats block (and desynchronizing subsequent parsing). Bump PGSTAT_FILE_FORMAT_ID. Unlike catversion.h this stamp is the author's responsibility. Confidence: high.
📄 src/include/storage/buf_internals.h (L136-L137)
BUF_STATE_GET_USAGECOUNT now has no callers anywhere in the tree -- every consumer was switched to BUF_STATE_GET_COOLSTATE. Leaving the old accessor behind is dead code and a footgun: the field it reads now mixes the cooling bit (bit 0) and the ref bit (bit 1), so any new caller that reaches for the familiar name gets a 0..3 value conflating two orthogonal flags. Remove it. (high confidence)
📄 src/include/storage/buf_internals.h (L139-L140)
pgindent-nonconforming multi-line comment: the block opens with text on the same line as /* and closes with an inline */, unlike every other multi-line comment in this header (which puts /* on its own line with asterisk-aligned continuation). This will not survive pgindent cleanly. (moderate confidence)
💡 Suggested change
Before:
/* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so
* the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. */
After:
/*
* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so
* the second-chance ref bit (bit 1) does not perturb the HOT/COOL test.
*/
📄 src/include/storage/buf_internals.h (L191-L191)
BM_MAX_USAGE_COUNT now means "the HOT cool-state value" (=1), not a usage-count maximum, yet it is still consumed as a numeric range bound -- e.g. pg_buffercache sizes usage_counts[BM_MAX_USAGE_COUNT + 1] and iterates over it. Retaining a count-named macro for a boolean cool-state violates POLA and invites a future reader to treat it as a real max count. Prefer a name that reflects the new semantics (e.g. BM_MAX_COOLSTATE) and update call sites, rather than aliasing the old name for the single expression that happens to read naturally. (moderate confidence)
| @@ -0,0 +1,41 @@ | |||
| # LiteLLM proxy config — bridges Open Code Review (OpenAI protocol) to AWS Bedrock. | |||
There was a problem hiding this comment.
Non-ASCII em-dash character (—) in a source/config comment. The project's contribution standards require ASCII-only in source and diffs (no smart quotes, em-dashes, or ellipsis characters). Replace with an ASCII - (or --).
| # LiteLLM proxy config — bridges Open Code Review (OpenAI protocol) to AWS Bedrock. | |
| # LiteLLM proxy config - bridges Open Code Review (OpenAI protocol) to AWS Bedrock. |
| aws_region_name: os.environ/AWS_REGION | ||
|
|
||
| # "High effort" review. Claude Opus 4.8 on Bedrock uses *adaptive* thinking | ||
| # controlled by output_config.effort. Set it DIRECTLY here — NOT via |
There was a problem hiding this comment.
Non-ASCII em-dash character (—) in this comment. Replace with an ASCII - to comply with the ASCII-only rule for source and diffs.
| # controlled by output_config.effort. Set it DIRECTLY here — NOT via | |
| # controlled by output_config.effort. Set it DIRECTLY here - NOT via |
| import boto3 | ||
| from botocore.config import Config |
There was a problem hiding this comment.
The module docstring promises the script "exits 0 even on soft failures (writes a note)", and every other failure path here honors that (MCP unreachable at line 168, no tools at line 172, Bedrock error at line 212 all write a note to OUT and return). But import boto3 / from botocore.config import Config are not guarded: if the SDK is missing/unimportable this raises an uncaught ImportError, exits non-zero, and writes NO note to PG_HISTORY_OUT, leaving the downstream "Upsert PR comment" step with empty output. The workflow's || true masks the exit code but not the missing note. Wrap the import in the same try/except-and-write-note pattern used for the MCP setup for consistency with the documented contract. (moderate confidence)
| import boto3 | |
| from botocore.config import Config | |
| try: | |
| import boto3 | |
| from botocore.config import Config | |
| except Exception as e: | |
| open(OUT, "w").write(f"_pg-history: boto3/botocore unavailable: {e}_\n") | |
| print(f"boto3 unavailable: {e}") | |
| return |
| open(OUT, "w").write(body) | ||
| print(body) |
There was a problem hiding this comment.
open(OUT, "w").write(...) leaves the file handle to be closed by refcount-based GC, so written bytes are only flushed at interpreter cleanup. This pattern is repeated at every write site (lines 158, 168, 172, 220). On CPython in this short-lived script it usually works, but it is not guaranteed on other interpreters and violates the tree's resource-management/portability expectations. Prefer with open(OUT, "w") as f: f.write(...). (low confidence, minor)
| check-model: | ||
| runs-on: ubuntu-latest | ||
| steps: |
There was a problem hiding this comment.
This job has no timeout-minutes, unlike every job in .github/workflows/pg-ci.yml (which sets it on all jobs). The OIDC credential step and the inline Python subprocess invoking the AWS CLI can hang on network/throttling/OIDC issues, letting the job run to the default 6-hour runner limit. Add a short explicit timeout for this scheduled maintenance job.
| check-model: | |
| runs-on: ubuntu-latest | |
| steps: | |
| check-model: | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 10 | |
| steps: |
| sync: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: write | ||
| issues: write |
There was a problem hiding this comment.
The sync job has no timeout-minutes. git fetch upstream master against the full-history PostgreSQL repository plus a rebase can hang on network problems, and with an hourly schedule stalled runs can pile up and burn runner minutes indefinitely. Other workflows in this repo (pg-ci.yml) set timeout-minutes; add an explicit timeout here for consistency and safety.
| sync: | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: write | |
| issues: write | |
| sync: | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 30 | |
| permissions: | |
| contents: write | |
| issues: write |
| echo "⚠️ Local master has $DIVERGED commits not in upstream" | ||
|
|
||
| # Check commit messages for "dev setup" or "dev v" pattern | ||
| DEV_SETUP_COMMITS=$(git log --format=%s upstream/master..origin/master | grep -iE "^dev (setup|v[0-9])" | wc -l) |
There was a problem hiding this comment.
The decision to rebase and force-push over master rests on brittle string matching: grep -iE "^dev (setup|v[0-9])" on commit subjects and grep -v "^\.github/" on changed paths. A squashed commit that mixes .github/ and non-.github/ changes, a merge commit, or a commit whose subject coincidentally starts with "dev v1" can be misclassified. A false positive here permits git rebase upstream/master followed by --force-with-lease, which can silently drop legitimate local commits on master. This is a data-loss footgun — the guard protecting destructive history rewriting should be more robust than subject-line pattern matching (e.g., fail closed on any non-.github/ change rather than trusting the commit subject).
| git config user.name "github-actions[bot]" | ||
| git config user.email "github-actions[bot]@users.noreply.github.com" | ||
|
|
||
| if git rebase upstream/master; then |
There was a problem hiding this comment.
git config user.name/user.email is already set in the earlier "Configure Git" step, and git config persists across steps within a job on the runner. Re-setting it here is redundant and can be removed.
| git config user.name "github-actions[bot]" | |
| git config user.email "github-actions[bot]@users.noreply.github.com" | |
| if git rebase upstream/master; then | |
| if git rebase upstream/master; then |
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
This change alters the user-visible semantics of the usage_count column exposed by all three functions (pg_buffercache_pages, pg_buffercache_summary, pg_buffercache_usage_counts). BUF_STATE_GET_USAGECOUNT returned the full 0..N usage count, whereas BUF_STATE_GET_COOLSTATE returns only the 0/1 cooling state (bit 0 of the field). The documentation in doc/src/sgml/pgbuffercache.sgml still describes and shows a multi-valued "usage count" (e.g. the sample output table and the column descriptions "A possible buffer usage count"), and is not updated in this change. A user-visible SQL-interface change without matching docs is incomplete for a pgsql-hackers patch. Update pgbuffercache.sgml to reflect the new HOT/COOL cooling-state semantics (and consider whether the exposed column/variable naming should still be usage_count). (high confidence)
| allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0; | ||
| } | ||
| else | ||
| allow_strat = allow_sync = false; | ||
|
|
||
| if (allow_strat) | ||
| { | ||
| /* During a rescan, keep the previous strategy object. */ | ||
| if (scan->rs_strategy == NULL) | ||
| scan->rs_strategy = GetAccessStrategy(BAS_BULKREAD); | ||
| } | ||
| else | ||
| { | ||
| if (scan->rs_strategy != NULL) | ||
| FreeAccessStrategy(scan->rs_strategy); | ||
| scan->rs_strategy = NULL; | ||
| } | ||
| allow_sync = false; |
There was a problem hiding this comment.
This diff removes the bulk-read access strategy from initscan(), but the comment above this block (lines 383-390) still reads "use a bulk-read access strategy and enable synchronized scanning ... only two behaviors to tune rather than four ... some callers need to be able to disable one or both of these behaviors". After this change there is only one behavior left here (sync scan); the bulk-read strategy no longer exists. The comment now describes code that is gone and should be updated to mention only synchronized scanning, otherwise it will mislead readers into looking for strategy handling that isn't there. (moderate confidence)
e495986 to
d7cd5d6
Compare
576fa38 to
70ceeea
Compare
There was a problem hiding this comment.
🔍 OCR found 26 issue(s).
- 25 inline, 1 in summary (inline capped at 25)
📄 src/backend/storage/buffer/freelist.c (L24-L25)
Include is out of alphabetical order. PostgreSQL sorts includes alphabetically within a block; port/pg_numa.h belongs immediately after port/atomics.h (line 19), before the storage/* headers, not appended at the end. Reviewers on -hackers routinely flag this.
💡 Suggested change
Before:
#include "storage/subsystems.h"
#include "port/pg_numa.h"
After:
#include "storage/subsystems.h"
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
This entire file should not be part of the patch. PostgreSQL's own top-level .gitignore states the policy explicitly: "Auxiliary files from local workflows, your preferred editor, etc. should be ignored locally using $GIT_DIR/info/exclude or ~/.gitexclude." A .clangd file is precisely such an editor/tooling artifact (as are the companion .gdbinit and pg-aliases.sh files in this changeset). Committing personal IDE configuration is unrelated churn that will draw an immediate rejection on pgsql-hackers. Keep it in a local exclude instead of tracking it. [high confidence]
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
Correctness / backward-compat (high confidence): this silently changes the semantics of the user-visible usage_count column of pg_buffercache_pages. Previously it reported 0..BM_MAX_USAGE_COUNT (0..5); it now reports only the HOT/COOL bit (0 or 1), because BUF_STATE_GET_COOLSTATE masks bit 0 only and discards the second-chance ref bit (bit 1). Existing queries/monitoring that interpret usage_count as the old clock-sweep counter break with no error and no cue. A user-visible interface change like this needs the column renamed (e.g. cool_state) or at least documented in pg_buffercache.sgml, plus regression-test updates; none are in this patch.
| { | ||
| buffers_used++; | ||
| usagecount_total += BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount_total += BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
Correctness (high confidence): usage_count_avg in pg_buffercache_summary now averages a 0/1 cooling bit instead of the 0..5 usage counter, so the column silently becomes "fraction of HOT buffers" while keeping its old name and documented meaning. This is a POLA violation for anyone consuming this column. Rename/redefine the column and update the docs, or keep the reported quantity consistent with its name.
| CHECK_FOR_INTERRUPTS(); | ||
|
|
||
| usage_count = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usage_count = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
Consistency (moderate confidence): pg_buffercache_usage_counts now only ever emits rows for counts 0 and 1 (BM_MAX_USAGE_COUNT is redefined to BUF_COOLSTATE_HOT = 1), whereas the documented/expected behavior is one row per usage count 0..5. Array indexing is safe (2-element arrays sized by BM_MAX_USAGE_COUNT+1), but the SQL result shape changes silently. Update the function's docs and its regression test expected output, and consider whether the usage_count column here should be renamed to reflect the cooling-state semantics.
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
This entire file is a personal developer debugging artifact, not part of any feature. A patch destined for pgsql-hackers/commitfest must be minimal and contain only what the change requires. Committing a top-level .gdbinit (alongside .clangd and pg-aliases.sh) pollutes the tree, makes the patch "do more than one thing," and is an immediate rejection reason. It also has security implications: a .gdbinit in a working directory auto-executes GDB commands, and shipping one in-tree is a footgun. Remove it from the patch and keep it local (e.g. via a personal, un-tracked gitignore).
| SpinLockAcquire(&StrategyControl->buffer_strategy_lock); | ||
| StrategyControl->completePasses++; | ||
| SpinLockRelease(&StrategyControl->buffer_strategy_lock); |
There was a problem hiding this comment.
completePasses is double-counted when batch_size > 1. When a fetch-add lands with start < NBuffers but the batch straddles the boundary (start + batch_size > NBuffers), this branch increments completePasses. But that same fetch-add pushes nextVictimBuffer past NBuffers, so the next backend to fetch-add gets start >= NBuffers and takes the wrap-CAS branch above, which increments completePasses again for the same physical wrap. Trace (NBuffers=100, batch=16): backend A start=90 -> straddle branch -> completePasses++ (counter now 106); backend B start=106 -> wrap branch -> CAS 106->6, completePasses++. One physical wrap, two increments. StrategySyncStart returns this to BgBufferSync, where an inflated passes_delta inflates strategy_delta (Assert(strategy_delta >= 0) and lap arithmetic), mispacing the LRU writer. The straddle-crossing is already implicitly accounted for by nextVictimBuffer / NBuffers in StrategySyncStart plus the eventual wrap-CAS, so this extra increment is redundant/wrong.
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
This one-line change (execl -> execlp) is unrelated to the rest of the patch, which is a buffer-manager cooling-stage change, and does not belong in this commit. On -hackers a patch that bundles an unrelated change is a top rejection reason; drop this hunk (and the .clangd/.gdbinit/pg-aliases.sh dev artifacts) so the change reads as one logical unit.
Beyond patch hygiene, the switch is a behavioral loosening with no benefit for the in-tree use. shellprog is the compiled-in SHELLPROG, which is $(SHELL) from make or /bin/sh from meson -- an absolute path. execl invokes exactly that binary. execlp additionally performs a PATH search when the name contains no slash, so if SHELLPROG is ever a bare name the test harness will exec whatever sh sits earliest in the caller's PATH instead of the intended shell -- a reproducibility/security footgun in a test runner. There is no accompanying rationale, test, or commit-message justification. Revert to execl unless there is a documented need, in which case it should be a standalone patch. (high confidence)
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
| * max") reads naturally. | ||
| */ | ||
| #define BM_MAX_USAGE_COUNT 5 | ||
| #define BM_MAX_USAGE_COUNT BUF_COOLSTATE_HOT |
There was a problem hiding this comment.
BM_MAX_USAGE_COUNT is redefined from 5 to 1 while keeping the historical name. This is a backward-compatibility footgun: the macro is visible to extensions (it lives in buf_internals.h and is used by contrib/pg_buffercache, which sizes usage_counts[BM_MAX_USAGE_COUNT + 1] on it). Any out-of-tree consumer that still assumes the 0..5 range now silently gets a 2-element domain, with no compile-time signal. The macro name no longer describes anything ("max usage count" when there is no usage count), which the comment itself concedes ("reads naturally"). Prefer renaming to something that reflects the new meaning (e.g. reference the cooling state directly, BUF_COOLSTATE_HOT) at the call sites and dropping the misleading alias, rather than overloading a semi-public name with new semantics.
| /* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so | ||
| * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. */ |
There was a problem hiding this comment.
PostgreSQL multi-line comment style puts the opening /* on its own line; here comment text starts on the same line as /*. This will not survive pgindent unchanged and is inconsistent with the surrounding comments in this file. s//* Cooling state...//*\n * Cooling state.../
| /* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so | |
| * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. */ | |
| /* | |
| * Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so | |
| * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. | |
| */ |
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
This shebang change breaks tree-wide consistency and is not minimal. Every other Perl script in the tree uses #!/usr/bin/perl (genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, and ~30 others). Changing only pgindent to /usr/bin/env perl is an unrelated, standalone edit with no functional change requiring it, so it will be flagged as churn on pgsql-hackers. It also weakens reproducibility: /usr/bin/env perl picks up whatever perl is first on PATH (e.g. a perlbrew/plenv build) rather than the system perl the tree standardizes on, which can change pgindent's output and is a POLA violation for a formatting tool. Revert to the canonical shebang.
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
2746528 to
3f6fff3
Compare
There was a problem hiding this comment.
🔍 OCR found 28 issue(s).
- 25 inline, 3 in summary (inline capped at 25)
📄 src/include/storage/buf_internals.h (L141-L142)
BUF_STATE_GET_USAGECOUNT is now dead: after this change every caller reads BUF_STATE_GET_COOLSTATE instead, and a codebase-wide search finds no remaining users of BUF_STATE_GET_USAGECOUNT (only this definition). Since the accessor no longer reflects a real field (the former usagecount is now a 1-bit cool state + 1-bit ref bit), leaving it in place is a footgun: a future caller could resurrect it expecting the old 0..5 count and instead get the raw 2-bit cool|ref field (0..3). Either remove it or repoint it explicitly. Confidence: high.
📄 src/include/storage/buf_internals.h (L139-L140)
Multi-line block comments in this file (and the tree convention) put /* alone on its own line. This opens with text on the /* line. Minor style nit; align with the surrounding comments in this header. Confidence: medium.
💡 Suggested change
Before:
+/* Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so
+ * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test. */
After:
+/*
+ * Cooling state (HOT/COOL) from buffer state -- bit 0 of the field only, so
+ * the second-chance ref bit (bit 1) does not perturb the HOT/COOL test.
+ */
📄 src/include/storage/buf_internals.h (L191-L191)
Retaining the name BM_MAX_USAGE_COUNT for a value that is now the single-bit HOT constant (1) is a POLA violation: a maintainer reading usage_counts[BM_MAX_USAGE_COUNT + 1] in pg_buffercache, or a loop bounded by it, will assume a 0..5 multi-value count and be misled. The pin fast path already uses BUF_COOLSTATE_HOT directly, so the "reads naturally" justification is thin. Prefer removing this alias and using BUF_COOLSTATE_HOT at the (few) sites, or renaming, to avoid a lingering misnomer in the core buffer manager. Confidence: medium.
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
This .clangd file is personal, developer-local editor tooling that must not be part of a PostgreSQL patch. It is unrelated to the actual change (the HOT indexed updates / buffer manager work in the other files) and does more than the patch's stated purpose. A few concrete problems:
- It is not minimal-diff: adding a per-developer LSP config alongside
.gdbinitandpg-aliases.shis exactly the kind of unrelated scaffolding that gets a patch rejected on -hackers. The tree has no.clangd, and adding one imposes one contributor's editor setup on everyone. - It hardcodes a relative include path
-I../../../../src/include, which only resolves from a specific nested subdirectory and is meaningless at the repository root where this file lives. -DPGDLLIMPORT=silently blanks outPGDLLIMPORT, which would hide exactly the Windows/MSVC export-annotation issues the project cares about (a footgun for anyone relying on clangd diagnostics).
If local clangd config is desired, it belongs in the developer's own environment (or .git/info/exclude / a personal gitignore), not committed to the tree. Recommend dropping this file (and .gdbinit, pg-aliases.sh) from the patch entirely.
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
This silently repurposes a user-visible column. pg_buffercache_pages still exposes this value through the usage_count int2 column (its name and docs unchanged), but it will now report a 0/1 cooling-state bit instead of the documented 0..5 clock-sweep usage count. That is a backward-incompatible semantic change to a stable extension view with no accompanying pg_buffercache SQL/version bump or doc update, so existing monitoring queries silently break. Either keep the column reporting a true count, or rename/redocument the column (new extension version) and rename the misleading local usagecount variable to reflect that it now holds a HOT/COOL state. high confidence.
| { | ||
| buffers_used++; | ||
| usagecount_total += BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount_total += BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
usagecount_total now accumulates the 0/1 cooling-state bit, so the view's usagecount_avg column (name/docs unchanged) silently changes meaning from an average 0..5 usage count to an average HOT/COOL bit. Same backward-compatibility concern as in pg_buffercache_pages: needs a column rename + doc/version update, not a silent substitution. high confidence.
| CHECK_FOR_INTERRUPTS(); | ||
|
|
||
| usage_count = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usage_count = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
pg_buffercache_usage_counts groups rows by usage_count (0..5). With BM_MAX_USAGE_COUNT now 1, this SRF collapses to just two rows (0 and 1) whose meaning is a HOT/COOL bit, not a usage count, while the output column is still named usage_count. This is a user-visible, backward-incompatible change to the view output with no doc/SQL update and a now-misleading usage_count local variable name. high confidence.
| # HOT Indexed Updates — GDB breakpoints for code review | ||
| # | ||
| # Usage: gdb -x .gdbinit <postgres-binary> | ||
| # Or from gdb: source .gdbinit |
There was a problem hiding this comment.
This .gdbinit is a personal debugging artifact and should not be part of a PostgreSQL contribution. It is not referenced by any build/test/gitignore infrastructure (confirmed: no references anywhere in the tree), is required by none of the actually-changed files (bufmgr/freelist/localbuf/pg_buffercache/pg_regress/pgindent/.clangd/pg-aliases.sh are all unrelated to HOT indexed updates), and placing an auto-loading .gdbinit at the repo root is a footgun: any developer running gdb in this directory silently gets these breakpoints (and gdb may refuse to auto-load or warn, depending on auto-load safe-path). Drop this file from the patch entirely.
Severity: high confidence.
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
This execl -> execlp change is unrelated to the rest of this changeset (buffer-manager work in bufmgr.c/freelist.c/localbuf.c) and does not belong in this patch. Split it out per minimal-diff discipline.
More importantly, it is a functional no-op as written: shellprog is SHELLPROG, defined as $(SHELL) in GNUmakefile and hardcoded to /bin/sh in meson.build -- always an absolute path. execlp only performs a PATH search when the program name contains no slash, so with an absolute path it behaves identically to execl. The change buys nothing.
If the intent is to let shellprog be resolved via PATH, that is a real behavioral change (and a security footgun -- searching PATH for the shell that runs test commands) that needs justification, tests, and a matching change to how SHELLPROG is defined. Otherwise revert this hunk. (high confidence)
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
| current = pg_atomic_read_u32(&StrategyControl->nextVictimBuffer); | ||
| if (current >= (uint32) NBuffers) | ||
| { | ||
| wrapped = current % NBuffers; |
There was a problem hiding this comment.
With batching, nextVictimBuffer is only wrapped opportunistically: the CAS uses the re-read current, and racing backends can leave the shared counter well above NBuffers (2*NBuffers or more). StrategySyncStart() then computes *complete_passes += nextVictimBuffer / NBuffers, which is no longer bounded by 1 as the stock code assumes, and StrategySyncStart's result = nextVictimBuffer % NBuffers is the only thing keeping the returned index in range. Combined with the double-increment above, the completed-pass count reported to BgBufferSync becomes unreliable. The invariant "nextVictimBuffer stays within [0, NBuffers) except for a transient single wrap" is broken; the wrap logic needs to guarantee the counter is brought back below NBuffers deterministically before other backends advance it further.
| #include "storage/proc.h" | ||
| #include "storage/shmem.h" | ||
| #include "storage/subsystems.h" | ||
| #include "port/pg_numa.h" |
There was a problem hiding this comment.
Include ordering: PostgreSQL groups and alphabetizes headers within a block. port/pg_numa.h is appended after the storage/* includes; port/ sorts before storage/. Move it up to keep the block ordered and pgindent/reviewer-clean.
| * grace before eviction. | ||
| */ | ||
| if (BUF_STATE_GET_REFBIT(local_buf_state)) | ||
| local_buf_state &= ~BUF_REFBIT; /* second chance: clear ref, stay HOT */ |
There was a problem hiding this comment.
These column-aligned trailing comments on the assignment statements are likely to be reflowed by pgindent (which manages tab alignment) and produce spurious diff churn / possible trailing-whitespace. Prefer a single explanatory comment above the if/else rather than aligned inline comments on each branch.
| * position above the cooling-state bit. PinBuffer sets it on every access. | ||
| * The bgwriter's pre-cooling gives a HOT buffer a second chance: the first | ||
| * time it passes a HOT buffer whose ref bit is set, it clears the ref bit and | ||
| * leaves the buffer HOT; only a HOT buffer whose ref bit is already clear (not | ||
| * re-accessed since the previous bgwriter pass) is demoted to COOL. This | ||
| * keeps genuinely-hot pages out of the COOL stage (protecting the working set | ||
| * from being cooled under scan pressure) while leaving the foreground sweep a | ||
| * single-pass search over the pre-staged COOL buffers. A separate bit (not a | ||
| * count) so it stays a plain masked store under the header lock. |
There was a problem hiding this comment.
This comment misdescribes the shipped behavior (stale/aspirational design). It states the bgwriter's pre-cooling applies the second chance and pre-stages COOL buffers for a "single-pass" foreground sweep. But the shipped code does the opposite:
- The foreground sweep in freelist.c is a "COOL-IN-PLACE" sweep that demotes HOT->COOL on every tick and applies the second-chance ref-bit logic itself (freelist.c:
if (BUF_STATE_GET_REFBIT(...)) local_buf_state &= ~BUF_REFBIT;), not the bgwriter. - BgBufferSync calls
SyncOneBuffer(next_to_clean, true, false, wb_context)withcool_if_hot = false, and its own comment says the bgwriter does "no background pre-cooling" because the sweep handles all cooling.
So the described bgwriter pre-cooling path is not exercised, and this header comment attributes the second-chance mechanism to the wrong actor. Rewrite it to describe what the foreground sweep actually does (per the review standard that comments describe current behavior, not a superseded design). Confidence: high.
There was a problem hiding this comment.
🔍 OCR found 31 issue(s).
- 25 inline, 6 in summary (inline capped at 25)
📄 src/tools/pgindent/pgindent (L1-L1)
This shebang change diverges from the project-wide convention: every other Perl script in the tree (genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, etc.) uses #!/usr/bin/perl. Switching only pgindent to #!/usr/bin/env perl is an unrelated, inconsistent change that does not belong in a functional patch. Such a convention change would need its own -hackers discussion applied tree-wide, not a one-off here. Revert to keep the diff minimal and consistent.
💡 Suggested change
Before:
#!/usr/bin/env perl
After:
#!/usr/bin/perl
📄 src/include/storage/buf_internals.h (L100-L102)
The atomicity claim here is inaccurate for the primary hot path. PinBuffer() in bufmgr.c sets BUF_REFBIT inside a lock-free CAS loop (pg_atomic_compare_exchange_u64) precisely because it is not allowed to touch the refcount while holding the header spinlock (see the comment at bufmgr.c: "We're not allowed to increase the refcount while the buffer header spinlock is held"). The ref bit is only cleared under the header lock (SyncOneBuffer via UnlockBufHdrExt). Stating it "stays a plain masked store under the header lock" misdescribes the mechanism and is a footgun: a future reader may assume all BUF_REFBIT mutations are lock-protected and drop the CAS. Reword to reflect that the set path is a CAS-loop store folded into the existing pin CAS, and only the clear path runs under the header lock. (high confidence)
📄 src/backend/storage/buffer/freelist.c (L195-L197)
completePasses accounting is inconsistent with StrategySyncStart under batching, and drifts in both directions.
StrategySyncStart() returns completePasses + nextVictimBuffer / NBuffers. The stock invariant is that every time the counter crosses a multiple of NBuffers, exactly one pass is folded into completePasses. With batching two problems appear:
-
Undercount: a single fetch-add of
batch_sizecan push nextVictimBuffer across the boundary while landing >= NBuffers withcurrent / NBuffers > 1; the wrap path folds in onlycompletePasses++(one pass) and setswrapped = current % NBuffers, discarding the extra whole passes. -
Double relative to the transient term: this
else ifbranch increments completePasses for a sub-NBuffers batch that straddles the wrap, but it leaves nextVictimBuffer un-wrapped (>= NBuffers). StrategySyncStart then ALSO addsnextVictimBuffer / NBuffersfor the same crossing, counting it twice until the next >=NBuffers fetch-add wraps the counter.
The comment concedes this feeds only bgwriter pacing, but the drift is unbounded (grows with concurrency x batch), which can materially misestimate BgBufferSync's scan-ahead. At minimum this needs the fold to account for current / NBuffers passes and the two paths to be reconciled with the StrategySyncStart transient term. (high confidence)
📄 src/backend/storage/buffer/freelist.c (L24-L25)
Include is out of order. The storage/* group is alphabetized (buf_internals, bufmgr, proc, shmem, subsystems); port/pg_numa.h belongs in the port/ group right after port/atomics.h, not appended at the end. pgindent won't fix include ordering, so a committer will bounce this.
💡 Suggested change
Before:
#include "storage/subsystems.h"
#include "port/pg_numa.h"
After:
#include "port/atomics.h"
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proc.h"
#include "storage/shmem.h"
#include "storage/subsystems.h"
📄 src/backend/storage/buffer/freelist.c (L496-L499)
This diff bundles two independent, user-visible changes: (1) NUMA-gated batched clock sweep and (2) a full replacement-policy rewrite from multi-value usage_count to a 1-bit cool-state (BUF_COOLSTATE/BUF_REFBIT). Per pgsql-hackers norms these must be split into separate, independently-committable patches. Neither ships regression/TAP tests or docs, and both change buffer-eviction behavior. Additionally the changeset carries developer-environment cruft (.clangd, .gdbinit, flake.nix, shell.nix, pg-aliases.sh) that must not be part of the patch. As-is this draws immediate rejection on the list. (high confidence)
📄 src/backend/storage/buffer/freelist.c (L288-L291)
Comment overstates the shipped design and contradicts the sibling changes. It claims the sweep "never needs a background pre-cooler" and "cannot get stuck in a 'no COOL victim anywhere' state that forces a foreground escalation pass." But bufmgr.c's SyncOneBuffer() still implements a cool_if_hot pre-cooling path, and buf_internals.h's BUF_REFBIT comment describes "the bgwriter's pre-cooling gives a HOT buffer a second chance." Even though BgBufferSync currently calls SyncOneBuffer(..., cool_if_hot=false), the dead pre-cooling machinery plus these mutually contradictory comments will confuse reviewers. Either wire it in or remove it, and make the comments describe what the code does now. (moderate confidence)
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
This .clangd file should not be committed. It is an editor/language-server auxiliary file, and the repository's top-level .gitignore explicitly documents the policy: "Auxiliary files from local workflows, your preferred editor, etc. should be ignored locally using $GIT_DIR/info/exclude or ~/.gitexclude." This change is also unrelated to the buffer-manager work in the rest of the patch (bufmgr.c/freelist.c/localbuf.c) and bundles personal tooling into a functional patch. A patch posted to pgsql-hackers containing personal IDE config will be rejected as out-of-scope. Remove it from the commit and exclude it locally instead.
| - -I. | ||
| - -I../../../../src/include |
There was a problem hiding this comment.
The include path -I../../../../src/include is hardcoded to a relative depth that only resolves when clangd is invoked from one specific subdirectory nesting (four levels below the tree root). Combined with CompilationDatabase: build/, this reflects a single developer's local layout and won't work generally. This further confirms the file is a personal, non-portable local artifact that does not belong in the shared repository.
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
This is a user-visible behavior change that leaks a private buffer-manager rename into a monitoring extension's public API. usage_count/usagecount here now only ever reports 0 (COOL) or 1 (HOT) instead of the documented 0..5 range, while the SQL-visible column names (usage_count int4 in pg_buffercache--1.3--1.4.sql) and the pg_buffercache.pages usage_count column are unchanged. Consumers filtering/grouping on usage_count (e.g. ... WHERE usage_count = 5) silently break.
Also, the semantics no longer match the name: BUF_STATE_GET_COOLSTATE returns a HOT/COOL bit, not a usage count. Wiring it into a variable/column still called usagecount is misleading (POLA).
At minimum this change needs: (1) doc updates in contrib/pg_buffercache's SGML describing the new range/meaning, (2) updated column/naming so the exposed value is not called usage_count if it is now a cooling bit, and (3) this should not ride along silently with the buf_internals change -- it is a separate, documented, user-facing decision. Tag: high confidence on the behavior/doc drift.
| CHECK_FOR_INTERRUPTS(); | ||
|
|
||
| usage_count = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usage_count = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
In pg_buffercache_usage_counts the array bounds happen to stay safe (BM_MAX_USAGE_COUNT is now BUF_COOLSTATE_HOT == 1, so usage_counts/dirty/pinned are size 2 and BUF_STATE_GET_COOLSTATE returns 0 or 1). But the emitted result set now has only two rows (usage_count 0 and 1) where callers expect six (0..5). The existing regress test only asserts count(*) > 0, so this contract change is untested. Add a regression test that pins the expected row count/values, and update the docs. Tag: high confidence.
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath is a NixOS module option, not a valid flake output. Because it sits inside the eachDefaultSystem closure, it becomes environment.<system>.localBinInPath in the flake outputs, which the flake schema does not recognize and which has no effect on the dev shells. It is dead config that should be removed (the option only makes sense inside a NixOS configuration.nix / module, not here).
| write_limit = bgwriter_lru_maxpages; | ||
| if (upcoming_alloc_est > write_limit) | ||
| write_limit = upcoming_alloc_est; |
There was a problem hiding this comment.
This silently overrides the user-facing GUC bgwriter_lru_maxpages, which is documented as the maximum number of pages the bgwriter may write per round. Raising the cap to upcoming_alloc_est violates POLA and the GUC's contract, changes the observable I/O footprint under bulk workloads, and ships with no doc update, no benchmark, and no design-thread reference. Note also that maxwritten_clean is incremented at line ~4127 when this raised limit is reached, mislabeling the stat that is meant to count hits of the maxpages cap. If ahead-of-demand cleaning is desired, this needs to be justified as a behavior change (or gated), not folded in silently. Moderate confidence.
| * a shorter window than the strategy proxy, tracking a burst of | ||
| * probationary/scan COOL pages within a cycle or two instead of lagging it. | ||
| */ | ||
| float cleaner_smoothing_samples = 4; |
There was a problem hiding this comment.
cleaner_smoothing_samples = 4 is an unexplained magic constant that changes long-standing bgwriter LRU pacing (the original used smoothing_samples = 16 here). No GUC, no benchmark, no design-discussion reference for the 16 -> 4 change in adaptivity. Justify the value or make it a named/derived constant with rationale; a large behavioral tuning change should not ride in on a bare literal. Moderate confidence.
| set_bits |= BM_TAG_VALID; | ||
| /* Admit the newly loaded page COOL (probation); a second access via | ||
| * PinBuffer promotes it to HOT. This is what makes a one-touch scan | ||
| * self-evicting -- see the cooling-state notes in buf_internals.h. */ |
There was a problem hiding this comment.
PostgreSQL block comments start with the text on the line after the opening /* (see the surrounding house style). This same-line, multi-line form and the paragraph-length narration of the algorithm will not survive pgindent cleanly and describe WHAT more than WHY. Reflow to standard style and trim to the rationale. Also verify -- here and elsewhere in the new comments is ASCII hyphen-minus, not a smart/em dash (source must be ASCII-only). Low confidence.
| set_bits |= BM_TAG_VALID; | |
| /* Admit the newly loaded page COOL (probation); a second access via | |
| * PinBuffer promotes it to HOT. This is what makes a one-touch scan | |
| * self-evicting -- see the cooling-state notes in buf_internals.h. */ | |
| set_bits |= BM_TAG_VALID; | |
| /* | |
| * Admit the newly loaded page COOL (probation); a second access via | |
| * PinBuffer promotes it to HOT. This is what makes a one-touch scan | |
| * self-evicting; see the cooling-state notes in buf_internals.h. | |
| */ |
| if (BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL) | ||
| { | ||
| buf_state -= BUF_USAGECOUNT_ONE; | ||
| /* HOT: give it a second chance, cool it and keep scanning. */ |
There was a problem hiding this comment.
The comment "give it a second chance" is inaccurate for the local sweep and drifts from what the code does. In the shared-buffer sweep (freelist.c) the "second chance" is the second-chance reference bit (BUF_REFBIT): a HOT buffer with the ref bit set is spared (ref bit cleared, stays HOT) and only demoted on the next pass. Here there is no ref-bit logic at all: a HOT local buffer is demoted HOT -> COOL unconditionally on the first pass this sweep passes it. Local buffers never set BUF_REFBIT (PinLocalBuffer does not set it), so the reprieve the phrase implies does not exist. Reword to describe the actual behavior, e.g. "HOT: demote to COOL and keep scanning; it becomes a candidate on the next pass." (moderate confidence)
| /* HOT: give it a second chance, cool it and keep scanning. */ | |
| /* HOT: demote to COOL and keep scanning; a later pass may evict it. */ |
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
This execl -> execlp change is a behavioral change with no visible justification, no comment, and no test. In practice shellprog is SHELLPROG, i.e. /bin/sh (meson.build) or $(SHELL) (GNUmakefile) — normally an absolute path. Per POSIX, execlp performs a PATH search only when the file argument contains no slash; with a slash it behaves exactly like execl. So for standard builds this is a no-op, and the only case it changes is when shellprog has no directory component, where it now resolves the shell via the runtime PATH.
Two problems for a pgsql-hackers patch:
- The rationale is absent. If the intent is to support a
$(SHELL)value without a directory component, state it in the commit message and add a comment here; otherwise this reads as an unrelated, unexplained diff (minimal-diff discipline). Confidence: high. - Resolving the interpreter through
PATHinstead of a build-configured absolute path is a footgun (non-deterministic/less secure resolution). If that is not the goal, keepexecl. Confidence: moderate.
Either add justification + a comment (and ideally a test exercising the intended new behavior), or revert to execl.
There was a problem hiding this comment.
🔍 OCR found 40 issue(s).
- 25 inline, 15 in summary (inline capped at 25)
📄 pg-aliases.sh (L417-L417)
The regression port is hardcoded to --port=40099 with no override. Two concurrent pg-test-one runs, or a leftover backend still bound to 40099, will collide and fail non-obviously. PostgreSQL's own test harness derives a unique port; at minimum make this configurable (e.g. --port="${PG_TEST_PORT:-40099}") or pick a random free port.
📄 src/tools/pgindent/pgindent (L1-L1)
This shebang change is unrelated to the PR's stated purpose (buffer-manager work) and breaks a tree-wide convention: every other Perl script in the source tree uses #!/usr/bin/perl (verified: genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, and ~30 others). Making pgindent the sole #!/usr/bin/env perl outlier is inconsistent, and env perl performs a PATH search that resolves to whatever perl happens to be first on PATH (a footgun in mixed environments / with local perlbrew installs). This is a local-convenience change that does not belong in a patch destined for -hackers; drop it to keep the diff minimal. (high confidence)
💡 Suggested change
Before:
#!/usr/bin/env perl
After:
#!/usr/bin/perl
📄 src/test/regress/pg_regress.c (L1246-L1246)
Switching execl -> execlp is unrelated to the PR's stated purpose and changes behavior subtly: execlp performs a PATH search when the program name contains no slash. With the default SHELLPROG ($(SHELL) via autoconf, "/bin/sh" via meson) the value is an absolute path, so today this is behavior-neutral; but if $(SHELL) were ever a bare name, the shell would be resolved from PATH instead of the configured path -- a non-obvious footgun in a test harness that should run a deterministic shell. There is no justification in this PR for the change. Recommend reverting to keep the diff minimal, or, if intentional, splitting it into its own commit with a rationale. (moderate confidence)
💡 Suggested change
Before:
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
After:
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
📄 pg-aliases.sh (L508-L508)
modified_files is assigned without local, so it leaks into the interactive shell that sourced pg-aliases.sh and persists after pg-format returns. The sibling helpers pg-tidy and pg-spell correctly declare local files. Declare it local for consistency and to avoid polluting the caller's environment.
💡 Suggested change
Before:
modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$")
After:
local modified_files
modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$")
📄 shell.nix (L389-L389)
TRANSACTIONS ($3) is captured, printed in the config banner and written into the results header, but it is never passed to pgbench: the run is driven purely by -T "$DURATION" (time-based). A user who passes a transaction count (e.g. via pg-bench-run 10 2 5000 ...) will see it echoed but it has no effect on the benchmark. Either pass it through with -t "$TRANSACTIONS" (mutually exclusive with -T) or drop the parameter to avoid the misleading knob.
📄 src/backend/storage/buffer/bufmgr.c (L4231-L4248)
Dead code: cool_if_hot is passed false from both call sites (BufferSync line 3790, BgBufferSync line 4112), so this entire pre-cooling block never executes. BUF_COOLED is set here but never read by any caller (callers only test BUF_WRITTEN/BUF_REUSABLE). Per the minimalism/YAGNI discipline, drop the cool_if_hot parameter, this block, and the BUF_COOLED flag entirely, or wire up a caller that actually uses them. Shipping the unused path also carries the unlock/re-lock overhead in the signature and confuses readers about which cooling strategy is live. (high confidence)
📄 src/backend/storage/buffer/bufmgr.c (L3345-L3347)
Stale comment: it claims the ref bit lets "the bgwriter's next cooling pass spare this recently-used buffer", but the bgwriter never runs a cooling pass -- both SyncOneBuffer call sites pass cool_if_hot=false (see the comment at line 4107). The ref bit is actually consumed only by the foreground clock sweep in StrategyGetBuffer(). Update the comment to describe the live behavior. (medium confidence)
📄 src/backend/storage/buffer/bufmgr.c (L4095-L4097)
This silently overrides the documented meaning of bgwriter_lru_maxpages. config.sgml states "no more than this many buffers will be written by the background writer" per round; raising write_limit to upcoming_alloc_est breaks that contract under bulk-dirtying. A user who lowered the GUC to intentionally leave writes to backends no longer gets the cap they set. This is a user-visible GUC-semantics change shipped with no doc update and no regression test -- WIP by pgsql-hackers standards. Either document the new behavior (and ideally gate it behind an opt-in) or drop the override. (medium confidence)
📄 src/backend/storage/buffer/freelist.c (L195-L197)
completePasses can be over-counted per sweep pass. The start >= NBuffers branch increments it at most once (via CAS), but this else if branch increments unconditionally for every backend whose batch straddles the wrap point. With batching and many concurrent backends, several batches can straddle NBuffers in the same pass (and this can co-occur with the wrap-CAS increment), so completePasses can advance by more than one per real pass. StrategySyncStart also adds nextVictimBuffer/NBuffers on top. Since completePasses feeds bgwriter pacing, confirm the resulting skew is acceptable, or count the wrap crossing exactly once. (medium confidence)
📄 src/backend/storage/buffer/freelist.c (L24-L25)
Include is out of order. PostgreSQL sorts includes alphabetically within the group; "port/pg_numa.h" belongs right after "port/atomics.h", not after the storage/* block. pgindent/committer style will flag this. (low confidence)
💡 Suggested change
Before:
#include "storage/subsystems.h"
#include "port/pg_numa.h"
After:
#include "port/atomics.h"
#include "port/pg_numa.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proc.h"
#include "storage/shmem.h"
#include "storage/subsystems.h"
📄 src/include/storage/buf_internals.h (L93-L96)
Stale comment: it describes "the bgwriter's pre-cooling" giving a HOT buffer a second chance, but the bgwriter's SyncOneBuffer calls all pass cool_if_hot=false, so no bgwriter pre-cooling pass exists. The second-chance ref bit is consumed by the foreground clock sweep in StrategyGetBuffer(). Rewrite so the comment describes the actual (foreground-sweep) behavior rather than the dead bgwriter path. (medium confidence)
📄 contrib/pg_buffercache/pg_buffercache_pages.c (L164-L164)
User-visible semantic change without a doc/version update. This column previously reported the 0..5 usage_count; it now reports the 0/1 cooling state. pg_buffercache_usage_counts() will now only emit rows for usage_count 0 and 1 (the [BM_MAX_USAGE_COUNT+1] arrays are correctly resized to [2], so no OOB), and pg_buffercache.usagecount changes meaning. The pg_buffercache documentation (pgbuffercache.sgml) still describes a 0..5 usage count and needs updating; the expected regression output (expected/pg_buffercache*.out) will also change. Ship the doc/test updates with this change. (medium confidence)
📄 src/backend/storage/buffer/bufmgr.c (L2338-L2340)
Non-standard block comment format: PostgreSQL style (and pgindent) require the opening /* to be alone on its own line for a multi-line block comment. Text on the same line as /* will be reflowed by pgindent, creating diff churn against the committed tree. Same issue at the ExtendBufferedRelShared admission site below. (low confidence / style)
💡 Suggested change
Before:
/* Admit the newly loaded page COOL (probation); a second access via
* PinBuffer promotes it to HOT. This is what makes a one-touch scan
* self-evicting -- see the cooling-state notes in buf_internals.h. */
After:
/*
* Admit the newly loaded page COOL (probation); a second access via
* PinBuffer promotes it to HOT. This is what makes a one-touch scan
* self-evicting -- see the cooling-state notes in buf_internals.h.
*/
📄 src/backend/storage/buffer/bufmgr.c (L3010-L3011)
Same non-standard block comment format as the other admission site: the opening /* must be alone on its own line. pgindent will reformat this. (low confidence / style)
💡 Suggested change
Before:
/* Admit COOL (probation); see the comment at the other admission
* site and the cooling-state notes in buf_internals.h. */
After:
/*
* Admit COOL (probation); see the comment at the other admission
* site and the cooling-state notes in buf_internals.h.
*/
📄 src/backend/storage/buffer/localbuf.c (L253-L254)
Comment is inaccurate for the local-buffer path. Unlike the shared clock sweep in StrategyGetBuffer(), this path implements no second-chance reference bit: PinLocalBuffer() never sets BUF_REFBIT, and this branch unconditionally demotes HOT -> COOL via buf_state -= BUF_COOLSTATE_ONE. There is no "second chance" here. The comment should describe the actual single-bit HOT->COOL demotion. (moderate confidence)
💡 Suggested change
Before:
/* HOT: give it a second chance, cool it and keep scanning. */
buf_state -= BUF_COOLSTATE_ONE;
After:
/* HOT: demote to COOL and keep scanning. */
buf_state -= BUF_COOLSTATE_ONE;
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
This .gdbinit is a personal debugger artifact that must not be committed. The repository's own root .gitignore (lines 1-3) states policy explicitly: "Auxiliary files from local workflows, your preferred editor, etc. should be ignored locally using $GIT_DIR/info/exclude or ~/.gitexclude." A debugger config for one developer's session belongs in $GIT_DIR/info/exclude, not in a patch destined for pgsql-hackers/commitfest.
Beyond scope, this is a security footgun: GDB auto-loads .gdbinit from the current working directory, so any developer who runs gdb from the repo root silently executes its contents -- a well-known code-execution vector. It also defeats the auto-load safe-path protections GDB ships to warn against exactly this. [high confidence] Recommend removing this file from the change entirely.
| # ========================================================================= | ||
|
|
||
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
The hardcoded source line breakpoints do not match the current tree and reference nonexistent code. Verified against the repo:
break heapam.c:3210is a comment forsimple_heap_delete, notheap_update(the file has 9524 lines; the comment here is wrong).break heapam.c:4019/4024/4033point to the deadlock-retry / no-TOAST branch of the existing update path, not any "pure HOT vs HOT indexed" decision.break heapam_indexscan.c:250/297and:182reference HOT-indexed accumulator logic, but that file is only 299 lines and contains no such code.
Furthermore, the function-name breakpoints (heap_hot_indexed_create_tuple, heap_hot_indexed_serialize_bitmap, heap_hot_indexed_read_bitmap, heap_xlog_indexed_update, etc.) and the HEAP_INDEXED_UPDATED flag do not exist anywhere in the tree (code_search returned no matches). The entire "HOT indexed updates" series this file debugs is absent from both this review group and the other-changed-files list. This makes the file dead/misleading on its own commit and violates the atomic/bisectable-commit requirement: it documents and depends on code that was never added. [high confidence]
| ParameterNames: true | ||
| DeducedTypes: true | ||
| CompileFlags: | ||
| CompilationDatabase: build/ # Search build/ directory for compile_commands.json |
There was a problem hiding this comment.
This .clangd LSP config is a personal editor artifact and should not be committed, per the repository's own root .gitignore policy (lines 1-3: "Auxiliary files from ... your preferred editor ... should be ignored locally using $GIT_DIR/info/exclude or ~/.gitexclude"). It is out of scope for a pgsql-hackers/commitfest patch and pollutes the diff. [high confidence]
Additionally the config is not portable to other contributors: CompilationDatabase: build/ and -I../../../../src/include encode one developer's build layout and assumed working directory (no build/ compilation database exists or is generated at a standard tracked location in this tree). As a repo-root config it silently applies -DDEBUG, -DLOCAL, -DPGDLLIMPORT= and other flags to every file opened in any clangd-enabled editor, which can mask real diagnostics (e.g. defining PGDLLIMPORT to empty hides Windows export-annotation issues) for everyone who clones the repo. Recommend removing this file from the change.
| ParameterNames: true | ||
| DeducedTypes: true | ||
| CompileFlags: | ||
| CompilationDatabase: build/ # Search build/ directory for compile_commands.json |
There was a problem hiding this comment.
Non-ASCII smart quote / em-dash characters appear in the added comments (this line uses —; .gdbinit also uses — in its header). PostgreSQL sources and diffs must be ASCII only -- no smart quotes, em-dashes, or ellipsis characters. Replace with a plain hyphen if these files are kept at all (though the primary recommendation is to drop both files). [high confidence]
|
|
||
| if [ "$DIVERGED" -gt 0 ]; then | ||
| # Check commit messages for "dev setup" or "dev v" pattern | ||
| DEV_SETUP_COMMITS=$(git log --format=%s upstream/master...origin/master | grep -iE "^dev (setup|v[0-9])" | wc -l) |
There was a problem hiding this comment.
Range operator inconsistency for DEV_SETUP_COMMITS. git log A...B (three-dot) computes the symmetric difference — commits reachable from either upstream/master or origin/master but not both. That includes upstream's new commits, so this count is inflated/misclassified and no longer reflects only local commits. To count local-only commits you want two-dot upstream/master..origin/master (as sync-upstream.yml line 61 correctly does). Since this count feeds the gating in the Attempt merge step that ultimately permits a --force-with-lease push, the mismatch between the two workflows is a real correctness hazard.
| DEV_SETUP_COMMITS=$(git log --format=%s upstream/master...origin/master | grep -iE "^dev (setup|v[0-9])" | wc -l) | |
| DEV_SETUP_COMMITS=$(git log --format=%s upstream/master..origin/master | grep -iE "^dev (setup|v[0-9])" | wc -l) |
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath = true; is a NixOS module option, not a valid flake output. Here it is placed inside the per-system attribute set returned by flake-utils.lib.eachDefaultSystem, so it is dead: nothing consumes it and it has no effect. Remove it.
| ulimit -c unlimited | ||
|
|
||
| # Local git excludes | ||
| git config core.excludesFile .local-gitignore 2>/dev/null || true |
There was a problem hiding this comment.
commonHookTail rewrites the user's repo-local git config (core.excludesFile) as a side effect of merely entering the dev shell, and points it at .local-gitignore, which is not committed to the repository. This silently mutates the checkout's git configuration on every nix develop, and references a file that does not exist. Either commit .local-gitignore and make this opt-in, or drop the automatic git config mutation.
| -P 5 \ | ||
| --log \ | ||
| --log-prefix="$OUTPUT_DIR/pgbench_$TIMESTAMP" \ | ||
| $BENCH_ARGS \ |
There was a problem hiding this comment.
$BENCH_ARGS is expanded unquoted under set -euo pipefail. It relies on word-splitting to pass multiple pgbench flags (e.g. -b select-only@70 -b tpcb-like@30), which works but is fragile and easy to break. Prefer a bash array (BENCH_ARGS=(); ...; "$PGBENCH" ... "${BENCH_ARGS[@]}" ...) so the intent is explicit and quoting-safe.
| if [ "$last_compiler" != "$current_compiler" ] && [ "$last_compiler" != "unknown" ]; then | ||
| echo "Detected compiler change from $last_compiler to $current_compiler" | ||
| echo "Cleaning build directory..." | ||
| trash "$build_dir" 2>/dev/null || rm -rf "$build_dir" |
There was a problem hiding this comment.
trash is used as the primary destructive tool here (and in pg-full-clean, pg-init, and the valgrind wrapper cleanup) with an || rm -rf fallback, but trash/trash-cli is not among the packages declared in shell.nix getPostgreSQLDeps. Inside the dev shell trash will therefore never be found and the code always falls through to the irreversible rm -rf, contradicting the safety intent of using a trash bin. Either add trash-cli to the dependency list or drop the trash indirection and document that removal is permanent.
| ulimit -c unlimited | ||
| if ! [ -w /proc/sys/kernel/core_pattern ]; then | ||
| echo "Setting kernel.core_pattern (requires sudo)..." | ||
| echo "core.%p" | sudo tee /proc/sys/kernel/core_pattern >/dev/null || { |
There was a problem hiding this comment.
pg-enable-cores/pg-disable-cores mutate the system-wide sysctl /proc/sys/kernel/core_pattern via sudo tee. This is a host-global side effect from a dev-shell helper: it changes core-dump handling for every process on the machine, and pg-disable-cores unconditionally 'restores' the pattern to a hardcoded core/core.%p, which may not match the system's original value (silently clobbering a distro or systemd-coredump configuration). Combined with ulimit -c unlimited set on every shell entry, this is an easy-to-misuse footgun. Consider dropping the sysctl writes entirely (leave core_pattern to the operator) or at least save and restore the prior value instead of assuming a default.
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
This execl -> execlp change is unrelated to the patch's purpose and should be dropped (minimal-diff discipline). shellprog is SHELLPROG, which is defined as $(SHELL) in the Makefiles and /bin/sh in meson.build -- always an absolute path. For an absolute path execlp behaves identically to execl, so this provides no functional benefit. Worse, execlp searches PATH when the program name contains no slash; if SHELLPROG were ever configured as a bare name, the test harness would resolve and exec a shell from a possibly attacker-influenced PATH -- a security regression in code that execs shell commands. Revert to execl. (high confidence)
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
Unrelated shebang change; drop it to keep the diff minimal. Every other Perl script in the tree (e.g. src/tools/copyright.pl, src/tools/git_changelog, src/tools/mark_pgdllimport.pl) uses #!/usr/bin/perl. Switching only pgindent to #!/usr/bin/env perl makes it the sole outlier and does not belong in this patch -- this looks like a local dev-environment convenience (consistent with the added flake.nix/shell.nix). Revert to #!/usr/bin/perl. (high confidence)
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
This entire file does not belong in a commitfest-bound PR. Two independent, verifiable defects make it actively misleading:
-
The hardcoded line-number breakpoints have already drifted.
heapam.c:3210is the closing brace ofheap_delete()in the current tree, notheap_update()as this comment claims. Line-number breakpoints silently point at the wrong code the moment any surrounding line changes. -
Several function-name breakpoints below reference symbols that do not exist anywhere in the tree (
heap_hot_indexed_create_tuple,heap_xlog_indexed_update, etc. -- see the comments on those lines). GDB will error on each unresolved breakpoint when the file is sourced.
Per the top-level .gitignore policy, local-workflow/editor scaffolding should be ignored via $GIT_DIR/info/exclude, not committed. Please drop this file from the patch.
| # Create augmented tuple with embedded modified-column bitmap | ||
| break heap_hot_indexed_create_tuple |
There was a problem hiding this comment.
heap_hot_indexed_create_tuple does not exist anywhere in the source tree (verified via search). The surrounding heap_hot_indexed_* breakpoints in this block are likewise unresolved symbols. Sourcing this .gdbinit will produce an error for each one. These names appear to come from an unmerged feature branch, so the file is non-functional as committed.
| # WAL replay for XLOG_HEAP2_INDEXED_UPDATE | ||
| break heap_xlog_indexed_update |
There was a problem hiding this comment.
Neither heap_xlog_indexed_update nor the WAL opcode XLOG_HEAP2_INDEXED_UPDATE exists in the tree (verified via search). This breakpoint will fail to resolve on load.
| * max") reads naturally. | ||
| */ | ||
| #define BM_MAX_USAGE_COUNT 5 | ||
| #define BM_MAX_USAGE_COUNT BUF_COOLSTATE_HOT |
There was a problem hiding this comment.
Retaining the name BM_MAX_USAGE_COUNT for what is now simply BUF_COOLSTATE_HOT (value 1) is misleading: it is no longer a usage count nor the maximum of a counter, yet it is still used to size arrays in pg_buffercache (usage_counts[BM_MAX_USAGE_COUNT + 1]) and the loop that emits usage-count buckets. The StaticAssertDecl message was also reworded to "cooling state doesn't fit..." while the asserted macro still carries the old name, so the message and the symbol disagree. Prefer referencing BUF_COOLSTATE_HOT directly at the use sites rather than keeping a historical alias whose name no longer describes its value. Confidence: moderate.
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
This is a user-visible contract change, not a mechanical rename. usagecount previously ranged over 0..5 and is documented in pgbuffercache.sgml as "Clock-sweep access count"; it now only ever reports 0 or 1. Monitoring queries, histograms, and dashboards built on the 0..5 domain will silently break. The same applies to pg_buffercache_summary.usagecount_avg (now bounded to [0,1]) and pg_buffercache_usage_counts (now only ever emits buckets 0 and 1). The pgbuffercache docs are not updated in this change and still describe the old semantics. A user-visible change like this needs the documentation updated to describe the new HOT/COOL domain. Confidence: high.
| /* Admit the newly loaded page COOL (probation); a second access via | ||
| * PinBuffer promotes it to HOT. This is what makes a one-touch scan | ||
| * self-evicting -- see the cooling-state notes in buf_internals.h. */ |
There was a problem hiding this comment.
Multi-line block comment does not follow the PostgreSQL comment style: the opening /* must be on its own line with the text starting on the next line (see the comment immediately above this hunk). As written this will not survive pgindent cleanly. Confidence: high.
| /* Admit the newly loaded page COOL (probation); a second access via | |
| * PinBuffer promotes it to HOT. This is what makes a one-touch scan | |
| * self-evicting -- see the cooling-state notes in buf_internals.h. */ | |
| /* | |
| * Admit the newly loaded page COOL (probation); a second access via | |
| * PinBuffer promotes it to HOT. This is what makes a one-touch scan | |
| * self-evicting; see the cooling-state notes in buf_internals.h. | |
| */ |
| /* Admit COOL (probation); see the comment at the other admission | ||
| * site and the cooling-state notes in buf_internals.h. */ |
There was a problem hiding this comment.
Same block-comment style problem here: /* should be on its own line for a multi-line comment, per surrounding convention and pgindent. Confidence: high.
| /* Admit COOL (probation); see the comment at the other admission | |
| * site and the cooling-state notes in buf_internals.h. */ | |
| /* | |
| * Admit COOL (probation); see the comment at the other admission | |
| * site and the cooling-state notes in buf_internals.h. | |
| */ |
| set_bits |= BM_TAG_VALID; | ||
| /* Admit the newly loaded page COOL (probation); a second access via |
There was a problem hiding this comment.
Behavioral regression / policy change (high confidence): stock code admits a demand-loaded page with usage_count=1, giving it one full clock-sweep pass of grace before it becomes an eviction candidate. Admitting COOL (0) makes every newly read-in page immediately reclaimable on the very first sweep tick that reaches it, before any opportunity for a second access to promote it. This affects ALL workloads, not just one-touch scans: a page that is read in, used once, and needed again shortly after can now be evicted a tick later and re-read. This is a fundamental buffer-replacement policy change. Per pgsql-hackers norms it needs (a) a reproducible benchmark showing it does not regress non-scan workloads, (b) regression/TAP tests, and (c) a reference to the design thread. Without those this reads as WIP, not commit-ready.
There was a problem hiding this comment.
🔍 OCR found 20 issue(s).
- 19 inline, 1 in summary
📄 .gitignore
[high confidence] This whole change set (.clangd, .gdbinit, flake.nix, shell.nix, pg-aliases.sh, and the glibc-no-fortify-warning.patch it pulls in) is personal developer-environment tooling. The repository's own policy, stated right here at the top of .gitignore, is that "Auxiliary files from local workflows, your preferred editor, etc. should be ignored locally using $GIT_DIR/info/exclude or ~/.gitexclude." None of these files belong in a patch destined for pgsql-hackers/commitfest; they add non-minimal, unrelated content and will draw immediate rejection. Keep them in a private $GIT_DIR/info/exclude instead. (This comment is anchored here only because .gitignore is the stable file documenting the policy; the files themselves should simply not be committed.)
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
This execl->execlp change is a functional no-op given the current build config, and is likely unwanted. shellprog comes from SHELLPROG, which is $(SHELL) (an absolute path like /bin/sh) in the make builds and the literal /bin/sh in the meson build. Per POSIX, execlp only performs a PATH search when the program name contains no slash; since shellprog always contains a slash here, execlp behaves identically to execl. So this does not change behavior for the shipped configuration, while introducing a footgun: if SHELLPROG were ever a bare name, the regression harness would start resolving the shell via $PATH, which is a less predictable/less secure choice for a test driver. (high confidence) Unless there is a specific need to allow a PATH-resolved shell (which should be justified with a -hackers reference), revert this to execl to keep the diff minimal.
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
This shebang change makes pgindent the only Perl script in the tree using #!/usr/bin/env perl; every other .pl/Perl tool (e.g. genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, and ~30 others) uses #!/usr/bin/perl. The change is inconsistent with the established project-wide convention and is unrelated to any functional fix here. (high confidence) Revert to keep the diff minimal and consistent; if switching to env perl is intended, it is a separate, tree-wide decision that belongs on -hackers, not a one-off edit to this file.
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| Diagnostics: | ||
| MissingIncludes: None |
There was a problem hiding this comment.
[high confidence] This is personal developer-environment config and must not be committed to a patch destined for pgsql-hackers/commitfest. The repository's top-level .gitignore explicitly directs such files to a local $GIT_DIR/info/exclude or ~/.gitexclude. Beyond that policy, -I../../../../src/include and CompilationDatabase: build/ assume a fixed working-directory depth and build layout, so this is brittle and environment-specific. Drop this file from the patch.
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
[high confidence] This developer-only debugging file must not be in a commitfest patch (see the local-workflow policy in .gitignore). Worse, it hardcodes line-number breakpoints such as heapam.c:3210, heapam.c:4019, and pruneheap.c:1802 that are tied to the exact feature diff. These silently rot the moment any of that code shifts by a line, so even as a dev aid they are a maintenance trap. Remove this file from the patch and keep it in a private local location.
| { | ||
| description = "PostgreSQL development environment"; |
There was a problem hiding this comment.
[high confidence] The Nix flake is local developer-environment setup and does not belong in a pgsql-hackers patch (see the local-workflow policy in the repo's .gitignore). It also bundles unrelated concern(s) with the actual feature work in this PR (the buffer cooling-state changes in bufmgr.c). Remove this file from the patch.
| if (strategy == NULL && | ||
| BUF_STATE_GET_COOLSTATE(buf_state) < BUF_COOLSTATE_HOT) | ||
| buf_state += BUF_COOLSTATE_ONE; |
There was a problem hiding this comment.
This replaces the 0..5 clock-sweep usage_count with a single HOT/COOL bit, a fundamental change to the buffer replacement algorithm. A HOT buffer now survives exactly one sweep pass before becoming a victim, versus up to 5+ full clock cycles of retention before. This drastically shortens working-set retention and risks cache thrashing under mixed workloads. For a core replacement-algorithm change like this, pgsql-hackers will require: a reproducible benchmark demonstrating the claimed scan-resistance benefit and no regression on OLTP/working-set-heavy workloads, a reference to the design thread (Message-Id), and ideally the ability to compare against the existing algorithm. As submitted there is no benchmark, no GUC, and no regression test exercising the new eviction behavior. (high confidence)
| * max") reads naturally. | ||
| */ | ||
| #define BM_MAX_USAGE_COUNT 5 | ||
| #define BM_MAX_USAGE_COUNT BUF_COOLSTATE_HOT |
There was a problem hiding this comment.
Retaining the name BM_MAX_USAGE_COUNT for a value that is now the max cooling state (1, not 5) is misleading, and the surrounding code mixes the old BUF_USAGECOUNT_* nomenclature (used to clear the cooling bit via &= ~BUF_USAGECOUNT_MASK) with the new BUF_COOLSTATE_* names. This dual vocabulary makes the subsystem harder to follow. Also note the comment at UnlockBufHdrExt() (around line 506) still reads "this approach would not trivially work for usagecount, since we need to cap the usagecount at BM_MAX_USAGE_COUNT" -- that reasoning is now stated in stale usagecount terms. Prefer renaming the field/macros consistently to the cooling-state vocabulary (or keep usagecount throughout); avoid aliasing one onto the other. If any out-of-tree extension reads BM_MAX_USAGE_COUNT, the silent redefinition from 5 to 1 is also a surprise. (moderate confidence)
| * The field historically used for the 0..5 usage_count now holds a single | ||
| * cooling-state bit: HOT (recently accessed, not an eviction candidate) or | ||
| * COOL (an eviction candidate). We reuse BUF_USAGECOUNT_ONE as the unit so | ||
| * the buffer-state bit geography -- refcount, flag, and lock offsets, and the |
There was a problem hiding this comment.
PostgreSQL source requires ASCII-only comments and the house style is WHY-not-WHAT. These new comment blocks use "--" as em-dash-like separators in running prose ("bit geography -- refcount...", "a sequential scan -- therefore") and are long narrative descriptions of what the algorithm does. Tighten to concise WHY comments and drop the em-dash-style "--" usage. (low confidence)
| local_buf_state -= BUF_USAGECOUNT_ONE; | ||
| /* | ||
| * HOT buffer: cool it in place this tick and keep scanning. We | ||
| * do NOT claim it now -- a demoted buffer only becomes a victim |
There was a problem hiding this comment.
Same ASCII/comment-style note as buf_internals.h: these new comments use em-dash-like "--" in prose ("we\n * do NOT claim it now --", "HOT working set instead -- tell the") and describe WHAT at length. Keep comments ASCII and WHY-focused per the tree's conventions. (low confidence)
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath is a NixOS module option and has no meaning as a flake output. Inside flake-utils.lib.eachDefaultSystem it becomes <system>.environment.localBinInPath, which is not a recognized flake output schema attribute and does nothing. This looks copy-pasted from a configuration.nix; remove it. (This is in addition to the already-confirmed finding that the whole flake is dev-only tooling that doesn't belong in a commitfest patch.)
There was a problem hiding this comment.
🔍 OCR found 28 issue(s).
- 25 inline, 3 in summary (inline capped at 25)
📄 src/include/storage/buf_internals.h (L103-L107)
The comment's derivation does not match the arithmetic. PG_CACHE_LINE_SIZE / sizeof(uint32) (= 128/4 = 32) is the number of uint32s in a cache line, not "one cache line of buffer descriptors' worth of demotions" -- a BufferDesc is ~64 bytes, so a cache line holds far fewer than 32 of them, and the field this acts on is 64-bit buffer state, not a bare uint32. The stated hardware justification is spurious; either pick a value with a real rationale or drop the cache-line framing. Confidence: high.
📄 src/include/storage/buf_internals.h (L106-L107)
Design concern: this threshold is a fixed 32 regardless of NBuffers, yet the comment itself argues the starvation window scales with NBuffers (pass time T grows with the pool). On a large shared_buffers, a single allocation abandons the probation rule after cooling only 32 buffers out of potentially millions -- so under even brief, localized pressure the clock sweep starts claiming still-HOT buffers almost immediately, defeating the scan-resistance the whole cooling design is meant to provide. The giving-up point should scale with NBuffers (or be justified against it) rather than being a tiny hardware-derived constant. Needs a benchmark on a large pool to show HOT working-set pages are not prematurely evicted. Confidence: moderate.
📄 src/include/storage/bufmgr.h (L34-L37)
This new block comment is stacked directly on top of the pre-existing block comment for the same enum (the "Possible arguments for GetAccessStrategy()" comment just above), leaving two adjacent comment blocks describing one declaration. Beyond the awkward stacking, the content is a design-rationale and manual-testing essay ("To test whether vacuum's ring is still earning its keep, set vacuum_buffer_usage_limit = 0 ...") that explains why rings were kept and how to experiment with removing them -- that belongs in the commit message / -hackers thread, not in a header comment above a type definition. A header comment should say what BufferAccessStrategyType is. Fold any still-relevant note into the existing comment and move the rationale/testing prose out. Confidence: high.
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
This is a silent, user-visible semantic change to the usage_count column of the pg_buffercache view. BUF_STATE_GET_COOLSTATE() returns only 0 (COOL) or 1 (HOT), whereas usage_count previously ranged 0..5. The column name and the local usagecount now misrepresent what is reported (a cool/hot bit, not a usage count) - a POLA violation for existing users and tooling that queries this column. The documented column description and example output in doc/src/sgml/pgbuffercache.sgml are now stale. Either rename the column/field to reflect the cool/hot state and update the docs, or document the new 0/1 semantics. Confidence: high.
| { | ||
| buffers_used++; | ||
| usagecount_total += BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount_total += BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
usagecount_total now accumulates the cool/hot bit (0/1), so the derived usagecount_avg column of pg_buffercache_summary() becomes the fraction of HOT buffers (0.0-1.0) rather than the former average usage count (0..~5, e.g. the documented 3.141129 example). This is a silent behavior change under an unchanged column name. The accompanying docs/example in pgbuffercache.sgml are not updated. Rename/redefine the column or document the new meaning. Confidence: high.
| # These breakpoints cover the major code paths introduced or modified by | ||
| # the HOT indexed updates patch series. They are organized by subsystem |
There was a problem hiding this comment.
This entire file targets a "HOT indexed updates" feature whose symbols do not exist anywhere in this tree. I verified that heap_hot_indexed_tuple_size, heap_hot_indexed_create_tuple, heap_hot_search_buffer's indexed-update variants, heap_xlog_indexed_update, and HEAP_INDEXED_UPDATED return zero matches under src/. The actual source change in this PR is buffer-cooling state (BUF_COOLED, BUF_COOLSTATE_HOT, StrategyCoolClaims) in bufmgr.c/freelist.c -- a completely different, unrelated feature. These breakpoints are orphaned scaffolding from a different patch series and would mislead or fail to resolve for anyone sourcing the file. This file should not be committed (and none of this developer-local tooling belongs in an upstream patch).
| break heapam.c:4019 | ||
| break heapam.c:4024 | ||
| break heapam.c:4033 |
There was a problem hiding this comment.
Hardcoded line-number breakpoints are already wrong against this tree. For example, the comment claims heapam.c:4019 is the "pure HOT (no indexed columns changed)" decision, but line 4019 of heapam.c in this repo is the generic re-fit check if (newtupsize > pagefree || ...). Line-number breakpoints are inherently brittle -- any edit shifts them so they silently point at unrelated code. Prefer function-name breakpoints (as used elsewhere in this file) over file:line form.
| alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"' | ||
|
|
||
| # Database management | ||
| alias pg-init='trash "$PG_DATA_DIR" 2>/dev/null || rm -rf "$PG_DATA_DIR"; "$PG_INSTALL_DIR/bin/initdb" --debug --no-clean "$PG_DATA_DIR"' |
There was a problem hiding this comment.
trash is the primary deletion command here but is never declared in shell.nix buildInputs (I confirmed it does not appear in shell.nix/flake.nix). On systems without trash, this falls through to an unconditional rm -rf "$PG_DATA_DIR". Since PG_DATA_DIR is set via the dev-shell hook (/tmp/test-db-$(basename $PWD)), sourcing these aliases outside the dev shell leaves it unset, making this rm -rf "" -- a data-loss footgun. Either add trash (trash-cli) to the dev-shell inputs, or guard the deletions with [ -n "$PG_DATA_DIR" ].
| pg_clean_for_compiler() { | ||
| local current_compiler="$(basename $CC)" | ||
| local build_dir="${1:-$PG_BUILD_DIR}" |
There was a problem hiding this comment.
The compiler-detection regex [gc]cc matches gcc and ccc, but not a plain cc symlink, and the first alternative also never matches a bare clang++. More importantly, basename $CC is unquoted (word-splits/globs if $CC ever contains spaces), and grep -o ... | head -1 | xargs basename will pick whichever absolute compiler path appears first in compile_commands.json, which is not necessarily the C compiler (it may be a C++ entry). This can both miss a real compiler switch (leaving a stale, mismatched build) and spuriously trigger a full rm -rf of the build dir. Consider comparing against the $build_dir/.compiler_used marker you already write instead of grepping the compile DB.
| ASAN_OPTIONS="halt_on_error=0:abort_on_error=1:detect_leaks=0:print_summary=1:print_stacktrace=1" \ | ||
| UBSAN_OPTIONS="halt_on_error=1:abort_on_error=1:print_stacktrace=1:print_summary=1" \ |
There was a problem hiding this comment.
pg-asan-regress builds the backend with -fsanitize=address but does not set LD_PRELOAD/verify_asan_link_order or ASAN_OPTIONS=verify_asan_link_order=0, and more critically runs the suite without detect_leaks disabled only for ASan while leaving UBSan halt_on_error=1. The bigger practical problem: ASan under PostgreSQL commonly aborts on shmget/mmap interception and on the fork()-heavy postmaster unless ASAN_OPTIONS includes abort_on_error=1 together with handling for detect_leaks across child processes; with halt_on_error=0 for ASan but halt_on_error=1 for UBSan, a single UBSan report in a backend kills that backend mid-test and desynchronizes pg_regress, producing confusing diffs rather than a clean sanitizer report. Please align the two option sets (and document the known-needed ASan options for a postmaster).
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | ||
| local_buf_state += BUF_REFCOUNT_ONE; | ||
|
|
||
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | ||
| local_buf_state)) |
There was a problem hiding this comment.
The numCoolClaims increment happens before the CAS, but the CAS can fail (another backend changed buf->state concurrently). On failure the inner loop falls through to the bottom and retries line 267 with the refreshed old_buf_state; if the buffer is still HOT and we are still at/above the threshold, we increment numCoolClaims again. Every CAS retry over-counts. Since BgBufferSync() keys its pre-cooling decision purely on the delta cool_claims > prev_cool_claims, this inflates the signal and can trigger background cooling under contention even when no allocation actually failed to find a COOL victim. Move the increment to only fire on a successful claim (after the CAS succeeds). Confidence: high.
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | |
| local_buf_state)) | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | |
| local_buf_state)) |
| if (cool_if_hot && | ||
| BUF_STATE_GET_REFCOUNT(buf_state) == 0 && | ||
| BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL) | ||
| { | ||
| UnlockBufHdrExt(bufHdr, buf_state, 0, BUF_USAGECOUNT_MASK, 0); | ||
| buf_state = LockBufHdr(bufHdr); | ||
| result |= BUF_COOLED; | ||
| } | ||
|
|
||
| if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 && | ||
| BUF_STATE_GET_USAGECOUNT(buf_state) == 0) | ||
| BUF_STATE_GET_COOLSTATE(buf_state) == BUF_COOLSTATE_COOL) | ||
| { | ||
| result |= BUF_REUSABLE; | ||
| } |
There was a problem hiding this comment.
Double-count: when cool_if_hot demotes a buffer above (lines 4237-4244), buf_state is re-read via LockBufHdr and is now COOL with refcount 0, so this block also sets BUF_REUSABLE on the same buffer. Back in BgBufferSync() the buffer then increments reusable_buffers twice -- once for BUF_REUSABLE and once for BUF_COOLED. This compounds the already-flagged problem of counting a just-cooled buffer as reusable: it is counted twice, further inflating reusable_buffers and causing the LRU scan to stop prematurely. A freshly demoted buffer is not an eviction candidate yet (an intervening access re-promotes it), so it should not satisfy the BUF_REUSABLE test in the same call that cooled it. Confidence: high.
| return true; | ||
| } |
There was a problem hiding this comment.
Spurious trailing blank line added at end of file. git diff --check will flag this and it is unrelated whitespace churn; remove it. Confidence: high.
| return true; | |
| } | |
| return true; | |
| } |
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
This makes pgindent the only Perl script in the tree not using the project's #!/usr/bin/perl convention; 30+ other .pl/Perl tools (genbki.pl, gen_node_support.pl, copyright.pl, etc.) still use the FHS absolute path. If the intent is non-FHS (e.g. Nix) portability, a single-file change doesn't achieve it while introducing an inconsistency. Either drop this hunk or convert the whole tree under a dedicated -hackers discussion. Confidence: moderate.
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
Switching to execlp has no effect for any in-tree build: SHELLPROG is $(SHELL) in the makefiles and /bin/sh in meson.build, all of which contain a slash. execlp only triggers a PATH search when the program name has no slash, so it behaves identically to execl here. This change only matters if SHELLPROG is a bare command name (e.g. sh), which is the Nix/non-FHS scenario implied by the other files in this changeset. As a standalone diff it is speculative scaffolding with no in-tree configuration exercising the new path. If the PATH-lookup behavior is actually intended/needed, the commit message should state why and ideally be accompanied by the build-config change that sets SHELLPROG to a bare name; otherwise this hunk is not minimal. (low confidence)
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
This entire file is personal developer scaffolding and does not belong in a patch destined for pgsql-hackers/commitfest. It references a feature that is not present in this change: a codebase search confirms none of the heap_hot_indexed_* symbols (e.g. heap_hot_indexed_create_tuple, heap_hot_indexed_read_bitmap) nor XLOG_HEAP2_INDEXED_UPDATE / heap_xlog_indexed_update exist anywhere under src/. A .gdbinit tied to a nonexistent feature is dead weight that will draw immediate rejection on-list. This file (together with .clangd, pg-aliases.sh, flake.nix, shell.nix) should be kept out of the tree (e.g. via a personal/global gitignore), not committed.
| # ========================================================================= | ||
|
|
||
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
These hardcoded source-line breakpoints are stale/wrong and actively misleading. Verified against the current tree: heapam.c:3210 is the closing brace of heap_delete (not the heap_update entry); heapam.c:4019 is a PageGetHeapFreeSpace check; heapam.c:4024 is inside a comment; heapam.c:4033 is a break;. None match the annotated HOT-indexed logic, and all line-number breakpoints silently drift after any edit to these files. If this file is kept at all, use function-name breakpoints (as the rest of the file already does) instead of file:line.
| # ============================================================ | ||
| pg_clean_for_compiler() { | ||
| local current_compiler="$(basename $CC)" | ||
| local build_dir="${1:-$PG_BUILD_DIR}" |
There was a problem hiding this comment.
Destructive footgun: build_dir defaults to ${1:-$PG_BUILD_DIR}, and if PG_BUILD_DIR is unset/empty the subsequent trash "$build_dir" / rm -rf "$build_dir" operates on an empty string. Guard against an empty path before deleting, e.g. [ -n "$build_dir" ] || { echo "build_dir unset" >&2; return 1; }.
| local build_dir="${1:-$PG_BUILD_DIR}" | |
| local build_dir="${1:-$PG_BUILD_DIR}" | |
| if [ -z "$build_dir" ]; then | |
| echo "Error: build_dir is empty (PG_BUILD_DIR unset?)" >&2 | |
| return 1 | |
| fi |
| usage_count = BUF_STATE_GET_COOLSTATE(buf_state); | ||
| usage_counts[usage_count]++; |
There was a problem hiding this comment.
User-visible/documented behavior change with no doc or test update. These three functions previously reported usage_count over the documented 0..5 range; after switching to BUF_STATE_GET_COOLSTATE they report only 0 or 1. pg_buffercache_usage_counts() in particular changes the SQL-visible result set: the loop at usage_counts[usage_count]++ and the output loop are bounded by BM_MAX_USAGE_COUNT+1, which is now 2 instead of 6, so the view silently drops rows for counts 2..5. doc/src/sgml/pgbuffercache.sgml (describing usage_count) and the expected regression output are not updated. Per pgsql-hackers standards this is WIP: a documented view's semantics cannot change without updating the docs and regression tests, and ideally without a column-meaning rename to avoid a silent POLA violation.
| return true; | ||
| } |
There was a problem hiding this comment.
Stray trailing blank line added at end of file; git diff --check will flag this. Remove the extra blank line to keep the diff minimal.
| #define BUF_COOLSTATE_COOL 0 | ||
| #define BUF_COOLSTATE_HOT 1 | ||
| #define BUF_COOLSTATE_ONE BUF_USAGECOUNT_ONE |
There was a problem hiding this comment.
This is a core buffer-replacement policy change (clock-sweep 0..5 usage_count replaced by a single-bit HOT/COOL 2Q-style scheme) on the buffer-manager hot path, plus a new background pre-cooling heuristic. It ships with no regression/TAP tests and no documentation, and the performance claims baked into these comments (self-evicting one-touch scans, reduced foreground starvation) are asserted without any reproducible benchmark or reference to a design discussion/-hackers thread. Per project standards this is WIP, not commit-ready: the behavior and performance change needs tests covering the eviction/scan-resistance edge paths, documentation, and a Message-Id reference for the design.
| * workloads never reach it. One cache line of buffer descriptors' worth of | ||
| * demotions is a cheap, hardware-derived choice. | ||
| */ | ||
| #define BUF_COOL_CLAIM_THRESHOLD \ | ||
| (PG_CACHE_LINE_SIZE / sizeof(uint32)) |
There was a problem hiding this comment.
Misleading unit in the rationale. BUF_COOL_CLAIM_THRESHOLD expands to PG_CACHE_LINE_SIZE / sizeof(uint32) = 128/4 = 32, i.e. one cache line's worth of uint32s, not of buffer descriptors. A BufferDesc is far larger than 4 bytes (and cache-line aligned), so "one cache line of buffer descriptors' worth of demotions" is wrong by ~an order of magnitude. Either correct the wording or derive the threshold from sizeof(BufferDesc) if descriptors were actually intended. (high confidence)
| /* | ||
| * Buffer access strategies. | ||
| * | ||
| * Note on the cooling-stage evictor: admitting a demand-loaded page COOL makes |
There was a problem hiding this comment.
This adds a second comment block immediately after the existing enum comment (lines 28-33), leaving two stacked comments before the typedef. The new block is a mailing-list-style design essay (why the rings are kept, how to test removing vacuum's ring) rather than a description of what BufferAccessStrategyType is. Per PostgreSQL comment discipline, header comments should explain the declared type's contract, not carry the patch's design justification; move this rationale to the commit message / -hackers thread and keep the header comment minimal. (moderate confidence)
There was a problem hiding this comment.
🔍 OCR found 26 issue(s).
- 25 inline, 1 in summary (inline capped at 25)
📄 src/include/storage/buf_internals.h (L106-L107)
The rationale for this magic number is self-contradictory. PG_CACHE_LINE_SIZE / sizeof(uint32) evaluates to 32 (128/4), but the comment calls it "one cache line of buffer descriptors' worth of demotions." A BufferDesc is padded to BUFFERDESC_PAD_TO_SIZE (64 bytes on 64-bit), so a cache line holds at most 2 descriptors, not 32 -- and sizeof(uint32) is not the size of anything being demoted here. Either derive the threshold from a quantity it actually represents (and fix the comment), or justify 32 directly instead of dressing it up as a hardware-derived value.
| # Predict augmented tuple size (returns 0 if t_hoff would overflow) | ||
| break heap_hot_indexed_tuple_size | ||
|
|
||
| # Create augmented tuple with embedded modified-column bitmap | ||
| break heap_hot_indexed_create_tuple |
There was a problem hiding this comment.
This .gdbinit (and the whole review group: .clangd, flake.nix, shell.nix, pg-aliases.sh) is personal developer-environment scaffolding unrelated to the actual change under review, which is a buffer-manager cooling-state rework (bufmgr.c SyncOneBuffer/cool_if_hot, buf_internals.h BUF_COOLSTATE_*, freelist.c StrategyCoolClaims). For a patch destined for pgsql-hackers/commitfest these files violate the minimal-diff rule and will draw immediate rejection; they should not be committed.
Worse, the breakpoints here target a completely different, nonexistent feature: the heap_hot_indexed_* functions (heap_hot_indexed_tuple_size, heap_hot_indexed_create_tuple, heap_hot_indexed_serialize_bitmap, etc.) do not exist anywhere in src/backend/access/heap/*.c in this tree. GDB will reject these as unresolved symbols. This confirms the file is stale scaffolding copied from an unrelated "HOT indexed updates" patch series and bears no relation to the buffer-manager change actually being reviewed. Remove the file from the patch.
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
The hardcoded absolute source line numbers (heapam.c:3210, heapam.c:4019, heapam.c:4033, heapam.c:4101, pruneheap.c:1802, execIndexing.c:370, etc.) will silently drift and attach to the wrong lines as the source evolves. Line-number breakpoints are inherently unmaintainable in a shared repo; combined with the stale heap_hot_indexed_* symbol references, this file is misleading. (Low severity because it is tooling, not shipped code -- the primary issue is that the file should not be in the patch at all.)
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath is a NixOS module option, not a valid per-system flake output. Placing it alongside formatter/devShells/packages in the eachDefaultSystem result set is meaningless here -- nothing consumes it (it is referenced only at this line), and it does not configure PATH for the dev shells as presumably intended. It is dead/misplaced output and should be removed (or moved into an actual NixOS module if that behavior is wanted).
| local current_compiler="$(basename $CC)" | ||
| local build_dir="${1:-$PG_BUILD_DIR}" | ||
|
|
||
| if [ -f "$build_dir/compile_commands.json" ]; then |
There was a problem hiding this comment.
pg_clean_for_compiler takes build_dir="${1:-$PG_BUILD_DIR}" and later runs rm -rf "$build_dir" (line 16). If both the argument and $PG_BUILD_DIR are empty/unset, build_dir is empty and the trash/rm -rf "" fallback is a footgun in a destructive-command path. The same unguarded pattern applies to pg-full-clean (rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR") and pg-init (rm -rf "$PG_DATA_DIR"): an unset variable makes these operate on nothing or the wrong path. Guard against empty/unset variables (e.g. [ -n "$build_dir" ] || return 1) before any destructive deletion. (The deeper issue remains that this helper script should not be part of a pgsql-hackers patch.)
| ulimit -c unlimited | ||
| if ! [ -w /proc/sys/kernel/core_pattern ]; then | ||
| echo "Setting kernel.core_pattern (requires sudo)..." | ||
| echo "core.%p" | sudo tee /proc/sys/kernel/core_pattern >/dev/null || { |
There was a problem hiding this comment.
pg-enable-cores/pg-disable-cores mutate the system-wide kernel.core_pattern sysctl via sudo tee, a global side effect triggered from a shell helper. Together with the developer-specific absolute/temp paths embedded elsewhere (/tmp/test-db-$(basename $PWD), ~/.gdb_history_postgres, $HOME/.ccache/pg/...), this is inappropriate for a shared repository. (Low severity as it is opt-in tooling, but it reinforces that these files do not belong in the patch.)
| /* Cool-claim count as of our previous cycle, to get a rate not a total */ | ||
| static uint64 prev_cool_claims = 0; |
There was a problem hiding this comment.
prev_cool_claims is a per-process function-level static while numCoolClaims is a cluster-wide shared counter that is only zeroed at StrategyCtlShmemInit(). After a bgwriter crash/restart the static resets to 0 but the shared counter persists at a large value, so the first post-restart cycle sees cool_claims > prev_cool_claims and enables pre-cooling regardless of actual pressure. Self-correcting after one cycle, but worth a comment or seeding prev from the shared value on entry. [moderate confidence]
| return true; | ||
| } |
There was a problem hiding this comment.
Trailing blank line added at EOF. This is whitespace churn unrelated to the change and git diff --check / pgindent will flag it. Remove it. [high confidence]
| return true; | |
| } | |
| return true; | |
| } |
| /* | ||
| * Buffer access strategies. | ||
| * | ||
| * Note on the cooling-stage evictor: admitting a demand-loaded page COOL makes |
There was a problem hiding this comment.
Two problems with this comment block. (1) It is a second, adjacent block comment immediately following the existing one at lines 28-33 on the same enum -- merge them rather than stacking two headers. (2) The content is design-discussion/aspirational narrative ("Measurement is the only way to settle whether their rings still earn their keep", suggested future experiments with vacuum_buffer_usage_limit) that describes work not done in this patch. Per project convention, comments describe what the code does now and explain WHY; this rationale belongs in the commit message / buffer README, not an enum header. [moderate confidence]
| * max") reads naturally. | ||
| */ | ||
| #define BM_MAX_USAGE_COUNT 5 | ||
| #define BM_MAX_USAGE_COUNT BUF_COOLSTATE_HOT |
There was a problem hiding this comment.
This redefines the public symbol BM_MAX_USAGE_COUNT from 5 to BUF_COOLSTATE_HOT (=1) while keeping the name. That collapses the user-visible usage_count field (exposed via pg_buffercache_pages / pg_buffercache_usage_counts / pg_buffercache_summary) from the documented 0..5 range to 0/1. This is a user-visible behavior change: the pgbuffercache docs (doc/src/sgml/pgbuffercache.sgml example output still shows usage_count rows 0..5) and any regression output encoding usage_count semantics become stale, and no doc/test update accompanies this header change. A fundamental buffer-replacement-algorithm change like this needs tests and documentation to be commit-ready. [high confidence]
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | ||
| local_buf_state += BUF_REFCOUNT_ONE; | ||
|
|
||
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | ||
| local_buf_state)) | ||
| { | ||
| trycounter = NBuffers; | ||
| break; | ||
| /* Found a usable buffer */ |
There was a problem hiding this comment.
numCoolClaims is incremented before the CAS. If the CAS fails (a concurrent backend pinned, cooled, or re-promoted the buffer), the counter was already bumped even though no HOT buffer was actually claimed, and the loop retries -- potentially bumping it again. Under header-state contention this over-counts the very signal BgBufferSync() reads via StrategyCoolClaims() to set cool_if_hot, biasing the background writer toward aggressive pre-cooling when real claim pressure is lower. Move the increment inside the successful-CAS branch so it counts actual claims only.
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | |
| local_buf_state)) | |
| { | |
| trycounter = NBuffers; | |
| break; | |
| /* Found a usable buffer */ | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | |
| local_buf_state)) | |
| { | |
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | |
| /* Found a usable buffer */ |
| break heapam.c:4019 | ||
| break heapam.c:4024 | ||
| break heapam.c:4033 |
There was a problem hiding this comment.
These GDB breakpoints target a "HOT indexed updates" patch that is NOT present in this tree. Verified: heap_hot_indexed_create_tuple, heap_hot_indexed_tuple_size, heap_hot_indexed_serialize_bitmap, and heap_xlog_indexed_update do not exist anywhere in the source, so these function breakpoints will fail to resolve. The hardcoded line-number breakpoints are also wrong against the current tree: heapam_indexscan.c:182 is a PageGetItemId() call (not "redirect-with-data bitmap accumulator"), heapam_indexscan.c:250 is skip = false; (not "accumulate bitmap from INDEXED_UPDATED tuple"), and heapam_indexscan.c:297 is inside an unrelated static callback (not "stale entry detection"). This file is local developer scaffolding tightly coupled to one working-tree state and must not be committed to a patch destined for pgsql-hackers. (high confidence)
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath = true; is a NixOS module option, not a flake output. Placed here among the per-system eachDefaultSystem outputs it is meaningless and silently ignored; it does not enable anything. Remove it.
| issue_comment: | ||
| types: [created] |
There was a problem hiding this comment.
The issue_comment trigger fires on any comment to a PR, and pull_request fires on synchronize/reopened from any contributor's branch. The reusable workflow is then invoked with the OIDC-federated AWS role (aws_role_arn) and the repo GITHUB_TOKEN. There is no actor/author-association gate (e.g. a check that github.event.comment.user or the PR author is OWNER/MEMBER/COLLABORATOR). An unprivileged user can therefore drive Bedrock spend at will, and — if gburd/ci-workflows/.../ocr-review.yml checks out PR head code before using the AWS role — this is a classic poisoned-pipeline-execution (PPE) path leaking a cloud credential to untrusted code. Add an author_association/actor restriction to the if: for the comment path (and ideally for forked-PR events), and confirm the called workflow does not expose the role to attacker-controlled checkout. (high confidence on the missing gate; moderate on exploitability since the called workflow is not in this repo to inspect.)
| (startsWith(github.event.comment.body, '/open-code-review') || | ||
| startsWith(github.event.comment.body, '@open-code-review') || | ||
| startsWith(github.event.comment.body, '/pg-history'))) | ||
| uses: gburd/ci-workflows/.github/workflows/ocr-review.yml@v1 |
There was a problem hiding this comment.
The reusable workflow is pinned to the mutable tag @v1. Whoever controls gburd/ci-workflows (or an attacker who compromises it) can re-point v1 to arbitrary code that then runs with this repo's OIDC-federated AWS role and GITHUB_TOKEN. Pin to a full commit SHA (optionally with a # v1 trailing comment) so the referenced code is immutable.
| uses: gburd/ci-workflows/.github/workflows/ocr-review.yml@v1 | |
| uses: gburd/ci-workflows/.github/workflows/ocr-review.yml@<full-commit-sha> # v1 |
| # Needs repo variables AWS_ROLE_ARN, AWS_REGION, OCR_BEDROCK_MODEL. Auth is | ||
| # GitHub OIDC; no static AWS keys. See the ci-workflows README. | ||
|
|
||
| name: OCR AI Review |
There was a problem hiding this comment.
No permissions: block is declared at workflow or job level, so the default GITHUB_TOKEN scope (which depends on repo settings and can be broad) is passed to the called workflow. An AI review that posts PR comments needs at most pull-requests: write and contents: read. Declare least-privilege permissions explicitly rather than relying on repo defaults.
| name: OCR AI Review | |
| name: OCR AI Review | |
| permissions: | |
| contents: read | |
| pull-requests: write |
|
|
||
| cmdline2 = psprintf("exec %s", cmdline); | ||
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | ||
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
There was a problem hiding this comment.
Unrelated change and a behavioral regression. shellprog comes from SHELLPROG, which the build sets to $(SHELL) (make) or the hardcoded absolute path /bin/sh (meson/src/test/regress/meson.build). The original execl runs exactly that path. execlp adds a $PATH search whenever the string contains no slash, so the test harness would resolve its shell via $PATH instead of the build-fixed path -- an unnecessary footgun (PATH injection / picking up an unintended shell) in a security-sensitive exec path.
This hunk is also unrelated to the buffer-manager work this PR centers on; per minimal-diff discipline it should not be bundled here. Unless there is a documented reason the regress runner must PATH-search for its shell, keep execl. (confidence: high)
| execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL); | |
| execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL); |
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
Unrelated change that breaks tree-wide consistency. Every other Perl script in the tree (30+ files, e.g. genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl) uses #!/usr/bin/perl; this edit makes pgindent the sole outlier using env perl. It is also unrelated to the buffer-manager changes this PR group concerns, so it is diff churn that will draw rejection on -hackers. Revert to the tree convention. (confidence: high)
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | ||
| local_buf_state += BUF_REFCOUNT_ONE; | ||
|
|
||
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | ||
| local_buf_state)) | ||
| { | ||
| trycounter = NBuffers; | ||
| break; | ||
| /* Found a usable buffer */ |
There was a problem hiding this comment.
numCoolClaims is incremented before the CAS that actually claims the buffer. On CAS failure the inner for (;;) loop reloads old_buf_state and retries, so a single buffer under contention can bump the counter multiple times; worse, if the retry observes the buffer is now pinned (refcount != 0) we break out via the refcount branch without claiming anything, yet the counter was already incremented. This inflates the cool-claim rate that BgBufferSync() reads via StrategyCoolClaims() to set cool_if_hot, biasing the background writer toward pool-wide cooling under pressure that did not actually occur. Increment the counter only inside the successful-CAS branch, after the claim is committed.
(high confidence)
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | |
| local_buf_state)) | |
| { | |
| trycounter = NBuffers; | |
| break; | |
| /* Found a usable buffer */ | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, | |
| local_buf_state)) | |
| { | |
| pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1); | |
| /* Found a usable buffer */ |
| return true; | ||
| } |
There was a problem hiding this comment.
Spurious trailing blank line added at end of file. git diff --check / pgindent will flag this as a whitespace defect; remove the extra blank line.
(high confidence)
| if (cool_if_hot && | ||
| BUF_STATE_GET_REFCOUNT(buf_state) == 0 && | ||
| BUF_STATE_GET_COOLSTATE(buf_state) != BUF_COOLSTATE_COOL) | ||
| { | ||
| UnlockBufHdrExt(bufHdr, buf_state, 0, BUF_USAGECOUNT_MASK, 0); | ||
| buf_state = LockBufHdr(bufHdr); | ||
| result |= BUF_COOLED; | ||
| } |
There was a problem hiding this comment.
The code already holds the header spinlock (from LockBufHdr above), yet to demote the buffer it calls UnlockBufHdrExt() -- which runs its own CAS retry loop and releases the lock -- and then re-acquires it with LockBufHdr(). That is two atomic operations plus a lock round-trip, directly contradicting the comment here ("this costs one masked store and no atomic retry") and the comment in BgBufferSync ("staging costs a masked store") and the function header ("a plain masked store rather than a CAS"). Since the spinlock is held, the bits can be cleared with a plain masked store on the already-read buf_state while keeping the lock, avoiding the unlock/relock entirely. As written, either the comments are wrong or the implementation is doing unnecessary work on the cleaning scan's path.
(high confidence)
left a comment
There was a problem hiding this comment.
🔍 OCR found 35 issue(s).
- 25 inline, 10 in summary (inline capped at 25)
📄 src/test/regress/pg_regress.c (L1246-L1246)
This execl -> execlp switch is an unrelated change bundled into a patch whose stated purpose is the buffer-manager work. On -hackers this is a top rejection reason ("patch does more than one thing"); it should be split into its own commit with a clear justification, or dropped.
On the substance: execlp() only differs from execl() when shellprog contains no slash, in which case it performs a PATH search. In the normal build SHELLPROG is /bin/sh (meson) or $(SHELL) (make, defaults to /bin/sh), both of which contain a slash, so behavior is identical. The only effect is to allow a bare program name in SHELLPROG to be resolved via PATH. If that is actually the goal, say so and update the surrounding comment (lines 1239-1242 explain why the shell is exec'd directly but nothing about PATH lookup). Note this also introduces a mild footgun: an untrusted PATH could cause a different sh to be executed than the one the build selected. (moderate confidence)
📄 contrib/pg_buffercache/pg_buffercache_pages.c (L164-L164)
Stale identifier after the mechanical substitution. The local is still named usagecount (and its column type is int2), but it now receives a HOT/COOL cooling-state bit (0 or 1), not a 0..5 usage count. The name actively misleads future readers into assuming the historical count semantics. Either keep the SQL column reporting a real count, or rename the variable to reflect that it now carries the cooling state so the code reads as if written that way. (moderate confidence)
📄 src/tools/pgindent/pgindent (L1-L1)
This change breaks the tree-wide shebang convention. Every other Perl script in the PostgreSQL source (genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, and ~30 more) uses #!/usr/bin/perl. Making pgindent the sole file to use #!/usr/bin/env perl is inconsistent, unrelated to any functional change, and would be rejected on -hackers as gratuitous churn. The in-tree ./configure/build process also rewrites the Perl path, so env provides no portability benefit here. Revert to match the established convention.
💡 Suggested change
Before:
#!/usr/bin/env perl
After:
#!/usr/bin/perl
📄 src/backend/storage/buffer/freelist.c (L340-L346)
numCoolClaims is incremented before the CAS, so it is over-counted on CAS failure. When the compare_exchange at the next line fails (concurrent modification), the inner for(;;) loop retries this same buffer; if it is still unpinned and HOT with cooled >= BUF_COOL_CLAIM_THRESHOLD, control reaches this fetch_add again and increments the counter a second (or Nth) time for a single eventual claim. StrategyCoolClaims() is read by BgBufferSync() purely as a rate signal (cool_claims > prev_cool_claims), so these spurious increments trigger spurious full-pool pre-cooling. Move the increment to the CAS-success branch, as is done for the pin in both the COOL branch and stock code. (high confidence)
💡 Suggested change
Before:
pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1);
local_buf_state += BUF_REFCOUNT_ONE;
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
{
/* Found a usable buffer */
After:
local_buf_state += BUF_REFCOUNT_ONE;
if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
local_buf_state))
{
pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1);
/* Found a usable buffer */
📄 src/backend/storage/buffer/freelist.c (L847-L848)
Stray trailing blank line added at EOF; git diff --check / pgindent will flag this whitespace defect. Remove it. (high confidence)
💡 Suggested change
Before:
return true;
}
+
After:
return true;
}
📄 src/backend/storage/buffer/bufmgr.c (L4108-L4110)
prev_cool_claims is a function-local static uint64 initialized to 0, but StrategyCoolClaims() reads a process-wide shared atomic that may already be large (and never resets). On a bgwriter (re)start, prev_cool_claims == 0 while cool_claims is large, so cool_claims > prev_cool_claims is true on the first post-restart cycle, triggering a spurious cool-the-whole-pool pass even under no pressure. The existing code guards its first-cycle rate math with saved_info_valid; this new rate signal has no equivalent guard. At minimum seed prev_cool_claims from the current reading on the first cycle (e.g. gate it under saved_info_valid like the other prev_* state). (moderate confidence)
📄 src/backend/storage/buffer/bufmgr.c (L4241-L4243)
Unnecessary header-lock drop and re-acquire. UnlockBufHdrExt() runs its own CAS loop to clear the usagecount and release BM_LOCKED, then LockBufHdr() re-acquires the spinlock immediately. The buffer is already locked here; prefer clearing the cooling bits in-place (buf_state &= ~BUF_USAGECOUNT_MASK; pg_atomic_write to the still-locked state, or fold it into the existing UnlockBufHdr at function end) to avoid an extra atomic CAS plus a fresh spinlock acquire on this bgwriter path. Functionally it is correct -- the BUF_REUSABLE test below re-reads the freshly re-locked buf_state, so a concurrent PinBuffer() that promotes/pins between unlock and relock is handled -- but the drop/relock is wasteful. (moderate confidence)
📄 src/include/storage/buf_internals.h (L106-L107)
Comment is misleading about the derivation. BUF_COOL_CLAIM_THRESHOLD = PG_CACHE_LINE_SIZE / sizeof(uint32) = 32, but the comment calls this "one cache line of buffer descriptors' worth of demotions." A BufferDesc is far larger than a uint32, so one cache line does not hold 32 descriptors; the divisor is the historical state-field width, not sizeof(BufferDesc). Either fix the rationale text or pick a value whose justification matches the arithmetic. (low confidence)
📄 src/include/storage/buf_internals.h (L194-L194)
This replaces the 0..5 clock-sweep usage_count with a single HOT/COOL bit, which is a user-visible change: pg_buffercache's usagecount column and pg_buffercache_usage_counts() now only ever report 0 or 1 (never 2..5), and usagecount_avg collapses to a 0..1 range. No documentation update or regression/TAP test accompanies this replacement-policy change in the patch, and the scan-resistance / pre-cooling performance claims in the comments have no referenced reproducible benchmark or -hackers design thread. Per project standards this is WIP-level: add docs for the changed semantics and tests covering the new eviction/promotion/claim paths. (moderate confidence)
📄 src/backend/storage/buffer/bufmgr.c (L4143-L4144)
Double-counting of reusable_buffers for demoted buffers. A buffer that SyncOneBuffer() demotes via cool_if_hot returns BUF_COOLED, but after demotion it is unpinned and COOL, so SyncOneBuffer() also sets BUF_REUSABLE (bufmgr.c:4246-4249), or BUF_WRITTEN if it was dirty. Because this if (sync_state & BUF_COOLED) is a separate branch rather than part of the existing else if chain, every cooled buffer increments reusable_buffers twice: once at line 4127/4134 and again here. The over-count makes the LRU scan hit reusable_buffers < upcoming_alloc_est early and stop before it has staged enough candidates, and it also corrupts new_recent_alloc = reusable_buffers - reusable_buffers_est (line 4167), skewing the self-tuning smoothed_density for subsequent cycles. A BUF_COOLED buffer is already counted as reusable, so this extra increment should be removed (or the three outcomes made mutually exclusive). (high confidence)
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
These breakpoints target a "HOT indexed updates" patch series that is not present in this repository. Verified against the tree: the symbols heap_hot_indexed_*, HEAP_INDEXED_UPDATED, and heap_xlog_indexed_update do not exist (no matches), and the hardcoded line numbers do not match the referenced files. For example heapam_indexscan.c is only 588 lines, so heapam_indexscan.c:182 is lp = PageGetItemId(page, offnum);, :250 is a blank line, and :297 is a return in heapam_index_plain_tuple_getnext_slot() -- none of which match the comments ("initialize bitmap accumulator", "accumulate bitmap from INDEXED_UPDATED tuple", "stale entry detection"). As written, every symbol breakpoint fails to resolve and every line-number breakpoint lands on unrelated code, making this file misleading and non-functional for anyone debugging this tree.
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
GDB auto-sources ./.gdbinit from the working directory (subject to auto-load safe-path). A committed .gdbinit that silently installs dozens of breakpoints -- most at hardcoded line numbers that don't match this tree -- is a footgun: a contributor debugging an unrelated issue will have execution halt at unexpected places or see breakpoint-resolution errors on startup. This is developer-local tooling that does not belong in a committable PostgreSQL patch; it should be kept out of the tree (e.g. via $GIT_DIR/info/exclude as the project's own .gitignore header recommends) rather than committed.
| inherit (shellConfig) gdbConfig flameGraphScript pgbenchScript; | ||
| }; | ||
|
|
||
| environment.localBinInPath = true; |
There was a problem hiding this comment.
environment.localBinInPath is a NixOS module option, not a flake output. Placed here it becomes a per-system flake output attribute (environment.localBinInPath = true) produced by eachDefaultSystem, which nothing consumes -- flakes do not read environment.*. It is dead/misplaced config that only clutters the output schema; remove it.
| jobs: | ||
| review: |
There was a problem hiding this comment.
This caller declares no permissions: block, so the job inherits the repository's default GITHUB_TOKEN permissions (potentially broad). The repo's own convention (pg-ci.yml) explicitly restricts to contents: read. For a reusable-workflow caller, permissions must be set at the caller (workflow or job) level and are passed to the called workflow, which can only further restrict them. An AI review job that posts comments typically needs pull-requests: write (and contents: read); declare the least-privilege set explicitly here rather than relying on repo defaults.
| jobs: | |
| review: | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| jobs: | |
| review: |
| jobs: | ||
| check: |
There was a problem hiding this comment.
No permissions: block is declared, so this inherits the repository default GITHUB_TOKEN permissions. The comment says the job opens/updates an issue, which requires issues: write. Declare least-privilege permissions explicitly (consistent with pg-ci.yml's contents: read) instead of relying on repo defaults.
| jobs: | |
| check: | |
| permissions: | |
| contents: read | |
| issues: write | |
| jobs: | |
| check: |
| alias pg-debug=' | ||
| if command -v gdb >/dev/null 2>&1; then | ||
| pg-debug-gdb | ||
| elif command -v lldb >/dev/null 2>&1; then | ||
| pg-debug-lldb | ||
| else | ||
| echo "No debugger available (gdb or lldb required)" | ||
| fi' |
There was a problem hiding this comment.
These dispatcher aliases invoke other aliases (pg-debug-gdb / pg-debug-lldb, and below pg-attach-gdb / pg-attach-lldb) from inside an alias body. Aliases are only expanded on the first word of each simple command and alias expansion is not recursive within another alias's expansded body, so when pg-debug runs, the names pg-debug-gdb/pg-debug-lldb are treated as plain commands and will produce 'command not found' rather than launching the debugger. Define these as shell functions (which are resolved at call time) instead of aliases if the dispatch is intended to work.
| uses: gburd/ci-workflows/.github/workflows/ocr-model-check.yml@v1 | ||
| with: | ||
| aws_role_arn: ${{ vars.AWS_ROLE_ARN }} |
There was a problem hiding this comment.
Like the review workflow, this caller passes an aws_role_arn for a Bedrock check, which per the fork's design authenticates via GitHub OIDC. OIDC role assumption requires id-token: write on the caller's GITHUB_TOKEN, and a reusable workflow cannot grant itself that scope. With no permissions: block here, the inherited default may lack id-token: write, breaking auth. The separately-raised least-privilege finding also notes issues: write is needed to open/update the notification issue - so the required permissions are id-token: write + issues: write, not merely contents: read. (moderate confidence)
|
|
||
| jobs: | ||
| check: | ||
| uses: gburd/ci-workflows/.github/workflows/ocr-model-check.yml@v1 |
There was a problem hiding this comment.
The reusable workflow is pinned to the mutable tag @v1. The upstream owner (or anyone who compromises that repo) can repoint v1 to any commit, and this job assumes an AWS OIDC role - repointed logic would run with your AWS credentials. Pin to a full commit SHA (e.g. ...@<40-char-sha> # v1) for third-party reusable workflows. (high confidence)
| # Python UV | ||
| UV_PYTHON_DOWNLOADS=never |
There was a problem hiding this comment.
UV_PYTHON_DOWNLOADS=never is missing export, unlike every other variable in this shellHook. uv reads UV_PYTHON_DOWNLOADS from the process environment, so a non-exported shell variable has no effect and uv will still attempt managed Python downloads -- the intended isolation is silently a no-op. (moderate confidence)
| # Python UV | |
| UV_PYTHON_DOWNLOADS=never | |
| # Python UV | |
| export UV_PYTHON_DOWNLOADS=never |
| forknum = BufTagGetForkNum(&bufHdr->tag); | ||
| blocknum = bufHdr->tag.blockNum; | ||
| usagecount = BUF_STATE_GET_USAGECOUNT(buf_state); | ||
| usagecount = BUF_STATE_GET_COOLSTATE(buf_state); |
There was a problem hiding this comment.
Backward-compatibility / POLA concern (high confidence). This mechanical substitution silently changes the user-visible contract: the usage_count column of pg_buffercache (and pg_buffercache_usage_counts(), plus usagecount_avg of pg_buffercache_summary()) historically ranged 0..5 under clock-sweep. BUF_STATE_GET_COOLSTATE() now returns only 0 (COOL) or 1 (HOT), so these SQL functions start reporting a two-value cooling state under an unchanged column name. The variable here is still named usagecount and the output column is still usage_count, so a consumer parsing these views gets a different meaning with no signal. This needs to be surfaced to the author: either (a) keep exposing a compatible value, or (b) rename the user-facing columns and bump the pg_buffercache extension version, and in all cases update doc/src/sgml/pgbuffercache.sgml, which still documents the 0..5 range and shows a 0..5 example output. As-is the docs and the behavior now disagree.
left a comment
There was a problem hiding this comment.
🔍 OCR found 36 issue(s).
- 25 inline, 11 in summary (inline capped at 25)
📄 pg-aliases.sh
The heredoc expands $PG_SOURCE_DIR, $PG_BENCH_DIR, and $bindir into the generated wrapper unquoted. Note PG_SOURCE_DIR/pgSourceDir is $PWD and PG_DATA_DIR/benchmark dirs are derived from $(basename $PWD) (see shell.nix), so if the checkout path contains a space the emitted --suppressions=, --log-file=, and the final postgres invocation break (unterminated arguments / wrong binary path). Quote these when writing the wrapper, e.g. --suppressions="$PG_SOURCE_DIR/src/tools/valgrind.supp" and "$bindir/postgres" "\$@".
📄 shell.nix (L264-L268)
The print_proc macro dereferences $proc->waiting, but PGPROC (src/include/storage/proc.h) has no waiting member — that boolean field was removed; the current fields are lwWaiting and the ProcWaitStatus waitStatus enum. This macro will fail to evaluate in GDB. (Same broken-GDB-macro class as the confirmed print_mcxt/print_relcache findings.) High confidence.
📄 shell.nix (L233-L234)
The print_slot macro reads $slot->tts_empty and $slot->tts_shouldFree, but TupleTableSlot (src/include/executor/tuptable.h) has no such members. Those states live in the tts_flags bitmask (TTS_FLAG_EMPTY/TTS_FLAG_SHOULDFREE, exposed via TTS_EMPTY()/TTS_SHOULDFREE()), so this macro will fail to evaluate in GDB. (Same broken-GDB-macro class as the confirmed print_mcxt/print_relcache findings.) High confidence.
📄 shell.nix (L217-L218)
The print_tupdesc macro indexes $desc->attrs[$i], but TupleDescData (src/include/access/tupdesc.h) has no attrs member. The attribute array is no longer an embedded attrs field; Form_pg_attribute entries are stored after the struct and accessed via TupleDescAttr(desc, i) / TupleDescAttrAddress(), while the struct only exposes compact_attrs. This macro will fail to evaluate in GDB. (Same broken-GDB-macro class as the confirmed print_mcxt/print_relcache findings.) High confidence.
📄 src/test/regress/pg_regress.c (L1246-L1246)
Changing execl to execlp makes the regression driver resolve the shell via $PATH whenever shellprog lacks a slash. In practice SHELLPROG is always an absolute path ($(SHELL) in GNUmakefile/ecpg, hardcoded /bin/sh in meson.build), so this has no effect for the normal build and is a no-op there. It only matters if someone overrides SHELLPROG with a bare name, in which case it introduces a PATH-dependent shell lookup -- a footgun that makes test execution depend on the ambient environment and a potential hijack vector. This hunk is unrelated to any stated feature; for an upstream-bound patch it is unneeded churn and should be dropped unless the PR specifically justifies and documents the need (e.g. non-standard install prefixes). Confidence: high.
💡 Suggested change
Before:
execlp(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
After:
execl(shellprog, shellprog, "-c", cmdline2, (char *) NULL);
📄 src/tools/pgindent/pgindent (L1-L1)
This makes pgindent the only Perl script in the entire tree using #!/usr/bin/env perl; every other script (~30+, e.g. genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, all generate-*.pl) uses #!/usr/bin/perl. For an upstream-destined PostgreSQL patch this is an inconsistent, unrelated hygiene change that violates minimal-diff discipline and breaks the project's established convention. If a PATH-based perl is needed for a local (e.g. Nix) environment, that belongs in the developer's environment, not in a committed shebang. Revert to keep it consistent with the rest of the tree. Confidence: high.
💡 Suggested change
Before:
#!/usr/bin/env perl
After:
#!/usr/bin/perl
📄 src/backend/storage/buffer/bufmgr.c (L4143-L4144)
Double-counting bug: a buffer demoted here always also passes the refcount==0 && COOL test immediately below, so SyncOneBuffer returns BUF_COOLED | BUF_REUSABLE (or BUF_COOLED | BUF_WRITTEN when dirty). In BgBufferSync the BUF_WRITTEN/BUF_REUSABLE increment and this standalone if (sync_state & BUF_COOLED) increment are not mutually exclusive, so the same physical buffer bumps reusable_buffers twice. That makes the LRU scan terminate roughly twice as early as upcoming_alloc_est intends, under-cleaning and under-cooling the pool exactly when pressure (cool_if_hot) is on. Count a cooled buffer only when it was not already counted as reusable/written.
💡 Suggested change
Before:
if (sync_state & BUF_COOLED)
reusable_buffers++;
After:
else if (sync_state & BUF_COOLED)
reusable_buffers++;
📄 src/backend/storage/buffer/freelist.c (L847-L848)
Trailing blank line added at EOF. git diff --check flags this as 'new blank line at end of file' - unrelated whitespace churn that will draw a review nit on -hackers. Remove it.
📄 src/backend/storage/buffer/bufmgr.c (L4241-L4243)
BUF_COOLED is set unconditionally after the lock is dropped and re-acquired. UnlockBufHdrExt() releases the header spinlock; between it and LockBufHdr() a concurrent PinBuffer() can pin and re-promote the buffer. The BUF_REUSABLE check below correctly re-reads the fresh state and will fail in that case, but BUF_COOLED is still returned and still counted toward reusable_buffers in BgBufferSync, overstating staged supply. Consider only reporting BUF_COOLED after confirming the re-locked state is still COOL/unpinned, or demote with a direct masked store without dropping the lock to avoid the race window entirely.
📄 src/backend/storage/buffer/freelist.c (L340-L341)
The numCoolClaims counter is incremented before the CAS that actually claims the buffer. If the CAS fails (a concurrent backend touched the buffer, which is exactly the HOT-pressure scenario where contention is likely), the counter was already bumped but no HOT buffer was claimed, and the loop retries, bumping it again. This inflates the metric that BgBufferSync() reads via StrategyCoolClaims() to decide cool_if_hot, so a burst of CAS retries can falsely signal eviction pressure. Increment only on CAS success, inside the if block (next to TrackNewBufferPin), mirroring how the function returns a buffer only after a successful CAS.
(moderate confidence)
💡 Suggested change
Before:
pg_atomic_fetch_add_u64(&StrategyControl->numCoolClaims, 1);
local_buf_state += BUF_REFCOUNT_ONE;
After:
local_buf_state += BUF_REFCOUNT_ONE;
📄 src/include/storage/buf_internals.h (L191-L192)
This rationale is now inaccurate. The pin fast paths no longer reference BM_MAX_USAGE_COUNT: PinBuffer() (bufmgr.c) and PinLocalBuffer() (localbuf.c) both test BUF_STATE_GET_COOLSTATE(buf_state) < BUF_COOLSTATE_HOT directly. The actual remaining consumer of this macro is contrib/pg_buffercache (array sizing). Fix the comment to describe what truly keeps the macro alive, or the next reader will chase a pin fast path that doesn't use it.
(high confidence)
| # Create augmented tuple with embedded modified-column bitmap | ||
| break heap_hot_indexed_create_tuple |
There was a problem hiding this comment.
This entire file is developer-local scaffolding and should not be committed to a patch destined for pgsql-hackers/commitfest. It is unrelated to the HOT-indexed-updates feature itself and violates the minimal-diff discipline. Beyond that, it is broken as committed: the function-name breakpoints reference symbols that do not exist in this tree. heap_hot_indexed_create_tuple, heap_hot_indexed_serialize_bitmap, heap_xlog_indexed_update, etc. return no matches in the source, so GDB will reject these break commands as unresolved. Keep this in a local, un-tracked file (e.g. via .gitignore) instead.
| break heapam.c:4019 | ||
| break heapam.c:4024 | ||
| break heapam.c:4033 |
There was a problem hiding this comment.
Hardcoded absolute line-number breakpoints (heapam.c:4019, heapam.c:4024, heapam.c:4033, pruneheap.c:1802, indexam.c:299, execIndexing.c:370, etc.) are extremely fragile: they silently drift to the wrong statement the moment any of these files change, which is a footgun during review/debugging. If this file is kept at all, prefer function-name + offset breakpoints over raw line numbers so they survive edits.
| pkgs, | ||
| pkgs-unstable, | ||
| system, | ||
| }: let |
There was a problem hiding this comment.
The system argument is destructured here but never referenced anywhere in the file (it only appears in comments/strings). Drop it from the function header to avoid an unused parameter, or use it where intended.
| if $context | ||
| printf "MemoryContext: %s\n", $context->name | ||
| printf " type: %s, parent: %p\n", $context->methods->name, $context->parent | ||
| printf " total: %zu, free: %zu\n", $context->mem_allocated, $context->freep - $context->freeptr |
There was a problem hiding this comment.
print_mcxt dereferences $context->freep and $context->freeptr, but MemoryContextData (src/include/nodes/memnodes.h) has no such fields. The macro will fail to evaluate in GDB. The only size-related field available is mem_allocated; there is no free-pointer pair to subtract.
| define print_relcache | ||
| set $rel = (Relation)$arg0 | ||
| if $rel | ||
| printf "Relation: %s.%s (OID: %u)\n", $rel->rd_rel->relnamespace, $rel->rd_rel->relname.data, $rel->rd_id |
There was a problem hiding this comment.
rd_rel->relnamespace is an OID (a 32-bit unsigned integer), not a string, so printing it with %s will dereference garbage. Use %u for the namespace OID.
| echo "Error: pgindent not found at $PG_SOURCE_DIR/src/tools/pgindent/pgindent" | ||
| else | ||
|
|
||
| modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$") |
There was a problem hiding this comment.
modified_files is assigned without local, so it leaks into the caller's interactive shell as a global. The sibling helpers pg-tidy/pg-spell correctly declare local files. Declare it local here too.
| define print_list | ||
| set $list = (List*)$arg0 | ||
| if $list | ||
| printf "List length: %d\n", $list->length | ||
| set $cell = $list->head | ||
| set $i = 0 | ||
| while $cell && $i < $list->length | ||
| printf " [%d]: ", $i | ||
| print_node $cell->data.ptr_value | ||
| set $cell = $cell->next | ||
| set $i = $i + 1 | ||
| end |
There was a problem hiding this comment.
The print_list GDB macro uses the pre-v13 cons-cell List layout: $list->head, $cell->next, and $cell->data.ptr_value. Since PostgreSQL 13 the List struct (src/include/nodes/pg_list.h) is array-based — it has elements/initial_elements and length/max_length, with no head field; ListCell is now a union with ptr_value directly (no .data sub-struct) and no next pointer. This macro will fail to evaluate against the tree it ships with. (Same broken-GDB-macro class as the confirmed print_mcxt/print_relcache findings.) High confidence.
| # Requires: texlive-latex-recommended texlive-latex-extra texlive-publishers | ||
| # texlive-fonts-recommended texlive-fonts-extra cm-super | ||
| # texlive-bibtex-extra texlive-pictures latexmk |
There was a problem hiding this comment.
[high confidence] This entire paper/ directory (a LaTeX/texlive build for an ACM paper) is out of scope for a PostgreSQL patch. The change is destined for pgsql-hackers/commitfest, where the top rejection reason is a non-minimal diff that "does more than one thing." A paper build system has nothing to do with the clock-sweep buffer manager code change and should be removed from the patch (keep it in a separate repo/branch). Everything below is secondary to this.
| clean: | ||
| latexmk -C |
There was a problem hiding this comment.
[high confidence] If this Makefile is kept at all, it does not follow PostgreSQL makefile conventions: it does not include $(top_builddir)/src/Makefile.global, uses no standard PG variables, and provides only clean (missing distclean and maintainer-clean). latexmk -C also removes the final PDF, so there is no separation between intermediate aux files and the committed/distributed output. It is additionally not wired into any parent Makefile or meson.build, making it orphaned scaffolding.
| bounded-clock-sweep.pdf: bounded-clock-sweep.tex refs.bib $(wildcard *.dat) | ||
| latexmk -pdf -interaction=nonstopmode $< |
There was a problem hiding this comment.
[moderate confidence] The LaTeX build produces artifacts (bounded-clock-sweep.pdf, .aux, .log, .bbl, .blg, .fls, .fdb_latexmk, .out) that are not covered by the top-level .gitignore and there is no paper/.gitignore. These will show up as untracked files after a build and risk being accidentally committed.
Squashes the four CI commits this fork carried on top of upstream into
one, and cuts the carry from 3776 lines to ~1000.
Removed:
- .github/{README,QUICKSTART}.md and .github/docs/* (2273 lines). Beyond
being bulk we rebase hourly, they had drifted into being wrong: they
described a daily (not hourly) sync, Claude 3.5 Sonnet via the direct
Anthropic API (we run Opus on Bedrock), a static IAM user (we use
OIDC), and a Windows dependency-builder workflow that the same commit
which documented it had already deleted. 16 referenced paths did not
exist. An AI reviewer reading this repo ingests that as truth.
- sync-upstream-manual.yml (252 lines). sync-upstream.yml already has
workflow_dispatch; the sole behavioral difference was a force_push
toggle whose "false" setting cannot work after a rebase anyway.
- .github/.gitignore, which only ignored scripts/ai-review/, a directory
that does not exist here.
Rewritten:
- sync-upstream.yml -> fork-sync-upstream.yml, 260 lines to 47. git
rebase already handles the fast-forward, no-op and diverged cases that
the ahead/behind arithmetic hand-rolled, so all that goes away, as do
the issue open/comment/close machinery (a scheduled-run failure is
already emailed to the one consumer, and the run log carries what the
issue body would) and the "dev setup|dev v[0-9]" commit allowlist,
which matched zero commits on master and had since it was written.
Sync drops to every three hours: hourly meant 24 full-history clones
of an 840MB repo per day, each force-push also re-triggering pg-ci.yml
on master, to collect a handful of upstream commits.
Renamed:
- ocr-review.yml, ocr-model-check.yml -> fork-*.yml. Upstream owns
.github/ too (it has touched pg-ci.yml three times this year), so
fork-owned workflows now live under a name prefix upstream will not
collide with, making the rebase structurally conflict-free and letting
the sync guard match fork-owned paths exactly rather than allowing all
of .github/.
Unchanged: the OCR review workflows' logic and .github/ocr/* (rule.json,
context.md, litellm.yaml, pg-history.py) - the part that carries value.
The OCR review workflows and their config were the last bulk this fork carried on top of upstream: ~970 lines rebased onto postgres/postgres every few hours, editable only by amending a commit that gets force- pushed. They are now reusable (workflow_call) workflows in the sidecar repo gburd/ci-workflows, and master carries two caller stubs instead. Carry after this: 3 files, ~110 lines, one commit. The sidecar is also where the value is: ocr/rule.json and ocr/context.md (the PostgreSQL committer-grade review corpus) become normal reviewable files with normal PRs and history, rather than a blob inside a commit that is rewritten on every sync. Mechanics worth knowing: - The review jobs check ci-workflows out at github.job_workflow_sha, so the rules always come from the same commit as the logic using them, and pinning @v1 (or a sha) in the stub pins both. This replaces the old "git show origin/<default_branch>:.github/ocr/..." dance, which existed only because the config had to be read off a branch other than the PR's. - AWS settings are passed as explicit inputs rather than read from vars inside the reusable workflow. vars does resolve across the boundary, but the contract stays visible at the call site. - A reusable workflow does not change the OIDC subject: it stays repo:gburd/postgres:*, so the ocr-bedrock-ci trust policy is unchanged. - ci-workflows must stay public; callers read it with GITHUB_TOKEN. The sync guard narrows to .github/workflows/fork-* accordingly, and sync-upstream keeps needing SYNC_TOKEN: the stubs are still workflow files, which GITHUB_TOKEN may not push.
This isn't a commit that will be submitting for review, it is purely for local developer tooling while developing this patch set. Ignore it.
A buffer becomes a candidate for eviction only once its usage_count has been decremented to zero, and PinBuffer() saturates the count on every access. With a pass time of T, a buffer must therefore go roughly BM_MAX_USAGE_COUNT * T without being touched before the sweep can take it. T grows with NBuffers, so on a large pool a workload that touches much of the pool more often than that re-promotes buffers faster than the hand decrements them. The population of candidates collapses, and because a decrement counts as progress and resets trycounter, StrategyGetBuffer() has no bound on the work it will perform: a single buffer allocation can sweep the entire pool. The header has documented the possibility for years -- "it can take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the clock-sweep hand to find a free buffer" -- but it is reached in practice, not only in theory, and the cost scales linearly with shared_buffers. Bound it. A sweep that has decremented BUF_DECREMENT_CLAIM_THRESHOLD buffers without finding a single one at zero has established, by observation rather than by guesswork, that the pool is hotter than the hand can grind down. From that point in the same call it claims the next unpinned buffer directly instead of decrementing it. usage_count is cleared when a victim is reused (InvalidateVictimBuffer), so the claim needs no separate step to drive the count to zero. The cost is evicting a buffer that has not been passed over the full BM_MAX_USAGE_COUNT times. Both sides of that trade are measured: the worst-case advances an allocation performs is the threshold plus about two, and on a workload with a hot set re-read by most queries a threshold of 4 evicts thousands of hot pages for 1.8 points of hot-set hit ratio, 32 evicts about ten for 0.03 points, and 128 evicts none for no measurable cost. The default of 128 therefore bounds the tail four orders of magnitude below an unbounded scan without measurably degrading replacement quality. A control loop that adapted the threshold was tried and proved worse than a fixed value on both axes. The counter is call-local, so the common path takes no additional atomic. Under a workload the hand can keep up with, the threshold is never reached and behaviour is exactly as before. StrategyControl gains numForcedClaims, counting allocations that had to resort to this. A growing value is the signal that the pool is undersized for its access rate. This changes only the termination rule of the sweep. It does not alter the 0..5 usage_count, the meaning of any buffer state bit, the background writer, or the BufferAccessStrategy rings, and it is independent of any change to the replacement state encoding.
left a comment
There was a problem hiding this comment.
🔍 OCR found 16 issue(s).
- 14 inline, 2 in summary
📄 .clangd
.clangd is personal editor/LSP configuration and should not be committed to a PostgreSQL patch. Additionally, the hardcoded relative include -I../../../../src/include assumes a specific nested working directory and will not resolve from the repo root or for most translation units, defeating the config's purpose. Recommend removing this file from the patch (keep it git-ignored locally).
📄 src/include/storage/buf_internals.h
The claim "The worst-case advances an allocation performs is almost exactly this threshold plus two" is only true under the targeted hot-pool workload. The true upper bound on advances per allocation remains NBuffers: after the threshold is reached, the sweep still iterates over pinned buffers (refcount != 0 path decrements trycounter without claiming) and over usage_count==0 buffers whose CAS loses to a concurrent pinner, until trycounter hits zero. Recommend qualifying this as an observed/typical worst case rather than an absolute one, to avoid misleading a future reader into assuming a hard threshold+2 bound.
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
This shebang change is an unrelated, non-minimal diff that deviates from the project-wide convention. Every other Perl script in the tree (e.g. src/tools/copyright.pl, mark_pgdllimport.pl, src/backend/catalog/genbki.pl, and ~40 others) uses #!/usr/bin/perl; this makes pgindent the sole outlier. PostgreSQL's build system locates the Perl interpreter explicitly (via $(PERL) / perl.path), so /usr/bin/env perl provides no benefit here and only introduces inconsistency. Unless the surrounding change actually requires a different interpreter resolution, revert this to keep the diff minimal and consistent with the rest of the tree.
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
This entire file is developer-local debugging state that does not belong in a patch destined for pgsql-hackers. Beyond the minimal-diff objection, it is actively broken: the breakpoints hardcode absolute source line numbers (heapam.c:4019, pruneheap.c:1802, pruneheap.c:2936, etc.) and symbol names (heap_hot_indexed_create_tuple, heap_hot_search_buffer, ...) that do not exist in this tree. A codebase search for heap_hot_indexed_ returns zero matches, and the referenced heap/pruneheap changes are not part of this change set. These breakpoints point at nothing and will rot silently the moment the source shifts by a line. Recommend removing this file from the patch and keeping it in a personal, git-ignored location.
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
Non-ASCII characters (em-dash —) appear in this file. PostgreSQL requires ASCII-only in source and committed files.
| alias pg-full-clean='trash "$PG_BUILD_DIR" "$PG_INSTALL_DIR" 2>/dev/null || rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR"; echo "Build and install directories cleaned"' | ||
|
|
||
| # Database management | ||
| alias pg-init='trash "$PG_DATA_DIR" 2>/dev/null || rm -rf "$PG_DATA_DIR"; "$PG_INSTALL_DIR/bin/initdb" --debug --no-clean "$PG_DATA_DIR"' |
There was a problem hiding this comment.
rm -rf "$PG_DATA_DIR" (and pg-full-clean's rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR") are a data-loss footgun when the variable is unset/empty. If this file is sourced outside the Nix shellHook that exports these vars, rm -rf "" degrades to deleting relative to the current directory. Guard each destructive deletion with a non-empty check (e.g. [ -n "$PG_DATA_DIR" ] && ...). More fundamentally, this personal alias library does not belong in a pgsql-hackers patch.
| if [ "$last_compiler" != "$current_compiler" ] && [ "$last_compiler" != "unknown" ]; then | ||
| echo "Detected compiler change from $last_compiler to $current_compiler" | ||
| echo "Cleaning build directory..." | ||
| trash "$build_dir" 2>/dev/null || rm -rf "$build_dir" |
There was a problem hiding this comment.
trash is not a POSIX/coreutils tool and is not provided by getPostgreSQLDeps in shell.nix, so on a clean dev shell the || rm -rf fallback is always taken. Either add trash/trash-cli to the Nix buildInputs or drop the trash indirection to keep the documented workflow reproducible.
| * StrategyGetBuffer). A growing value means the pool is hotter than the | ||
| * clock hand can keep up with. | ||
| */ | ||
| pg_atomic_uint64 numForcedClaims; |
There was a problem hiding this comment.
numForcedClaims is write-only dead instrumentation. Unlike its sibling numBufferAllocs, which is consumed by StrategySyncStart() (feeding the bgwriter) and reset there, numForcedClaims is only initialized (StrategyCtlShmemInit) and incremented (StrategyGetBuffer) but has no reader anywhere in the tree. It is therefore not observable, contradicting the comment's premise ("A growing value means the pool is hotter than the clock hand can keep up with") -- nothing can see that value grow. Either wire it up to a consumer (e.g. pg_stat_bgwriter / a pg_buffercache summary / StrategySyncStart out-param) with tests, or drop the field and its atomic add from the hot path. As-is it bloats the shared BufferStrategyControl struct and adds a cross-core atomic on the pressure path for no benefit (YAGNI).
| * when the victim is reused (see InvalidateVictimBuffer), so | ||
| * there is no need to drive it to zero first. | ||
| */ | ||
| pg_atomic_fetch_add_u64(&StrategyControl->numForcedClaims, 1); |
There was a problem hiding this comment.
This atomic add sits on the eviction pressure path and feeds a counter that is never read (see numForcedClaims field). It is pure overhead: a 64-bit RMW on the StrategyControl cache line (shared with numBufferAllocs/completePasses) under exactly the contended, buffer-starved workload this feature targets. If the counter is kept, it should at least be justified by an observable consumer; otherwise remove this line along with the field.
| * A control loop that adapted this value was tried and was worse than a fixed | ||
| * one on both axes; see the commit message for that experiment. |
There was a problem hiding this comment.
"see the commit message for that experiment" is a fragile cross-reference: once committed/squashed/rebased the commit message is not accessible from the source tree, and readers of the code can't follow it. PostgreSQL comments should be self-contained. Either state the conclusion inline (adaptive control was tried and performed worse on both axes, so a fixed threshold is used) and drop the pointer, or reference the pgsql-hackers thread Message-Id. Also, this block reads as a benchmark narrative; it explains WHY well but is unusually long for buf_internals.h -- consider trimming the per-threshold measurement figures (4/32/128) to the essential conclusion.
| if (decremented < BUF_DECREMENT_CLAIM_THRESHOLD) | ||
| { | ||
| local_buf_state -= BUF_USAGECOUNT_ONE; |
There was a problem hiding this comment.
This changes the buffer-eviction policy under pressure (claiming a buffer with non-zero usage_count after BUF_DECREMENT_CLAIM_THRESHOLD fruitless decrements), yet the patch adds no regression/TAP coverage for the new path. A user-visible behavior change in the clock-sweep victim selection needs a test that exercises the forced-claim branch (e.g. a small NBuffers pool kept hot enough that the threshold is reached), otherwise this is WIP, not commit-ready. Confidence: moderate.
| * The value trades the tail against hit ratio, and both sides have been | ||
| * measured. The worst-case advances an allocation performs is almost exactly | ||
| * this threshold plus two, so the tail is a direct and predictable function of | ||
| * it. The cost is that a claimed buffer is evicted without having been passed | ||
| * over the full number of times: on a workload with a hot set re-read by most | ||
| * queries, a threshold of 4 evicts thousands of genuinely hot pages and costs | ||
| * 1.8 percentage points of hot-set hit ratio, a threshold of 32 evicts about | ||
| * ten and costs 0.03 points, and at 128 nothing hot is evicted and the cost is | ||
| * not measurable. 128 therefore buys a bound four orders of magnitude below an | ||
| * unbounded scan for no measurable loss of replacement quality, which is the | ||
| * right place to sit. |
There was a problem hiding this comment.
This header comment embeds very specific, unverifiable benchmark numbers ("1.8 percentage points", "0.03 points", "four orders of magnitude") with no reference to a reproducible benchmark or pgsql-hackers thread/Message-Id. PostgreSQL comments should explain the invariant/why, not carry a results table that will silently rot as the code and workloads change. Trim this to the design rationale (bounded work vs. replacement quality) and move the measurements to the commit message / -hackers thread. Confidence: moderate.
Confirms the clock-sweep tail across three pool sizes on storage fast enough to expose it: Fedora 44, XFS on RAID-0 instance-local NVMe at 429k random read IOPS, no network-attached storage in the path. The stock worst case is exactly one full pass of the buffer pool at every size measured (N+5, N+5, N+38; tail/N = 1.0000 to five figures), so it scales directly with shared_buffers. The bound holds it flat at 130-134 advances across an eightfold pool range, a 3,971x / 7,825x / 32,017x reduction, with the factor growing precisely because the bounded quantity is the one that scales. Cost is nil on the other two axes: throughput within 0.4%, amplification within 0.019 advances per victim, hit ratio within 0.0014 points, all inside run-to-run variation. Also records why an earlier attempt on a 6,000 IOPS volume measured nothing: when every miss waits on storage the allocation rate is throttled and the clock hand keeps up, so fast local storage is a precondition for the regime regardless of skew or working-set size. And that several hundred thousand buffers at usage_count zero -- true in every arm, including stock -- is not evidence the tail is absent. Data in .agent/nvme-2026-10-07/ (not committed).
left a comment
There was a problem hiding this comment.
🔍 OCR found 26 issue(s).
- 25 inline, 1 in summary (inline capped at 25)
📄 pg-aliases.sh (L508-L508)
modified_files is assigned without local, so it leaks into the caller's shell environment. Since this file is sourced into interactive shells (see shell.nix source ./pg-aliases.sh), the function clobbers any pre-existing modified_files variable the user may have and leaves it set afterwards. Every other helper here correctly declares its working variables local. Declare it local (ideally at the top of the function alongside since).
💡 Suggested change
Before:
modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$")
After:
local modified_files
modified_files=$(git diff --name-only "${since}" | grep -E "\.c$|\.h$")
| @@ -0,0 +1,156 @@ | |||
| # HOT Indexed Updates — GDB breakpoints for code review | |||
There was a problem hiding this comment.
This developer-only .gdbinit does not belong in a PostgreSQL patch and would be rejected on pgsql-hackers as unrelated noise. Worse, it is scaffolding for an unmerged feature: the breakpoints reference heap_hot_indexed_create_tuple, heap_hot_indexed_serialize_bitmap, heap_xlog_indexed_update, XLOG_HEAP2_INDEXED_UPDATE, and indexed_attrs (indexam.c) — none of which exist anywhere in this tree (verified via code search, zero matches). A source .gdbinit against today's binary will fail to set these breakpoints. Keep this in a developer's local, git-ignored environment, not in the source tree. [high confidence]
| # Main entry: heap_update | ||
| break heapam.c:3210 |
There was a problem hiding this comment.
Breakpoints pinned to absolute source line numbers are brittle and have already drifted. heapam.c:3210 is the closing brace of heap_delete — heap_update actually begins at line 3269. The accompanying prose ("Line 4019: pure HOT", "Line 4024: HOT indexed path", etc.) will desync from the file on any edit, making this file actively misleading. Function-name breakpoints (as used elsewhere in this file) are the only maintainable form; the numeric ones should be dropped. [high confidence]
| CompileFlags: | ||
| CompilationDatabase: build/ # Search build/ directory for compile_commands.json |
There was a problem hiding this comment.
This is a per-developer IDE environment file (CompilationDatabase: build/, personal flags like -DLOCAL, -O2, -std=c11, and hard-coded relative includes -I. / -I../../../../src/include). Committing it imposes one contributor's local layout on everyone and is unrelated to any functional change in this series — it would be flagged as non-minimal diff on pgsql-hackers. Such IDE/build artifacts belong in a personal, git-ignored config, not the source tree. [high confidence]
| name: OCR AI Review | ||
|
|
||
| on: |
There was a problem hiding this comment.
This job declares no permissions block, so GITHUB_TOKEN defaults to the repo's broad scope. The sibling workflows pg-ci.yml and fork-sync-upstream.yml both pin permissions: contents: read; this one does not. That is especially risky here because the issue_comment trigger (lines 14-15) lets anyone who can comment on a PR invoke a review run. Add an explicit least-privilege permissions block at the top level (or on the job) so the caller only grants what the reusable workflow actually needs.
| name: OCR AI Review | |
| on: | |
| name: OCR AI Review | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| on: |
|
|
||
| on: | ||
| pull_request: | ||
| # Note: no draft filter — drafts are reviewed too. |
There was a problem hiding this comment.
ASCII-only is required in source. This comment uses an em-dash (U+2014). Replace it with an ASCII hyphen/dash.
| # Note: no draft filter — drafts are reviewed too. | |
| # Note: no draft filter - drafts are reviewed too. |
|
|
||
| alias pg-start='ulimit -c unlimited && "$PG_INSTALL_DIR/bin/postgres" -D "$PG_DATA_DIR" -k "$PG_DATA_DIR"' | ||
|
|
||
| alias pg-stop='pkill -f "postgres.*-D.*$PG_DATA_DIR" || true' |
There was a problem hiding this comment.
pg-init and pg-full-clean fall back to rm -rf "$PG_DATA_DIR" / rm -rf "$PG_BUILD_DIR" "$PG_INSTALL_DIR". These are safe only because shell.nix always exports these vars; if this script is sourced standalone (its only guard is the warning in shell.nix, not the script itself) the vars are empty and these expand to rm -rf "" (harmless) but pg-stop/pg-status build the pkill -f "postgres.*-D.*$PG_DATA_DIR" pattern which, with an empty var, becomes postgres.*-D.* and kills every postgres on the host. Add a non-empty guard on PG_DATA_DIR before the destructive/pkill operations so the script fails safe outside the nix shell.
| --dlpath="$bdir/src/test/regress" \ | ||
| --outputdir="$outdir" \ | ||
| --temp-instance="$outdir/tmp" \ | ||
| --port=40099 \ |
There was a problem hiding this comment.
The hardcoded --port=40099 means two concurrent pg-test-one invocations (or anything already bound to 40099) collide and fail non-obviously. Consider deriving the port from the PID/PWD or letting pg_regress pick a free port, to avoid surprising failures on a shared dev box or parallel runs.
| @@ -1,4 +1,4 @@ | |||
| #!/usr/bin/perl | |||
| #!/usr/bin/env perl | |||
There was a problem hiding this comment.
This changes pgindent's shebang to #!/usr/bin/env perl, making it the only Perl script in the entire tree that doesn't use #!/usr/bin/perl. All other ~35 Perl scripts/tools (genbki.pl, gen_node_support.pl, copyright.pl, mark_pgdllimport.pl, etc.) use the fixed #!/usr/bin/perl path. This one-line deviation is unrelated to any functional change in this PR, breaks tree-wide consistency, and will be rejected on the minimal-diff / consistency grounds the project enforces. If a #!/usr/bin/env convention were desired it would need to be a separate, tree-wide discussion on -hackers, not a single-file change. Revert to keep the diff minimal and consistent. (high confidence)
| #!/usr/bin/env perl | |
| #!/usr/bin/perl |
| * StrategyGetBuffer). A growing value means the pool is hotter than the | ||
| * clock hand can keep up with. | ||
| */ | ||
| pg_atomic_uint64 numForcedClaims; |
There was a problem hiding this comment.
numForcedClaims is write-only dead state. It is incremented here and initialized in StrategyCtlShmemInit, but no code anywhere in the tree ever reads or exports it (only the .tex paper references the name), and unlike its sibling numBufferAllocs it is never reset or returned by StrategySyncStart. It therefore just enlarges the shared BufferStrategyControl struct and adds an atomic RMW on the hot allocation path for no observable benefit. Either wire it into an observable path (e.g. a pg_stat_bgwriter-style counter with matching reset/read handling) or drop it. Minimal patches should not leave unused scaffolding. [high confidence]
| pg_atomic_fetch_add_u64(&StrategyControl->numForcedClaims, 1); | ||
| local_buf_state += BUF_REFCOUNT_ONE; |
There was a problem hiding this comment.
The counter is incremented before the CAS below, so it counts claim attempts, not successful forced claims. When the CAS at the following line fails (concurrent modification of buf->state -- precisely the contended/hot-pool scenario this path is designed for), the inner for (;;) re-reads old_buf_state and re-enters this branch, incrementing numForcedClaims again without any buffer having been claimed. Move the increment to inside the successful-CAS block so it tracks actual forced claims. [medium confidence]
| pg_atomic_fetch_add_u64(&StrategyControl->numForcedClaims, 1); | |
| local_buf_state += BUF_REFCOUNT_ONE; | |
| local_buf_state += BUF_REFCOUNT_ONE; |
Cooling-stage clock sweep for the buffer manager — CI check.
This PR is against gburd/postgres:master (my fork), not upstream, purely
to exercise CI on the three-patch series before posting to pgsql-hackers.
Net +510/-1642 over 83 files (almost all deletion is patch 3).
Each commit builds -Werror + cassert + injection_points and passes
regress/regress(245 tests) independently; verified on both the fork baseand a clean upstream/master rebase. See the draft cover letter for the
design reasoning and the m6i / r8i (6-node NUMA) / huge-pages / local-NVMe
real-IO benchmark data.
Not for merge — this is the review/CI vehicle for the -hackers submission.