docs: clarify current and best state under strict gating - #301
Open
Bogdan (Dan) Baciu (bogdanbaciu21) wants to merge 1 commit into
Open
Bogdan (Dan) Baciu (bogdanbaciu21) wants to merge 1 commit into
Bogdan (Dan) Baciu (bogdanbaciu21) wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Problem This Solves
Refs #113. The training-loop guide explains candidate gating but does not explain why the paper keeps separate current and best skills when they remain equal in Algorithm 1. The nested best-score check can appear to affect the result even though it is redundant on that strict path.
Why This Change Was Made
Adds one subsection to
docs/guide/training-loop.mdexplaining the invariant and the optional implementation modes that can break it: force-accepted candidates with the gate disabled and ungated slow updates.Project Fit
Documents the behavior already confirmed in issue #113 and covered by the existing gate regression. The change is documentation-only.
User Impact
Readers can distinguish the paper's strict procedure from optional modes where the current skill and the historical validation-best snapshot can diverge.
Proof
Current and best start with the same skill and score. Rejection changes neither. A candidate that beats current must also beat best while those scores are equal, so both advance together. The existing invariant test executes improving and regressing candidates and checks the equality after every transition; the gate branches implement that contract.
Exact candidate:
1c48d52106c026fbca7553743bf87e3d64a4683a, based on upstreamf02c6fce16e958c185b57ebb66e241a8ab2a7b76. Its entire diff is 14 added documentation lines in one file.Academic Support
Issue #113 provides the algorithm question and accepted explanation. The linked implementation and executable invariant are the primary evidence for this clarification. No new method or performance claim is introduced.
Testing
Every fork-runner job asserted the exact candidate SHA before testing. These are contributor-run receipts, not official upstream CI. The matrix runs strict docs on Python 3.12 for each operating system; Python 3.10 and 3.11 add test-suite coverage only.
The Windows full-suite failures reproduce unchanged on the exact upstream base: the home-as-git-root test and five unchanged-staging subcases fail with the same assertions and totals. Neither file changes in this PR. No test was deselected or failure masked in either full-suite run.
Strict docs passed locally and on Ubuntu/macOS Python 3.12. A separate Windows run executed the docs build despite the known full-suite failures and passed that step; its overall job correctly remains failed.
git diff --checkpassed.Limitations & Negative Results
This clarifies the repository guide; it does not revise the paper or change runtime behavior. It does not fix the existing Windows test failures. Equality is claimed only for the strict Algorithm 1 path from equal initial state, without optional operations that modify current separately.
Reproduce It Yourself
Linux or macOS, shell from a checkout of this PR at the candidate SHA above:
python3 -m venv .venv .venv/bin/python -m pip install -e '.[dev,docs]' .venv/bin/python -m pytest tests/test_gate.py -q .venv/bin/python -m pytest -q .venv/bin/python -m mkdocs build --strict git diff --check