Skip to content

docs: clarify current and best state under strict gating - #301

Open
Bogdan (Dan) Baciu (bogdanbaciu21) wants to merge 1 commit into
microsoft:mainfrom
bogdanbaciu21:docs-113-current-best-invariant
Open

Bogdan (Dan) Baciu (bogdanbaciu21) wants to merge 1 commit into
microsoft:mainfrom
bogdanbaciu21:docs-113-current-best-invariant

Conversation

@bogdanbaciu21

Copy link
Copy Markdown
Contributor

What Problem This Solves

Refs #113. The training-loop guide explains candidate gating but does not explain why the paper keeps separate current and best skills when they remain equal in Algorithm 1. The nested best-score check can appear to affect the result even though it is redundant on that strict path.

Why This Change Was Made

Adds one subsection to docs/guide/training-loop.md explaining the invariant and the optional implementation modes that can break it: force-accepted candidates with the gate disabled and ungated slow updates.

Project Fit

Documents the behavior already confirmed in issue #113 and covered by the existing gate regression. The change is documentation-only.

User Impact

Readers can distinguish the paper's strict procedure from optional modes where the current skill and the historical validation-best snapshot can diverge.

Proof

Current and best start with the same skill and score. Rejection changes neither. A candidate that beats current must also beat best while those scores are equal, so both advance together. The existing invariant test executes improving and regressing candidates and checks the equality after every transition; the gate branches implement that contract.

Exact candidate: 1c48d52106c026fbca7553743bf87e3d64a4683a, based on upstream f02c6fce16e958c185b57ebb66e241a8ab2a7b76. Its entire diff is 14 added documentation lines in one file.

Academic Support

Issue #113 provides the algorithm question and accepted explanation. The linked implementation and executable invariant are the primary evidence for this clarification. No new method or performance claim is introduced.

Testing

Every fork-runner job asserted the exact candidate SHA before testing. These are contributor-run receipts, not official upstream CI. The matrix runs strict docs on Python 3.12 for each operating system; Python 3.10 and 3.11 add test-suite coverage only.

Platform Python Gate tests Full suite Strict docs
Ubuntu 3.10 22 passed 1,558 passed; 12 skipped; 359 subtests not run
Ubuntu 3.11 22 passed 1,558 passed; 12 skipped; 359 subtests not run
Ubuntu 3.12 22 passed 1,558 passed; 12 skipped; 359 subtests passed
macOS arm64 3.12 22 passed 1,558 passed; 12 skipped; 359 subtests passed
Windows 3.12 22 passed 6 failed; 1,510 passed; 59 skipped; 354 subtests passed

The Windows full-suite failures reproduce unchanged on the exact upstream base: the home-as-git-root test and five unchanged-staging subcases fail with the same assertions and totals. Neither file changes in this PR. No test was deselected or failure masked in either full-suite run.

Ubuntu / Python 3.12
actual_sha=1c48d52106c026fbca7553743bf87e3d64a4683a
1558 passed, 12 skipped, 8 warnings, 359 subtests passed in 43.31s
INFO    -  Documentation built in 0.87 seconds

macOS arm64 / Python 3.12
actual_sha=1c48d52106c026fbca7553743bf87e3d64a4683a
1558 passed, 12 skipped, 8 warnings, 359 subtests passed in 32.91s
INFO    -  Documentation built in 0.62 seconds

Windows / Python 3.12
candidate actual_sha=1c48d52106c026fbca7553743bf87e3d64a4683a
6 failed, 1510 passed, 59 skipped, 354 subtests passed

baseline actual_sha=f02c6fce16e958c185b57ebb66e241a8ab2a7b76
6 failed, 1510 passed, 59 skipped, 354 subtests passed

Strict docs passed locally and on Ubuntu/macOS Python 3.12. A separate Windows run executed the docs build despite the known full-suite failures and passed that step; its overall job correctly remains failed. git diff --check passed.

Limitations & Negative Results

This clarifies the repository guide; it does not revise the paper or change runtime behavior. It does not fix the existing Windows test failures. Equality is claimed only for the strict Algorithm 1 path from equal initial state, without optional operations that modify current separately.

Reproduce It Yourself

Linux or macOS, shell from a checkout of this PR at the candidate SHA above:

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev,docs]'
.venv/bin/python -m pytest tests/test_gate.py -q
.venv/bin/python -m pytest -q
.venv/bin/python -m mkdocs build --strict
git diff --check

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant