Document YAML configuration schemas and guard against schema drift

configs/README.md

@@ -17,7 +17,7 @@ Put repeated language `evidence` and `note` under `language_defaults` in the eva
 
 Each profile or set needs a distinct `name` within its directory. To change a selector's startup choice, edit that directory's `default.txt` to name one YAML file. The catalogue is selected at build time with `--catalogue`, or temporarily loaded in the browser.
 
-The [configuration reference](../docs/configuration.md) describes all three formats with small examples. [examples/](examples/) contains the fictional catalogue and weights used by [examples/scores.csv](../examples/scores.csv).
+The [configuration reference](../docs/configuration.md) lists every field's type, required/optional status, allowed values, and default behavior. Start with [eval sets](../docs/configuration.md#eval-sets), [weighting profiles](../docs/configuration.md#weighting-profiles), or [per-eval definitions](../docs/configuration.md#per-eval-definitions). It also covers catalogue manifests, portable catalogues, language assignments, and startup selections. [examples/](examples/) contains the fictional catalogue and weights used by [examples/scores.csv](../examples/scores.csv).
 
 Submit changes as a PR, then check the generated dashboard:
 

docs/development.md

@@ -16,6 +16,22 @@ Run commands from the repository root. Install the Python package with `python -
 | `docs/` | Configuration and contributor documentation. |
 | `data/`, `output/` | Ignored local inputs and generated files. |
 
+## Keep configuration docs in sync
+
+The public checks validate the schema tables and YAML examples in
+[the configuration reference](configuration.md). Invisible `config-schema` labels
+identify field tables; `config-example` labels identify executable examples.
+Keep those labels when moving sections or rewriting prose. Every YAML block
+needs an example label and a validation context in `tests/config_docs.py`.
+
+`tests/test_config_docs.py` observes the Python validators' field checks to
+compare allowed fields and unconditional requirements with the tables. It also
+loads the manifest and startup examples from disk. `tests/test_engines.py` runs
+the scoring examples through both engines. These checks do not snapshot prose,
+line numbers, or heading order. Conditional requirements, defaults, and numeric
+semantics still need behavioral regressions when the corresponding contract
+changes; the tests cannot prove that every explanatory sentence is correct.
+
 ## Build and inspect a change
 
 ```sh

docs/python-api.md

@@ -95,16 +95,18 @@ scopes are resolved during loading; local metadata overrides are preserved. The
 complete in-memory catalogue, so later analysis does not access those files.
 Existing single-file catalogues remain supported. When passing a dictionary
 instead of a filename, supply the complete catalogue; a filesystem manifest needs
-a filename to resolve its directory. See [editing an eval](configuration.md#edit-one-eval).
+a filename to resolve its directory. See the [per-eval YAML format](configuration.md#per-eval-definitions).
 
 ## Expected settings and relaxed comparisons
 
 Both `analyze()` and `compare()` accept `matching="strict"` (the default) or
 `matching="relaxed"`. They first apply the eval set's optional whole-eval `metric`,
 `metric_filter`, and `shots` overrides to the catalogue defaults. Set-wide and
 per-eval `exclude_languages` remove results from membership and required coverage.
-See the [configuration contract](configuration.md#strict-and-relaxed-matching)
-for selection, ambiguity, and component-group rules.
+See the [set field reference](configuration.md#set-eval-entries) for `shots`,
+`metric`, and `metric_filter` override syntax, and the
+[matching rules](configuration.md#strict-and-relaxed-matching) for selection,
+ambiguity, and component-group behavior.
 
 ```python
 comparison = compare(

tests/check.py

@@ -103,7 +103,8 @@ def main():
     # Browser tests invoke python3; use this command's Python environment for builds.
     os.environ['PATH'] = str(Path(sys.executable).parent) + os.pathsep + os.environ.get('PATH', '')
     subprocess.run([sys.executable, '-m', 'unittest', 'tests.test_check',
-                    'tests.test_data', 'tests.test_engines', 'tests.test_pages_preview'], cwd=ROOT, check=True)
+                    'tests.test_data', 'tests.test_engines', 'tests.test_config_docs',
+                    'tests.test_pages_preview'], cwd=ROOT, check=True)
     subprocess.run(['node', '--test'] + ['tests/' + name for name in NODE_TESTS], cwd=ROOT, check=True)
     run_browser(chrome)
     print('All public checks passed, including the browser.')

tests/config_docs.py

@@ -0,0 +1,81 @@
+"""Read stable schema/example labels without depending on prose or heading order."""
+
+from pathlib import Path
+import re
+
+from quickdash.io import parse_yaml
+from quickdash.config import load_catalogue
+
+ROOT = Path(__file__).resolve().parent.parent
+REFERENCE = ROOT / "docs/configuration.md"
+
+
+def examples(source=None):
+    source = REFERENCE.read_text() if source is None else source
+    entries = re.findall(
+        r"<!--\s*config-example:\s*([\w-]+)\s*-->\s*```yaml\s*\n(.*?)\n```",
+        source, re.S,
+    )
+    if len(entries) != len(re.findall(r"(?m)^```yaml\s*$", source)):
+        raise AssertionError("Every YAML example needs a config-example label")
+    result = {name: parse_yaml(body) for name, body in entries}
+    if len(result) != len(entries):
+        raise AssertionError("Duplicate config-example label")
+    return result
+
+
+def schemas(source=None):
+    source = REFERENCE.read_text() if source is None else source
+    result = {}
+    for name, table in re.findall(
+        r"<!--\s*config-schema:\s*([\w-]+)\s*-->\s*((?:\|[^\n]*\n)+)", source
+    ):
+        if name in result:
+            raise AssertionError("Duplicate config-schema label: " + name)
+        fields = {}
+        for row in table.splitlines():
+            match = re.match(r"\|\s*`(\w+)`\s*\|\s*([^|]+)\|", row)
+            if match:
+                key, requirement = match.groups()
+                if key in fields:
+                    raise AssertionError("Duplicate documented field: " + name + "." + key)
+                fields[key] = requirement.strip()
+        if not fields:
+            raise AssertionError("Missing field table for " + name)
+        result[name] = fields
+    return result
+
+
+def engine_cases():
+    """Exercise every scoring example in its documented catalogue context."""
+    docs = examples()
+    production = load_catalogue(ROOT / "configs/catalogue.yaml")
+    available = dict(version=1, name="Documentation checks", mode="available")
+    base = dict(catalogue=docs["catalogue"], profile=docs["profile"], suite=available)
+    metadata = {k: v for k, v in docs["manifest"].items() if k != "evals_dir"}
+    cases = {}
+    for name, example in docs.items():
+        config = dict(base)
+        case = dict(config=config, rows=[])
+        if name.startswith("set-"):
+            config.update(catalogue=docs["catalogue"] if name == "set-pinned" else production,
+                          suite=example)
+        elif name in {"catalogue", "profile"}:
+            config[name] = example
+        elif name in {"manifest", "eval", "language-defaults", "translation", "components"}:
+            definition = dict(docs["eval"])
+            if name == "language-defaults":
+                definition.update(example)
+            elif name == "translation":
+                definition.update(match=dict(name=example["tasks"][0]), languages=[example])
+            elif name == "components":
+                definition = parse_yaml((ROOT / "configs/evals/polymath.yaml").read_text())
+                definition.update(example)
+            config["catalogue"] = metadata
+            case["eval_definitions"] = [definition]
+        elif name == "startup":
+            continue  # Builder-only format; covered in test_config_docs.
+        else:
+            raise AssertionError("Add a validation context for config-example: " + name)
+        cases[name] = case
+    return cases

tests/test_config_docs.py

@@ -0,0 +1,107 @@
+"""Keep schema field tables and executable examples aligned with the validators."""
+
+from pathlib import Path
+import tempfile
+import unittest
+from unittest.mock import patch
+import yaml
+
+from app.build import default_comparison
+from quickdash import config
+from tests.config_docs import examples, schemas
+
+
+class ConfigurationDocs(unittest.TestCase):
+    def test_schema_fields_follow_live_validators(self):
+        docs = schemas()
+        # Fictional inputs exercise every nested mapping. Field inventories come
+        # from object_keys calls in the validators, not a second schema in tests.
+        match = dict(regex="example_(en|fr)")
+        language = dict(tasks=["example_en"], scope="single", language="eng_Latn")
+        component = dict(name="only", match=match, relative_weight=1)
+        rule = dict(name="Example", category="C", match=match, metric="acc",
+                    metric_filter="none", score=dict(scale=1),
+                    normalize=dict(min=0, max=1),
+                    aggregation=dict(components=[component]))
+        catalogue = dict(version=1, name="Example catalogue", evals=[rule], languages=[language])
+        variant = dict(task="example_en")
+        entry = dict(name="Example", variants=[variant])
+        suite = dict(version=1, name="Example set", mode="fixed", evals=[entry])
+        profile = dict(version=1, name="Example weights", weights=dict(C=1))
+        defaults = dict(note="Language rationale")
+        definition = dict(rule, languages=[language], language_defaults=defaults)
+        metadata = dict(version=1, name="Example manifest")
+        manifest = dict(metadata, evals_dir="evals")
+        targets = dict(catalogue=catalogue, eval=rule, match=match, score=rule["score"],
+                       normalize=rule["normalize"], aggregation=rule["aggregation"],
+                       component=component, language=language, set=suite,
+                       profile=profile, manifest=manifest)
+        targets.update({"set-eval": entry, "variant": variant, "language-defaults": defaults})
+        observed = {}
+        original = config.object_keys
+
+        def record(value, allowed, required=()):
+            original(value, allowed, required)
+            for name, target in targets.items():
+                if value == target:
+                    signature = (set(allowed), set(required))
+                    if name in observed:
+                        self.assertEqual(observed[name], signature, name)
+                    observed[name] = signature
+
+        with patch.object(config, "object_keys", side_effect=record):
+            config.validate_catalogue(catalogue)
+            config.validate_suite(suite)
+            config.validate_profile(profile)
+            config.assemble_catalogue(metadata, [definition])
+            with tempfile.TemporaryDirectory() as directory:
+                path = Path(directory)
+                (path / "evals").mkdir()
+                (path / "evals/example.yaml").write_text(yaml.safe_dump(definition))
+                (path / "catalogue.yaml").write_text(yaml.safe_dump(manifest))
+                config.load_catalogue(path / "catalogue.yaml")
+
+        self.assertEqual(set(observed), set(targets), "Every documented mapping must exercise its validator")
+        # Per-eval assembly moves these two file-level fields outside the rule.
+        observed["eval"][0].update(definition.keys() - rule.keys())
+        observed["eval"][1].add("languages")
+        self.assertEqual(set(docs), set(observed) | {"startup"})
+        for name, (allowed, required) in observed.items():
+            with self.subTest(schema=name):
+                self.assertEqual(set(docs[name]), allowed, "Document every allowed field, and no obsolete fields")
+                for field in required:
+                    self.assertIn("required", docs[name][field].lower(), field)
+                for field, status in docs[name].items():
+                    if status.lower() == "required":
+                        self.assertIn(field, required, field)
+
+    def test_examples_load_from_disk_with_the_documented_layout(self):
+        docs = examples()
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory)
+            (path / "evals").mkdir()
+            (path / "catalogue.yaml").write_text(yaml.safe_dump(docs["manifest"]))
+            (path / "evals/example.yaml").write_text(yaml.safe_dump(docs["eval"]))
+            (path / "weights.yaml").write_text(yaml.safe_dump(docs["profile"]))
+            (path / "set.yaml").write_text(yaml.safe_dump(docs["set-pinned"]))
+            loaded = config.load_config(catalogue=path / "catalogue.yaml",
+                                        weights=path / "weights.yaml", eval_set=path / "set.yaml")
+            self.assertEqual(loaded["catalogue"]["evals"][0]["name"], docs["eval"]["name"])
+            (path / "default.yaml").write_text(yaml.safe_dump(docs["startup"]))
+            labels = set(docs["startup"].values())
+            self.assertEqual(default_comparison(path, labels), docs["startup"])
+            self.assertEqual(set(schemas()["startup"]), set(docs["startup"]))
+            for field in docs["startup"]:
+                invalid = {k: v for k, v in docs["startup"].items() if k != field}
+                (path / "default.yaml").write_text(yaml.safe_dump(invalid))
+                with self.assertRaises(ValueError):
+                    default_comparison(path, labels)
+
+    def test_markers_allow_prose_and_table_formatting_changes(self):
+        self.assertEqual(schemas("Intro\n<!-- config-schema: demo -->\n\n"
+                                 "| Field | Required? | Details |\n|:---|:---|---|\n"
+                                 "| `name` | Required | Wording can change. |\n"),
+                         {"demo": {"name": "Required"}})
+        self.assertEqual(examples("New heading\n<!-- config-example: demo -->\n\n"
+                                  "```yaml\nname: Example # A comment\n```\n"),
+                         {"demo": {"name": "Example"}})

tests/test_engines.py

@@ -250,17 +250,14 @@ def both(self, cases):
             results.append((py, other))
         return results
 
-    def test_documented_eval_sets_resolve_in_both_engines(self):
-        section = (ROOT / "docs/configuration.md").read_text().split("## Strict and relaxed matching")[0]
-        examples = [parse_yaml(block) for block in re.findall(r"```yaml\n(.*?)```", section, re.S)]
-        self.assertTrue(examples, "The eval-set guide must contain executable examples")
-        base = load_config(catalogue=ROOT / "configs/catalogue.yaml",
-                           weights=ROOT / "configs/weights/oellm.yaml")
-        rows = parse_csv((ROOT / "examples/sample-evals.csv").read_text())
-        cases = [dict(config={**base, "suite": suite}, rows=rows) for suite in examples]
-        for results in self.both(cases):
-            for result in results:
-                self.assertNotIn("error", result)
+    def test_documented_configurations_resolve_in_both_engines(self):
+        from tests.config_docs import engine_cases
+        cases = engine_cases()
+        self.assertTrue(cases, "The configuration guide must contain executable examples")
+        for name, results in zip(cases, self.both(list(cases.values()))):
+            with self.subTest(example=name):
+                for result in results:
+                    self.assertNotIn("error", result)
 
     def test_per_eval_catalogue_assembly(self):
         original = fixture()