diff --git a/SPEC.md b/SPEC.md index 77d59ea..a94d2dd 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1128,6 +1128,36 @@ while half the fleet was still broken. Collector → `e2e/harness/_artifacts.sh` harnesses; prune → `e2e/ci/prune-artifacts.sh`, invoked by both workflows. A reusable workflow can call the script from the consumer's vendored devkit path, so there is still exactly one definition. +**DEC-077 — Plugin documentation is generated from the plugin's own e2e run.** +Every green e2e already captures one screenshot per BDD scenario, and each scenario title is a +plain-English statement of a behaviour the plugin guarantees. Those two artefacts together *are* the +documentation, but they were being thrown away: the screenshots expired as CI artifacts and the +feature files were read only by developers. The result was a fleet where one plugin had **zero** +images in its README and the rest had hand-captured ones that quietly go stale. +`make docs` joins them into `docs/BEHAVIOUR.md` + `docs/screenshots/`. Because the source is a +*passing* run, a screenshot can never depict a state the plugin no longer produces, and a deleted or +renamed scenario drops out of the docs automatically. Captions are the scenario titles verbatim — the +generator invents no prose, and a scenario with no captured screenshot says so rather than being +silently omitted. +Implementation note worth keeping: Playwright **truncates** long scenario titles in its result +directory names and inserts a hash, so matching a screenshot back to its scenario cannot be an exact +comparison. It matches on the longest shared run of *trailing* tokens (the tail survives truncation) +and requires at least two, since a single shared token is meaningless in a feature file where every +scenario mentions the same nouns. The first implementation missed this and produced a document in +which all twelve scenarios reported "no screenshot" — a plausible-looking empty result rather than an +obvious failure, which is why `test_docs_from_e2e.py` pins this behaviour with real artifact names. + +**DEC-078 — Every plugin README carries the same, verifiable badge block.** +The e2e suite is the strongest quality claim these plugins have — nested real A0, real browser, and a +seam-off red-proof that fails the build if the scenarios could pass with the plugin *not* installed. +None of it was visible to a reader: badge coverage across the fleet was 1, 0, 0, 1 (and 5 on the +devkit itself). `make badges` splices a standard block under the H1, idempotently, between +`BADGES:START`/`BADGES:END` markers. +Constraint on what may appear there: every badge must be independently checkable by clicking it. +Workflow status comes from the live Actions badge, the release from the tag list, the licence from +the repo. The one non-GitHub badge ("BDD e2e") is a static label whose link lands on the actual +feature directory. No invented coverage percentages. + **DEC-079 — devkit-sync is on-demand, and never deletes its own branch.** Two corrections to the sync mechanism, one of policy and one of implementation. @@ -1148,7 +1178,7 @@ e2e had passed and the result was discarded and re-queued for nothing. The same alone is sufficient and non-destructive: an open PR follows its head ref and picks up the new commit. > **Numbering note:** DEC-070–072 were introduced as code comments only and were never written up -> here. This section documents 073–076; backfilling 070–072 is tracked separately. +> here. This section documents 073–079; backfilling 070–072 is tracked separately. --- diff --git a/e2e/Makefile.devkit b/e2e/Makefile.devkit index 3629a7f..36d3d91 100644 --- a/e2e/Makefile.devkit +++ b/e2e/Makefile.devkit @@ -30,7 +30,7 @@ PODMAN_E2E_FLAGS := --rm --device /dev/fuse --security-opt label=disable \ # Riding along here means the standard one-line include gives them for free. -include $(DEVKIT_DIR)/e2e/make/bdd.mk -.PHONY: build package up down e2e conformance update-devkit link-workflows link-gemini link-skills link-devkit help-devkit +.PHONY: build package up down e2e conformance update-devkit link-workflows link-gemini link-skills link-devkit help-devkit docs badges # Which devkit ref to bump to. Empty (default) = the latest release tag within your # MAJOR channel (.devkit.yml `devkit_major`, default 1) — a new MAJOR (e.g. v2, which @@ -115,9 +115,24 @@ up: ## boot A0 nested for manual/agent exploration (inside the devcontainer) -v $(CURDIR):/workspace:ro -e A0_IMAGE=$(A0_IMAGE) \ $(DEVKIT_IMAGE) -lc 'bash /workspace/$(DEVKIT_DIR)/e2e/harness/a0-up.sh && echo "A0 on :80 — exec in to explore" && sleep infinity' +docs: ## regenerate docs/BEHAVIOUR.md + docs/screenshots/ from the last e2e run (DEC-077) + @# ARTIFACTS defaults to the local run's output; in CI the workflow points it + @# at the collected artifact dir. Never invents screenshots — a scenario with + @# no captured shot is reported as such in the generated doc. + python3 $(DEVKIT_DIR)/e2e/scripts/docs-from-e2e.py \ + --artifacts $(or $(ARTIFACTS),artifacts) \ + --features tests/e2e/features \ + --plugin $(PLUGIN_NAME) \ + --out docs + conformance: ## assert this repo exposes the frozen target set + allowed assets @for t in build package up e2e; do \ $(MAKE) -n $$t >/dev/null 2>&1 || { echo "missing target: $$t"; exit 1; }; \ done @test -f tests/e2e/hooks/verify-installed || echo "note: no tests/e2e/hooks/verify-installed (optional)" @echo "conformance OK" + +badges: ## refresh the standard badge block in README.md (DEC-078) + python3 $(DEVKIT_DIR)/e2e/scripts/readme-badges.py \ + --repo $(shell git config --get remote.origin.url | sed -E 's#.*github.com[:/]([^/]+/[^/]+?)(\.git)?$$#\1#') \ + --plugin $(PLUGIN_NAME) --apply README.md diff --git a/e2e/scripts/docs-from-e2e.py b/e2e/scripts/docs-from-e2e.py new file mode 100755 index 0000000..7c6d8ae --- /dev/null +++ b/e2e/scripts/docs-from-e2e.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python3 +""" +Build a plugin's visual behaviour documentation from its own e2e run (DEC-077). + +WHY THIS EXISTS + Every green e2e already captures a screenshot per BDD scenario, and each + scenario is a plain-English statement of a behaviour the plugin guarantees. + Those two facts together ARE the documentation — but the screenshots were + only ever uploaded as a CI artifact that expires, and the feature files were + only read by developers. So every repo either had no screenshots at all + (chat-comments) or hand-captured ones that silently go stale. + + This joins them: scenario name -> its screenshot -> docs/BEHAVIOUR.md. The + docs are therefore regenerated from a passing test run, so a screenshot can + never show a state the plugin no longer produces. If a scenario is deleted + or renamed, its screenshot disappears from the docs on the next run. + +WHAT IT DOES NOT DO + It does not invent prose. The feature file's Scenario line is the caption, + verbatim. If that reads badly, fix the feature file — the same text is the + contract the test asserts. + +Usage: + docs-from-e2e.py --artifacts --features --plugin + --out docs [--title "Display Name"] +""" +from __future__ import annotations + +import argparse +import re +import shutil +import sys +from pathlib import Path + +# Playwright's per-scenario dir is flattened into the artifact filename by the +# collector (_artifacts.sh), e.g. +# chat_comments-tests-e2e-features-10-comm-6bdad-an-be-commented-BEH-5-BEH-8--test-finished-1.png +# The middle is a TRUNCATED, slugified scenario title with a hash — Playwright +# shortens long titles, so an exact match back to the feature file is impossible. +# Match on the longest shared token-suffix instead (see _match_scenario). +SHOT_RE = re.compile(r"^(?P.+?)-(?P.+?)--(?P[a-z-]+-\d+)\.png$") + + +def _slug_tokens(text: str) -> list[str]: + """Lowercase alphanumeric tokens — the common ground between a feature-file + scenario title and Playwright's slugified, truncated directory name.""" + return [t for t in re.split(r"[^a-z0-9]+", text.lower()) if t] + + +def parse_features(features_dir: Path) -> list[dict]: + """Extract scenarios in file order. Keeps the tag comment (# BEH-5, BEH-8) + because it is the traceability link back to the spec.""" + scenarios: list[dict] = [] + for feature in sorted(features_dir.glob("*.feature")): + feature_title = "" + for raw in feature.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if line.startswith("Feature:"): + feature_title = line.split(":", 1)[1].strip() + elif line.startswith("Scenario:"): + body = line.split(":", 1)[1] + title, _, comment = body.partition("#") + # Match on title AND the trailing "# BEH-5, BEH-8" comment: + # Playwright's directory name is built from the whole Scenario + # line, so the refs are part of the slug's surviving tail. + scenarios.append( + { + "feature": feature_title, + "title": title.strip(), + "refs": comment.strip(), + "tokens": _slug_tokens(f"{title} {comment}"), + "shot": None, + } + ) + return scenarios + + +def _match_scenario(scenarios: list[dict], slug: str) -> dict | None: + """Playwright truncates the middle of long titles, so the artifact slug is a + prefix+hash+SUFFIX of the real title. The suffix survives intact, so score on + the longest matching run of trailing tokens and require at least two — one + shared token ("chat", "comment") is far too weak across a feature file where + every scenario mentions the same nouns.""" + slug_tokens = _slug_tokens(slug) + best, best_score = None, 0 + for sc in scenarios: + if sc["shot"] or not sc["tokens"]: + continue + score = 0 + for a, b in zip(reversed(sc["tokens"]), reversed(slug_tokens)): + if a != b: + break + score += 1 + if score > best_score: + best, best_score = sc, score + return best if best_score >= 2 else None + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--artifacts", required=True, type=Path) + ap.add_argument("--features", required=True, type=Path) + ap.add_argument("--plugin", required=True) + ap.add_argument("--out", required=True, type=Path) + ap.add_argument("--title", default=None) + args = ap.parse_args() + + title = args.title or args.plugin.replace("_", " ").title() + + if not args.features.is_dir(): + print(f"::warning::no features dir at {args.features} — nothing to document") + return 0 + + scenarios = parse_features(args.features) + if not scenarios: + print(f"::warning::no scenarios found in {args.features}") + return 0 + + shots_dir = args.out / "screenshots" + shots_dir.mkdir(parents=True, exist_ok=True) + + # Only the final screenshot of each scenario: it shows the asserted end state, + # which is exactly what the caption claims. Intermediate frames would show + # states the scenario does not guarantee. + matched = 0 + for png in sorted(args.artifacts.rglob("*.png")): + m = SHOT_RE.match(png.name) + if not m or not m.group("kind").startswith("test-finished"): + continue + sc = _match_scenario(scenarios, m.group("slug")) + if sc is None: + continue + dest_name = re.sub(r"[^a-z0-9]+", "-", sc["title"].lower()).strip("-") + ".png" + shutil.copyfile(png, shots_dir / dest_name) + sc["shot"] = f"screenshots/{dest_name}" + matched += 1 + + lines = [ + f"# {title} — behaviour", + "", + "", + "", + f"Each section is one scenario from `tests/e2e/features/`, with the screenshot the", + f"test captured at its final assertion. {matched} of {len(scenarios)} scenarios have one.", + "", + ] + + current_feature = None + for sc in scenarios: + if sc["feature"] != current_feature: + current_feature = sc["feature"] + lines += [f"## {current_feature}", ""] + lines.append(f"### {sc['title']}") + if sc["refs"]: + lines.append(f"{sc['refs']}") + lines.append("") + if sc["shot"]: + lines.append(f"![{sc['title']}]({sc['shot']})") + else: + # Honest gap rather than a silently missing section. + lines.append("_No screenshot captured for this scenario in the last run._") + lines.append("") + + out_file = args.out / "BEHAVIOUR.md" + out_file.write_text("\n".join(lines), encoding="utf-8") + print(f"[docs-from-e2e] {out_file}: {matched}/{len(scenarios)} scenarios illustrated") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/e2e/scripts/readme-badges.py b/e2e/scripts/readme-badges.py new file mode 100755 index 0000000..dd9e6f1 --- /dev/null +++ b/e2e/scripts/readme-badges.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +""" +Emit the standard plugin badge block, and splice it into README.md (DEC-078). + +WHY + The e2e suite is the strongest quality claim these plugins have — a nested + real A0, a real browser, a seam-off red-proof that fails the build if the + scenarios could pass without the plugin installed. None of that was visible + to anyone reading the README: badge coverage across the fleet was 1 badge, + 0 badges, 0 badges, 1 badge and 5 (the devkit itself). + +WHAT IS AND ISN'T CLAIMED + Every badge here is backed by something a reader can click and verify: + workflow status comes from the live Actions badge, the licence from the repo, + the release from the tag list. The one badge that is not GitHub-generated + ("BDD e2e") is a static label pointing at the feature directory, so a reader + lands on the actual scenarios. No invented percentages, no "100% coverage". + +Usage: + readme-badges.py --repo owner/name --plugin chat_comments [--apply README.md] +""" +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +START = "" +END = "" + + +def build(repo: str, workflow: str, features_path: str) -> str: + base = f"https://github.com/{repo}" + rows = [ + f"[![plugin-e2e]({base}/actions/workflows/{workflow}/badge.svg?branch=main)]" + f"({base}/actions/workflows/{workflow})", + f"[![BDD scenarios](https://img.shields.io/badge/tests-BDD%20e2e-brightgreen)]" + f"({base}/tree/main/{features_path})", + f"[![Release](https://img.shields.io/github/v/tag/{repo}?label=release&sort=semver)]" + f"({base}/tags)", + f"[![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)]" + f"({base}/blob/main/LICENSE)", + f"[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)]" + f"({base}/blob/main/CONTRIBUTING.md)", + ] + return "\n".join([START, "", " ".join(rows), "", END]) + + +def apply(readme: Path, block: str) -> str: + text = readme.read_text(encoding="utf-8") + if START in text and END in text: + return re.sub( + re.escape(START) + r".*?" + re.escape(END), block, text, flags=re.S + ) + # First insert: place directly under the H1 so it is the first thing seen. + lines = text.splitlines() + for i, line in enumerate(lines): + if line.startswith("# "): + return "\n".join(lines[: i + 1] + ["", block] + lines[i + 1 :]) + return block + "\n\n" + text + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--repo", required=True, help="owner/name") + ap.add_argument("--plugin", required=True) + ap.add_argument("--workflow", default="plugin-e2e.yml") + ap.add_argument("--features", default="tests/e2e/features") + ap.add_argument("--apply", type=Path, help="README.md to splice into") + args = ap.parse_args() + + block = build(args.repo, args.workflow, args.features) + if args.apply: + args.apply.write_text(apply(args.apply, block), encoding="utf-8") + print(f"[readme-badges] spliced badge block into {args.apply}") + else: + print(block) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/e2e/scripts/test_docs_from_e2e.py b/e2e/scripts/test_docs_from_e2e.py new file mode 100755 index 0000000..d2f44ba --- /dev/null +++ b/e2e/scripts/test_docs_from_e2e.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +"""Self-check for docs-from-e2e.py. Run directly: `python3 test_docs_from_e2e.py`. + +The one thing worth testing here is the scenario<->screenshot match, because +Playwright TRUNCATES long scenario titles in its directory names and inserts a +hash. A naive exact match silently produces a document where every scenario says +"no screenshot captured" — which is exactly what the first version of this script +did, and it looked like a plausible empty result rather than a bug. +""" +import subprocess +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).parent +SCRIPT = HERE / "docs-from-e2e.py" + +FEATURE = """\ +Feature: Commenting on a chat + + Scenario: The comments control is present # BEH-1, UI-1 + Given I am in a chat + + Scenario: A comment can be attached to selected message text # BEH-5, BEH-7, BEH-8 + Given I am in a chat + + Scenario: A comment can be deleted # BEH-10 + Given I am in a chat + + Scenario: This one never ran # BEH-99 + Given I am in a chat +""" + +# Real artifact names from run 30866557253 — note the truncated middles and the +# 5-hex-char hash Playwright inserts. +SHOTS = [ + "chat_comments-tests-e2e-features-10-comm-0c7ed-ntrol-is-present-BEH-1-UI-1--test-finished-1.png", + "chat_comments-tests-e2e-features-10-comm-a57f8-sage-text-BEH-5-BEH-7-BEH-8--test-finished-1.png", + "chat_comments-tests-e2e-features-10-comm-f5b9e-mment-can-be-deleted-BEH-10--test-finished-1.png", + # A video must never be picked up as a screenshot. + "chat_comments-tests-e2e-features-10-comm-0c7ed-ntrol-is-present-BEH-1-UI-1--video.webm", +] + + +def main() -> int: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / "features").mkdir() + (root / "features" / "10-comments.feature").write_text(FEATURE) + (root / "artifacts").mkdir() + for name in SHOTS: + (root / "artifacts" / name).write_bytes(b"\x89PNG fake") + + out = subprocess.run( + [sys.executable, str(SCRIPT), "--artifacts", str(root / "artifacts"), + "--features", str(root / "features"), "--plugin", "chat_comments", + "--out", str(root / "docs")], + capture_output=True, text=True, check=True, + ) + print(out.stdout.strip()) + + doc = (root / "docs" / "BEHAVIOUR.md").read_text() + shots = sorted(p.name for p in (root / "docs" / "screenshots").glob("*")) + + assert "3 of 4 scenarios have one" in doc, doc[:400] + assert shots == [ + "a-comment-can-be-attached-to-selected-message-text.png", + "a-comment-can-be-deleted.png", + "the-comments-control-is-present.png", + ], shots + # The scenario with no run must say so rather than be omitted. + assert "### This one never ran" in doc + assert "_No screenshot captured" in doc + # Traceability refs survive into the doc. + assert "BEH-5, BEH-7, BEH-8" in doc + # A one-token overlap must NOT match (every scenario says "comment"). + assert "a-comment-can-be-attached" in " ".join(shots) + + print("PASS: truncated Playwright slugs matched back to their scenarios") + return 0 + + +if __name__ == "__main__": + sys.exit(main())