Skip to content

feat(docs): generate plugin docs + badges from the e2e run - #53

Merged
omar-nahhas merged 1 commit into
mainfrom
feat/docs-from-e2e
Aug 4, 2026
Merged

omar-nahhas merged 1 commit into
mainfrom
feat/docs-from-e2e

Conversation

@omar-nahhas

Copy link
Copy Markdown
Contributor

Two new devkit targets, so this is built once in the devkit rather than hand-made four times.

make docs — DEC-077

Every green e2e already captures one screenshot per BDD scenario, and each scenario title is a plain-English statement of a behaviour the plugin guarantees. Both were being thrown away: screenshots expired as CI artifacts, feature files were read only by developers.

Fleet state today: chat-comments has zero images in its README; the other three have hand-captured ones that go stale with no signal.

make docs joins them into docs/BEHAVIOUR.md + docs/screenshots/. Because the source is a passing run, a screenshot cannot depict a state the plugin no longer produces, and a renamed or deleted scenario drops out of the docs automatically. Captions are scenario titles verbatim — the generator invents no prose, and a scenario with no captured screenshot says so rather than being silently omitted.

The subtle part. Playwright truncates long scenario titles in its result directory names and inserts a hash:

chat_comments-tests-e2e-features-10-comm-6bdad-an-be-commented-BEH-5-BEH-8--test-finished-1.png

so matching back to the feature file cannot be an exact comparison. It matches on the longest shared run of trailing tokens (the tail survives truncation), requiring at least two — one shared token is meaningless in a file where every scenario says "comment". My first version missed this and produced a document where all twelve scenarios reported "no screenshot" — a plausible-looking empty result rather than an obvious failure. test_docs_from_e2e.py pins this with real artifact names.

Verified against the real chat-comments artifact and feature file: 11/12 scenarios illustrated (the 12th is the devkit's own lifecycle scenario, correctly absent from the plugin's feature file).

make badges — DEC-078

The e2e suite is the strongest quality claim these plugins have — nested real A0, real browser, and a seam-off red-proof that fails the build if the scenarios could pass with the plugin not installed. None of it was visible to a reader:

repo badges today
chat-comments 0 (the one match was a Mermaid node, not a badge)
diff-visualizer 0
mermaid-diagrams 0
share-chat 1
development-testkit 5

make badges splices a standard block under the H1, idempotently, between BADGES:START/BADGES:END markers (verified: applying twice yields one block).

Constraint on the content: every badge must be independently checkable by clicking it. Workflow status, release tag and licence are all GitHub-generated. The single static label ("BDD e2e") links to the actual feature directory. No invented coverage percentages.

Follow-up, not in this PR

  • Applying both targets across the four plugin repos.
  • Marketplace cards: none of the four plugins is currently listed in the index at all — omar-nahhas/a0-plugins/plugins/ has no entry for them. The schema supports up to 5 screenshots URLs plus a ≤20 KB square thumbnail, so the generated screenshots feed it directly once the entries exist.

DEC-077 — docs from e2e. Every green run already captures a screenshot per BDD
scenario, and each scenario title is a plain-English behaviour guarantee. Both
were being discarded: screenshots expired as CI artifacts, feature files were
read only by developers. Fleet state today: chat-comments has ZERO images in its
README; the others have hand-captured ones that go stale silently.
`make docs` joins them into docs/BEHAVIOUR.md + docs/screenshots/. Sourced from a
PASSING run, so a screenshot cannot show a state the plugin no longer produces,
and a renamed/deleted scenario drops out automatically. Captions are scenario
titles verbatim — no invented prose; a scenario with no shot says so.

The matching is the subtle part: Playwright TRUNCATES long titles in its result
dir names and inserts a hash, so an exact match is impossible. Matching is on the
longest shared run of trailing tokens, minimum two — one shared token is
meaningless where every scenario says "comment". My first version missed this and
produced a doc where all 12 scenarios reported "no screenshot": a plausible empty
result rather than an obvious failure. test_docs_from_e2e.py pins it with real
artifact names from run 30866557253.

DEC-078 — badges. The e2e suite (nested real A0, real browser, seam-off red-proof
that fails the build if scenarios could pass with the plugin absent) was invisible
to readers: badge coverage was 1, 0, 0, 1 across the four plugins. `make badges`
splices a standard block under the H1, idempotently, between BADGES markers.
Every badge is click-verifiable — workflow status, release tag, licence all
GitHub-generated; the one static label links to the actual feature directory.
No invented coverage numbers.

Verified: generator run against the real chat-comments artifact and feature file
illustrates 11/12 scenarios (the 12th is the devkit lifecycle scenario, correctly
not in the plugin feature file); badge splice is idempotent across two runs.
@omar-nahhas
omar-nahhas merged commit 3aaf74c into main Aug 4, 2026
1 check passed
@omar-nahhas
omar-nahhas deleted the feat/docs-from-e2e branch August 4, 2026 22:51
@omar-nahhas omar-nahhas mentioned this pull request Aug 4, 2026
omar-nahhas added a commit that referenced this pull request Aug 4, 2026
v2.1.5 was tagged BEFORE the docs/badges tooling merged (#53), so consumers
pinning v2.1.5 lack docs-from-e2e.py / readme-badges.py and `make docs` cannot
run for them. This tag delivers DEC-077/078 plus the DEC-079 devkit-sync fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant