Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/superpowers/plans/2026-11-14-frame-cadence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Browser Frame Cadence Implementation Plan

> Execute with an implementation agent and independent specification/quality review; user authorized the separate benchmark correction.

**Goal:** Correct the frame interval gate while preserving processing and correctness budgets.
**Architecture:** Independent bounded idle calibration, persisted evidence, dynamic frame-only threshold.
**Tech Stack:** Node.js, Playwright, node:test.

- [x] Verify clean main baseline: existing six gate tests pass.
- [x] Add failing tests in `scripts/__tests__/bench-row-model-gate.test.mjs` for calibration, provenance, frame rejection and unchanged processing gates.
- [x] Add `scripts/frame-cadence.mjs` for pure calibration validation and browser collection. Integrate with `scripts/bench-row-model-gate.mjs`; persist evidence before workloads and in report. Share explicit gate context with `playwright.config.ts`, use a bounded calibration timeout with browser cleanup and run gate browser checks with one worker. Reject nonfinite metric values.
- [x] Run focused tests then all root infrastructure tests with local-server access; formatting and affected lint.
- [x] Run real gate, inspect evidence and review changes independently; prepare separate PR with validation and limitations.

Validation so far: original 6 gate tests passed. New regression tests failed on the old hard ceiling, missing/invalid calibration, and NaN acceptance before implementation. Final 12 focused tests and 487 root infrastructure tests pass; scoped ESLint, Prettier and diff checks pass. Independent plan, specification-compliance and final quality reviews passed. Real four-scenario gate completed: 112 browser assertions passed, 4 expected skips. Calibration measured 16.7 ms median and derived a 16.95 ms frame budget; both grouped frame p95 values (16.7/16.8 ms) meet it. Overall performance gate correctly failed the unchanged 8 ms limits: local-max grouped commit p95 9.8 ms and rebuild max 10.6 ms while host load exceeded 150. This verifies calibration end to end, not overall performance acceptance. Logs: `/private/tmp/pretable-frame-cadence-gate.log`; raw calibration and scenario artifacts retained in ignored `status/`.

Correction is isolated from totals PR #622. A temporary integration branch combines the correction with totals for validation; neither PR branch is rewritten or merged. Independent final quality review approved with no findings.
19 changes: 19 additions & 0 deletions docs/superpowers/specs/2026-11-14-frame-cadence-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Browser cadence for the row-model performance gate

The approved separate fix replaces the impossible 16 ms animation-frame interval ceiling on a 60 Hz browser, retaining all 8 ms processing and correctness limits. The measured defect: an empty Chromium page has 16.7 ms p95, as do passing streaming runs. Main exhibits the same failure.

## Approach

Before measuring the four runs, launch the same installed default headless Chromium used by the gate. Measure 120 animation-frame intervals on a blank page after 30 warm-up frames. Record raw intervals, browser version, timestamp and viewport in a calibration artifact. Use the median interval plus 0.25 ms for timestamp quantization as the grouped frame-p95 budget. Never derive the budget from the application workload.

Reject missing, stale, mismatched, nonfinite or nonpositive calibration evidence. Require at least 120 samples, median between 4 ms and 1000/60 + 0.25 ms, and p95 minus p05 at most 0.5 ms. Thus a uniformly slow 30 Hz or noisy host cannot bless itself. Faster browser cadences receive proportionally tighter budgets; a doubled p95 interval fails (isolated dropped frames may fall outside p95). This is an explicit environment eligibility rule, not a measurement of display hardware refresh rate.

Run benchmarks serially as today. Retain raw application metrics unchanged. Validate calibration freshness and browser version, viewport, device scale and default profile against all four artifacts; retain the derived budget and calibration in the successful gate report, and save calibration before workloads so failed runs remain diagnosable. Keep commit/rebuild ceilings of 8 ms, patch checksums, responsive rebuild, group/row counts, zero long tasks and drift unchanged. Reject nonfinite measured metrics rather than accidentally accepting NaN.

A fixed 17 ms ceiling is simpler but ignores faster cadences. Using the workload's own median could normalize a regression away. Independent, bounded idle calibration is preferred.

## Validation

Tests cover 60/120 Hz, quantization, dropped frames, slow/noisy/invalid/short calibration, mismatched browser and stale evidence, preservation of processing/correctness checks and nonfinite metrics. Run the existing gate suite first, observe new tests fail, then implement. Run root infrastructure tests, formatting/lint for affected files, and the real four-scenario gate. Separate PR from totals; do not merge either PR automatically. If the host violates eligibility or processing limits, preserve evidence and report failure without relaxing limits.

Calibration has a bounded timeout and closes its owned browser in finally. The gate explicitly uses the shared calibration context (1440×900, device scale 1) for workload browsers as well as calibration, rather than relying on Playwright viewport defaults. Gate browser assertions run with one worker to avoid parallel fixture work contaminating timings. These settings apply only to the gate, not ordinary browser test runs.
7 changes: 6 additions & 1 deletion playwright.config.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import { defineConfig, devices } from "@playwright/test";

import { ROW_MODEL_GATE_CONTEXT } from "./scripts/frame-cadence.mjs";

const baseURL = process.env.PRETABLE_BENCH_BASE_URL ?? "http://127.0.0.1:4173";
const useExternalServer = process.env.PRETABLE_BENCH_EXTERNAL_SERVER === "1";

Expand Down Expand Up @@ -36,7 +38,10 @@ export default defineConfig({
projects: [
{
name: "chromium",
use: { ...devices["Desktop Chrome"] },
use:
process.env.PRETABLE_BENCH_ROW_MODEL_GATE === "1"
? ROW_MODEL_GATE_CONTEXT
: { ...devices["Desktop Chrome"] },
},
],
});
155 changes: 150 additions & 5 deletions scripts/__tests__/bench-row-model-gate.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ test("accepts a complete comparable four-run milestone without making a performa
const report = validateRowModelGateSummaries(passingSummaries(), {
seed: 91_337,
startedAt,
calibration: calibrationFixture(),
});

assert.equal(report.status, "ready-for-performance-gate");
Expand All @@ -128,6 +129,7 @@ test("rejects missing, mismatched, incomplete, and stale summaries", () => {
validateRowModelGateSummaries(passingSummaries().slice(0, 3), {
seed: 91_337,
startedAt,
calibration: calibrationFixture(),
}),
/missing/i,
);
Expand All @@ -137,7 +139,7 @@ test("rejects missing, mismatched, incomplete, and stale summaries", () => {
passingSummaries().map((summary, index) =>
index === 3 ? { ...summary, browserVersion: "different" } : summary,
),
{ seed: 91_337, startedAt },
{ seed: 91_337, startedAt, calibration: calibrationFixture() },
),
/comparable/i,
);
Expand All @@ -147,7 +149,7 @@ test("rejects missing, mismatched, incomplete, and stale summaries", () => {
passingSummaries().map((summary, index) =>
index === 0 ? { ...summary, status: "partial" } : summary,
),
{ seed: 91_337, startedAt },
{ seed: 91_337, startedAt, calibration: calibrationFixture() },
),
/completed/i,
);
Expand All @@ -159,7 +161,7 @@ test("rejects missing, mismatched, incomplete, and stale summaries", () => {
? { ...summary, timestamp: "2026-08-11T17:59:59.000Z" }
: summary,
),
{ seed: 91_337, startedAt },
{ seed: 91_337, startedAt, calibration: calibrationFixture() },
),
/stale/i,
);
Expand All @@ -173,7 +175,7 @@ test("rejects a uniformly wrong permanent-job identity", () => {
...summary,
adapterId: "not-pretable",
})),
{ seed: 91_337, startedAt },
{ seed: 91_337, startedAt, calibration: calibrationFixture() },
),
/adapter/i,
);
Expand All @@ -182,7 +184,7 @@ test("rejects a uniformly wrong permanent-job identity", () => {
test("enforces commit and grouped frame/rebuild gates", () => {
const mutations = [
["row_model_commit_p95_ms", 8.01],
["scroll_frame_p95_ms", 16.01],
["scroll_frame_p95_ms", 17.01],
["long_tasks_count", 1],
["scroll_position_drift_px", 1],
["visible_row_count_drift", 1],
Expand All @@ -200,6 +202,7 @@ test("enforces commit and grouped frame/rebuild gates", () => {
validateRowModelGateSummaries(summaries, {
seed: 91_337,
startedAt,
calibration: calibrationFixture(),
}),
new RegExp(String(metric)),
);
Expand Down Expand Up @@ -227,7 +230,149 @@ test("requires a completed responsive rebuild and a checksum containing every ca
validateRowModelGateSummaries(summaries, {
seed: 91_337,
startedAt,
calibration: calibrationFixture(),
}),
/responsive|checksum|patch/i,
);
});

function calibrationFixture(interval = 1000 / 60, overrides = {}) {
return {
timestamp: "2026-08-11T18:00:10.000Z",
browserName: "chromium",
browserVersion: "140.0.0",
profile: "default",
viewport: { width: 1440, height: 900 },
deviceScaleFactor: 1,
intervalsMs: Array.from(
{ length: 120 },
(_, i) => interval + (i % 2 ? 0.05 : -0.05),
),
...overrides,
};
}

function validateWithCadence(calibration, interval = 16.7) {
const summaries = passingSummaries();
for (const summary of summaries)
summary.metrics.scroll_frame_p95_ms = interval;
return validateRowModelGateSummaries(summaries, {
seed: 91_337,
startedAt,
calibration,
});
}

test("uses independent 60/120 Hz calibration plus quantization allowance", () => {
for (const interval of [1000 / 60, 1000 / 120]) {
const calibration = calibrationFixture(interval);
const report = validateWithCadence(calibration, interval + 0.2);
assert.equal(report.frameCadence.budgetMs, interval + 0.25);
assert.deepEqual(report.frameCadence.calibration, calibration);
assert.throws(
() => validateWithCadence(calibration, interval * 2),
/scroll_frame_p95_ms/,
);
}
});

test("requires eligible independent calibration evidence", () => {
for (const calibration of [
undefined,
calibrationFixture(1000 / 30),
calibrationFixture(3),
calibrationFixture(16.7, { intervalsMs: Array(120) }),
calibrationFixture(16.7, { intervalsMs: Array(119).fill(16.7) }),
...[NaN, Infinity, 0, -1].map((value) =>
calibrationFixture(16.7, {
intervalsMs: [value, ...Array(119).fill(16.7)],
}),
),
calibrationFixture(16.7, {
intervalsMs: Array.from({ length: 120 }, (_, i) => (i % 2 ? 16 : 17)),
}),
])
assert.throws(() => validateWithCadence(calibration, 12), /calibration/i);
});

test("rejects stale, future, and mismatched calibration provenance", () => {
for (const overrides of [
{ timestamp: "2026-08-11T17:59:59.000Z" },
{ timestamp: "2026-08-11T18:02:00.000Z" },
{ timestamp: "invalid" },
{ browserVersion: "different" },
{ browserVersion: "" },
{ browserName: "firefox" },
{ profile: "cpu-throttled" },
{ viewport: { width: 1280, height: 720 } },
{ deviceScaleFactor: 2 },
])
assert.throws(
() => validateWithCadence(calibrationFixture(16.7, overrides), 12),
/calibration/i,
);
});

test("rejects all nonfinite processing and frame metrics", () => {
for (const metric of [
"row_model_commit_p95_ms",
"rebuild_slice_max_ms",
"scroll_frame_p95_ms",
]) {
for (const value of [NaN, Infinity, -Infinity]) {
const summaries = passingSummaries();
summaries[1].metrics[metric] = value;
assert.throws(
() =>
validateRowModelGateSummaries(summaries, {
seed: 91_337,
startedAt,
calibration: calibrationFixture(),
}),
new RegExp(metric),
);
}
}
});

test("isolated dropped calibration frames do not change the median budget", () => {
const calibration = calibrationFixture();
calibration.intervalsMs[0] = 1000 / 30;
assert.ok(validateWithCadence(calibration).frameCadence.budgetMs < 17);
});

test("preserves independent responsive rebuild and deterministic checksum checks", () => {
const mutations = [
(rowModel) => {
rowModel.finalChecksum = "incorrect";
},
(rowModel) => {
rowModel.acceptedPatchCount = 2999;
},
(rowModel) => {
rowModel.rebuild.responsive = false;
},
(rowModel) => {
rowModel.rebuild.streamCommitsObserved = 0;
},
(rowModel) => {
rowModel.rebuild.sourceRowCountAfter = 1;
},
(rowModel) => {
rowModel.rebuild.groupCountAfter = 1;
},
];
for (const mutate of mutations) {
const summaries = passingSummaries();
mutate(summaries[1].rowModel);
assert.throws(
() =>
validateRowModelGateSummaries(summaries, {
seed: 91_337,
startedAt,
calibration: calibrationFixture(),
}),
/checksum|patch|responsive|row count|group count/i,
);
}
});
27 changes: 22 additions & 5 deletions scripts/bench-row-model-gate.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ import {
waitForOwnedServer,
} from "./owned-preview.mjs";

import { collectFrameCadence, validateFrameCadence } from "./frame-cadence.mjs";

const DEFAULT_SEED = 505;
const EXPECTED_PATCHES = 3_000;

Expand All @@ -32,7 +34,7 @@ export function createRowModelGateEntries({ seed = DEFAULT_SEED } = {}) {

export function validateRowModelGateSummaries(
summaries,
{ seed = DEFAULT_SEED, startedAt },
{ seed = DEFAULT_SEED, startedAt, calibration },
) {
if (!Array.isArray(summaries)) throw new TypeError("Summaries are required.");
const expectedEntries = createRowModelGateEntries({ seed });
Expand Down Expand Up @@ -149,10 +151,15 @@ export function validateRowModelGateSummaries(
}
}

const frameCadence = validateFrameCadence(calibration, {
startedAt,
summaries: selected,
});

for (const summary of selected.filter(
(candidate) => candidate.scriptName === "updates-grouped",
)) {
assertAtMost(summary, "scroll_frame_p95_ms", 16);
assertAtMost(summary, "scroll_frame_p95_ms", frameCadence.budgetMs);
assertEqual(summary, "long_tasks_count", 0);
assertEqual(summary, "scroll_position_drift_px", 0);
assertEqual(summary, "visible_row_count_drift", 0);
Expand Down Expand Up @@ -184,6 +191,7 @@ export function validateRowModelGateSummaries(
return Object.freeze({
status: "ready-for-performance-gate",
seed,
frameCadence,
summary:
"Deterministic harness artifacts are complete; Task 22 evaluates performance claims.",
runs: Object.freeze(
Expand All @@ -201,7 +209,7 @@ export function validateRowModelGateSummaries(

function assertAtMost(summary, metricId, threshold) {
const value = summary.metrics?.[metricId];
if (typeof value !== "number" || value > threshold) {
if (!Number.isFinite(value) || value > threshold) {
throw new Error(
`${metricId} must be <= ${threshold}; received ${String(value)}.`,
);
Expand Down Expand Up @@ -238,14 +246,22 @@ async function run() {
const summaries = [];
try {
await waitForOwnedServer(baseUrl, server);
const calibration = await collectFrameCadence();
const reportDir = path.join(workspaceDir, "status", "runsets");
await mkdir(reportDir, { recursive: true });
await writeFile(
path.join(reportDir, "row-model-frame-cadence.json"),
`${JSON.stringify(calibration, null, 2)}\n`,
);
validateFrameCadence(calibration, { startedAt, summaries: [] });
for (const entry of createRowModelGateEntries({ seed })) {
summaries.push(await runEntry(entry, workspaceDir, baseUrl));
}
const report = validateRowModelGateSummaries(summaries, {
seed,
startedAt,
calibration,
});
const reportDir = path.join(workspaceDir, "status", "runsets");
const reportPath = path.join(reportDir, "row-model-gate.json");
await mkdir(reportDir, { recursive: true });
await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`);
Expand All @@ -262,10 +278,11 @@ async function runEntry(entry, workspaceDir, baseUrl) {
const before = new Set(await collectSummaries(statusDir));
await runCommand({
command: "pnpm",
args: ["bench:e2e"],
args: ["bench:e2e", "--workers=1"],
cwd: workspaceDir,
env: {
...process.env,
PRETABLE_BENCH_ROW_MODEL_GATE: "1",
PRETABLE_BENCH_BASE_URL: baseUrl,
PRETABLE_BENCH_EXTERNAL_SERVER: "1",
PRETABLE_BENCH_ADAPTER: entry.adapterId,
Expand Down
Loading
Loading