Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ real third-party repositories.

## Current state

Language expansion: initial Vue SFC and Go package-level analysis is implemented through repository
Language expansion: initial Vue SFC, Go package-level, and conventional Maven reactor analysis is implemented through repository
adapters. See [the support matrix and setup requirements](docs/language-support.md) for exact scope,
fallback behavior, and validation boundaries.

Expand Down
20 changes: 17 additions & 3 deletions docs/language-support.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,37 @@
# Language and framework support

DiffCI now has an extensible repository-adapter boundary and initial Vue and Go support.
DiffCI now has an extensible repository-adapter boundary and initial Vue, Go, and Maven support.
These additions propose selections in the existing observer; they do not enable production CI skipping.

| Surface | Implemented scope | Boundaries |
| --- | --- | --- |
| JavaScript / TypeScript | Existing dependency analysis and ten existing test-runner command mappings | Runner recognition is not a guarantee of complete framework semantics |
| Vue | SFC parsing with `@vue/compiler-sfc`; script imports, literal Options API component registrations, empty script fixtures, compiled template asset imports; propagation into importing JS/TS tests; protected runtime-dependent tests in verified isolated scopes | Runtime uncertainty requires full validation unless protected by the isolated-suite policy below; preprocessors, external/custom SFC blocks, style URLs/imports, Nuxt conventions and glob imports require full validation |
| Go | One root module; optional explicit root-module scope in a repository containing nested modules; native metadata, package-level transitive selection, embeds and Go test commands | Workspaces, changes inside excluded modules, inactive Go files, cgo/native objects, plugins, generation/linkname and incomplete metadata require full validation; root scoping rejects local replacements |
| Maven / Java / Kotlin | Conventional multi-module reactors with `src/main/{java,kotlin}` and `src/test/{java,kotlin}`; module dependency propagation; Surefire-style test names; proposed reactor commands | Nonstandard layouts, generated sources, integration-test conventions, and POM changes may require full validation. Maven commands are observations and must match the repository's CI lifecycle and profiles. |
| Mixed Go and JS/TS | Detected | Full validation until cross-language relationships are declared and modeled |
| Python, Svelte, Astro, Java/Kotlin, C#, Rust | No new semantic support in this release | Require additional adapters and qualification |
| Python, Svelte, Astro, C#, Rust | No new semantic support in this release | Require additional adapters and qualification |

## Using the observer

Use the existing `observe` command and commit range options. The observer now admits a root
`go.mod` or Vue SFCs as well as a TypeScript project. Vue projects without a tsconfig have their
`go.mod`, Vue SFCs, or a conventional Maven reactor as well as a TypeScript project. Vue projects without a tsconfig have their
JS/TS files parsed alongside their components. Existing TS-only corpus eligibility remains separate:
this release does not silently enroll Go repositories in historical JS/TS research cohorts.

For Maven, DiffCI proposes module-level commands but does not run them or change CI. The default
goal is `test`. If the repository's CI runs `verify` with profiles, declare that command shape in
`diffci.json` (or the `diffci` key in `package.json`):

```json
{ "maven": { "goal": "verify", "profiles": ["run-its"] } }
```

This produces a command such as `mvn -pl tools -am verify -P run-its`. The `-am` option can also run
tests in upstream modules. Check the proposed command against the actual CI workflow before using
it for execution. A single Cloudflare Resolver run showed a candidate saving of 8.6% with a
CI-aligned `verify -P run-its` command; it does not establish repeatable savings across repositories.

For Go, install a compatible Go toolchain on the **same host and build environment** used for analysis
and testing, and prepare the repository's dependencies first using its normal setup procedure.
DiffCI invokes `go list -mod=readonly -deps -test -json ./...` with `GOTOOLCHAIN=local`, `GOPROXY=off`,
Expand Down
26 changes: 26 additions & 0 deletions docs/research/2026-09-22-maven-ci-aligned-proof.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# CI-aligned Maven selection proof on Cloudflare

Date: 2026-09-22. This follows the [PR #3 probe](./2026-09-22-maven-pr3-cloudflare-probe.md) and [runtime diagnosis](./2026-09-22-maven-pr3-cloudflare-runtime.md). Both commands ran in the same Cloudflare `standard-4` Sandbox Container with OpenJDK 21 and Maven 3.9.11. The checkout was reset and cleaned between runs; the Maven dependency cache remained shared. Selected ran first, so it had the colder cache. These are single observations, not repeated performance estimates.

## Apache Maven Resolver: valid candidate saving

The analyzed source-only commit was [`a90b7f7`](https://github.com/apache/maven-resolver/commit/a90b7f7f8a05b961c2880ac48ca880b3640b3545), which changed one production source file and its test in `maven-resolver-tools`. Core selected that module without a graph fallback. Resolver's [CI workflow](https://github.com/apache/maven-resolver/blob/a90b7f7f8a05b961c2880ac48ca880b3640b3545/.github/workflows/maven-verify.yml) uses the Apache shared workflow, whose [default verification goal](https://github.com/apache/maven-gh-actions-shared/blob/v5/.github/workflows/maven-verify.yml) is `-P run-its verify`.

| Command | Cloudflare wall time | Result |
| --- | ---: | --- |
| `mvn -pl maven-resolver-tools -am verify -P run-its` | 339.224 s | Pass |
| `mvn verify -P run-its` | 371.154 s | Pass |

**Observed candidate saving: 31.930 s, or 8.6% of the full command's wall time.** Maven's selected reactor contained 26 modules because `-am` brought in upstream dependencies. The earlier 1-of-206 test-file result was not a runtime-savings estimate.

This is **not a saving delivered by PR #3 as merged**. Its planner currently emits `mvn -pl maven-resolver-tools -am test`; that command failed on this repository because a reactor artifact needed packaging. The passing selected command above manually substituted the repository's CI verification goal. The benchmark used Maven 3.9.11 and the Java 21 CI matrix member, while the shared workflow may choose a different Maven version. The single pair is enough to demonstrate a viable candidate, not enough to estimate stable savings across commits or CI runners.

## Jicofo: faster manual command was unsafe

For Jicofo commit [`076a4c7`](https://github.com/jitsi/jicofo/commit/076a4c79cc6907f7db740e4da96ed8d58a8fee68), manual `mvn -pl jicofo-selector -am verify -Pcoverage` passed in 232.654 s and full `mvn verify -Pcoverage` passed in 256.548 s. **Do not count the 23.894 s difference as DiffCI savings.** A subsequent core graph analysis selected tests in both `jicofo-selector` and downstream `jicofo`. The manual command omitted the latter. Core also required fallback because of an unrelated Python script and an unclassified changed `reference.conf` resource. A safe command would have to include the downstream module, erasing this apparent module reduction.

## Consequence for core

The Maven planner needs to derive or accept the repository's validated CI goal and options before it can emit a selective command. It should refuse selection when it cannot prove execution parity. Module-level coverage must include downstream tests; benchmark denominators must reflect the modules and tests actually run by Maven with `-am`.

The dedicated Cloudflare Worker and container application were deleted after the run. Raw Resolver status responses are in `.scratch/maven-proof/` in this checkout.
25 changes: 25 additions & 0 deletions docs/research/2026-09-22-maven-pr3-cloudflare-probe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# PR #3 Maven support: five repository Cloudflare probe

Date: 2026-09-22. All clones, installation, graph analysis, and impact planning ran inside a Cloudflare Sandbox Container (`docker.io/cloudflare/sandbox:0.12.5`, `standard-2`) through a temporary Worker. The desktop only deployed the Worker and retrieved its JSON responses. The container cloned `DiffCI/core` after PR #3 merged and ran `npm ci` before each probe.

Each row analyzes the latest commit against its first parent at the time of the probe. “Tests” means discovered test **files**, not test methods. A fallback means the selective command is not usable; the effective plan is full validation. No Maven test command was executed, so compute, wall time, and money savings are **unmeasured**.

| Repository | Head | Discovered tests | Actual commit result | Actionable test-file reduction |
| --- | --- | ---: | --- | ---: |
| [jitsi/jicofo](https://github.com/jitsi/jicofo) | `6ed82c9` | 70 | Full-validation fallback: unmodeled languages; 32 files identified but no selective command | 0 verified |
| [apache/maven-resolver](https://github.com/apache/maven-resolver) | `a90b7f7` | 206 | 1 file selected; `mvn -pl maven-resolver-tools -am test`; no fallback | 205 files (99.5% of discovered files) potentially excluded |
| [apache/maven-surefire](https://github.com/apache/maven-surefire) | `54522a7` | 726 | Workflow change forces full validation | 0 verified |
| [google/guava](https://github.com/google/guava) | `2a11c2a` | 0 | Unsupported nonstandard Maven source and test paths; unsafe graph and full-validation fallback | 0 verified |
| [apache/commons-text](https://github.com/apache/commons-text) | `00be782` | 103 | Workflow changes force full validation | 0 verified |

The resolver result is a **planning** reduction only. Maven's `-am` also builds required upstream modules, and this probe did not measure which test goals run there. Thus 99.5% is neither runtime nor compute savings. Across these five actual commits, the only usable selective command was for resolver; measured execution savings remain unknown.

The additional one-file scenarios showed that PR #3 can produce a command for Surefire (`mvn -pl maven-failsafe-plugin,surefire-its -am test`, 238 of 726 discovered test files) but the sampled latest commit changed CI configuration and correctly fell back. For Commons Text, a source-only scenario selected all 103 test files and `mvn test`, giving no test-file reduction. Jicofo still fell back. These scenarios were diagnostic and are excluded from the actual-commit table.

Findings for follow-up:

- Guava uses paths such as `guava/src/...` and `guava-tests/test/...`, outside the adapter's conventional `src/main` and `src/test` matching.
- Surefire's adapter scan counted 229 directories as modules because it includes POMs under `surefire-its/src/test/resources`; this count includes fixture projects, so it must not be used as the real reactor size or as a savings denominator.
- The temporary Cloudflare probe had intermittent Sandbox startup and interruption errors; failed attempts were retried. The five table rows are completed Cloudflare analyses, not desktop fallbacks.

Raw responses and the probe source are in `.scratch/maven-cloud-probe/` in this checkout. The dedicated temporary Cloudflare Worker and container application were deleted after the run.
27 changes: 27 additions & 0 deletions docs/research/2026-09-22-maven-pr3-cloudflare-runtime.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Maven PR #3: Cloudflare runtime comparison

Date: 2026-09-22. This follows the [five repository planning probe](./2026-09-22-maven-pr3-cloudflare-probe.md). All Maven commands below ran in Cloudflare Sandbox Containers using `docker.io/cloudflare/sandbox:0.12.5`, OpenJDK 21 and Maven 3.9.11. Timings are command wall time inside the container; clone and tool installation are excluded. The first attempts used `standard-2`; Commons Text and Guava were repeated on `standard-4` after Java heap failures. A command is a valid timing baseline only when it exits successfully. **These `mvn test` probes are not verified reproductions of each repository's actual CI suite.**

| Repository | Root `mvn test` probe | DiffCI command for sampled commit | Valid time saving |
| --- | --- | --- | --- |
| jitsi/jicofo | 403.7 s, passed (`standard-2`) | Full `mvn test` fallback | 0%; identical command |
| apache/maven-resolver | 202.4 s, failed on reactor packaging (`standard-2`) | `mvn -pl maven-resolver-tools -am test`: 191.1 s, failed on the same reactor packaging requirement | Unmeasurable; neither run passed |
| apache/maven-surefire | 47.0 s, failed at compilation under OpenJDK 21 (`standard-2`) | Full `mvn test` fallback | 0% command reduction; no successful runtime baseline |
| google/guava | 701.9 s, failed: one test assumes an unprivileged user (`standard-4`) | Full `mvn test` fallback | 0% command reduction; no successful runtime baseline |
| apache/commons-text | 50.7 s, passed (`standard-4`) | Full `mvn test` fallback | 0%; identical command |

Resolver is the critical finding: its only selective plan from PR #3 is not executable on this repository with the `test` goal because a reactor dependency's `unpack-dependencies` execution requires a packaged artifact. Root `mvn test` also failed. Resolver's [CI workflow](https://github.com/apache/maven-resolver/blob/a90b7f7f8a05b961c2880ac48ca880b3640b3545/.github/workflows/maven-verify.yml) delegates to the Apache shared workflow, whose [default verification goal](https://github.com/apache/maven-gh-actions-shared/blob/v5/.github/workflows/maven-verify.yml) is `-P run-its verify`; the PR #3 planner hardcodes `test` instead. The difference between two failure durations is **not** a savings measurement. The adapter needs a workflow-compatible command or a conservative fallback for this reactor shape before a runtime savings claim.

## Why none of the five showed savings

1. **Sampled commits that require full validation.** The latest Surefire and Commons Text commits changed only GitHub workflows, so core's `WORKFLOW_GLOBAL` rule deliberately requires a full run. Jicofo's latest commit also changed the root `pom.xml`, which is a configuration change. These are poor candidates for demonstrating selective CI, even with a capable Maven adapter.
2. **A repository-wide adapter blocker.** Jicofo contains `script/graceful_shutdown.py`. Core currently marks any repository with an active adapter and any `.py`, `.rs`, `.cs`, `.svelte`, or `.astro` file unsafe, regardless of whether that file participates in Maven tests. The root-POM change independently forces a full run on this sampled commit.
3. **Unsupported Maven layout.** Guava stores code and tests under paths such as `guava/src/...` and `guava-tests/test/...`. The adapter recognizes only conventional `src/main/{java,kotlin}` and `src/test/{java,kotlin}` paths, so it discovered zero Guava test files and refused a selective plan.
4. **The only selective command did not execute successfully.** Resolver's `mvn -pl maven-resolver-tools -am test` reaches `maven-resolver-transport-jdk`, where `unpack-dependencies` needs a packaged reactor artifact. The command failed after 191.1 s. Full `mvn test` failed after 202.4 s on the same reactor requirement, so this environment cannot establish a successful runtime baseline either.
5. **File selection is not Maven test execution.** The adapter models every file within a Maven module as mutually connected and emits a module-level `mvn -pl ... -am test` command. Maven will run tests in selected modules and potentially in upstream modules brought in by `-am`; the earlier 1-of-206 file count for Resolver was not an executable 99.5% time reduction.

The fastest path to a meaningful savings measurement is to validate an execution-safe command against a real multi-module CI workflow, then benchmark a source-only commit in a supported repository with matched Java/Maven versions, user permissions, cache state, and successful full and selective runs. The four fallback cases cannot show a time saving for their sampled commits because their DiffCI command is identical to the full command.

Surefire failed before test execution with `ObjectUtils.JAVA_RECENT` missing. Its [CI workflow](https://github.com/apache/maven-surefire/blob/54522a785b017355638f525c69893edfa60848a9/.github/workflows/maven-verify.yml) supplies a JDK 25 toolchain and runs `clean install -nsu -P run-its`, unlike this OpenJDK 21 `mvn test` probe. On the smaller container, Commons Text and Guava both failed with `Java heap space`; those failed durations are not test-suite timings. On the larger container, Guava ran 865,592 tests but failed `MoreFilesTest.testCreateParentDirectories_noPermission` because the root-running Sandbox did not trigger the expected permission error. Maven commands were run once per condition, so even the successful durations are observations rather than stable performance estimates. For fallback repositories, DiffCI and the root `mvn test` probe are the same command, making command-level time savings zero by construction.

The temporary Cloudflare Worker and container application were dedicated to this benchmark and were deleted after the run.
10 changes: 5 additions & 5 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@diffci.com/diffci",
"version": "0.1.11",
"version": "0.1.12",
"description": "DiffCI - deterministic change-aware CI planning",
"keywords": [
"ci",
Expand Down Expand Up @@ -125,11 +125,11 @@
"check:oss-boundary": "node scripts/check-oss-boundary.mjs"
},
"dependencies": {
"@diffci.com/core": "git+https://github.com/DiffCI/core.git#db1d637dfcca8d8f7fa047500dfe0027d0d018a7",
"@types/node": "^22.15.12",
"@vue/compiler-sfc": "3.5.42",
"typescript": "^5.8.3",
"yaml": "^2.9.0",
"@diffci.com/core": "git+https://github.com/DiffCI/core.git#27afb990e76f764c8bd2087df5bb60ba5913042f"
"yaml": "^2.9.0"
},
"devDependencies": {
"@cloudflare/containers": "^0.3.7",
Expand Down
Loading
Loading