diff --git a/README.md b/README.md index 35440b792..713a2fc88 100644 --- a/README.md +++ b/README.md @@ -1,36 +1,16 @@ # diffr -Structural diffs with a streaming API and an interactive terminal frontend. -diffr is derived from [difftastic](https://github.com/Wilfred/difftastic) -(MIT, Wilfred Hughes) and its terminal UI from -[hunk](https://github.com/modem-dev/hunk) (MIT, Modem Labs). See `NOTICE` -for every upstream and its license. +> diffr is experimental and in alpha. API breakages are possible at any time, although we will do our best to warn you of them. -## Installation - -CLI and terminal UI (Apple Silicon macOS and x64 Linux): - -```sh -brew install devdotfast/tap/diffr -``` - -The CLI, from crates.io (prebuilt via [cargo-binstall](https://github.com/cargo-bins/cargo-binstall), or compiled): +diffr is a Rust-based structural diffing (AST-aware) library. It is also packaged as a CLI with a built-in TUI, and it has a WASM plugin system for extensibility. -```sh -cargo binstall diffr-cli -cargo install diffr-cli --locked -``` +## Features -From source: `cargo xtask install` (requires Rust and [Bun](https://bun.sh)). - -## Configuration - -To turn on semantic diff summarization: - -1. Run `diffr config` -2. Search for 'summarization' - - Enable in the dropdown - - Add your API key for Gemini if you don't already have on your path +1. Interactive TUI with AST-aware code folding +3. Sane defaults for AI coding + - Summarize long changes as pseudocode + - Collapse tests and docs +3. WASM-based plugin system ## Usage @@ -42,105 +22,50 @@ diffr --cached # staged changes diffr main...HEAD -- src/ # merge-base comparison ``` -You can also use `diffr` in streaming mode, which is useful for TUI or GUI applications: +## Installation + +CLI and terminal UI (Apple Silicon macOS and x64 Linux): ```sh -diffr main HEAD --format ndjson +brew install devdotfast/tap/diffr ``` -## Architecture - -When you run `diffr ${commit_range_exp}`, the following happens: - -1. Commits loaded from git -2. Plugins (explained in more detail later) load -3. Each file is parsed via tree-sitter & diffed using difftastic's ast/ast diffing algorithm - - This produces an alignment of file / file - - Note: because of known upstream limitations, the diffing algorithm is quite CPU/Mem intensive. - We fall back to a textual diffing algorithm in case of issue -4. Plugins define which AST nodes are present in the API + folded by default. - -### Plugin API - -`diffr` has a powerful, wasm-based plugin API which customizes how it presents changed files. +The CLI, from crates.io (prebuilt via [cargo-binstall](https://github.com/cargo-bins/cargo-binstall), or compiled): -For example, the following are all implemented as plugins: +```sh +cargo binstall diffr-cli +cargo install diffr-cli --locked +``` -- Context Folding: showing relevant context, like a function signature + closing brace (if applicable) -- Algorithm Summarization: using an LLM to summarize long algorithms into pseudocode -- Comment collapsing: collapsing long LLM comments + function bodies & expanding both at once -- Collapsing tests by default +From source: `cargo xtask install` (requires Rust and [Bun](https://bun.sh)). -The [Rust SDK's `Plugin` trait](crates/diffr-plugin-sdk/src/lib.rs) exposes -four important methods: +## History -```rust -// rust bindings of underlying WASM plugin API -use diffr_plugin_sdk::{FileEntry, Move, Pairing, QuerySource, Source}; +diffr is a fork of the lovely [difftastic](https://github.com/Wilfred/difftastic)(MIT, Wilfred Hughes) and its terminal UI from +[hunk](https://github.com/modem-dev/hunk) (MIT, Modem Labs). -pub trait Plugin: Sized { - /// Options are configuration for the plugin - type Options: serde::de::DeserializeOwned; +We forked [difftastic](https://github.com/Wilfred/difftastic)(MIT, Wilfred Hughes) because of the following 3 technical reasons: - /// new loads the plugin from its configuration; this is to allow plugins to fail - /// early if user config isn't set correctly - fn new(options: Self::Options) -> anyhow::Result; +1. TUI affordance +2. AST-based fold matching +3. Plugin System - /// queries return tree-sitter queries to add metadata to the tree-sitter tree - /// this means that the plugins can backpack off of the tree-sitter parse that the - /// diffing algorithm does. - fn queries(&self) -> anyhow::Result>; +We hope in the future that we can find some way to upstream some / all of these features. Obviously everyone works differently and this is quite an opinionated stance on things, so we imagine this may take time. - /// classify (bad name lol) runs classification of files into generated, test, etc. - /// Useful to prevent wasteful semantic diffing for things users will skip. - /// emits tags that clients can make use of - fn classify(&self, file: &FileEntry) -> anyhow::Result>; +## Known Limitations - /// mutate emits a series of structured mutations ('Moves') to the parsed diff type - /// (e.g., fold X function body, show Y lines of context around it, etc.) - fn mutate(&self, file: &FileEntry, sides: &Pairing) - -> anyhow::Result>; -} -``` +1. Semantic diff summarization is only supported for the Gemini class of models right now. Turn it on via: + a. Through the TUI: + - Run `diffr config` + - Search for 'summarization' + - Enable in the dropdown + - Add your API key for Gemini if you don't already have on your path + b. Through the [Whiteboard app](https://github.com/devdotfast/whiteboard). +2. The plugin API is a bit awkward and will be simplified radically in the coming releases. -```mermaid -sequenceDiagram - participant D as diffr - participant P as Plugins - participant G as Git - participant E as Diff engine - participant C as UI / API consumer - - D->>P: new(options), queries() - P-->>D: Plugin instances and query sources - D->>G: Load changed files for comparison - G-->>D: Before and after versions - loop Each changed file - D->>P: classify(file) - P-->>D: File tags - D->>E: Compare versions using tags and queries - alt Structural comparison available - E->>E: Parse with tree-sitter and diff with difftastic - else Generated file or structural fallback - E->>E: Compute line diff - end - E-->>D: Aligned regions and folds - loop Each enabled plugin in order - D->>P: mutate(file, sides) - P-->>D: Presentation moves - D->>D: Apply moves to regions and fold state - end - D-->>C: Diff with initial fold state - end -``` +### Plugin API -The WASM interface is defined in [`wit/plugin.wit`](wit/plugin.wit). -The SDK's [`export!` macro](crates/diffr-plugin-sdk/src/lib.rs) and -[guest adapter](crates/diffr-plugin-sdk/src/guest.rs) expose a Rust plugin -as a WASM component; the [Wasmtime runner](src/plugin/wasm.rs) loads and -calls it. See the [context plugin](plugins/context/src/lib.rs) and its -[Rust query](plugins/context/queries/rust.scm) for a concrete implementation, -or [Writing plugins](docs/plugins.md) for the full guide. +`diffr` has a powerful, wasm-based plugin API which customizes how it presents changed files. For more details, read the [docs](./docs/plugin.md). ## License diff --git a/docs/cli.md b/docs/cli.md deleted file mode 100644 index a904ba31f..000000000 --- a/docs/cli.md +++ /dev/null @@ -1,48 +0,0 @@ -# Git-style CLI - -```sh -diffr # index -> working tree -diffr --cached # HEAD -> index (empty tree on an unborn branch) -diffr HEAD # HEAD -> working tree -diffr main HEAD # two revisions -diffr main...HEAD # merge-base(main, HEAD) -> HEAD -diffr main..HEAD # main -> HEAD -diffr --merge-base main # merge-base(main, HEAD) -> working tree -diffr HEAD -- src/ '*.rs' # repository selection, with pathspecs -diffr --no-index before.rs after.rs -``` - -`--repo DIR` selects a repository; otherwise discovery starts in the current -directory. Paths are relative to that directory. Use `--` to separate paths -from revisions. `-R` reverses the comparison. `--staged` aliases `--cached`. - -`--name-only`, `--name-status`, `--stat`, `--shortstat`, and `--numstat` use -libgit2 without compiling syntax configuration or matching syntax. Counts are -textual. Name status uses A/D/M/R/T/U; rename similarity scores are not exposed. -`-z` supports name-only and name-status output for unambiguous path delimiters. -Text output does not implement Git's path quoting or compact rename formatting. -`-M` enables rename detection (the default); `--no-renames` disables it. - -`--quiet` suppresses output and exits 1 for changes. `--exit-code` also exits 1 -for changes; ordinary output exits 0. Errors exit 2. `--no-index` supports two -files, implies change exit status, and does not yet support metadata options. - -A comparison opens the terminal UI (`tui/`), which needs a terminal on stdin -and stdout; without one diffr exits 2. `--format ndjson` instead writes the -event stream described in [streaming.md](streaming.md), diffing `--jobs N` -files at once (default 16) and emitting each as it finishes; the terminal UI -reads the same stream. `ndjson` is the only format. -`-U N` sets the unchanged lines kept around each change, overriding -`plugins.bundled.context.lines` (default 3). Matching limits and `--ignore-comments` -remain configurable; `--width` sets the columns of `--stat`. See `--help`. - -Configuration comes from the global file, command-line flags and git -attributes; `--config PATH` replaces the global file. `diffr config` opens the -settings screen, and `diffr config schema`, `show` and `set` are the commands -frontends use. See [config.md](config.md). - -This is a subset of git diff, not full flag parity: unsupported options and Git -magic pathspecs fail explicitly. Untracked files are excluded as in git diff. -`diffr debug` keeps difftastic's syntax dumps (`--dump-ts`, `--dump-syntax`, -`--dump-syntax-dot`) and `--list-languages` for diagnostics. The stdout event -contract is in [streaming.md](streaming.md). diff --git a/docs/config.md b/docs/config.md deleted file mode 100644 index 216f43c99..000000000 --- a/docs/config.md +++ /dev/null @@ -1,322 +0,0 @@ -# Configuration - -diffr's configuration comes from exactly three places: - -1. The global file, `$XDG_CONFIG_HOME/diffr/config.toml` (usually - `~/.config/diffr/config.toml`). `--config PATH` replaces it and must exist; - a missing global file means every default. -2. Command-line flags, for one run. `--byte-limit`, `--graph-limit` and - `--parse-error-limit` override `[diff]`, and `-U` overrides - `plugins.bundled.context.lines`. -3. Git attributes, for facts about files in a repository (`.gitattributes` - and git's user-wide attributes file); see [File tags](#file-tags). - -There is no repository configuration file and no environment-variable layer. -The embedded [default config](../src/config/default.toml) provides fresh-install -settings. Config files use `version = 1`; unsupported versions are rejected. -Keys the file omits keep their defaults. An unknown key or a mistyped value is -an error that names the file and the key's dotted path, for example -`config.toml: diff.typo: unknown field`. - -## Commands - -```sh -diffr config # settings screen in the terminal frontend -diffr config theme # the same, searching for "theme" -diffr config schema # JSON Schema: title, group, description and default per key, plugins included -diffr config show [--json] # the resolved configuration -diffr config set diff.graph_limit 5000000 -diffr config set theme.name default-light -diffr config set plugins.bundled.deleted-bodies.min_lines 20 -``` - -On the first successful edit, `set` writes the complete resolved config, -including `version = 1`, plugin order, and option defaults. Later edits preserve -existing values and comments. Existing partial files are filled in on edit too; -explicit plugin lists remain authoritative. This pins today's defaults until -the user edits them or a future version migration changes them. - -To add an external plugin, edit the config file: add its -`[plugins.external.NAME]` entry with a `path`, and include `external.NAME` -at the desired position in `plugins.order`. - -`set` writes one key into the global file (or the `--config` file). The value -is read as the type the schema gives the key, and only that type: a string key takes the text as it is -(`diffr config set theme.name 1234` writes `"1234"`), an integer key a TOML -integer (`12`), a number key a TOML number (`1.5`), a boolean key `true` or -`false`, an array key a TOML array (`'["a", "b"]'`), and an enum key one of -its choices. A plugin option has the type its `plugin.toml` declares, a WASM -plugin's folder included, and an entry's `path` is a string. Anything else, an -unknown key, or a value the configuration rejects (such as a negative limit) -is an error naming the key and what it expected, for example -`diff.graph_limit: expected an integer, got "abc"`; diffr exits 2 and the file -is not touched. Frontends drive their settings pages through these three -commands; the schema is the only contract. Each setting in the schema carries -a `title` and an `x-group` that settings screens show in place of the dotted -key; keys marked `"x-settings": false` are not settings. - -## Keys - -```toml -version = 1 - -[diff] -byte_limit = 1000000 # larger files on either side get a line diff -graph_limit = 3000000 # the largest AST matching graph explored per file -parse_error_limit = 0 # more tree-sitter parse errors than this: line diff - -[theme] -name = "default-dark" # a bundled terminal theme -path = "/path/to/theme.toml" # or a Helix-style theme file - -[plugins] # see "Plugins" below -order = ["bundled.context", "bundled.hide-files", "bundled.deleted-bodies", "bundled.summarize", "bundled.test-bodies", "bundled.removed-runs", "bundled.group"] -``` - -When a file exceeds a `[diff]` limit it falls back to a line diff: the file's -`stats` carries a `fallback` with code `too_large`, `too_complex` or -`parse_error` and a message naming the key to raise. `--byte-limit`, -`--graph-limit` and `--parse-error-limit` override the file for one run. The -defaults are difftastic's. A large rewrite can exceed the graph limit and fall -back to a line diff; raising the limit trades memory for it across every file -diffed in parallel, so prefer narrowing the comparison or leaving the fallback. - -## Plugins - -After a file is diffed and its regions are built, plugins decide how it starts -out on screen: which files are hidden, which regions start collapsed and with -what label, which regions open and close together, and which collapsed -regions are grouped. They run in `plugins.order`, each on the region trees the -one before it left. [streaming.md](streaming.md#plugins) describes what they -produce on the wire. - -```toml -[plugins] -order = ["bundled.context", "bundled.hide-files", "bundled.deleted-bodies", "bundled.summarize", "bundled.test-bodies", "bundled.removed-runs", "bundled.group"] - -[plugins.bundled.context] # unchanged lines far from any change collapse -enabled = true -lines = 3 # kept on either side of a change; -U overrides it - -[plugins.bundled.hide-files] -enabled = true -tags = ["generated", "vendored", "test"] # the first listed tag a file carries names the reason -deleted = true # hide deleted files whatever their tags - -[plugins.bundled.deleted-bodies] # deleted function bodies, "12 lines removed" -enabled = true -min_lines = 12 - -[plugins.bundled.test-bodies] # test functions and test modules, on both sides -enabled = true -min_lines = 3 - -[plugins.bundled.removed-runs] # the middle of long removed stretches -enabled = true -min_lines = 5 - -[plugins.bundled.summarize] # pseudocode for large new function bodies and right-side tests -enabled = false # off by default: it needs an API key -provider = "gemini" -model = "gemini-3.8-flash" -min_lines = 20 -tests = true # include right-side tests even when not newly added -test_min_lines = 20 -api_key = "…" # or GEMINI_API_KEY / GOOGLE_API_KEY -endpoint = "https://…" # optional base URL override -request_timeout_ms = 60000 -max_concurrency = 16 # legacy option; ignored by the WASM summarizer -retries = 3 -system_prompt = """…""" # the model's system instruction; defaults to diffr's own - -[plugins.bundled.group] # adjacent collapsed regions under one row -enabled = true -``` - -Without an explicit `plugins.order`, bundled defaults are available and the -file may override individual settings. With an explicit order, only the -listed bundled plugins and declared entries participate: diffr does not add -other bundled plugins. Every declared entry must appear exactly once. - -Entries live in `plugins.bundled` or `plugins.external`; order uses qualified -references such as `bundled.context` and `external.mine`. These references do -not change plugin names or query tags. A bundled entry cannot set `path`. -An external entry must set `path` to a folder containing `plugin.toml` and -`plugin.wasm`, relative to the configuration file or absolute. Missing WASM -components are errors; external entries never fall back to native code. -Only one enabled implementation may use a given plugin name. - -`enabled` and `path` belong to diffr. Other keys are validated against the -plugin's option schema and filled with defaults. Unknown options and wrong -types are errors naming the entry and option. - -Every plugin that is on is made (its `new` runs) when diffr starts, before -any output, and one that cannot be made stops diffr with an error naming it. -The summarizer cannot be made without an API key, which is why it ships off: turn it on with -`enabled = true` once `api_key`, `GEMINI_API_KEY` or `GOOGLE_API_KEY` holds a -key. - -```toml -[plugins] -order = ["bundled.context", "bundled.hide-files", "bundled.deleted-bodies", "bundled.summarize", "bundled.test-bodies", "bundled.removed-runs", "bundled.group", "external.fixtures"] - -[plugins.external.fixtures] -path = "plugins/fixtures" -``` - -`summarize`'s `system_prompt` is the system instruction sent with every -request. The per-file message (the file's numbered lines and the folds to -summarize) is built by diffr. `diffr config show` prints the default prompt. - -### Plugin folders - -Every bundled plugin is a folder under `plugins/`, laid out the way any plugin -is: - -```text -plugins/deleted-bodies/ - plugin.toml # name, title, options - queries/rust.scm # one query file per language - Cargo.toml, src/lib.rs # the plugin's code, a crate using the plugin SDK - plugin.wasm # the same code built as a WASM component (not committed) -``` - -diffr embeds a bundled plugin's `plugin.toml` and query files and compiles -its code in. Any other plugin's folder holds `plugin.wasm`. - -`plugin.toml` is static: - -```toml -name = "deleted-bodies" # the entry name, and the prefix of its tags -title = "Deleted function bodies" # the group settings screens list its settings under -description = "Collapse function bodies that were deleted: …" - -[enabled] # how settings screens show the switch -title = "Collapse deleted function bodies" -description = "…" -default = true # whether the plugin is on unless an entry says otherwise - -[options.min_lines] # each option is a JSON Schema, written in TOML -type = "integer" -minimum = 0 -title = "Shortest body to collapse (lines)" -description = "Deleted bodies shorter than this stay open." -default = 12 -``` - -Every option needs a `title`; one with a `default` is pre-filled in -the plugin's bundled or external entry, and one without starts unset. Options -are listed in the settings schema in the order `plugin.toml` declares them. - -### Queries - -Tree-sitter queries decide which folds exist and what they are, as tags; -plugins decide how they are shown. Each plugin returns named source text from -`queries()`, called once on its instance during setup. Each source has a -`language` (a lowercase language name such as `rust` or `typescripttsx`), -`name` for imports and diagnostics, and `text`. Rust plugins can embed their -`.scm` files with `include_str!`; no `[queries]` section is used in -`plugin.toml`, and queries are not configured in `[plugins]`. - -For each language, diffr concatenates the sources of every enabled plugin, -in `order`, into one compiled and validated query before the stream starts. -A source whose first line is `; inherits: shared.scm` (several names separated -by commas) includes its dependencies first. Relative imports resolve against -the importing source's name. Return those sources from `queries()` too; -source names should include the plugin name to avoid accidental collisions. -Returned sources take precedence over files. An unresolved `builtin:` import -loads a bundled file, and an absolute-path import loads a file on disk. -Relative imports from an absolute source name resolve beside that file. -Each source is included at most once per language; import cycles and different -text returned under the same name are setup errors. - -The structure every bundled plugin shares (blocks, collections, imports, -strings) lives in `plugins/shared/queries/.scm`, which is not a -plugin: the bundled queries import it as -`builtin:shared/queries/.scm`. - -Docstrings live beside it, in `plugins/shared/queries/-docstrings.scm`: -comments directly above a function (Rust `///` and `//` runs and `/** */` -doc comments, Go comment groups, JavaScript and TypeScript comments and JSDoc) -and Python's leading string. The plugins that collapse function bodies -(`deleted-bodies`, `test-bodies`, `summarize`) import it, and each links a -body it collapses to its docstring, so the two open and close together. -Because a file is included once however many plugins import it, each -docstring pattern sets one tag per importer (`deleted-bodies:docstring`, -`test-bodies:docstring`, `summarize:docstring`), and a docstring carries all -three whenever any of those plugins is enabled. A docstring is tagged only -when its function's body spans lines: a comment above a one-line function or -a plain declaration still folds, untagged. - -`src/config/README.md` describes the capture conventions (`@fold`, -`@fold.open`, `@fold.close` and `#set! tag`). Every `@fold` capture in one match -forms one fold, so a quantified run such as `(comment)+ @fold` folds as one -region spanning the run. Tags name a plugin and are written in the query: -`(#set! tag "deleted-bodies:function")`. A tag must have the form -`:` with a plugin from `plugins.order`, or the configuration -does not compile; a plugin reads only the tags its own queries set. A -pattern may set no tag at all, to make a fold exist without meaning anything -to a plugin. - -Patterns from different files that capture the same syntax node merge into -one fold with the union of their tags when they agree on its range. When they -capture it with different ranges, that file is not diffed: its record carries -an error with code `query_conflict` naming both files, for example -`src/lib.rs:42: builtin:deleted-bodies/queries/rust.scm and -builtin:summarize/queries/rust.scm capture the same function_item with -different fold ranges`. Other files proceed. - -A query file that does not compile (a syntax error, an unknown node, an -unsupported capture or directive, a malformed tag) is an error naming the -file, and diffr exits 2 before the stream starts. - -### Settings screen - -`diffr config schema` builds the `plugins` part of the schema from each -plugin's `plugin.toml`. Every scalar setting under `plugins` has a `title` -and an `x-group`: the group is the plugin's `title` (`Context`, `Hidden files`, -`Deleted function bodies`, `Test bodies`, `Removed stretches`, -`Summaries`, `Groups`), and each `enabled` is titled after what that plugin -does. `plugins.order`, every list option such as `hide-files.tags`, and -`summarize.system_prompt` (a multi-line value) are marked `"x-settings": -false`: the schema describes them, a settings screen does not edit them, and -they are set in the file or with `diffr config set`. `diffr config show` redacts `plugins.bundled.summarize.api_key` unless -`--reveal` is given. - -## File tags - -Every file in a repository comparison carries `tags` in the stream manifest: -what the file is, such as `generated`, `vendored`, `docs` or `test`. There is -no configuration key for them. In order of precedence, lowest first: - -1. Bundled rules. `generated`, `vendored` and `docs` follow GitHub Linguist: - `generated` ports `lib/linguist/generated.rb` (lockfiles and other names - and paths, headers such as `// Code generated ... DO NOT EDIT.` in a - file's first lines, and minified JavaScript and CSS by average line - length), and `vendored` and `docs` match the path against Linguist's - `vendor.yml` and `documentation.yml`, bundled under `src/tags/linguist/` - with Linguist's MIT license. `test` comes from diffr's own path rules: - `tests/`, `test/`, `__tests__/`, `spec/`, `*_test.go`, `*.test.*`, - `*.spec.*`, `test_*.py`, `*_test.py`, `conftest.py`, `tests.rs`, `test.rs`, - `*_test.rs`, `*_tests.rs`. A file carries every tag that matches. -2. Git attributes, with git's own precedence between `$GIT_DIR/info/attributes`, - `.gitattributes` files and the user-wide file (`core.attributesFile`, - default `~/.config/git/attributes`). Attributes always beat the bundled - rules. `linguist-generated`, `linguist-vendored` and - `linguist-documentation` add `generated`, `vendored` and `docs` when set, - and remove them when unset or `false`, whatever the rules said. - `diffr-tags=a,b` adds tags: lowercase letters, digits, `-` and `_`, - separated by commas. A malformed value is an error naming the path, and - diffr exits 2 before the stream starts. - -```gitattributes -Cargo.lock linguist-generated=false -web/schema.json linguist-generated diffr-tags=schema -fixtures/** diffr-tags=fixture -``` - -Header and minified checks read only the first 8 KiB of a file's after side -(its before side when deleted), and only when no name, path or attribute -has already decided `generated`. A file tagged `generated` always gets a line -diff without parsing, and its `stats.fallback` has code `generated`. -Comparisons with `--no-index` are outside any repository and carry no tags. diff --git a/docs/plugin.md b/docs/plugin.md new file mode 100644 index 000000000..f961dd66a --- /dev/null +++ b/docs/plugin.md @@ -0,0 +1,96 @@ +# Plugin Architecture + +`diffr` has a powerful, wasm-based plugin API which customizes how it presents changed files. For more details, read the [docs](./docs/plugin.md). + +For example, the following are all implemented as plugins: + +- Context Folding: showing relevant context, like a function signature + closing brace (if applicable) +- Algorithm Summarization: using an LLM to summarize long algorithms into pseudocode +- Comment collapsing: collapsing long LLM comments + function bodies & expanding both at once +- Collapsing tests by default + +## Architecture + +When you run `diffr ${commit_range_exp}`, the following happens: + +1. Commits loaded from git +2. Plugins (explained in more detail later) load +3. Each file is parsed via tree-sitter & diffed using difftastic's ast/ast diffing algorithm + - This produces an alignment of file / file + - Note: because of known upstream limitations, the diffing algorithm is quite CPU/Mem intensive. + We fall back to a textual diffing algorithm in case of issue +4. Plugins define which AST nodes are present in the API + folded by default. + +## Plugin Interface + +The [Rust SDK's `Plugin` trait](../crates/diffr-plugin-sdk/src/lib.rs) exposes +four important methods: + +```rust +// rust bindings of underlying WASM plugin API +use diffr_plugin_sdk::{FileEntry, Move, Pairing, QuerySource, Source}; + +pub trait Plugin: Sized { + /// Options are configuration for the plugin + type Options: serde::de::DeserializeOwned; + + /// new loads the plugin from its configuration; this is to allow plugins to fail + /// early if user config isn't set correctly + fn new(options: Self::Options) -> anyhow::Result; + + /// queries return tree-sitter queries to add metadata to the tree-sitter tree + /// this means that the plugins can backpack off of the tree-sitter parse that the + /// diffing algorithm does. + fn queries(&self) -> anyhow::Result>; + + /// classify (bad name lol) runs classification of files into generated, test, etc. + /// Useful to prevent wasteful semantic diffing for things users will skip. + /// emits tags that clients can make use of + fn classify(&self, file: &FileEntry) -> anyhow::Result>; + + /// mutate emits a series of structured mutations ('Moves') to the parsed diff type + /// (e.g., fold X function body, show Y lines of context around it, etc.) + fn mutate(&self, file: &FileEntry, sides: &Pairing) + -> anyhow::Result>; +} +``` + +> Note: the api above is subject to change / unstable at the moment. It's a bit overengineered for our taste and we are working to simplify it. For example: there are too many methods on it, and the trait uses [anyhow](https://github.com/dtolnay/anyhow) and really should be using [thiserror](https://github.com/dtolnay/thiserror). + +```mermaid +sequenceDiagram + participant D as diffr + participant P as Plugins + participant G as Git + participant E as Diff engine + participant C as UI / API consumer + + D->>P: new(options), queries() + P-->>D: Plugin instances and query sources + D->>G: Load changed files for comparison + G-->>D: Before and after versions + loop Each changed file + D->>P: classify(file) + P-->>D: File tags + D->>E: Compare versions using tags and queries + alt Structural comparison available + E->>E: Parse with tree-sitter and diff with difftastic + else Generated file or structural fallback + E->>E: Compute line diff + end + E-->>D: Aligned regions and folds + loop Each enabled plugin in order + D->>P: mutate(file, sides) + P-->>D: Presentation moves + D->>D: Apply moves to regions and fold state + end + D-->>C: Diff with initial fold state + end +``` + +The WASM interface is defined in [`wit/plugin.wit`](../wit/plugin.wit). +The SDK's [`export!` macro](../crates/diffr-plugin-sdk/src/lib.rs) and +[guest adapter](../crates/diffr-plugin-sdk/src/guest.rs) expose a Rust plugin +as a WASM component; the [Wasmtime runner](../src/plugin/wasm.rs) loads and +calls it. See the [context plugin](../plugins/context/src/lib.rs) and its +[Rust query](../plugins/context/queries/rust.scm) for a concrete implementation. \ No newline at end of file diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 76c8651f9..000000000 --- a/docs/plugins.md +++ /dev/null @@ -1,392 +0,0 @@ -# Plugins - -A plugin shapes how diffr shows a changed file. Every plugin implements the -same contract, [`wit/plugin.wit`](../wit/plugin.wit), and diffr loads and -runs every plugin the same way: the bundled ones (`plugins//`, compiled -into diffr) and WASM components in a folder the configuration points at. All -are configured the same way, in `[plugins]` ([config.md](config.md#plugins)), -and produce the same wire records ([streaming.md](streaming.md#plugins)). - -This is a research prototype: a WASM plugin runs with full access to the -machine (see [Access](#access)). Run only plugins you trust. - -## The plugin points - -A plugin is made once, then a file passes three points, each in -`plugins.order`: - -0. **New.** When diffr starts, before any output, it makes each enabled - plugin's one instance for the run: it deserializes the plugin's options - into the plugin's own options type and calls `new` with them. A plugin - that cannot be made (options that do not deserialize, a summarizer without - an API key, a component that does not compile or link) is a setup error: - diffr writes it to stderr, naming its bundled or external config entry, and exits - 2 before any record, for native and component plugins alike. -1. **Pre-process: `classify`.** Before the stream starts, each plugin's - `classify` returns tags to add to the file's manifest entry. They join the - tags from the bundled rules and git attributes - ([config.md](config.md#file-tags)), so they decide how the file is diffed - (a `generated` file gets a line diff), how `--order` ranks it, and what - later plugins see: each plugin sees the tags the ones before it added. An - error here, or a tag that is not lowercase letters, digits, `-` and `_`, - is a setup error. -2. **During: queries.** The named tree-sitter source text returned by a - plugin's `queries()` joins the per-language fold query the diff runs with - ([config.md](config.md#queries)). They decide which folds exist and tag - them `:`. diffr collects and validates the sources once - during setup, before the stream starts. Tags are the only syntax a plugin - sees: the `context` plugin reads the scopes its queries - tag `context:scope`. -3. **Post-process: `mutate`.** After the file is diffed and its region trees - are built, each plugin's `mutate` reads the file and both sides and - returns moves, which diffr carries out before the next plugin runs. An - error, a trap, or a move that cannot be carried out aborts the run with - `mutation_failed` naming the plugin. - -## A plugin folder - -```text -examples/plugins/fixtures/ - plugin.toml # name, title, options schema - plugin.wasm # the component - queries/rust.scm # optional: one query file per language -``` - -`plugin.toml` is the same file the bundled plugins carry -([config.md](config.md#plugin-folders)): `name` (the entry name in -`[plugins]`, and the prefix of every tag its queries set), `title`, -`description`, `[enabled]` (with `default`, whether the plugin is on unless -its entry says otherwise), `[options.]` (each a JSON Schema with a -`title`). Query sources belong to the plugin's code, typically embedded -with `include_str!`, and are returned by `queries()`; there is no `[queries]` -manifest section. See [query sources and imports](config.md#queries). - -An external plugin is explicitly selected in the user's config: - -```toml -[plugins] -order = ["bundled.context", "external.fixtures", "bundled.group"] - -[plugins.external.fixtures] -path = "plugins/fixtures" -fail = false -``` - -The path is relative to the config file's directory, or absolute. The folder's -manifest must name `fixtures`, and `plugin.wasm` must exist when the plugin -is loaded. A missing component is an error even if a native plugin has the -same name. Plugins keep their unqualified names and tag prefixes regardless -of whether they are loaded through `bundled` or `external`. - -Bundled plugins use their registered implementations; external plugins use -the WASM interface. Config controls enabled state, options and order. An -explicit order does not silently add other bundled plugins. Adding an -external plugin never requires rebuilding diffr. - -## The contract - -[`wit/plugin.wit`](../wit/plugin.wit) is the contract: the package -`diffr:plugin@0.2.0`, world `plugin`. A plugin exports the interface `guest`, -which holds one resource, the plugin itself (the component model has no -optional exports; a plugin that does not classify returns an empty list): - -```wit -resource plugin { - new: static func(options: string) -> result; - queries: func() -> result, string>; - classify: func(file: file-entry) -> result, string>; - mutate: func(file: file-entry, sides: source-sides) -> result, string>; -} -``` - -`new` is a static function rather than a constructor, because a constructor -cannot fail. `queries` returns records with `language`, `name`, and `text` -strings; the Rust SDK defaults to an empty list. Existing WASM components -must be rebuilt to provide the new export. - -- `options`: the plugin's entry as a JSON object, defaults filled in and - already validated against `plugin.toml`. Only `new` gets it; what the - plugin keeps from it lasts the run. -- `file-entry`: `file`, `status` and `tags`. `file` is the manifest entry's - sides — `both`, `left-only` (a deletion) or `right-only` (an addition) — and - each side is a `file-ref` of `path`, `oid` (the blob; empty outside a - repository) and `mode`. `status` says how the two sides relate, which the - sides alone do not: a path that changed is `renamed`, an object kind that - changed `type-changed`. The SDK's `FileEntry::side()` is the side a file is - named by and `path()` its path. -- `source-sides`: the sides the diffed file has, shaped like `file-sides`. - The SDK rebuilds them as trees on the way in, so a plugin's `mutate` is - handed a `Pairing` rather than the flat records. -- `source`: one side's `text` and its `regions`, the tree flattened in - preorder: each `region` carries its `parent` (a fold's `id`, or `0` for a - top-level region), `id`, `fold-state-id`, `range`, `tags`, `visibility`, and - `kind`: `leaf(alignment-id, changed)` or `fold`. The fields mean what they - mean on the wire ([streaming.md](streaming.md#regions)). A binary file's - sides have empty text and no regions. -- `move`: `cut({region, at})`, `join-folds(ids)`, `link-fold-state(ids)`, - `set-collapsed((id, collapsed))`, `set-label((id, label))` and - `set-tags((id, tags))`, with the semantics of the moves in - [streaming.md](streaming.md#plugins). `0` names the file where a move - allows it. - -diffr builds these records once per call, from the file's manifest entry and -its trees as the plugins before left them, and hands the same records to a -native plugin and to a component. Nothing else reaches a plugin. - -By default diffr makes one instance of each plugin and serializes calls to it. -A plugin author can declare `parallel = true` at the top of `plugin.toml` to -permit independent deferred-work instances. This promises that calls do not depend on shared -mutable state, call order, or unique external side effects. Initial presentation and enrichment may use different instances. Plugin ordering within a file -is still sequential. - -Opted-in plugins expose a host-owned `instances` setting (default 4, range 1–64): - -```toml -[plugins.bundled.summarize] -instances = 4 -``` - -The pool is shared across every file and phase of that plugin, so the bound is -per comparison, not per file. A separate initial-presentation instance keeps -queries, classification, and mutation responsive while the deferred pool is busy. -With `instances = 1`, all phases share the same instance for serial debugging. The host compiles a WASM component once and creates -independent stores. Each store still executes only one call at a time. Leases -return on success and errors. Cancellation skips queued enrichment; active calls -retain their configured request timeouts. Native plugins use the same pool. -The summarizer batches selected folds into one request per file, so this -parallelizes files; it does not split a file's batch into per-fold requests. - -### Fresh ids - -A plugin often needs to name what its own moves create: the piece a cut -leaves, the fold a join adds. Ids are handed out deterministically, so a -plugin can predict them: - -- When a plugin's moves begin, the next fresh `id` is one above the largest - region `id` in the file (both sides), and the next fresh `alignment_id` one - above the largest leaf `alignment_id`. -- A **cut** takes one fresh `alignment_id`, which the second pieces share, - then one fresh `id` for the second piece on each side that holds the leaf - or the leaf paired with it, lhs first. Both second pieces take the lhs - piece's `id` (or the only piece's) as their `fold_state_id`. The first - piece keeps the leaf's ids. -- A **join** takes one fresh `id` for the new fold on each side that holds - the listed regions, lhs first; both share the lhs fold's `id` as their - `fold_state_id`. -- No other move takes an id. - -So cutting a paired leaf when the largest id is `n` names the lhs piece -`n + 1` and the rhs piece `n + 2`. diffr carries out every plugin's moves -with the SDK's applier (`crates/diffr-plugin-sdk/src/apply.rs`), the same -code a plugin predicts ids with. - -## Host imports - -Besides WASI, which only a component gets, diffr gives every plugin one -import, `diffr:plugin/host`: - -- `git(args) -> result`: runs `git` with these arguments in - the repository's working directory (the current directory for - `--no-index`), returning stdout on success and stderr otherwise. - -A host failure (git cannot be started, or its output is not UTF-8) is the -call's error, natively and in a component alike. - -A plugin does everything else itself. It reads files: a component has the -working directory preopened, and reads any side of a file the working tree -does not have with `git show`. And it prints what it wants to say to stderr, -rather than calling diffr to say it. - -### Access - -A component gets full access; diffr does not sandbox it: - -- WASI with the working directory preopened read-write as `.`. -- The environment inherited. -- The network open, with name lookup and standard WASI HTTP/HTTPS outgoing requests. -- Its stdout and stderr captured, and written to diffr's stderr a line at a - time, each line prefixed with `[] `. Neither may reach diffr's - stdout, which is the stream. A native plugin writes to diffr's stderr - directly, and is not prefixed. - -## Writing a plugin in Rust - -`crates/diffr-plugin-sdk` is the contract in Rust, and every bundled plugin is -written with it: - -- `types`: the contract's records, generated from `wit/plugin.wit` itself by - `wit_bindgen::generate!`, so there is one definition of each. A plugin - works with the same records natively and as a component. -- `Plugin`: the one trait, mirroring the `plugin` resource. A plugin is a - struct with an `Options` type (deserialized from the options JSON), and - implements `new` (make the plugin from its options; where it can fail), - `queries`, `classify` and `mutate`. `queries` defaults to an empty list; - a plugin that does not classify returns `Ok(Vec::new())`. -- `export!("my-plugin", MyPlugin)`: built for `wasm32`, exports the plugin as the - component's `plugin` resource: its `new` deserializes the options string - into `Options` and calls `Plugin::new`, and its `classify` and `mutate` - call the instance with the records it was given, since the guest bindings' - records are the contract's. Built for a native target, it exposes the name and - constructor as `DIFFR_PLUGIN`, collected into diffr's registry at build time. - The native registry deserializes the options and calls the trait - itself. The same source builds both ways. -- `host::git`: the host function, the same call natively and in a component. -- `tree::sides` rebuilds the contract's sides as region trees - (`Pairing`, each `Region` holding its children). The SDK calls it - itself, so `mutate` receives the trees; the module also holds the helpers - the bundled plugins read them with: `walk`, `OtherSide`, `one_sided`, - `docstring_of`, `before_and_after_ids`, and the rest. -- `Draft` carries moves out on a copy of the trees as the plugin makes them, - with diffr's applier, so `draft.cut_lines`, `draft.collapse`, `draft.link` - and `draft.group` can name what earlier moves created. - `apply::Fresh::of(&sides)` predicts ids without a draft. - -```toml -# Cargo.toml -[dependencies] -diffr-plugin-sdk = { path = "../../../crates/diffr-plugin-sdk" } -serde = { version = "1.0", features = ["derive"] } -``` - -```rust -use diffr_plugin_sdk::{anyhow, export, FileEntry, Move, Pairing, Plugin, Source, ROOT}; -use serde::Deserialize; - -#[derive(Deserialize)] -#[serde(deny_unknown_fields)] -pub struct Options {} - -pub struct HideAll; - -impl Plugin for HideAll { - type Options = Options; - - fn new(_: Options) -> anyhow::Result { - Ok(Self) - } - - fn classify(&self, _: &FileEntry) -> anyhow::Result> { - Ok(Vec::new()) - } - - fn mutate(&self, _: &FileEntry, _: &Pairing) -> anyhow::Result> { - Ok(vec![Move::SetCollapsed((ROOT, true))]) - } -} - -export!("hide-all", HideAll); -``` - -Build it as a component with the pinned toolchain and the `wasm32-wasip2` -target, which emits a component directly: - -```sh -rustup target add wasm32-wasip2 -cargo rustc --release --target wasm32-wasip2 --crate-type cdylib -cp target/wasm32-wasip2/release/hide_all.wasm plugin.wasm -``` - -`cargo xtask build-plugins` builds every plugin crate in this repository -that runs as a component into its folder's `plugin.wasm`. The bundled -summarizer's component is committed because normal builds embed it without -requiring a WASM toolchain; rebuild and commit it whenever its source, manifest, -queries or SDK changes. Other plugin build outputs are not committed. -diffr compiles each component once per run, with wasmtime's -compilation cache on disk, so an unchanged component is not recompiled on the -next run. - -## Examples - -- `examples/plugins/fixtures`: `classify` tags a file `fixture` when it is - under a `fixtures/` directory or its first line is `// fixture` (read from - the blob the file entry names, with `git cat-file`, so a deleted file is - read as readily as a new one); `mutate` hides a fixture behind the subject of the last - commit that touched it (from `git log -1 --format=%s -- `), and - collapses all but the first line of its first leaf, naming the cut's piece - by predicting its id. Its `fail` option makes `mutate` return an error. -- `plugins/context`, `plugins/hide-files`, `plugins/deleted-bodies`, - `plugins/test-bodies`, `plugins/removed-runs` and `plugins/group`: the - bundled plugins, which build as components too. Pointing their entries at - the bundled folders once the script has built them runs the components in - place of the native code, and produces the same stream (`tests/wasm.rs` - checks it, with `cargo test --features wasm-plugin-tests --test wasm`: the - tests build every plugin for `wasm32-wasip2`, so they are behind a feature - and run where that target is installed): - - ```toml - [plugins.bundled.deleted-bodies] - path = "/path/to/diffr/plugins/deleted-bodies" - ``` - -- `plugins/summarize`: a network-backed component example that runs as WASM - even when bundled. Its transport - uses `wasi:http/outgoing-handler` for HTTP/HTTPS, including request headers, - streaming bodies and timeouts. Build it with `cargo xtask build-plugins`, - then load it without rebuilding diffr: - - ```toml - [plugins] - order = ["external.summarize", "bundled.test-bodies", "bundled.group"] - - [plugins.external.summarize] - path = "/path/to/diffr/plugins/summarize" - enabled = true - # Reads GEMINI_API_KEY or GOOGLE_API_KEY from the inherited environment. - ``` - - `plugins/summarize/src/http.rs` is a complete outgoing-request example for - plugin authors. The host provides TLS; components do not need to embed a - TLS implementation. Each WASM instance processes calls serially; `instances` controls the host pool. - The legacy `max_concurrency` option is still accepted but has no effect. Timeouts apply to - connection, first-byte and between-byte waits. - `cargo test --features wasm-plugin-tests --bin diffr external_component_summarizes_over_http` - checks the external component against a local model endpoint, including retries. - -Native plugin crates register their name and constructor through the same -`export!("name", Type)` macro used for WASM exports. The linker gathers the -registrations; configuration still controls instantiation and execution order. -Native crates must be dependencies linked into the host. External WASM plugins -need no native registration and do not require rebuilding diffr. - -Repository tooling discovers plugin crates through `cargo metadata` and -`[package.metadata.diffr]` in each crate's Cargo.toml. `native = true` links a -bundled implementation into diffr; otherwise the plugin uses WASM. This does -not add a WASM requirement to `cargo install`. `cargo xtask test-plugins` -builds WASM variants and runs parity tests. The workspace's -`wasm-test-exclude` list can record test-only exceptions; currently all plugins build as WASM. - -The host also discovers bundled manifests and query assets from its plugin -dependencies, rather than maintaining a second file list. A dependency marked -`native = true` contributes its registration; otherwise its prebuilt -`plugin.wasm` is embedded alongside its manifest. The default configuration -chooses which bundled plugins run; discovering an asset does not enable it. - -### Test summaries and folding order - -Enable `summarize` to get pseudocode for new functions and right-side test -bodies. `tests = false` disables test summaries independently; `test_min_lines` -(default 20) controls their size threshold. Modified and unchanged tests in a -diffed file qualify too; deleted tests do not. A summary must fit in half the -body's nonblank lines or the ordinary test fold remains. - -The default order runs `summarize` before `test-bodies`, then `group`. -`test-bodies` preserves existing labels and still collapses modules and bodies. -Opening a module or aggregate fold reveals each test's pseudocode; opening an -individual body reveals its source. JS/TS `describe` suites remain containers, -with summaries on their individual `it`/`test` callbacks. An explicit custom -`plugins.order` is respected; update it to this order to match the defaults. - -## Deferred annotations - -The 0.2 plugin ABI adds `enrich(file, sides) -> list`. Rebuild -external components against the updated SDK (`cargo xtask build-plugins` rebuilds -bundled components). SDK plugins default to returning no annotations. - -`mutate` performs initial presentation only. After all mutations finish, `enrich` -may do slow work and return `{region-id, label}` records. IDs refer to the final -region trees and are validated by the host. Annotations cannot change ranges, -alignment, fold-state IDs, or collapsed state. The summarizer reserves its folds -and links docstrings during mutation; enrichment only attaches pseudocode. -Rejected or absent summaries leave the initial fold available to expand. - -The ordinary stream still applies both phases before emitting a file. diff --git a/docs/releases.md b/docs/releases.md deleted file mode 100644 index b3a4e8610..000000000 --- a/docs/releases.md +++ /dev/null @@ -1,29 +0,0 @@ -# Releases and Homebrew - -The release workflow builds `diffr` and the standalone `diffr-tui` on Apple Silicon -macOS and x86-64 Linux. Bun 1.3.10 is required only when building. Each archive -contains both executables at its root, plus license notices. Existing consumers -can continue extracting only `diffr` using the original asset naming convention. - -Before tagging, update the root package version in Cargo.toml and its Cargo.lock -entry. Keep plugin dependency versions unchanged unless those crates also change. -The tag must match the CLI version exactly (`X.Y.Z`). Do not replace old assets: -consumers pin their checksums. - -Release packaging runs on relevant pull requests and can be run manually without -publishing. Each platform checks an extracted archive in a fresh directory with an -empty PATH, exercising settings and interactive diff rendering without Bun or the -checkout. The workflow generates `SHA256SUMS` and `diffr.rb` from both archives. -Only after validation does a tag build upload a draft release and publish it. - -If upload fails, inspect and remove the incomplete draft before rerunning the -publish job. Never delete or overwrite an already public release to retry it. - -The public `devdotfast/homebrew-tap` repository checks the latest release hourly -and installs/tests its formula before committing an update. Its workflow also has -a manual trigger. This uses the tap's own GitHub token, with no cross-repository -credential. Releases predating the `diffr.rb` asset are ignored. - -Install with `brew install devdotfast/tap/diffr`; update with `brew update` followed -by `brew upgrade devdotfast/tap/diffr`. To install manually, verify the archive with -`SHA256SUMS`, extract it, and copy both executables into the same directory on PATH. diff --git a/docs/streaming.md b/docs/streaming.md deleted file mode 100644 index 1ee98af24..000000000 --- a/docs/streaming.md +++ /dev/null @@ -1,448 +0,0 @@ -# CLI diff streaming - -```sh -diffr main HEAD --format ndjson -diffr --cached --format ndjson --syntax --order test,docs -- src/ -diffr --no-index --format ndjson -- before.rs after.rs -``` - -Spawn one process per comparison and consume stdout line by line. The comparison -arguments are the same as the [ordinary CLI](cli.md). There is no HTTP server. -`--format ndjson` is the stream described here. It requires a repository -comparison or `--no-index`; it rejects `--quiet` and metadata output flags. - -The Rust types behind this document are `src/protocol/mod.rs`; the projection -from the internal diff is `src/protocol/project.rs`, the plugin host that -shapes it is `src/plugin/`, and the bundled plugins are under `plugins/`. -WASM component plugins, and the three points where a plugin shapes a file, -are described in [plugins.md](plugins.md). - -## Configuration and ordering - -Each invocation reads the global configuration file (or `--config PATH`), -applies command-line flags, and compiles the result once; see -[config.md](config.md). Omitted keys -retain defaults. The fold query for each language is assembled from the query -files of the enabled plugins; see [plugins](#plugins). - -File tags come from bundled GitHub Linguist rules and diffr's test path rules, -then git attributes (`linguist-generated`, `linguist-vendored`, -`linguist-documentation`, `diffr-tags`); [config.md](config.md#file-tags) has -the precedence. Renames are tagged by new path; deletions by old path. - -`--order test,docs` puts files carrying those tags first, ranked by the -earliest listed tag a file carries. It can also be repeated. Files with none of -the listed tags follow, with path order breaking ties. Without `--order`, use -path order. - -Paths after `--` accept libgit2 directory prefixes and wildcard patterns, not Git -magic pathspecs. Only changed files are selected. An unmatched path yields an empty -stream. Filtering precedes rename detection, so selecting one side of a rename -can appear as an addition or deletion. `--no-renames` skips rename detection. - -## Conventions - -Stdout carries UTF-8 newline-delimited JSON, one record per line. Read complete -lines; pipe reads can split records or contain several. - -- Every enum is tagged: `type` on records, snapshots and diffs; `kind` on regions. - Tags and enum values are `snake_case`. -- Optional, empty, and default fields are omitted, never `null`. Read a missing - `visibility` as open, a missing `tags` as none, a missing `changed` as none. -- Consumers ignore unknown fields and tolerate unknown enum strings. Additions - within a `version` never change the meaning of existing fields. -- Sides are `lhs` (before) and `rhs` (after). Anything that can exist on one side - only is a *pairing*, written by presence: `{"lhs": …, "rhs": …}`, `{"lhs": …}`, - or `{"rhs": …}`. -- Lines are 0-based and split on `\n` only: an empty file has zero lines, and a - file without a trailing newline still counts its last line. Columns are 0-based - byte offsets into the UTF-8 text on the wire. Ranges are half-open. - -## Records - -| Record | Fields | Consumer action | -| --- | --- | --- | -| `start` | `version: 3`, `lhs`, `rhs`, `files` | Lay out every file up front. | -| `file` | `file`, optional `visibility`, then `diff` or `error` | Render a result, or mark the file failed. | -| `complete` | `succeeded`, `failed`, optional `aborted` | Mark complete; `aborted` means the run stopped early. | - -`lhs` and `rhs` on `start` say what is being compared: `{"type":"revision","rev":""}`, -`{"type":"index"}`, `{"type":"working_tree"}`, `{"type":"empty_tree"}` for an unborn -branch, or `{"type":"path","path":"…"}` for `--no-index`. Revisions resolve once. -Index content is pinned by blob id during discovery; working-tree files are read -as results are computed, not as an atomic snapshot. - -`files` lists every selected file in priority order: - -```jsonc -{"file": {"lhs": {"path": "src/a.rs", "oid": "3b18…", "mode": "100644"}, - "rhs": {"path": "src/a.rs", "oid": "9be2…", "mode": "100644"}}, - "status": "modified", // added | deleted | modified | renamed | copied | type_changed - "tags": ["generated", "vendored"]} // sorted; absent when none -``` - -`file` is git's delta: a deleted file has `lhs` only, an added file `rhs` only. The -path pair is the file's identity; each `file` record repeats it verbatim so the -record can be matched back to the manifest. Tags are known before `start`: -reading the start of a file for the header checks happens during discovery, -and if that read fails the file's record carries a `read_failed` error. Whether -a file starts hidden is not known yet: the plugins decide that after diffing, and -the `file` record carries it. Working-tree sides carry git's -all-zero oid. A `--no-index` comparison has empty `oid` and `mode`. - -A `file` record's `visibility` is how the file starts out, set by the plugins: -`{"collapsed": true, "label": "Generated file · hidden by default"}` for a -hidden file, absent for an open one. A file whose record is an `error` has no -`visibility`. - -Results arrive in completion order, not manifest order, since files are diffed -concurrently (`--jobs`, default 16). `--jobs 1` restores priority order. At -completion, `succeeded + failed` equals the manifest length unless `aborted` is -present. EOF without `complete` means the output was cut off. - -### Errors - -One shape everywhere: `{"code": "", "message": ""}`. - -- On a `file` record, `error` replaces `diff` and the run continues. Codes: - `not_utf8`, `unsupported_file_type` (symlinks, submodules), - `unmerged`, `read_failed`, `query_conflict` (two query files capture one - syntax node with different fold ranges; the message names the line and both - files), or `internal` for a failure diffr did not classify. -- On `complete`, `aborted` reports a run-level failure in a plugin: - `mutation_failed` when a plugin's `mutate` returns an error (the - summarizer's model call failing after its retries, say) or traps, or when a - plugin asks for a move that cannot be carried out. The message names the - plugin. diffr - stops pulling files, lets the ones in flight finish, and exits 2. Every - `file` record already written stays valid; the file that failed has no - record. -- Setup failures (bad revision, unreadable config, a query file that does not - compile, a malformed `diffr-tags` attribute, a plugin option that does not - match its schema, a plugin that cannot be made, such as a WASM plugin that - does not compile or link or a summarizer without an API key, or a - `classify` that fails or returns a malformed tag) write to stderr - and exit 2 before any record. - -Exit status is 0 on success, 1 with `--exit-code` when there are changes, 2 when -any file failed or the run aborted. - -## The diff - -```jsonc -{"type": "text", - "lhs": {"text": "…", "syntax": [...], "regions": [...]}, - "rhs": {"text": "…", "syntax": [...], "regions": [...]}, - "structural_changes": {"base": [[8, 9]], "head": [[10, 14]]}, - "stats": {"textual": {"added": 4, "removed": 1}, - "visible": {"added": 2, "removed": 1}}} // + "fallback": {code, message} on a line diff -``` - -A `binary` diff carries only `{"lhs": {"size": n}, "rhs": {"size": n}}`; either -side being binary makes the whole diff binary. Text sides are a pairing too: a -deleted file has `lhs` only. - -`text` is the complete source. `syntax` is present only with `--syntax`: every -token as `{line, start_column, end_column, capture}`, where `capture` is the -tree-sitter highlight capture name (`keyword`, `function.method`, …). Spans are -per line, sorted, and non-overlapping; where captures nest, the innermost wins. -Files that fell back to a line diff have no syntax. - -`structural_changes` records all structurally changed source lines, including -those inside collapsed regions. `base` refers to `lhs`, and `head` to `rhs`. -Each array contains zero-based, half-open `[start, end)` line ranges, sorted, -nonempty, non-overlapping, and merged when adjacent. Both arrays are present; -a missing or unchanged side has an empty array. - -A paired leaf contributes the distinct lines carrying `changed` spans. An -unpaired leaf contributes every line, including blank lines. Formatting-only -lines with no changed spans in paired leaves do not contribute. Fallback -files use the same rule over their line-diff leaves; `stats.fallback` still -identifies them. Binary diffs have no structural coverage. - -Consumers can intersect these ranges with a code selection and subtract viewed -ranges to measure remaining work. Folding must not affect that calculation. -The sum of range lengths gives complete structural counts, which may exceed -`stats.visible` and differ from `stats.textual`. - -`stats.textual` counts lines with any byte change. `stats.visible` counts the -changed lines still on screen under the default visibility: a line that carries -a `changed` span, or any line of a leaf that exists on one side only, unless it -sits inside a region that starts collapsed. It is computed after the plugins -run, so configuration changes it. It is an initial-visibility measurement, -not the complete structural total. A frontend can keep it fixed (as the TUI -does), or recompute visible counts as folds toggle. For fold-independent -progress, use `structural_changes` instead. - -`stats.fallback` is present when the AST match did not run: `unsupported_language`, -`too_large`, `too_complex`, `parse_error`, or `generated` for a file tagged -`generated`, which is always diffed by line without parsing. Its `message` is -the engine's own account of why, such as the size a file reached and the limit -it exceeded. A fallback diff is aligned by a line diff and its `changed` spans -are word-level, but the parse still stands: folds are present whenever the -language parsed (`too_complex`, `parse_error`). A line diff has no matcher, so -its folds pair with nothing; what is new inside them is still the leaves' -answer. Only `unsupported_language`, `too_large` and `generated` produce -leaves alone. - -### Regions - -Each side carries a tree of regions. A region is a line range on that side with -identities that never stand in for one another. `id` names the region: it is -unique within the file, across both sides, and it is what plugins address. -`fold_state_id` says what the region opens and closes with: regions sharing it -open and close together, on the same side or across sides. Paired leaves and -matched folds share it across sides, and a plugin that links regions (a -docstring with its function body, say) gives several regions one -`fold_state_id` while each keeps its own `id`. Only a leaf has an -`alignment_id`, and it is row alignment: the same value on the other side marks -the leaf whose rows line up with this one, line for line. Consumers key the row -zip by leaf `alignment_id`, collapse state by `fold_state_id`, and anything -about the region itself by `id`. -Today the bundled link is a docstring: a plugin that collapses a function body -(`deleted-bodies`, `test-bodies`, `summarize`) links the body's docstring, a -fold tagged `deleted-bodies:docstring`, `test-bodies:docstring` and (when the -summarizer is on) `summarize:docstring`, to it. The docstring then carries the body's -`fold_state_id` and starts collapsed, with an empty label. A docstring whose -body no plugin collapses is not linked. - -```jsonc -{"id": 7, "fold_state_id": 7, "kind": "fold", - "start": {"line": 18, "column": 0}, "end": {"line": 53, "column": 0}, - "tags": ["deleted-bodies:function", "removed-runs:function"], - "visibility": {"collapsed": false, "label": ""}, - "children": [ - {"id": 8, "fold_state_id": 8, "kind": "leaf", "alignment_id": 5, "start": {"line": 18, "column": 0}, "end": {"line": 30, "column": 0}}, - {"id": 9, "fold_state_id": 9, "kind": "leaf", "alignment_id": 6, "start": {"line": 30, "column": 0}, "end": {"line": 31, "column": 0}, - "changed": [{"line": 30, "start_column": 8, "end_column": 9}]}, - {"id": 10, "fold_state_id": 10, "kind": "leaf", "alignment_id": 7, "start": {"line": 31, "column": 0}, "end": {"line": 53, "column": 0}}]} -``` - -**Leaves** tile the file: read in order, their line ranges cover every line once. -They always start and end at column 0. A leaf whose `alignment_id` a leaf on the -other side also carries is paired: the two have the same line count and their -rows pair line for line. A -leaf on one side only has no counterpart, and the other side shows blank rows -against it. The row table is the walk over both sides' leaves, zipped by -`alignment_id`. Paired leaves come in the same order on both sides. - -`changed` holds the byte ranges inside a leaf that should be painted as changed: -a line that is entirely new carries one span covering it, a changed word inside -an otherwise matching line carries just that word. A line with no span in a leaf -that has spans is a changed line whose tokens all matched elsewhere. Blank -changed lines carry no span. - -**Folds** are regions with `children`. Their `start` and `end` are the hull of -their children, which tile it exactly, so a fold's range is whole lines: exactly -the lines collapsing it hides. A fold covers only the lines it holds whole. A -body fold starts on the line after the `{` or `:` that opens it, because the -header line holds code the fold does not cover; that line is the last line of -the leaf before the fold. The same at the end: a fold stops above the line its -`}` sits on, whether the brace is in column 0 or not. A fold whose node begins -its line, such as a comment block or an import, starts on that line. Regions form -a strict tree: every child lies inside its parent's range and siblings never -overlap; where the parser still hands over two folds that cross on one line, -the earlier one gives that line to the later. Two queries that fold the same lines, -from one plugin or two, make one fold carrying both their tags: a fold is a -region of the file, and it belongs to the innermost syntax node that covers -that region, whose pairing it takes. Two folds that share `fold_state_id` -across sides are matched: the syntax nodes they were built on are the same node -on both sides, as the matcher paired them, whose contents may differ. A fold is -paired exactly when a region on the other side shares its `fold_state_id`. A -syntactic region that -spans a single line is not a region: it hides nothing. - -`tags` name what a region is. A leaf carries none. A fold carries -the tags the fold queries set on it, each written `:` after the -plugin whose query set it (`deleted-bodies:function`, `test-bodies:test`, -`context:scope`); a fold only some query made exist without meaning -carries none. Tags describe syntax, and are the only syntax a plugin sees; -how a region starts out is `visibility`, -and frontends need no knowledge of tags to render it. `visibility` is -`collapsed` and the `label` to show while collapsed: a line count, a -pseudocode summary, or the count for a collapsed stretch of unchanged lines. -A label is the plugin's that collapsed the fold, so a fold no plugin -collapsed has none, and the empty label shows the source. Absent means open. - -**Context** comes from the `context` plugin; the projection itself hides -nothing. A line stays open when it is within `plugins.bundled.context.lines` -(`-U`, default 3) of a changed line, when it is paired with such a line, or -when it opens or closes a scope, such as the function a change -sits in, that contains a change. Scopes are folds the plugin's own queries -tag `context:scope`, on the construct's own node: a scope runs from the line -its signature starts on to the line that closes it, so keeping its first and -last line always shows where it opens and where it ends, however many lines -the signature takes. The body fold inside it covers the body alone and starts -a line later, so the two never share a range. Every other stretch of unchanged paired lines that -is at least three lines long collapses with a label such as -`"142 unchanged lines"`; shorter stretches stay open because a row would save -nothing. A file with no change collapses whole however short. - -Region edges cut a stretch. The part of it among one list of siblings -collapses when it is at least three lines long or is the whole stretch, so a -sliver at a fold edge stays open. A part inside one leaf is that leaf's lines, -cut out. A part that spans several siblings, folds among them, collapses under -one new fold with no tags, labelled with the part's line count. A fold collapses only when it -lies wholly inside the stretch, and so does every region on the other side in -its fold state, so a fold whose matched counterpart holds changes stays open. -Both folds of a matched pair take the label. The group is made only when the -two sides' siblings match one for one (leaves sharing an `alignment_id`, or -folds sharing a fold state), and then on both sides, the two group folds -sharing a fold state; -otherwise the regions collapse one by one. A stretch that crosses the end of a fold collapses on -each side of that edge. - -**Groups** come from the `group` plugin: a run of two or more sibling -regions that start collapsed, whichever plugin collapsed them, is wrapped in -one new fold, with no tags. Between two collapsed regions of a run there may -be open leaves spanning at most two lines in all, such as a blank line and the -next function's header. The label counts the collapsed regions and the lines -the group spans: `"3 collapsed regions · 42 lines"`. It starts collapsed; -expanding it reveals each child's own row. The wrapped children are untouched. -When the other side pairs a region of the run, the run is grouped only if the -other side holds a run that matches it region for region, and then both are -grouped with one fold state. - -Ids are per file. The projection numbers the regions as it builds them, in lhs -preorder then rhs preorder, from two counters: `id` starts at 1 and -`alignment_id` at 0. Every region takes the next `id`, and every leaf takes -the next `alignment_id` unless -its counterpart on the other side was numbered first, whose `alignment_id` it -takes instead. A region's `fold_state_id` is its own `id`, except that a paired -leaf or matched fold numbered second takes its counterpart's `fold_state_id`. -Plugins take `id`s and `alignment_id`s above every one already in the file. -No region has `id` 0: plugins use it to name the file itself. Ids mean nothing -across files or runs. - -## Computation and output - -Discovery and rename detection finish before `start`; syntax matching is lazy. -A pool of `--jobs` workers pulls files from the iterator: each worker reads the -next file's sources under a lock, then diffs them while other workers pull -further files. The calling thread serializes, writes and flushes each record. -A bounded queue holds one ready record, so computation overlaps slow writes -without collecting the entire comparison. In-flight files are bounded by the -pool size, not their individual size. - -Closing stdout stops production once the files in flight finish. Terminate the -process to cancel immediately. The CLI also retains its normal SIGPIPE behavior -on Unix. - -Regular UTF-8 text files are supported. Binary files containing NUL bytes -produce size-only binary diffs. Non-UTF-8 text files, symlinks, submodules and -unmerged index entries produce per-file errors. Non-UTF-8 paths fail discovery. - -## Plugins - -Each file goes through one pipeline before its record is written: - -1. Git lists the file and its tags ([config.md](config.md#file-tags)), and - each enabled plugin's `classify` adds tags of its own, before the - `start` header is written ([plugins.md](plugins.md#the-plugin-points)). -2. The diff runs. Its fold query for the file's language is assembled from - the query files of every enabled plugin ([config.md](config.md#queries)). -3. The projection builds the region trees above and pairs them. Nothing - starts collapsed yet. -4. The enabled plugins run in `plugins.order` on the finished trees, starting - with `context`. -5. Complete `structural_changes` and initially visible coverage are collected - together. The latter supplies `stats.visible`, and the record is written. - -Queries decide which regions exist; plugins decide how they start out. No -plugin matches one side to the other: pairing is the projection's alone. A -plugin counts a leaf as paired when the other side holds its `alignment_id`, -and a fold as paired when a region on the other side shares its -`fold_state_id`. Only the other side counts, since a link shares a fold state -within one side too. Whether a fold is new, deleted or unpaired is a -different question, and the leaves answer it: a fold is new when no line -inside it pairs with the other side, whatever the matcher made of the two -nodes, and however the file was diffed. - -A plugin's `mutate` step reads the file (its paths, status and tags) and both -sides, as the records of `wit/plugin.wit`, and returns moves. It never edits a -tree itself; one applier, the plugin SDK's, carries the moves out, in order, -and the next plugin sees the result. A move names a -region by its `id`, which is on one side only, or the file by `0`, the root -every region hangs from: - -- **Cut** a leaf at a line offset, relative to its first line and strictly - inside it. The leaf paired with it (the one on the other side with its - `alignment_id`) is cut at the same offset. The first piece on each side - keeps its leaf's ids; the second takes a fresh `id` of its own and a fresh - `alignment_id` shared with its counterpart, so the rows still pair, and the - lhs piece's `id` as its `fold_state_id`. A fold, or the file, cannot be - cut. -- **Join folds**: wrap consecutive sibling regions in a new fold, open, - unlabelled and without tags. Each side wraps the listed regions it holds, - so one move can list a run and the run matched with it on the other side; - the two new folds take their own `id`s and share one `fold_state_id`. -- **Link fold state**: every region in the listed regions' fold states, on - both sides, takes the first region's `fold_state_id` and whether it starts - collapsed, so they open and close together. -- **Set collapsed** on a region: every region sharing its `fold_state_id` - starts collapsed, or open. On the file, whether the file starts hidden: the - record's `visibility`. -- **Set label** on one region, or on the file: the label shown while it is - collapsed. -- **Set tags** on one region, replacing them. The file's `tags` are its - manifest entry's, known before any diff runs, so the file's cannot be set. - -Collapsing a region behind a label is a set collapsed and a set label (after -a cut, for some lines of a leaf); grouping is a join, a set collapsed and a -set label on each new fold; hiding a file is a set collapsed and a set label -on the file. When the bundled plugins collapse a region, the regions that -start collapsed with it and were open lose their labels, and the -leaf paired with a collapsed leaf takes its label. - -New `id`s are larger than every `id` already in the file, and new -`alignment_id`s larger than every leaf's. A move that cannot be carried out -(an unknown region, an offset outside a leaf, a cut on a fold, a join of -regions that are not consecutive siblings, or of a run one side holds only -one region of) is an error: the run aborts with `mutation_failed`. - -The bundled plugins, in their default order, each a folder under `plugins/`: - -| Plugin | What starts out differently | -| --- | --- | -| `context` | Unchanged lines far from any change and outside its enclosing headers collapse: `"142 unchanged lines"`. | -| `hide-files` | Files with a listed tag (`generated`, `vendored`, `test`), and deleted files, are hidden: `"Generated file · hidden by default"`. | -| `deleted-bodies` | A function body of at least 12 lines with nothing paired under it collapses: `"20 lines removed"`. Its docstring is linked to it. | -| `test-bodies` | Test function bodies (`"test body"`) and Rust `#[cfg(test)]` modules (`"test module"`) collapse on both sides. A test body's docstring is linked to it. | -| `removed-runs` | A one-sided leaf of at least 5 lines in removed (not rewritten) code keeps its first and last line open and collapses the rest: `"8 lines removed"`. | -| `summarize` | Off by default. A new function body of at least 20 lines collapses behind plain pseudocode from the configured model. The body is chosen as new before its docstring is linked to it; the pseudocode may quote the docstring. On without an API key, diffr stops before the stream starts. | -| `group` | Two or more adjacent collapsed regions are wrapped in one collapsed fold: `"3 collapsed regions · 42 lines"`. An open fold showing only collapsed regions and at most two open lines, such as a function's scope around its collapsed body and closing brace, counts as collapsed. | - -A binary diff has no regions, so only `hide-files` can change how it starts -out. - - -## Progressive annotations (opt-in v4) - -`diffr main HEAD --format ndjson --stream-annotations` emits `start.version = 4`. -Without the flag the v3 stream continues to emit fully enriched files, for -existing consumers including the TUI. - -In v4, each successful `file` contains the complete source, initial folds, -alignment, and authoritative structural changed-line ranges. It is flushed -before enrichment starts. Text files subsequently receive an `annotations` -event, identified by the same file pair: - -```json -{"type":"annotations","file":{"rhs":{"path":"a.rs","oid":"...","mode":"100644"}},"annotations":[{"region_id":12,"label":"validate input\nwrite result"}]} -``` - -Apply each label to the existing region. Do not rebuild the editor or replace -collapsed state, selection, or scroll position. The update never changes region -IDs, fold-state IDs, line ranges, alignment, or counts. An empty annotations list -is valid. If enrichment fails, the event instead has an empty list and an -`error` with code `enrichment_failed`; keep showing the initial file. - -Events for different files may interleave, but a file always precedes its -annotations. `complete` follows all annotation work. Its succeeded/failed totals -count initial file results, not annotation updates. An annotation error still -sets process exit status 2, independently of `--exit-code`. - -Parsing and enrichment use separate workers with a bounded queue. This keeps -slow model requests off parsing workers while bounding retained file trees; -when enrichment falls behind, the queue applies backpressure. Disconnecting -stops queued work; already-running plugin calls finish under their own timeouts.