Skip to content

Let people download a Whisper model, or not - #37

Closed
savvasioakeim wants to merge 2 commits into
feat/qa-capture-and-dictationfrom
feat/whisper-dictation
Closed

savvasioakeim wants to merge 2 commits into
feat/qa-capture-and-dictationfrom
feat/whisper-dictation

Conversation

@savvasioakeim

Copy link
Copy Markdown
Owner

Stacked on #34. Base is feat/qa-capture-and-dictation, so review that one first; this diff is only the Whisper work.

Dictation's Web Speech path is gone on Windows (WebView2 Runtime 153+) and it is not ours to fix, so recognition has to run on this machine. That means a model has to get here somehow, and the decision this PR encodes is that it is never the installer's call. Nothing is bundled. A model arrives only when someone presses a button, and every row states its size and its trade-off before the download rather than after.

Backend — src-tauri/src/whisper.rs

The four sizes OpenAI publishes as safetensors, with whisper_models, whisper_download and whisper_delete.

Deliberate choices worth reviewing:

  • Partial downloads cannot masquerade as installed. Each file lands as <name>.part and is renamed only when its stream ends; "installed" means all three real files exist. An interrupted download reads as absent rather than as a model that fails later at load time, where the cause would be far less obvious.
  • Progress uses real sizes. Totals come from a HEAD per file, not from the catalogue figure, so the bar cannot sit at 100% with bytes still arriving, or stop at 94%.
  • Events are throttled to roughly one per megabyte. Per chunk would be thousands of events for a 1GB file.
  • Double execution is refused, not tolerated. A second download of the same model would interleave two streams into the same .part files; it is rejected. Deletion is refused while a download runs. The in-flight claim is released by a Drop guard, so an early ? return cannot strand it.

Catalogue sizes are measured, not estimated

HEAD against the real URLs, all three files counted:

tiny base small medium
total 153 MB 292 MB 969 MB 3058 MB

These are fp32 weights, which is why they are several times the quantised GGUF files people may have seen. Quantised builds exist, but only tiny is published in a form candle reads today, and tiny is where multilingual accuracy collapses. The compression is available exactly where it is useless.

Frontend

The picker lives under Settings › Speech-to-text and appears only once Whisper is the selected engine. Delete asks first and names the size it reclaims. The progress listener is torn down on unmount, and a row's busy state follows the download promise rather than the event stream, which can stop arriving without the download being finished.

Not in this PR

Recognition itself: microphone capture and the candle decode loop. The settings copy says so plainly rather than implying the engine works, and the engine dropdown still marks Whisper as unavailable.

Verification

  • cargo check and cargo clippy clean, no warnings from whisper.rs.
  • npx tsc --noEmit and npm run build pass.
  • The download URLs and sizes in the table above were fetched live, which is also how two catalogue figures got corrected before commit.
  • Not yet exercised end to end: an actual multi-hundred-MB download, so the throttling and the .part rename are reasoned rather than observed.

🤖 Generated with Claude Code

savvasioakeim and others added 2 commits October 27, 2026 08:01
Dictation's Web Speech path is gone on Windows (WebView2 Runtime 153+), and
it is not ours to fix, so recognition has to run on this machine. That means
a model has to get here somehow, and the decision this commit encodes is
that it is never the installer's call. Nothing is bundled. A model arrives
only when someone presses a button, and every row states its size and its
trade-off before the download rather than after.

Backend (whisper.rs): the four sizes OpenAI publishes as safetensors, with
commands to list, download and delete. A few things were deliberate.

Each file lands as `<name>.part` and is renamed only when its stream ends,
and "installed" means all three real files exist. An interrupted download
therefore reads as absent rather than as a model that would fail later at
load time, when the cause would be far less obvious.

Progress is measured against sizes fetched by HEAD rather than against the
catalogue figure, so the bar cannot sit at 100% with bytes still arriving.
Events are throttled to roughly one per megabyte; per chunk would be
thousands for a 1GB file.

A second download of the same model is refused rather than allowed to
interleave two streams into the same .part files, and deletion is refused
while a download is running. The claim is released by a Drop guard so an
early return cannot strand it.

Sizes in the catalogue were measured against the real URLs, not estimated:
153/292/969/3058 MB for tiny/base/small/medium, all three files included.

Frontend: the picker lives under Settings › Speech-to-text and only appears
once Whisper is the selected engine. Delete asks first and names the size it
will reclaim. The progress listener is torn down on unmount, and a row's busy
state follows the download promise rather than the event stream, which can
stop arriving without the download being done.

Recognition itself is not here yet. The settings copy says so rather than
implying the engine works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the engine behind the model picker: capture, mel, encoder,
language detection and a greedy decode, all local. Audio never leaves the
machine and no service can withdraw it the way the webview's did.

mic.rs takes audio through cpal rather than getUserMedia. The webview has
already proved unreliable for speech, and a native device also avoids a
second permission prompt per window, since the QA window and the main window
are separate webviews asked separately. Channels are averaged in the
callback — keeping interleaved stereo would double the buffer for nothing,
Whisper being mono. A recording is capped at five minutes so a forgotten
button cannot grow the buffer until memory runs out.

Resampling to 16kHz is linear, which is enough: these devices run at 44.1k or
48k, so it is always downsampling, and Whisper's mel front end bins to 80
bands under 8kHz, far above any interpolation error. It is covered by tests
because an off-by-one would pitch-shift the audio and wreck every transcript
with nothing in the output pointing at the cause.

whisper.rs decodes greedily rather than running candle's temperature
fallback: dictation is someone speaking deliberately into a microphone, and
the fallback costs up to six decodes per window for cases that barely arise
here. Audio longer than 30 seconds is walked window by window, with the last
one zero-padded as the model was trained to expect. Language is detected once
on the first window and held, because re-detecting per window lets a quiet
stretch flip mid-sentence into another language.

One bug fixed before it ever ran. Language detection masked the logits from
<|startoftranscript|> to <|endoftext|>, but in Whisper's vocabulary eot is
50257 and sot is 50258 — the end sits BELOW the start, so the range was
always empty and detection would have failed every time. The bound is
<|translate|>, which sits immediately after the 99 language tokens. Deriving
the range from those two tokens also keeps this file from carrying a list of
99 language codes that could drift out of step with the vocabulary.

The model stays loaded between dictations: reading `small` off disk costs
seconds that would otherwise be paid per phrase. One model is cached, and
switching size evicts the previous rather than holding both, since the whole
point of choosing a size is a machine that cannot afford both. Recognition
runs on a blocking thread; on the async runtime it would stall every other
command.

Not yet verified: the decode loop has never run against real weights. The
test that does it exists and is ignored (OCTOSHELL_WHISPER_DIR), but the run
was killed when the machine ran out of memory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@savvasioakeim

Copy link
Copy Markdown
Owner Author

Recognition is now in this branch too (120d535)

The PR title and description above cover only the model picker; this commit adds the engine behind it. Capture → mel → encoder → language detection → greedy decode, all local.

mic.rs takes audio through cpal rather than getUserMedia. The webview has already proved unreliable for speech, and a native device also avoids a second permission prompt per window — the QA window and the main window are separate webviews, asked separately. Channels are averaged in the callback, and a recording is capped at five minutes so a forgotten button cannot grow the buffer until memory runs out.

Resampling to 16kHz is linear and covered by tests. An off-by-one there would pitch-shift the audio and wreck every transcript with nothing in the output pointing at the cause, which is exactly the kind of failure worth pinning down with a test rather than an eyeball.

whisper.rs decodes greedily rather than running candle's temperature fallback: dictation is someone speaking deliberately into a microphone, and the fallback costs up to six decodes per window for cases that barely arise here. Audio past 30 seconds is walked window by window with the last zero-padded. Language is detected once on the first window and held — re-detecting per window lets a quiet stretch flip mid-sentence into another language.

The model stays loaded between dictations, one at a time; switching size evicts the previous rather than holding both, since the point of choosing a size is a machine that cannot afford both. Recognition runs on a blocking thread.

A bug caught before it ever ran

Language detection masked the logits from <|startoftranscript|> to <|endoftext|>. In Whisper's vocabulary eot is 50257 and sot is 50258 — the end sits below the start, so the range was always empty and detection would have failed on every single run. The correct bound is <|translate|>, immediately after the 99 language tokens. Deriving the range from those two tokens also keeps this file from carrying a list of 99 language codes that could drift out of step with the vocabulary.

What is NOT verified

The decode loop has never run against real weights. cargo check, cargo clippy and the resampling tests are clean, and tiny was downloaded for the purpose, but the run that would have exercised the pipeline was killed when the machine ran out of memory while building in release.

The test exists and is ignored:

OCTOSHELL_WHISPER_DIR=".../whisper/tiny" cargo test --manifest-path src-tauri/Cargo.toml --release whisper::tests -- --ignored --nocapture

It drives a real tensor through every stage and prints the audio-to-wall-clock ratio, which is also the only honest way to answer how slow a given model size is on a given machine. Do not merge this until that has been run and a human has dictated a sentence through the UI. Tensor reshapes compile whatever the dimensions are; a wrong one surfaces at runtime, or worse, as fluent nonsense.

🤖 Generated with Claude Code

@savvasioakeim

Copy link
Copy Markdown
Owner Author

Closing this unmerged. The engine works — the decode path was finished and fixed today, and it does transcribe real speech — but it is not worth shipping.

What it took to get it running. Two bugs, both invisible to the compiler:

  • out.i((0, seq - 1)) dropped the batch and sequence axes and left a bare [384], so final_linear got [1, 384] against [1, 384, 51865]. Range-slicing instead of indexing keeps all three axes. The same mistake sat in detect_language, where the indexing was not needed at all.
  • Language detection picked <|fi|> for Greek and <|es|> for English. The token range was right (99 languages, 50259–50357, checked against the model's own tokenizer.json); the small models are simply poor at the guess, and the captured audio peaked at 0.03–0.19 of full scale, which is where detection fails first.

Why it is still not worth it. Measured on an i7-8750H, after moving the inference crates to opt-level = 3 in both profiles:

model audio wall clock ratio
tiny 10.0s 6.7s 0.67×
base 9.5s 12.8s 1.35×

Greek only becomes usable around small, which extrapolates past 4× realtime. The quality worth having sits beyond the point where the speed collapses, and no setting bridges that.

What replaced it. #38: the real bug was that we asked the speech service for navigator.language, so an English Windows requested English recognition for Greek speech. With the language named, the free web engine is accurate and immediate.

The branch stays if local recognition is ever worth revisiting — quantized weights, or CUDA, would change the arithmetic. The measurements above are the bar it would have to clear.

@github-actions github-actions Bot locked and limited conversation to collaborators Oct 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant