diff --git a/.gitignore b/.gitignore index e5e84b764..3dd2ea4a7 100644 --- a/.gitignore +++ b/.gitignore @@ -36,6 +36,7 @@ video-materials/ node_modules/ dist/ target/ +!/openless-all/app/src-tauri/src/ask_window/target/ # 本地从 CI / Actions 下载的 APK 及解压目录(非源码) ci-artifacts/ *.apk diff --git a/docs/ATTRIBUTION.md b/docs/ATTRIBUTION.md index d45898632..ca4bdd1ae 100644 --- a/docs/ATTRIBUTION.md +++ b/docs/ATTRIBUTION.md @@ -12,3 +12,14 @@ The overview yearly activity card is a port of Rare UI's GitHub Activity compone The port keeps the original geometry: 11px cells, 3px gap, 28px corner radius, 16px padding, week and month layout, the hover tooltip, and the expandable bottom panel. Local dictation counts replace GitHub contributions. The software with the most voice input (name and count) replaces repositories. Cells are not redrawn or resized to fit a narrow column. The in-app credit is Settings → About and updates → Component credits. The repository READMEs (`openless-all/README.md` and `openless-all/README.zh.md`) carry the same credit. + +## Rare UI Toast + +The standalone Ask window uses OpenLess's shared toast API with the [Rare UI Toast](https://www.rareui.com/components/toast) panel, glyphs, and tile transitions (Copyright (c) 2026 Swami Malode, [MIT + Commons Clause + Attribution](../openless-all/app/public/licenses/rare-ui-LICENSE.txt)). OpenLess uses its per-window interaction controller instead of Sonner's global store. + +## Rare UI Prompt Box + +The standalone Ask window adapts Rare UI's [Prompt Box demo](https://rareui.com/components/promptbox). OpenLess retains its layout and animations, adds a small attachment slot, and replaces scripted responses with the configured provider's responses. The upstream files in `openless-all/app/third_party/rare-ui` come from [`33f7ce51`](https://github.com/swamimalode07/rare-ui/tree/33f7ce51e5a41fb7caef66b7aad0812b5bb2ab17). They retain their upstream license and are included as part of the app, not offered as a component library. + +- Rare UI code: Copyright (c) 2026 Swami Malode, [MIT + Commons Clause + Attribution](../openless-all/app/public/licenses/rare-ui-LICENSE.txt). +- Open Runde fonts: [SIL Open Font License 1.1](../openless-all/app/public/licenses/open-runde-OFL.txt). diff --git a/docs/ask-standalone-contract.md b/docs/ask-standalone-contract.md new file mode 100644 index 000000000..f755e60e3 --- /dev/null +++ b/docs/ask-standalone-contract.md @@ -0,0 +1,26 @@ +# 「随便问」附件、上下文和工具契约 + +窗口、快捷键、提交时选区、意图分流和写回见 [统一入口契约](ask-unified-entry-contract.md)。本文保留附件、上下文和模型工具的共享约定。 + +## 后端端口与上下文来源 + +- **桌面 App 元数据 → ContextProvider**:在切到 Ask 前冻结显式唤起时的来源 `{ invocationId, capturedAt, foregroundApp?, desktopApps? }`。当前真正接收键盘输入的只有前台对象;后台用可见/最小化窗口所属 App 或已打开的用户 App 元数据表达,按平台能力区分 `unknown`。不把后台当成另一份前台焦点,不读取后台窗口内容、标题、浏览历史或所有进程;不假设能跨虚拟桌面枚举。Ask 已前台的后续轮使用该对话最近一次有效外部来源及时间,不把主窗/胶囊当成用户正在工作的 App。 +- **`ContextProvider`**:按轮提供 `staticInstructions`(固定且版本化)和带 `source/asOf` 的 `facts`,缺失单独标原因。用户“今天在做什么”只来自已授权的本地信息或用户指定的公开检索,搜索不等于能看见本机活动。时间使用设备时钟;天气只查询用户明确给定或授权的地点,保留观测/预报的时间与来源;无地点、禁止搜索或离线时不猜。现有用量记录未提供可用的 Ask 实际金额,首版显示「无数据」,不拿 token 乘现价,也不访问个人账单或外部账户。App/用量范围须有 Ask 专属授权;开启联网、保存历史或操作系统授予可读权限不算授权。被撤销的隐藏事实不能作为后续请求或 Compact 的输入;这不等于抹掉用户已看到的聊天消息。 +- **`SearchTool`**:输入限定为模型 `web_search` 工具调用中的短公开查询和调用 ID,每轮至多一次;全局搜索设置仍可关闭。默认「免费搜索」在客户端合并 FreeSerp 与 Keenable 的免 Key 结果,遵守端点限流,不通过 OpenLess 服务转发。「混合搜索」额外使用用户单独配置的 DeepSeek Search API Key,可能计费;原有 Bing 或支持 JSON 的 SearXNG 地址仍可按需选择。结果通过供应商协议的工具消息返回,不混入普通提示词;私有附件和应用上下文不参与搜索决策。复用超时、网络保护和取消;不把整个 Prompt、历史、选区、附件、账单或 App 清单发给搜索服务。实际发出请求才进入 `searching`,实际读取才进入 `reading`,只有成功取回可用正文才标 `read`。搜索发现的来源和读取状态随活会话回放;部分读取失败不抹掉已发现来源。返回给模型的证据正文有预算,不放进公开事件或额外的持久化记录。模型读取的网页、选区、PDF 与动态事实始终是数据,不是系统指令。 +- **`AttachmentReader`**:按 `{ conversationId, turnId, attachmentIds }` 一次性取得所有已就绪附件的只读引用,解析为选区原文、带真实页码(从 1 开始)的 PDF 文本或经验证的图像。选区完整保留到 4,000 个 Unicode 标量或 16 KiB;预览先显示前 280 个标量,展开看原文,超限提示、不静默截断。大图片优先在本机压缩;保留 48 MiB / 40 MP 的解码保护。PDF 本地处理最多 48 MiB / 100 页 / 512 KiB 提取文本,不上传原件。每轮最多 4 件。可恢复的操作错误复用主界面的弹窗 API。PDF 首版只本地提取有文字层的文档,解析器必须有可中止的资源限制;扫描件、加密、损坏和超限分开报错,不执行脚本、访问外链或做 OCR。图片首版验证 JPG/PNG,通过实际协议和模型能力双重校验后才能发送;能力不明也保留草稿并报告,不探测上传、不换渠道、不当空附件。首版可在内部按页读取,不把页范围放到公开 `submit`。文件使用选取时有界读取的内容,不在发送时重开源路径。结束对话清理临时资源,不自动删除用户原文件或既有 opt-in 历史行;「保存历史」沿用旧的文本记录范围,不新增长期保存附件原件、原文、缩略图或路径。 +- **模型和 Compact**:复用现有 QA 生命周期的所有权检查及供应商底层传输,旧单槽/选区润色接口不直接接新窗口。每轮冻结已配置渠道、模型及能力;不假冒深度思考、视觉和工具支持,不静默切换供应商。固定指令与稳定历史放在前,动态事实和本轮材料放在后;遵守 [现有缓存策略](prompt-cache-policy.md)。**256K 指总窗最多 256,000 tokens**,取与供应商确认的模型限额较小者,再扣除输出、未计入的工具结果和安全余量。未知窗口使用明确的保守限制,不把输出 `max_tokens` 当成总窗;搜索回填后再计量。Compact 只替换模型输入中的旧完整轮次,保留 UI 全量消息和来源、新近完成轮及当前轮。摘要有对话与版本边界,必须完整、更小且在当前轮有效时才替换;过大附件不静默裁剪,压缩失败不修改旧摘要。KV 命中率按 provider 可比实测 `Σ缓存读入 tokens / Σ输入 tokens`,在同服务/渠道/模型/协议内聚合;缺字段显示未知,不写死百分比,不为预热多发请求。按实际调用归档已知用量,不把答复、压缩等多次请求算作同一次;聊天视图可把辅助请求与答复分开。 +- **工具扩展**:聊天核心定义可取消、按 `{ conversationId, turnId, toolCallId }` 跟踪的类型化工具分派接口;常规任务注册 `web_search` 与 `calculator`,主模型工具判定阶段单独开放 `resolve_intent`。提示词优先核实时效性强或易变的公开事实;需要计算时先核实公开输入,再用本地 Rust 计算器。计算器只接受有界的数值表达式,返回精确整数或最简分数及最多 50 位小数,每轮最多三次,关闭联网仍可用;计算结果作为工具消息,不随搜索查询外发。不写锁屏、切用户、开 App 或切换主题的实现,不给模型通用 Shell、系统控制或调用特权。未来的系统操作需独立权限、用户确认及审计契约。 + +## PromptBox 内的附件呈现 + +新截图参考的是**附件在输入框里的呈现**,不是把录屏里的聊天布局复制过来。预览、引用标签和编辑区属于同一个 PromptBox 边框;不能把独立附件条放到整个输入框上方。 + +| 类型 | 框内呈现 | +| --- | --- | +| 图片 | 显示真实图片缩略图,角标如 `#1`,右上角有移除按钮;编辑区前可见对应的「图片 #1」引用标签,光标可以接着输入。图片加载失败时显示原因,不把空白方框当作成功预览。 | +| PDF | 用同一框内样式显示 PDF 图标、文件名和移除按钮,已知时可显示页数;编辑区的引用标签能识别这是哪个 PDF,不冒充图片预览。 | +| 选中文字 | 经提交时验证且本轮确实使用的原文可作为本轮私有材料;不在唤起时自动添加草稿卡片,也不把原文塞进可编辑的问题草稿。普通粘贴不变。 | + +标签是附件的视觉引用,不是发给模型的字面文本;提交仍用 `attachmentIds`。卡片与标签指向同一附件,移除后两者同步消失,同时保留草稿与光标。布局可以加高框内内容,但不改变独立窗口尺寸。 + +同一个 PromptBox 边框内提供预览和引用,优先保留原组件自身的边框与样式。拖入图片时只接收浏览器提供的文件内容,不接受或转发本机路径;后端与原生选择器共用验证和登记逻辑。拖放只添加附件,不发送,不打开本地文件网页。纯附件用同一个发送入口;中文输入法选词回车不触发。不显示联网按钮或确认面板。模型调用工具时在答复旁显示 Websearch;完成后仍可查看来源。用户消息下不显示档位标签,开启思考后按钮始终为蓝色。 diff --git a/docs/ask-unified-entry-contract.md b/docs/ask-unified-entry-contract.md new file mode 100644 index 000000000..6b38456d5 --- /dev/null +++ b/docs/ask-unified-entry-contract.md @@ -0,0 +1,51 @@ +# Ask Anything unified entry: desktop contract + +This contract supersedes the previous standalone behavior for windows, hotkeys, selection, submission and delivery. The existing [standalone contract](ask-standalone-contract.md) remains the reference for attachments, search, model/cache limits and context consent where compatible. Scope: macOS and Windows Ask only; no new HTTP API, no legacy popup, no changes to ordinary dictation or non-Ask editing. + +## User flow + +The current recording binding plus Space opens or focuses the `ask-anything` window with only the Rare UI PromptBox visible. The collapsed composer has no enclosing backdrop or close button, even if the external app has selected text. After an explicit submission is acknowledged and its authoritative route is `answer` or `selectionAnswer`, the same native window expands around the same input. The underlay starts at the measured prompt bounds and grows from the bottom on a curved transition to the original single-window 800 × 920 chat layout; reduced motion uses the final layout directly. Write, edit and clarify do not trigger expansion. Escape dismisses and cancels pending voice input without clearing the draft or cancelling an accepted turn; the expanded surface also has the platform-positioned close button. Reopening returns to the compact view, and replay cannot reopen a dismissed surface. + +After a composer-focus acknowledgement for the current window/editor generation, the hotkey may launch the existing independent capsule. `autoVoiceOnOpen` defaults to true (including missing keys); it is applied only to the recording-key chord and never to a settings/menu open. Single-key dictation while the Ask composer is focused also uses the existing capsule. Both are insert-only, scoped to recording ID, conversation, editor instance/revision and focus generation; preserve unrelated draft text and current IME behavior. Repeated chords during active or starting recording are a focus-only action, never a stop or send. Losing the editor, changing conversation, cancellation, or a stale focus acknowledgement cannot send or insert late audio/transcript. No upload of Ask text or selection to ASR. + +At explicit composer send, capture the latest verifiable external editor/document and selection, not an invocation-time selection. Remember external target identity at the focus boundary; if the user visits a different app/control before returning, use the newest eligible observation and do not fall back to an older control after an unreadable observation. Query the bound AX/UIA element at submit without activation or Clipboard/Copy. Keep native ownership, validation baseline, cursor/range and generation in the host. Do not report unavailable as no selection. Protected inputs and unsupported controls are unavailable; bounded capture can support read-only selection without authorizing it for writing. An unrelated selection is never appended to the draft or sent to the model. A selection may be included as an untrusted, ephemeral content item only for a resolved selected-text answer or edit. Existing App context consent and explicit attachment processing stay in effect. + +| Route (wire) | Definition and destination | +| --- | --- | +| `answer` | An answer independent of external selection. Uses the existing chat, with no external selection content. | +| `selectionAnswer` | Interpret, explain, summarize or translate relevant selected text for inspection. Chat output, no write. Explicit reference with no verified selected text becomes `clarify`. | +| `write` | Produce new ready-to-use body text. Insert at a verified collapsed external caret if still unchanged; otherwise retain in the composer and copy. In particular, never replace an incidental nonempty selection. | +| `edit` | Explicitly change existing selected text, including polite interrogatives. Replace only the verified original range. No verified selection becomes `clarify`; never edits a draft inside Ask. | +| `clarify` | Ask for selection or clarify ambiguous intent in the composer. Never changes transcript visibility or sends outside Ask. | + +A question mark is not a routing signal. “Make this shorter, please?” edits; “what does this mean?” answers. `translate and show me` answers; `translate and replace` edits. For “look up the exchange rate, calculate the price, then write a quote”, final deliverable is `write`. If distinct final deliverables are incompatible and user did not choose, clarify. Attachment-only requests retain the default answer behavior, subject to the chosen classifier and capability checks. + +## Frozen boundaries + +### Settings and admission + +Machine-local, persisted typed values: `{ autoVoiceOnOpen: boolean, intentMode: "semanticHybrid" | "separateModel" | "mainModelTool" }`. Defaults: `{true, semanticHybrid}`. There is one mode enum, not two authoritative booleans. Both Beta off means hybrid; enabling either disables the other; turning the selected Beta off restores hybrid. Main Settings only: `ask_get_entry_settings()` and `ask_set_entry_settings({ settings })`, returning committed settings. Coordinator owns command/ACL registration and loading the same preference state for host and turn preparation. Worker D owns the store and Settings rendering, not plumbing inside existing `chat.rs`/`lib.rs`. + +Keep composer `submit({ conversationId, clientRequestId, text, attachmentIds, options:{ deepThinking } }) -> { turnId }` and `cancel({conversationId,turnId})`; remove voice recording proof/auto-send from the final path. Same key and payload returns original ID before capture; same key and different payload conflicts. Reject busy subjects before reading any new target. Host-only lazy preparation runs inside the winning admission, captures the submit-time target once before model calls, and freezes mode, validated attachments and channel. Existing callers may retain a compatibility wrapper for the plain submit in non-desktop unit harnesses. Workers may choose internal Rust names consistent with wire. Coordinator owns IPC `chat.rs` / `wire.rs`, command registries and combining capture/delivery with the admission callback; no paths, raw native handles or target fingerprints cross IPC. + +### Host-only capture and delivery + +Worker B will supply `ask_window/target.rs` exposing a desktop-local `TargetTracker` with `new()`, owned `start()`/`stop()` lifecycle, async bounded `capture()` and delivery operations; it may choose its observer/platform submodule internals. `capture()` returns a `CapturedTarget` with `selection: ExternalSelection` (`None | Selected(String) | Unavailable`) and an optional opaque `WriteTarget`, separate from selection so readonly text is usable. Start and arm outside the low-level key callback; coordinator owns initializing it in shared desktop/host state. `WriteTarget` is immutable and safe to move into a blocking delivery job. Async delivery takes `Insert` vs `Replace`, text, and a one-shot authorization callback passed by the core (claim immediately before native write, after any reactivation/revalidation). Outcome distinguishes `Written`, `NotWritten`, `Unknown` and a declined claim; never fire Enter, refocus the then-current third-party editor, fall back to replace-all, or retry an uncertain write. Capture of protected/unknown/racy focus fails closed. If nonfocused AX/UIA read cannot demonstrably confirm the same original element, report unavailable; don't reactivate it just to sample. In normalized content, preserve original selected text/UTF-16 range semantics (not substring search) and line breaks. + +Core worker A exposes host-injectable lazy admission input (`intent_mode`, selection status and optional verified selected text, plus a closure/trait object carrying any opaque host target), and a writer port taking requested insert/replace, plain generated text and a one-shot claim function. Coordinator adapts these ports to B. Clipboard and transcript/window side effects are not in B. For a write/edit with no delivery or `NotWritten`/`Unknown`, first retain the complete non-chat result in the core's per-turn projection, then attempt clipboard copy exactly once via a separate host port; copy failure leaves the result. Only one delivery claim per turn and only while active. Cancel before claim prevents write/copy; after claim report the observed outcome, no blind retry or imaginary rollback. Never save a write/edit to chat history or use it as a later editable draft. No external write or clipboard activity for chat/clarify. + +### Projection/events and model + +Continue one ordered sequence per conversation, replay and snapshot. Add a non-chat submission state (`currentTurnId`, `route`, existing activity/search state, `resultText`, `delivery`, `copyStatus`, structured error); preserve last retained result until next submission or explicit session end. UI can restore this on reopen without reenacting any external effect. Broadcast `submitted` (non-message admission) and `routeResolved { route }` for all accepted turns; the existing message `accepted`/answer delta events are exclusively chat routes after routing. Delivery progress/result is a per-turn upsert (`delivery` event, same data as snapshot), statuses `pending`, `written`, `retained`, `unknown`; clipboard is `notNeeded`, `pending`, `copied`, `failed`. Existing `activity`/`search`/`source`/`terminal` events stay conversation- and turn-scoped. The single Ask window shows incremental replies only for chat. Its expansion gate requires an acknowledged submission from the current visibility generation and that same turn’s authoritative chat route. The scoped desktop size command validates the route before resizing. A recovered route can expand the current submission even if live events lag, but recovery cannot expand a dismissed request. CSS/viewport details belong to worker C; coordinator owns native shared event emit/allowlist and command registration. + +`semanticHybrid` uses conservative local intent cues (not web search) and calls the configured main model for a small typed decision only when uncertain. `separateModel` makes one isolated classifier request with no transcript output. `mainModelTool` requires the configured main model to call a registered local `resolve_intent` tool before any user-facing output and resumes the same logical model conversation with the validated routing result; protocol tool continuation may use subsequent requests but not a standalone classifier session. If classifier result/choice is invalid or protocol cannot support the mode, show a localized composer error; do not silently switch model strategy. Classifier inputs are user instruction and minimum selection-presence/reference signals, not raw external text, app content, attachments or history. Tool arguments are untrusted decisions, not permission to write. A hidden classification result never becomes a chat message or renderable draft. After routing, normal question/selection and write/edit generation share existing chat provider/tool limits; edit uses the actual selection as data, body-only output for writes. Preserve private-context search isolation and existing public-only search decision if required; never expose selected text/attachment/history/target to web search. `Web Search` indicator is driven by actual tool status in the active surface, not by routing or speculative planning. + +## Ownership and acceptance + +- A: Core standalone Ask runtime, prompt and provider adapters (not shared registration, settings, host or UI); tests for all routes/modes, tools and late/cancelled delivery. +- B: Native external focus observer, target capture/verification and actual insert/replace in `ask_window/target*` and strictly necessary native target helpers; no app/router/voice IPC or window work. +- C: Ask window frontend and native window/voice/hotkey modules. Keep `ask_window/chat.rs`, `wire.rs`, `lib.rs`/ACL to I1. Preserve typing, attachment, ordinary dictation and focus contracts. +- D: Ask entry setting store and Settings UI/localization, targeted settings tests; no Core/router or shared native registration. I1 registers both commands and shares the same store instance with capture/admission and key routing. +- I1: Cross-workstream IPC, registrations, result clipboard adapter, coordinator glue, integration/review, baseline lockfile and publication. No worker pushes or opens a PR. No archive snapshots are pushed with implementation. + +Acceptance: deterministic contracts for four routes and clarification, all three modes and failure/no fallback, unrelated choice not leaking selection, real search then numeric calculation then writing without opening a chat, duplicate/cancel/late callback and clipboard failure. macOS and Windows: rebind recording key, open bottom only, switch to an external editor and change selection, return and submit the new selection, confirm capsule only fills, verify caret insertion and selection replacement, mutate target during generation and verify retained/copy only. Existing ask answer, attachments, search and ordinary dictation must remain functional. Review native runtime, prompt and IPC together, commit and wait for required CI on PR #1469; no automatic merge. diff --git a/openless-all/app/Cargo.lock b/openless-all/app/Cargo.lock index 86ba7f195..cd7d5a03e 100644 --- a/openless-all/app/Cargo.lock +++ b/openless-all/app/Cargo.lock @@ -159,6 +159,21 @@ dependencies = [ "memchr", ] +[[package]] +name = "alloc-no-stdlib" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" + +[[package]] +name = "alloc-stdlib" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" +dependencies = [ + "alloc-no-stdlib", +] + [[package]] name = "allocator-api2" version = "0.2.21" @@ -736,6 +751,16 @@ dependencies = [ "piper", ] +[[package]] +name = "brotli-decompressor" +version = "5.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", +] + [[package]] name = "bumpalo" version = "3.20.3" @@ -1085,6 +1110,12 @@ dependencies = [ "libc", ] +[[package]] +name = "core_detect" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48" + [[package]] name = "coreaudio-rs" version = "0.14.2" @@ -1482,6 +1513,15 @@ dependencies = [ "rustfft", ] +[[package]] +name = "ecb" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26f2a8b3e564eba0877223dc343703ad0385794e882e6d13f3a4dd5c6b1f41ac" +dependencies = [ + "cipher", +] + [[package]] name = "ecdsa" version = "0.16.9" @@ -1680,6 +1720,20 @@ dependencies = [ "bytemuck", ] +[[package]] +name = "encoding_rs" +version = "0.8.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89253175e704ab869b04f9db224d38630e0211a53841a79181deae0889e169f5" +dependencies = [ + "cfg-if", + "core_detect", + "multiversion_no_op", + "rustversion", + "scopeguard", + "simdutf8", +] + [[package]] name = "endi" version = "1.1.1" @@ -2728,6 +2782,8 @@ dependencies = [ "num-traits", "png", "tiff", + "zune-core", + "zune-jpeg", ] [[package]] @@ -3067,6 +3123,33 @@ version = "0.4.34" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" +[[package]] +name = "lopdf" +version = "0.47.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6ad532ff9262dd4bfd6e085400cbb39959d48cb8d08518b01b68f77bdf6feca" +dependencies = [ + "aes", + "bitflags 2.13.2", + "brotli-decompressor", + "cbc", + "ecb", + "encoding_rs", + "flate2", + "getrandom 0.4.3", + "indexmap", + "itoa", + "log", + "md-5", + "nom 8.0.0", + "rand 0.10.3", + "rangemap", + "sha2 0.11.1", + "stringprep", + "thiserror 2.0.21", + "weezl 0.2.1", +] + [[package]] name = "lru-slab" version = "0.1.3" @@ -3207,6 +3290,12 @@ dependencies = [ "pxfm", ] +[[package]] +name = "multiversion_no_op" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d" + [[package]] name = "munge" version = "0.4.7" @@ -3912,10 +4001,15 @@ dependencies = [ "getrandom 0.4.3", "hex", "hmac 0.13.0", + "image", "jsonwebtoken", "log", + "lopdf", "md-5", "micyou-dsp", + "num-bigint", + "num-rational", + "num-traits", "once_cell", "parking_lot", "rand 0.8.8", @@ -4758,6 +4852,12 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "rangemap" +version = "1.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a611d15b50743feb4c76b7d03edcb0e64f399c26961e4efe6975bc398be6aa3d" + [[package]] name = "raw-window-handle" version = "0.6.2" @@ -5599,6 +5699,17 @@ dependencies = [ "float-cmp", ] +[[package]] +name = "stringprep" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" +dependencies = [ + "unicode-bidi", + "unicode-normalization", + "unicode-properties", +] + [[package]] name = "strsim" version = "0.11.1" @@ -5797,7 +5908,7 @@ dependencies = [ "flate2", "half", "quick-error", - "weezl", + "weezl 0.1.12", "zune-jpeg", ] @@ -6210,9 +6321,9 @@ dependencies = [ [[package]] name = "unicode-properties" -version = "0.1.4" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" +checksum = "993ad287eb20e8ba755312801cfe1413e0f25d604075863acaddca10b04c1833" [[package]] name = "unicode-script" @@ -6607,6 +6718,12 @@ version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a28ac98ddc8b9274cb41bb4d9d4d5c425b6020c50c46f25559911905610b4a88" +[[package]] +name = "weezl" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4ca08e5ef825b65b056d9efbd95c8750683f0a6d0466d02e96dc2e4e360f3d2" + [[package]] name = "wgpu" version = "30.0.1" diff --git a/openless-all/app/android/frontend/components/AndroidPermissionsPanel.tsx b/openless-all/app/android/frontend/components/AndroidPermissionsPanel.tsx index d728c2d4a..95dee60d8 100644 --- a/openless-all/app/android/frontend/components/AndroidPermissionsPanel.tsx +++ b/openless-all/app/android/frontend/components/AndroidPermissionsPanel.tsx @@ -621,7 +621,7 @@ export function AndroidPermissionsPanel({ mode = 'all' }: AndroidPermissionsPane up: 'cancel', down: 'none', left: 'translation', - right: 'qa', + right: 'none', }), [direction]: action, })); @@ -635,7 +635,6 @@ export function AndroidPermissionsPanel({ mode = 'all' }: AndroidPermissionsPane 'translation', 'style_pack', 'cancel', - 'qa', 'voice_edit', ] as const ).map((action) => ( diff --git a/openless-all/app/android/frontend/lib/androidTypes.ts b/openless-all/app/android/frontend/lib/androidTypes.ts index 14303fc22..b995fc90e 100644 --- a/openless-all/app/android/frontend/lib/androidTypes.ts +++ b/openless-all/app/android/frontend/lib/androidTypes.ts @@ -6,7 +6,7 @@ export type AndroidOverlayActivationMode = 'tap' | 'long_press'; export type AndroidOverlayLeftSwipeAction = 'translation' | 'style_pack'; export type AndroidOverlayCancelSwipeDirection = 'up' | 'down'; export type AndroidOverlayGestureAction = - 'none' | 'quick_note' | 'translation' | 'style_pack' | 'cancel' | 'qa' | 'voice_edit'; + 'none' | 'quick_note' | 'translation' | 'style_pack' | 'cancel' | 'voice_edit'; export interface AndroidOverlayGestureActions { up: AndroidOverlayGestureAction; diff --git a/openless-all/app/android/kotlin/OpenLessAndroidPreferences.kt b/openless-all/app/android/kotlin/OpenLessAndroidPreferences.kt index 4b15c0314..79f0a070b 100644 --- a/openless-all/app/android/kotlin/OpenLessAndroidPreferences.kt +++ b/openless-all/app/android/kotlin/OpenLessAndroidPreferences.kt @@ -25,7 +25,7 @@ object OpenLessAndroidPreferences { private val VALID_OVERLAY_LEFT_SWIPE_ACTIONS = setOf("translation", "style_pack") private val VALID_OVERLAY_CANCEL_SWIPE_DIRECTIONS = setOf("up", "down") private val VALID_OVERLAY_GESTURE_ACTIONS = - setOf("none", "quick_note", "translation", "style_pack", "cancel", "qa", "voice_edit") + setOf("none", "quick_note", "translation", "style_pack", "cancel", "voice_edit") fun overlayTriggerMode(context: Context): String? { val value = readPreferenceString(context, KEY_OVERLAY_TRIGGER) ?: return null @@ -70,7 +70,9 @@ object OpenLessAndroidPreferences { if (!file.isFile) continue try { val actions = JSONObject(file.readText()).optJSONObject(KEY_OVERLAY_GESTURE_ACTIONS) - val value = actions?.optString(direction, "")?.takeIf { + val stored = actions?.optString(direction, "") + if (stored == "qa") return "none" + val value = stored?.takeIf { it in VALID_OVERLAY_GESTURE_ACTIONS } if (value != null) return value @@ -84,7 +86,7 @@ object OpenLessAndroidPreferences { "up" -> if (legacyCancel == "up") "cancel" else "none" "down" -> if (legacyCancel == "down") "cancel" else "none" "left" -> legacyLeft - "right" -> "qa" + "right" -> "none" else -> "none" } } diff --git a/openless-all/app/android/kotlin/OpenLessNative.kt b/openless-all/app/android/kotlin/OpenLessNative.kt index 467904784..17ba7870f 100644 --- a/openless-all/app/android/kotlin/OpenLessNative.kt +++ b/openless-all/app/android/kotlin/OpenLessNative.kt @@ -118,7 +118,6 @@ object OpenLessNative { @JvmStatic external fun nativeSwitchStylePack() - @JvmStatic external fun nativeFinalizeQaFromOverlay() @JvmStatic external fun nativeFinalizeVoiceEditFromOverlay() diff --git a/openless-all/app/android/kotlin/OpenLessOverlayService.kt b/openless-all/app/android/kotlin/OpenLessOverlayService.kt index 473dcb049..2b9c184d0 100644 --- a/openless-all/app/android/kotlin/OpenLessOverlayService.kt +++ b/openless-all/app/android/kotlin/OpenLessOverlayService.kt @@ -658,7 +658,6 @@ class OpenLessOverlayService : Service(), OpenLessOverlayBridge.OverlayStateList "quick_note", "translation", "style_pack" -> applyVisualState(OverlayVisualState.Processing) - "qa", "voice_edit" -> applyVisualState(OverlayVisualState.Processing) "cancel" -> applyVisualState(OverlayVisualState.Error) else -> Unit @@ -674,7 +673,6 @@ class OpenLessOverlayService : Service(), OpenLessOverlayBridge.OverlayStateList switchStylePackFromOverlay() if (recording) stopRecordingFromOverlay() } - "qa" -> finalizeQaFromOverlay() "voice_edit" -> finalizeVoiceEditFromOverlay() "cancel" -> cancelRecordingFromOverlay(direction) else -> Unit @@ -711,22 +709,6 @@ class OpenLessOverlayService : Service(), OpenLessOverlayBridge.OverlayStateList } } - private fun finalizeQaFromOverlay() { - try { - Log.i(TAG, "finalize QA from overlay") - OpenLessNative.nativeFinalizeQaFromOverlay() - recording = false - processing = true - setArmed(false) - applyVisualState(OverlayVisualState.Processing) - } catch (error: Throwable) { - Log.w(TAG, "finalize QA bridge unavailable", error) - processing = false - applyVisualState(OverlayVisualState.Error) - showToast("问答服务未就绪,请打开 OpenLess 后重试") - } - } - private fun finalizeVoiceEditFromOverlay() { if (!recording) return try { diff --git a/openless-all/app/ask-window/ask-window.tsx b/openless-all/app/ask-window/ask-window.tsx new file mode 100644 index 000000000..a44c6f481 --- /dev/null +++ b/openless-all/app/ask-window/ask-window.tsx @@ -0,0 +1,391 @@ +import { ToastNotificationCenter } from '../src/ui/feedback/ToastNotificationCenter'; +import { AppViewport } from '../src/ui/layout/AppViewport'; +import { InteractionProvider } from '../src/ui/overlays/InteractionProvider'; +import { useCallback, useEffect, useRef, useState, type CSSProperties } from 'react'; +import { errorMessage, type AskClient, type AskFocus, type AskSnapshot } from './client'; +import { useComposer } from './composer'; +import { ConversationLayout, Sources, SubmissionResult } from './conversation'; +import { connectAskFeed } from './feed'; +import { useAskNotice } from './notices'; +import { watchTheme } from './theme'; +import './window.css'; + +type SessionProps = { + client: AskClient; + snapshot: AskSnapshot; + connected: boolean; + check: () => void; + focus?: AskFocus; + activation: number; + canDeliverVoice: () => boolean; + expanded: boolean; + captureSubmitReceipt: () => (turnId: string) => void; +}; + +type Expansion = { width: number; height: number; bottom: number }; + +function ComposerSession(props: SessionProps) { + const box = useComposer(props); + const { client, snapshot } = props; + const submission = snapshot.submission; + const current = submission?.currentTurnId === snapshot.activeTurnId; + const chatRoute = submission?.route === 'answer' || submission?.route === 'selectionAnswer'; + const [sourcesTurn, setSourcesTurn] = useState(); + const sourceTurns = [ + ...(props.expanded ? snapshot.turns : []), + ...(submission && !chatRoute + ? [ + { + turnId: submission.currentTurnId, + searchStatus: submission.searchStatus, + sources: submission.sources, + }, + ] + : []), + ]; + return ( + <> + + {submission && !chatRoute && } + {box} + + setSourcesTurn(undefined)} + /> + + ); +} + +export function AskWindow({ client }: { client: AskClient }) { + return ( + + + + + + ); +} + +function AskScreen({ client }: { client: AskClient }) { + const notice = useAskNotice(); + const feed = useRef>(); + const shown = useRef(true); + const hiding = useRef(false); + const reconnect = useRef<() => void>(() => {}); + const presentation = useRef(null); + const visibilityGeneration = useRef(0); + const expanding = useRef(); + const [receipt, setReceipt] = useState<{ conversationId: string; turnId: string }>(); + const [expansion, setExpansion] = useState(); + const [snapshot, setSnapshot] = useState(); + const [connected, setConnected] = useState(false); + const [windowFocused, setWindowFocused] = useState(() => document.hasFocus()); + const [focus, setFocus] = useState(); + const [activation, setActivation] = useState(0); + const [streamError, setStreamError] = useState(); + const [platform, setPlatform] = useState(); + const onWindows = platform ? platform === 'windows' : /Windows/i.test(navigator.userAgent); + + const suspend = useCallback(() => { + shown.current = false; + feed.current?.disconnect(); + feed.current = undefined; + setConnected(false); + }, []); + + const clearPresentation = useCallback(() => { + visibilityGeneration.current++; + expanding.current = undefined; + setReceipt(undefined); + setExpansion(undefined); + }, []); + + const dismiss = useCallback(() => { + if (hiding.current || !shown.current) return; + hiding.current = true; + visibilityGeneration.current++; + suspend(); + void client + .hide() + .then(clearPresentation) + .catch((cause) => { + shown.current = true; + notice(errorMessage(cause, 'Could not hide Ask.'), 'error', 'ask-hide'); + reconnect.current(); + }) + .finally(() => { + hiding.current = false; + }); + }, [clearPresentation, client, notice, suspend]); + + useEffect(() => { + function onEscape(event: KeyboardEvent) { + if ( + event.key !== 'Escape' || + event.defaultPrevented || + event.isComposing || + event.keyCode === 229 || + event.repeat + ) + return; + event.preventDefault(); + dismiss(); + } + window.addEventListener('keydown', onEscape); + return () => window.removeEventListener('keydown', onEscape); + }, [dismiss]); + + useEffect(() => { + let destroyed = false; + let connection = 0; + let unlisten: (() => void) | undefined; + const theme = watchTheme(); + const updateFocus = ({ focusEpoch, focusRequest }: Partial) => { + if (focusEpoch === undefined || focusRequest === undefined) return; + setFocus((previous) => + previous && + (previous.focusEpoch > focusEpoch || + (previous.focusEpoch === focusEpoch && previous.focusRequest >= focusRequest)) + ? previous + : { focusEpoch, focusRequest }, + ); + }; + function connect() { + const attempt = ++connection; + feed.current?.disconnect(); + setConnected(false); + feed.current = connectAskFeed( + { + ...client, + async current() { + const current = await client.current(); + if (!destroyed && shown.current && attempt === connection) { + if (current.themeMode) theme.setTheme(current.themeMode); + if (current.platform) setPlatform(current.platform); + if (current.notice) notice(current.notice.message, 'warning', 'ask-window-notice'); + updateFocus(current); + } + return current; + }, + }, + (view) => { + setSnapshot(view); + setStreamError(undefined); + setConnected(true); + }, + (cause) => { + setStreamError(errorMessage(cause, 'Could not connect to Ask.')); + setConnected(false); + }, + ); + } + reconnect.current = connect; + const resume = () => { + if (hiding.current || document.visibilityState === 'hidden') return; + setWindowFocused(document.hasFocus()); + if (!shown.current || !feed.current) { + shown.current = true; + connect(); + } else { + feed.current.check(); + } + }; + const preventFileNavigation = (event: DragEvent) => { + if (event.dataTransfer?.types.includes('Files')) event.preventDefault(); + }; + const onBlur = () => setWindowFocused(false); + function onVisibilityChange() { + if (document.visibilityState === 'hidden') suspend(); + else resume(); + } + void client + .onWindowEvent((event) => { + if (destroyed) return; + if (event.type === 'hidden') { + clearPresentation(); + suspend(); + } + if (event.type === 'activated' || event.type === 'sessionChanged') { + updateFocus(event); + if (event.type === 'activated') { + const wasHidden = !shown.current; + shown.current = true; + hiding.current = false; + setActivation((previous) => previous + 1); + setWindowFocused(document.hasFocus()); + if (wasHidden || !feed.current) connect(); + else feed.current.check(); + } else if (shown.current) { + connect(); + } + } + if (event.type === 'themeChanged') theme.setTheme(event.themeMode); + if (event.type === 'notice') notice(event.error.message, 'warning', 'ask-window-notice'); + }) + .then((stop) => { + if (destroyed) stop(); + else { + unlisten = stop; + if (!feed.current && shown.current) connect(); + } + }) + .catch((cause) => { + if (!destroyed) { + notice( + errorMessage(cause, 'Could not connect to the Ask window.'), + 'error', + 'ask-connection', + ); + if (!feed.current && shown.current) connect(); + } + }); + window.addEventListener('focus', resume); + window.addEventListener('blur', onBlur); + window.addEventListener('dragover', preventFileNavigation); + window.addEventListener('drop', preventFileNavigation); + document.addEventListener('visibilitychange', onVisibilityChange); + const checkTail = window.setInterval(() => { + if (document.visibilityState === 'visible') feed.current?.check(); + }, 3000); + return () => { + destroyed = true; + feed.current?.disconnect(); + feed.current = undefined; + unlisten?.(); + theme.stop(); + window.clearInterval(checkTail); + window.removeEventListener('focus', resume); + window.removeEventListener('blur', onBlur); + window.removeEventListener('dragover', preventFileNavigation); + window.removeEventListener('drop', preventFileNavigation); + document.removeEventListener('visibilitychange', onVisibilityChange); + }; + }, [clearPresentation, client, notice, suspend]); + + const route = snapshot?.submission?.route; + const routedTurn = snapshot?.submission?.currentTurnId; + const conversationId = snapshot?.conversationId; + useEffect(() => { + if ( + expansion || + !connected || + !shown.current || + !conversationId || + !receipt || + receipt.conversationId !== conversationId || + receipt.turnId !== routedTurn || + (route !== 'answer' && route !== 'selectionAnswer') || + expanding.current === routedTurn + ) + return; + const frame = presentation.current; + const box = frame?.querySelector('[data-slot="prompt-box"]'); + if (!frame || !box) return; + const start = box.getBoundingClientRect(); + const bounds = frame.getBoundingClientRect(); + const from = { width: start.width, height: start.height, bottom: bounds.bottom - start.bottom }; + const generation = visibilityGeneration.current; + expanding.current = receipt.turnId; + void (client.expandChat?.(conversationId, receipt.turnId) ?? Promise.resolve()) + .then(() => { + if (shown.current && visibilityGeneration.current === generation) setExpansion(from); + }) + .catch((cause) => { + if (shown.current && visibilityGeneration.current === generation) { + notice(errorMessage(cause, 'Could not expand Ask.'), 'error', 'ask-expand'); + } + }); + }, [client, connected, conversationId, expansion, notice, receipt, route, routedTurn]); + + function captureSubmitReceipt() { + const generation = visibilityGeneration.current; + return (turnId: string) => { + if (shown.current && visibilityGeneration.current === generation && conversationId) { + setReceipt({ conversationId, turnId }); + } + }; + } + + const expansionStyle = + expansion && + ({ + '--ask-start-width': `${expansion.width}px`, + '--ask-start-height': `${expansion.height}px`, + '--ask-start-bottom': `${Math.max(0, expansion.bottom)}px`, + } as CSSProperties); + return ( +
+ {expansion && + ); +} diff --git a/openless-all/app/ask-window/attachments.tsx b/openless-all/app/ask-window/attachments.tsx new file mode 100644 index 000000000..df9882bca --- /dev/null +++ b/openless-all/app/ask-window/attachments.tsx @@ -0,0 +1,246 @@ +import { AskNotice } from './notices'; +import { FileText, Image as ImageIcon, Quote, X } from 'lucide-react'; +import { useEffect, useLayoutEffect, useRef, useState } from 'react'; +import { errorMessage, type AskClient, type AttachmentInfo } from './client'; + +function AttachmentCard({ + client, + conversationId, + attachment, + label, + removing, + disabled, + remove, +}: { + client: AskClient; + conversationId: string; + attachment: AttachmentInfo; + label: string; + removing: boolean; + disabled: boolean; + remove: () => void; +}) { + const { attachmentId, kind, displayName, state, preview, error } = attachment; + const [image, setImage] = useState(); + const [loadedImage, setLoadedImage] = useState(); + const [expanded, setExpanded] = useState(false); + const [fullText, setFullText] = useState(); + const [failure, setFailure] = useState(); + const [loading, setLoading] = useState(false); + const hasThumbnail = preview?.kind === 'image' && preview.thumbnailAvailable; + const excerpt = preview?.kind === 'selection' ? preview : undefined; + const lifetime = useRef(false); + + useEffect(() => { + lifetime.current = true; + return () => { + lifetime.current = false; + }; + }, []); + + useEffect(() => { + if (kind !== 'image' || state !== 'ready' || !hasThumbnail) return; + let current = true; + void client + .getAttachmentPreview(conversationId, attachmentId) + .then((result) => { + if (!current) return; + if (result?.kind !== 'image' || !['image/png', 'image/jpeg'].includes(result.mimeType)) { + throw new Error('This thumbnail is not a supported image.'); + } + setImage(`data:${result.mimeType};base64,${result.dataBase64}`); + }) + .catch((cause) => { + if (current) setFailure(errorMessage(cause, 'Could not load the thumbnail.')); + }); + return () => { + current = false; + }; + }, [client, conversationId, attachmentId, kind, state, hasThumbnail]); + + async function toggleExcerpt() { + setExpanded(!expanded); + if (expanded || fullText !== undefined || loading) return; + setLoading(true); + setFailure(undefined); + try { + const result = await client.getAttachmentPreview(conversationId, attachmentId); + if (result?.kind !== 'selection') throw new Error('The original selection is unavailable.'); + if (lifetime.current) setFullText(result.text); + } catch (cause) { + if (lifetime.current) + setFailure(errorMessage(cause, 'Could not load the original selection.')); + } finally { + if (lifetime.current) setLoading(false); + } + } + + return ( +
+ + {kind === 'image' ? ( + <> +
+ ); +} + +export function AttachmentDeck({ + client, + conversationId, + attachments, + removing, + disabled, + remove, +}: { + client: AskClient; + conversationId: string; + attachments: AttachmentInfo[]; + removing: ReadonlySet; + disabled: boolean; + remove: (attachmentId: string) => void; +}) { + const numbers = useRef(new Map()); + const frame = useRef(null); + const content = useRef(null); + + useLayoutEffect(() => { + const outer = frame.current; + const inner = content.current; + if (!outer || !inner) return; + // Measure only the intrinsic content, never the frame whose height is animating. + const measure = () => { + outer.style.height = `${inner.offsetHeight}px`; + }; + measure(); + outer.dataset.measured = 'true'; + const observer = new ResizeObserver(measure); + observer.observe(inner); + return () => observer.disconnect(); + }, []); + + const labels = attachments.map((attachment) => { + if (attachment.kind === 'pdf') return `PDF: ${attachment.displayName}`; + if (attachment.kind === 'selection') return 'Selected text'; + if (!numbers.current.has(attachment.attachmentId)) { + numbers.current.set(attachment.attachmentId, numbers.current.size + 1); + } + return `Image #${numbers.current.get(attachment.attachmentId)}`; + }); + + return ( +
+
+ {attachments.length > 0 && ( + <> +
+ {attachments.map((attachment, index) => ( + remove(attachment.attachmentId)} + /> + ))} +
+
    + {attachments.map((attachment, index) => ( +
  • + {labels[index]} +
  • + ))} +
+ + )} +
+
+ ); +} diff --git a/openless-all/app/ask-window/client.ts b/openless-all/app/ask-window/client.ts new file mode 100644 index 000000000..2d066e1df --- /dev/null +++ b/openless-all/app/ask-window/client.ts @@ -0,0 +1,220 @@ +export type AskError = { code: string; message: string; reason?: string }; +export type AttachmentKind = 'image' | 'pdf' | 'selection'; + +export type AttachmentInfo = { + attachmentId: string; + kind: AttachmentKind; + displayName: string; + state: 'processing' | 'ready' | 'error'; + pageCount?: number; + sourceApp?: string; + preview?: + | { kind: 'image'; width: number; height: number; thumbnailAvailable: boolean } + | { kind: 'selection'; text: string; hasMore: boolean }; + error?: AskError; +}; + +export type AttachmentPreview = + | { kind: 'image'; mimeType: 'image/jpeg' | 'image/png'; dataBase64: string } + | { kind: 'selection'; text: string }; + +export type SearchStatus = + 'idle' | 'searching' | 'reading' | 'done' | 'no_results' | 'error' | 'cancelled'; + +export type AskSource = { + sourceId: string; + url: string; + title: string; + snippet?: string; + readStatus: 'not_attempted' | 'reading' | 'read' | 'blocked' | 'failed' | 'cancelled'; +}; + +export type AskOptions = { + deepThinking: boolean; +}; + +export type SubmitAsk = { + conversationId: string; + clientRequestId: string; + text: string; + attachmentIds: string[]; + options: AskOptions; +}; + +export type AskFocus = { focusEpoch: number; focusRequest: number }; + +export type AskEditor = { + conversationId: string; + mountId: string; + editorId: string; + revision: number; + editable: boolean; + selectionStart: number; + selectionEnd: number; +}; + +export type WorkingStatus = 'queued' | 'thinking' | 'writing' | 'compacting'; +export type TerminalStatus = 'completed' | 'cancelled' | 'failed'; +export type AskMessage = { messageId: string; text: string }; +export type AskRoute = 'answer' | 'selectionAnswer' | 'write' | 'edit' | 'clarify'; + +export type AskTurn = { + turnId: string; + userMessage: AskMessage; + assistantMessage?: AskMessage; + attachments: Pick[]; + deepThinking: boolean; + status: WorkingStatus | TerminalStatus; + searchStatus: SearchStatus; + searchError?: AskError; + sources: AskSource[]; + error?: AskError; +}; + +export type AskSnapshot = { + conversationId: string; + sequence: number; + activeTurnId: string | null; + turns: AskTurn[]; + attachments: AttachmentInfo[]; + submission?: AskSubmission | null; +}; + +export type AskSubmission = { + currentTurnId: string; + route?: AskRoute | null; + status: WorkingStatus | TerminalStatus; + searchStatus: SearchStatus; + searchError?: AskError | null; + sources: AskSource[]; + resultText?: string | null; + delivery?: 'pending' | 'written' | 'retained' | 'unknown' | null; + copyStatus: 'notNeeded' | 'pending' | 'copied' | 'failed'; + error?: AskError | null; +}; + +export type AskEvent = { conversationId: string; sequence: number } & ( + | { type: 'submitted'; submission: AskSubmission; attachmentIds?: string[] } + | { type: 'routeResolved'; turnId: string; route: AskRoute } + | { type: 'delivery'; submission: AskSubmission } + | { type: 'turnAccepted'; turn: AskTurn } + | { type: 'answerDelta'; turnId: string; messageId: string; text: string } + | { type: 'turnStatus'; turnId: string; status: WorkingStatus } + | { + type: 'searchChanged'; + turnId: string; + toolCallId?: string; + status: SearchStatus; + sources?: AskSource[]; + error?: AskError; + } + | { type: 'sourceChanged'; turnId: string; source: AskSource } + | { type: 'usageChanged'; turnId: string } + | { type: 'turnFinished'; turnId: string; status: TerminalStatus; error?: AskError } + | { type: 'attachmentChanged'; attachment: AttachmentInfo } + | { type: 'attachmentRemoved'; attachmentId: string } +); + +export type AskReplay = { events: AskEvent[]; latestSequence: number; truncated: boolean }; +export type AskTheme = 'light' | 'dark' | 'system'; +export type AskWindowEvent = + | ({ type: 'activated' | 'sessionChanged'; conversationId?: string | null } & Partial) + | { type: 'hidden' } + | { type: 'themeChanged'; themeMode: AskTheme } + | { type: 'notice'; error: AskError }; +export type AskVoiceEvent = Omit & { + recordingId: string; + focusEpoch: number; +} & ({ type: 'started' } | { type: 'cancelled' } | { type: 'transcribed'; text: string }); + +export interface AskClient { + current(): Promise< + Partial & { + conversationId: string | null; + themeMode?: AskTheme; + platform?: string; + notice?: AskError | null; + } + >; + newConversation(): Promise<{ conversationId: string }>; + endConversation(conversationId: string): Promise; + snapshot(conversationId: string): Promise; + events(conversationId: string, afterSequence: number): Promise; + submit(request: SubmitAsk): Promise<{ turnId: string }>; + cancel(conversationId: string, turnId: string): Promise; + addAttachment(conversationId: string): Promise; + addDroppedImage( + conversationId: string, + displayName: string, + dataBase64: string, + ): Promise; + removeAttachment(conversationId: string, attachmentId: string): Promise; + getAttachmentPreview( + conversationId: string, + attachmentId: string, + ): Promise; + openSource(conversationId: string, turnId: string, sourceId: string): Promise; + hide(): Promise; + expandChat?(conversationId: string, turnId: string): Promise; + syncEditor?(focus: AskFocus, editor: AskEditor): Promise; + acknowledgeVoice?(recordingId: string): Promise; + onEvent(handler: (event: AskEvent) => void): Promise<() => void>; + onWindowEvent(handler: (event: AskWindowEvent) => void): Promise<() => void>; + onVoiceEvent(handler: (event: AskVoiceEvent) => void): Promise<() => void>; +} + +export function errorMessage(error: unknown, fallback: string): string { + if ( + typeof error === 'object' && + error && + 'message' in error && + typeof error.message === 'string' + ) { + return error.message || fallback; + } + return typeof error === 'string' && error ? error : fallback; +} + +const INTENT_ERRORS: Record = { + askIntentInvalid: [ + 'Ask could not determine what to do. Try making your request more specific.', + 'Ask 无法确定该怎么处理。请试着把要求说得更具体一些。', + ], + askIntentToolUnsupported: [ + 'The configured model does not support tool-based intent detection. Change the model or the Ask intent mode in Settings.', + '当前模型不支持通过工具识别意图。请在设置中更换模型或调整 Ask 意图识别模式。', + ], + askIntentTimeout: [ + 'Ask took too long to determine what to do. Try again.', + 'Ask 判断请求意图超时,请重试。', + ], + askEntrySettingsUnavailable: [ + 'Ask could not read its settings. Open Ask settings and save your preferences again.', + 'Ask 无法读取设置。请打开 Ask 设置并重新保存偏好。', + ], +}; + +export function submissionErrorMessage(error?: AskError | null) { + if (!error) return undefined; + const localized = error.reason && INTENT_ERRORS[error.reason]; + if (!localized) return error.message; + let language = navigator.language; + try { + language = window.localStorage.getItem('ol.locale') ?? language; + } catch {} + return localized[/^zh\b/i.test(language) ? 1 : 0]; +} + +export function newRequestId(): string { + return Array.from(crypto.getRandomValues(new Uint8Array(16)), (byte) => + byte.toString(16).padStart(2, '0'), + ).join(''); +} + +export function requestFingerprint({ + text, + attachmentIds, + options, +}: Omit) { + return JSON.stringify([text, attachmentIds, options.deepThinking]); +} diff --git a/openless-all/app/ask-window/composer.tsx b/openless-all/app/ask-window/composer.tsx new file mode 100644 index 000000000..9cbde9024 --- /dev/null +++ b/openless-all/app/ask-window/composer.tsx @@ -0,0 +1,537 @@ +import { useAskNotice } from './notices'; +import { PromptBox, type PromptEffort } from '@/components/ui/prompt-box'; +import { Brain, Paperclip } from 'lucide-react'; +import { useEffect, useLayoutEffect, useRef, useState, type ReactNode } from 'react'; +import { AttachmentDeck } from './attachments'; +import { + errorMessage, + newRequestId, + requestFingerprint, + submissionErrorMessage, + type AskError, + type AskClient, + type AskFocus, + type AskSnapshot, + type AskVoiceEvent, + type SubmitAsk, +} from './client'; +import { readDroppedImage } from './drop-image'; + +const PLACEHOLDERS = [ + 'Ask anything', + 'Plan a weekend in Kyoto', + "Explain springs like I'm five", + 'Draft a launch tweet', +]; + +const EFFORTS: PromptEffort[] = [ + { id: 'quick', label: 'Quick' }, + { id: 'deep', label: 'Think deeply', color: '#F2552C' }, +]; + +function ActionButton({ + label, + pressed, + disabled, + onClick, + children, +}: { + label: string; + pressed?: boolean; + disabled?: boolean; + onClick: () => void; + children: ReactNode; +}) { + return ( + + ); +} + +type Selection = { + start: number; + end: number; + direction: HTMLTextAreaElement['selectionDirection']; +}; +type Draft = { text: string; selection?: Selection }; +type PendingAsk = Omit; +type Recording = { id: string; draft: string; input: HTMLTextAreaElement }; + +function selectionOf(editor: HTMLTextAreaElement | null): Selection | undefined { + if (!editor) return; + return { + start: editor.selectionStart, + end: editor.selectionEnd, + direction: editor.selectionDirection, + }; +} + +export function useComposer({ + client, + snapshot, + connected, + canDeliverVoice, + check, + focus, + activation, + captureSubmitReceipt, +}: { + client: AskClient; + snapshot: AskSnapshot; + connected: boolean; + canDeliverVoice: () => boolean; + check: () => void; + focus?: AskFocus; + activation: number; + captureSubmitReceipt: () => (turnId: string) => void; +}) { + const { conversationId, attachments } = snapshot; + const notice = useAskNotice(); + const sendNotice = useRef>(); + const editor = useRef(null); + const draftRef = useRef(''); + const lastClear = useRef(); + const nextSelection = useRef(); + const composing = useRef(false); + const draftIds = useRef(new Map()); + const pendingRef = useRef(false); + const activeRecording = useRef(); + const receiveVoice = useRef<(event: AskVoiceEvent) => void>(() => {}); + const currentVoiceEditor = useRef<(event: AskVoiceEvent) => HTMLTextAreaElement | undefined>(); + const syncSelection = useRef(() => {}); + const mountId = useRef(newRequestId()); + const editorRevision = useRef(0); + const editorFingerprint = useRef(''); + const focusSeen = useRef(''); + + const [draft, setDraft] = useState(''); + const [deepThinking, setDeepThinking] = useState(false); + const [pending, setPending] = useState(false); + const [acknowledgedTurn, setAcknowledgedTurn] = useState(); + const [voiceListening, setVoiceListening] = useState(false); + const [adding, setAdding] = useState(false); + const [removing, setRemoving] = useState>(new Set()); + const [cancelling, setCancelling] = useState(); + const [isComposing, setIsComposing] = useState(false); + + const turnId = + snapshot.activeTurnId ?? + (acknowledgedTurn && + snapshot.submission?.currentTurnId !== acknowledgedTurn && + !snapshot.turns.some((turn) => turn.turnId === acknowledgedTurn) + ? acknowledgedTurn + : undefined); + const attachmentsReady = attachments.every((item) => item.state === 'ready'); + const cannotSubmit = !connected || pending || adding || removing.size > 0 || !attachmentsReady; + + function replaceDraft(text: string, selection?: Selection) { + if (selection && text === draftRef.current && editor.current) { + editor.current.setSelectionRange( + selection.start, + selection.end, + selection.direction ?? undefined, + ); + } else { + nextSelection.current = selection; + } + draftRef.current = text; + setDraft(text); + } + + function syncEditor(editorBlurred = false) { + const input = editor.current; + if ( + !connected || + !input || + !focus || + !voiceListening || + !client.syncEditor || + !canDeliverVoice() + ) + return; + const editable = + !editorBlurred && document.activeElement === input && !composing.current && !input.readOnly; + const selectionStart = input.selectionStart; + const selectionEnd = input.selectionEnd; + const key = JSON.stringify([ + focus.focusEpoch, + editable, + selectionStart, + selectionEnd, + draftRef.current, + ]); + const changed = key !== editorFingerprint.current; + const requestKey = `${focus.focusEpoch}:${focus.focusRequest}:${activation}`; + if (!changed && requestKey === focusSeen.current) return; + if (changed) { + editorFingerprint.current = key; + editorRevision.current += 1; + activeRecording.current = undefined; + } + focusSeen.current = requestKey; + void client + .syncEditor(focus, { + conversationId, + mountId: mountId.current, + editorId: mountId.current, + revision: editorRevision.current, + editable, + selectionStart, + selectionEnd, + }) + .then((accepted) => { + if (!accepted && focusSeen.current === requestKey) focusSeen.current = ''; + }) + .catch(() => { + if (focusSeen.current === requestKey) focusSeen.current = ''; + }); + } + + const focused = useRef(''); + useLayoutEffect(() => { + const selection = nextSelection.current; + if (selection && editor.current) { + editor.current.setSelectionRange( + selection.start, + selection.end, + selection.direction ?? undefined, + ); + nextSelection.current = undefined; + } + const token = focus ? `${focus.focusEpoch}:${focus.focusRequest}` : ''; + if (connected && token !== focused.current && editor.current && canDeliverVoice()) { + focused.current = token; + editor.current.focus({ preventScroll: true }); + } + syncSelection.current = () => syncEditor(); + syncEditor(); + }); + + useEffect(() => { + const input = editor.current; + const onSelectionChanged = () => syncSelection.current(); + input?.addEventListener('select', onSelectionChanged); + document.addEventListener('selectionchange', onSelectionChanged); + return () => { + input?.removeEventListener('select', onSelectionChanged); + document.removeEventListener('selectionchange', onSelectionChanged); + }; + }, []); + + function launch(request: SubmitAsk, restore?: Draft) { + const attachmentIds = new Set( + attachments + .filter( + (attachment) => attachment.state === 'ready' && !removing.has(attachment.attachmentId), + ) + .map((attachment) => attachment.attachmentId), + ); + const blocked = + !connected || + pendingRef.current || + !!turnId || + adding || + request.attachmentIds.some((attachmentId) => !attachmentIds.has(attachmentId)); + if (blocked) { + if (restore) replaceDraft(restore.text, restore.selection); + return; + } + + pendingRef.current = true; + const reportAccepted = captureSubmitReceipt(); + sendNotice.current?.dismiss(); + sendNotice.current = undefined; + setPending(true); + void client + .submit(request) + .then(({ turnId: acceptedTurnId }) => { + setAcknowledgedTurn(acceptedTurnId); + reportAccepted(acceptedTurnId); + draftIds.current.delete(requestFingerprint(request)); + }) + .catch((cause) => { + const message = + typeof cause === 'object' && cause !== null && 'reason' in cause + ? (submissionErrorMessage(cause as AskError) ?? 'Ask could not send this message.') + : errorMessage(cause, 'Ask could not send this message.'); + const recoverable = + typeof cause === 'object' && + cause !== null && + 'code' in cause && + ['busy', 'invalid_argument', 'cancelled'].includes(String(cause.code)); + const severity = recoverable ? 'warning' : 'error'; + if (restore) replaceDraft(restore.text, restore.selection); + sendNotice.current = notice(message, severity, 'ask-send'); + }) + .finally(() => { + pendingRef.current = false; + setPending(false); + check(); + }); + } + + function submitDraft(text: string, effort: PromptEffort) { + const restore = lastClear.current; + lastClear.current = undefined; + const request: PendingAsk = { + conversationId, + text, + attachmentIds: attachments.map((attachment) => attachment.attachmentId), + options: { deepThinking: deepThinking || effort.id === 'deep' }, + }; + const fingerprint = requestFingerprint(request); + let clientRequestId = draftIds.current.get(fingerprint); + if (!clientRequestId) { + clientRequestId = newRequestId(); + draftIds.current.set(fingerprint, clientRequestId); + } + launch({ ...request, clientRequestId }, restore); + } + + function attach(add: () => Promise) { + setAdding(true); + void add() + .catch((cause) => + notice(errorMessage(cause, 'Could not add an attachment.'), 'warning', 'ask-attachment'), + ) + .finally(() => { + setAdding(false); + check(); + }); + } + + useLayoutEffect(() => { + currentVoiceEditor.current = (event) => { + const input = editor.current; + if ( + connected && + canDeliverVoice() && + event.conversationId === conversationId && + event.focusEpoch === focus?.focusEpoch && + event.mountId === mountId.current && + event.editorId === mountId.current && + event.revision === editorRevision.current && + input && + document.activeElement === input && + !input.readOnly && + !composing.current && + input.value === draftRef.current && + event.selectionStart === input.selectionStart && + event.selectionEnd === input.selectionEnd + ) + return input; + }; + receiveVoice.current = (event) => { + if (event.type === 'started') { + const input = currentVoiceEditor.current?.(event); + if (input) { + activeRecording.current = { id: event.recordingId, draft: draftRef.current, input }; + } + return; + } + const recording = activeRecording.current; + if (event.type === 'cancelled') { + if (recording?.id === event.recordingId) activeRecording.current = undefined; + return; + } + void client + .acknowledgeVoice?.(event.recordingId) + .then((claimed) => { + if (activeRecording.current !== recording || recording?.id !== event.recordingId) return; + activeRecording.current = undefined; + if ( + !claimed || + !event.text.trim() || + currentVoiceEditor.current?.(event) !== recording.input || + draftRef.current !== recording.draft + ) + return; + const offset = event.selectionEnd; + const caret = offset + event.text.length; + replaceDraft( + recording.draft.slice(0, offset) + event.text + recording.draft.slice(offset), + { start: caret, end: caret, direction: 'none' }, + ); + }) + .catch(() => {}); + }; + }); + + useEffect(() => { + let stopped = false; + let stop: (() => void) | undefined; + const clearRecording = () => { + activeRecording.current = undefined; + }; + window.addEventListener('blur', clearRecording); + void client + .onVoiceEvent((event) => { + if (!stopped) receiveVoice.current(event); + }) + .then((unsubscribe) => { + if (stopped) unsubscribe(); + else { + stop = unsubscribe; + setVoiceListening(true); + } + }) + .catch((cause) => { + if (!stopped) + notice( + errorMessage(cause, 'Could not connect to Ask voice input.'), + 'warning', + 'ask-voice', + ); + }); + return () => { + stopped = true; + activeRecording.current = undefined; + window.removeEventListener('blur', clearRecording); + stop?.(); + }; + }, [client, notice]); + + useLayoutEffect(() => { + if (!connected) activeRecording.current = undefined; + }, [connected]); + + const box = ( +
{ + if (!event.dataTransfer.types.includes('Files')) return; + event.preventDefault(); + event.dataTransfer.dropEffect = connected && !adding && !pending ? 'copy' : 'none'; + }} + onDrop={(event) => { + if (!event.dataTransfer.types.includes('Files')) return; + event.preventDefault(); + if (!connected || adding || pendingRef.current) return; + const files = event.dataTransfer.files; + if (files.length !== 1) { + notice('Drop one PNG or JPEG image at a time.', 'warning', 'ask-attachment'); + return; + } + const file = files[0]; + attach(async () => { + const { displayName, dataBase64 } = await readDroppedImage(file); + await client.addDroppedImage(conversationId, displayName, dataBase64); + }); + }} + > + 0 && attachmentsReady} + inputProps={{ + 'aria-describedby': attachments.length ? 'draft-references' : undefined, + readOnly: pending, + onFocus: () => syncEditor(), + onBlur: () => { + activeRecording.current = undefined; + syncEditor(true); + }, + onSelect: () => syncEditor(), + onCompositionStart: () => { + activeRecording.current = undefined; + composing.current = true; + syncEditor(); + setIsComposing(true); + }, + onCompositionEnd: () => { + composing.current = false; + setIsComposing(false); + }, + }} + onValueChange={(value) => { + if (pendingRef.current) return; + lastClear.current = + value === '' + ? { text: draftRef.current, selection: selectionOf(editor.current) } + : undefined; + if (value !== draftRef.current) activeRecording.current = undefined; + replaceDraft(value); + }} + onSubmit={submitDraft} + onStop={() => { + if (!connected || !turnId || cancelling === turnId) return; + setCancelling(turnId); + void client + .cancel(conversationId, turnId) + .catch((cause) => { + notice( + errorMessage(cause, 'Could not stop the current request.'), + 'error', + 'ask-stop', + ); + }) + .finally(() => { + setCancelling(undefined); + check(); + }); + }} + beforeInput={ + { + setRemoving((current) => new Set(current).add(attachmentId)); + void client + .removeAttachment(conversationId, attachmentId) + .catch((cause) => { + notice( + errorMessage(cause, 'Could not remove this attachment.'), + 'warning', + 'ask-attachment', + ); + }) + .finally(() => { + check(); + setRemoving( + (current) => new Set([...current].filter((id) => id !== attachmentId)), + ); + }); + }} + /> + } + actions={ + <> + attach(() => client.addAttachment(conversationId))} + > + + + setDeepThinking(!deepThinking)} + > + + + + } + /> +
+ ); + return box; +} diff --git a/openless-all/app/ask-window/conversation.tsx b/openless-all/app/ask-window/conversation.tsx new file mode 100644 index 000000000..45acb5546 --- /dev/null +++ b/openless-all/app/ask-window/conversation.tsx @@ -0,0 +1,299 @@ +import { AskNotice } from './notices'; +import { AiStatus, type AiStatusState } from '@/components/ui/ai-status'; +import { ArrowUpRight, BookOpen } from 'lucide-react'; +import { AnimatePresence, motion, useReducedMotion } from 'motion/react'; +import { useEffect, useRef, useState, type ReactNode } from 'react'; +import { + errorMessage, + submissionErrorMessage, + type AskClient, + type AskSource, + type AskSubmission, + type AskTurn, + type SearchStatus, +} from './client'; + +const FADE = + 'linear-gradient(to bottom, transparent 0, rgba(0,0,0,0.04) 20px, rgba(0,0,0,0.15) 48px, rgba(0,0,0,0.35) 80px, rgba(0,0,0,0.6) 112px, rgba(0,0,0,0.85) 144px, black 176px, black calc(100% - 28px), transparent 100%)'; + +export const SPRING = { type: 'spring', duration: 0.5, bounce: 0.15 } as const; + +const SEARCH_LABEL: Record = { + idle: 'Not searched', + searching: 'Websearch · Searching…', + reading: 'Websearch · Reading sources…', + done: 'Search complete', + no_results: 'No results', + error: 'Search failed', + cancelled: 'Search stopped', +}; + +const SOURCE_READ_LABEL: Record = { + not_attempted: 'Not opened', + reading: 'Reading', + read: 'Read', + blocked: 'Page blocked', + failed: 'Could not read', + cancelled: 'Reading stopped', +}; + +function statusFor(turn: Pick & { deepThinking?: boolean }): { + state: AiStatusState; + label?: string; +} { + if (turn.status === 'failed') return { state: 'error', label: 'Failed · Incomplete' }; + if (turn.status === 'cancelled') return { state: 'error', label: 'Stopped · Incomplete' }; + if (turn.status === 'completed') return { state: 'done' }; + if (turn.searchStatus === 'searching') + return { state: 'researching', label: SEARCH_LABEL.searching }; + if (turn.searchStatus === 'reading') return { state: 'analyzing', label: SEARCH_LABEL.reading }; + if (turn.status === 'writing') return { state: 'writing' }; + if (turn.status === 'thinking') { + return { state: 'thinking', label: turn.deepThinking ? 'Thinking deeply' : 'Working' }; + } + return { state: 'thinking', label: 'Working' }; +} + +export function ConversationLayout({ + expanded, + turns, + children, + showSources, +}: { + expanded: boolean; + turns: AskTurn[]; + children: ReactNode; + showSources: (turnId: string) => void; +}) { + const threadRef = useRef(null); + const follow = useRef(true); + const lastScroll = useRef(0); + const reduced = useReducedMotion(); + + useEffect(() => { + const thread = threadRef.current; + if (thread && follow.current) { + thread.scrollTo({ top: thread.scrollHeight, behavior: reduced ? 'auto' : 'smooth' }); + } + }, [expanded, turns, reduced]); + + const latest = turns.at(-1); + return ( +
+ {expanded && ( + + )} +
+ {expanded && turns.length > 0 && ( +
{ + const thread = event.currentTarget; + if (thread.scrollHeight - thread.scrollTop - thread.clientHeight < 72) + follow.current = true; + else if (thread.scrollTop < lastScroll.current) follow.current = false; + lastScroll.current = thread.scrollTop; + }} + > +
+ {turns.map((turn) => { + const { state, label } = statusFor(turn); + return ( + +
+
+ {turn.userMessage.text &&

{turn.userMessage.text}

} + {turn.attachments.length > 0 && ( +
    + {turn.attachments.map((attachment) => ( +
  • {attachment.displayName}
  • + ))} +
+ )} +
+
+
+ + + {turn.assistantMessage?.text && ( + + {turn.assistantMessage.text} + + )} + + + {turn.searchStatus !== 'idle' && ( + + )} +
+
+ ); + })} +
+
+ )} + +
{children}
+
+
+ ); +} + +export function SubmissionResult({ + submission, + active, +}: { + submission: AskSubmission; + active: boolean; +}) { + const text = + submission.delivery === 'retained' || + submission.delivery === 'unknown' || + submission.route === 'clarify' + ? submission.resultText + : undefined; + const error = submissionErrorMessage(submission.error) ?? submission.searchError?.message; + const { state, label } = statusFor(submission); + const delivery = + submission.delivery === 'written' + ? submission.route === 'edit' + ? 'Replaced in the original editor.' + : 'Inserted in the original editor.' + : submission.delivery === 'unknown' + ? 'Ask could not confirm whether the editor changed. Check it before pasting.' + : submission.delivery === 'retained' + ? 'Kept here because Ask could not write to the original editor.' + : undefined; + return ( +
+ {active && } + + {delivery &&

{delivery}

} + {submission.copyStatus === 'copied' &&

Copied to the clipboard.

} + {submission.copyStatus === 'failed' &&

Could not copy to the clipboard.

} + {!active && submission.status === 'cancelled' &&

Request stopped.

} + {text && + (submission.route === 'clarify' ? ( +

{text}

+ ) : ( +