Skip to content

feat(transcript): show Host image files referenced in assistant replies - #174

Merged
akemmanuel merged 2 commits into
akemmanuel:masterfrom
touch2be:feat/transcript-image-previews
Oct 23, 2026
Merged

akemmanuel merged 2 commits into
akemmanuel:masterfrom
touch2be:feat/transcript-image-previews

Conversation

@touch2be

Copy link
Copy Markdown
Collaborator

Builds on #173. It includes that commit (ebdde51) for the Desktop sidecar attachmentBaseUrl. Please merge #173 first; this diff then shrinks to the second commit.

Problem

When the agent creates an image and replies with ![Maus auf dem Mars](maus-auf-dem-mars.png), the transcript shows only the alt text. MarkdownRenderer blocks every markdown image so that tracking pixels can't load. A file on the Host is not a third-party origin, so it doesn't need that block.

Change

Markdown images in assistant replies

  • Project-relative paths, absolute paths, file: URLs, and inline raster data:image/* render as a preview. Clicking a preview opens the existing lightbox.
  • http(s), javascript:, SVG data URLs, and files without an image extension still render as alt text.
  • They load through the existing /api/fs/file route, so path grants, OPENGUI_ALLOWED_ROOTS, and realpath checks apply unchanged.
  • Without a MarkdownImageContext (any other MarkdownRenderer use), behaviour is unchanged.

Paths and permissions across Shells and Hosts

Setup Handling
Windows /api/fs/file uses the platform's isAbsolute, so C:\… and \\server\share are no longer joined onto the Project directory. The Frontend also treats UNC paths as absolute. A custom urlTransform keeps C:\… and file: image sources, which react-markdown's default would drop as unknown protocols. Percent-encoding (spaces, umlauts, %5C) is decoded before the path is resolved.
macOS realpath (/var → /private/var) is already enforced by the Host. Nothing new needed.
Desktop Local Previews use the sidecar URL (attachmentBaseUrl). ToolCallPartView and FilePartView still used the stored workspace URL and requested the wrong port.
Desktop → Remote Host (token) <img> can't send Authorization. The new HostImage component fetches with the bearer token and shows a blob URL. The Desktop Local credential is never sent to another Host. This also fixes uploaded-image thumbnails on token-authenticated Remote Hosts.
Hosted / Web (cookie session) Plain <img> requests, as before.
Restricted Hosts / missing file A path outside the grants returns 403 and shows an "Image unavailable" placeholder instead of a broken image.

Hardening /api/fs/file

  • X-Content-Type-Options: nosniff.
  • Content-Security-Policy: sandbox; default-src 'none'; …: SVG written by the agent can't run script on the Host origin when opened directly.
  • Cache-Control: no-cache: images the agent overwrites in place refresh in the transcript.

Other

  • System prompt: one line telling models to reference images as ![alt](path).
  • Tool image attachments also accept gif and webp.
  • i18n: attachments.openImage and attachments.imageUnavailable in en, de, and es.

Not in this PR (follow-up)

  • Images from MCP tools and the read tool are stored under the Host data directory (tool-output/), and the tool-result projection doesn't carry them into state.attachments yet. Restricted Hosts don't grant that directory, so the follow-up needs a session-scoped route (/api/host/sessions/:id/tool-output/:file, checked via readSession) plus the projection fix.
  • Showing image previews in collapsed tool groups, e.g. for write of *.png/*.svg.

Testing

  • vp check: clean
  • vp test: frontend 541, backend + harness 706, all passing
  • pnpm run slop-check: ok
  • New tests:
    • classifyMarkdownImageSource: POSIX, Windows, UNC, file:, percent-encoding, rejections
    • resolveAttachmentAuthToken
    • MarkdownRenderer: Host route URL, Windows path, remote still blocked, placeholder, no-context fallback
    • HostImage: direct load vs. bearer fetch vs. 403 placeholder
    • /api/fs/file: security headers and Project-relative path
  • Not yet tested by hand in the running app (Desktop Local, Web, Windows/macOS builds).

🤖 Generated with Claude Code

touch2be and others added 2 commits October 23, 2026 14:39
…hem via the sidecar

Desktop transcripts showed broken image tiles for uploaded screenshots while
hosted Web worked. Two Desktop-only causes:

- Transcript previews used the stored workspace URL (127.0.0.1:4096), but the
  Desktop Local Host runs as a sidecar on a per-launch port.
- Unrestricted uploads went to the OS temp dir, which is outside the default
  allowed roots (home) and is wiped on reboot, so /api/fs/file rejected them.

Uploads with a Project directory now always land in <project>/.opengui-uploads
(self-ignored by git), matching restricted Hosts. Transcript previews resolve
the sidecar URL for Desktop Local workspaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assistant replies like `![Maus auf dem Mars](maus-auf-dem-mars.png)` rendered
only their alt text: markdown images were blocked wholesale to avoid tracking
pixels. Host files are not a third-party origin, so render them through the
existing /api/fs/file route (path grants and allowed roots still apply) and
keep remote URLs blocked.

- Markdown images: Project-relative, absolute, and file: paths plus inline
  raster data URLs render with a lightbox; http(s) and active URLs stay alt
  text. A custom urlTransform keeps Windows drive paths that react-markdown
  would otherwise drop as an unknown protocol, and percent-encoding (spaces,
  umlauts, backslashes) is decoded before resolving.
- Windows: /api/fs/file uses the platform's isAbsolute, so `C:\...` and UNC
  paths are no longer joined onto the Project directory; the Frontend treats
  UNC paths as absolute too.
- Remote Hosts with a bearer token: `<img>` cannot send Authorization, so
  HostImage fetches with the token and shows a blob URL. Desktop Local and
  cookie sessions keep plain requests. Applies to uploads, file parts, tool
  images, and markdown images; failures show "Image unavailable".
- Tool and file-part previews use the Desktop sidecar URL (attachmentBaseUrl)
  instead of the stored workspace URL; gif/webp tool images are allowed.
- /api/fs/file sends nosniff, a sandboxing CSP (agent-written SVG cannot run
  script on the Host origin), and no-cache so regenerated images refresh.
- System prompt tells models to reference images as ![alt](path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@akemmanuel
akemmanuel merged commit 294b009 into akemmanuel:master Oct 23, 2026
2 checks passed
@akemmanuel akemmanuel mentioned this pull request Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants