Add gated native macOS computer use to Mothership desktop

[Sg312]

Summary

Adds native macOS computer use to Mothership behind the global mothership-computer-use feature flag, disabled by default. An opted-in desktop can inspect app accessibility trees and window screenshots, then interact with approved apps through the signed native helper. The desktop UI exposes permissions, app approvals, current activity, and immediate Stop.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• Other

Testing

• Desktop suite: 1,909 passing tests across 79 files; subsequent permission-onboarding change passed 28 focused service/transport tests and typecheck.
• Sim integration: 254 targeted tests, nine real-Postgres claim tests (12 concurrent requests yield one claim), 11 browser UI checks, typecheck, and strict API audit passed.
• Desktop IPC audit and exact generated-contract sync passed. Native Swift protocol, geometry, cancellation, and shortcut checks passed.
• Live native fixture verified screenshots, accessibility, secure-field suppression, Unicode text, semantic actions, foreground drag, and scrolling. Calculator background interaction produced 2 + 3 = 5 without switching foreground apps.
• Companion worker includes real Astra accessibility and visual trials. The visual trial consumed two screenshots and selected an unlabeled triangle, with exact fixture output verified.
• Remaining merge/rollout gate: the separately signed, normally launched Sim Computer Use Test.app must complete its bridge-to-native screenshot check after macOS grants the parent app Accessibility and Screen Recording. Nested-helper grants alone do not cover that identity.

Review authorization claims, late-response cancellation, snapshot ownership, and account cleanup across this PR and its companion. macOS 14+ only; coordinate input requires explicit foreground activation. These checks do not establish compatibility with every Mac app or full Codex parity.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced — existing toolchain/test warnings remain
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

Before: no desktop Computer Use settings or activity strip. After: a settings toggle, macOS permission status/actions, approved-app controls, and an active-action strip with Stop and the registered keyboard shortcut. These source UI states were visually checked locally; live native fixture images and trial reports are committed in the companion under worker/tools/bench/data/computer-use/.

Companion PR

Companion: https://github.com/simstudioai/mothership/pull/520

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 24, 2026 8:48pm UTC