Cell - Ta Computer Use

A built-in pi package for Ta (enabled by default, can be disabled in Web client Settings → Extensions): lets Ta use desktop applications on this machine via Cua Driver in the background. It lists apps and windows, reads window accessibility trees, performs clicks, types text, scrolls, and takes screenshots, without taking your mouse and keyboard. It's just one pi extension (src/extension.ts → extension/index.mjs), does not modify Ta's core, and pi can use it as well.

Ta Desktop comes with Cua Driver (fixed version) and works out of the box. For the terminal Ta, use your own installation:

/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"   # macOS, Linux
irm https://cua.ai/driver/install.ps1 | iex                    # Windows (PowerShell)

The /computer-use page shows which program is in use and checks what your system still needs (guiding you through setup like the first time); if nothing is installed, it shows the installation command.

How It Works

• Finding the program: first checks the environment variable TA_CUA_DRIVER (set by Ta Desktop when starting ta serve, pointing to its bundled copy), then checks PATH, ~/.local/bin (the default install location), and on macOS /Applications/CuaDriver.app and ~/Applications/CuaDriver.app, in order. It only calls stat, never runs anything. If not found, it checks again before each message, so installing mid-session doesn't need a restart.
• Hint: when found, adds a <computer_use> section to the system prompt: tool names are tools.mcp__cua_driver__*, suggests calling describeNamespace("mcp__cua_driver") first to see usage, plus platform limitations (macOS: don't prompt user again while waiting for permissions; Windows: can't operate windows running as administrator; Linux: needs desktop session and AT-SPI). Adds nothing if not found.
• On-demand startup: the MCP server cua-driver (exposed in codemode) is not registered at session start. The first time a codemode script mentions mcp__cua_driver (per pi's reachesTools; scripts using tools must write it out), it registers with pi.registerMcpServer and waits for tools to appear (up to 5 seconds). Pi connects all registered MCP servers at session start, creating a cua-driver mcp process per session (~10–24 MB); on-demand registration means sessions not using desktop apps have no extra process.
• System requirements and permissions (src/cua-driver-access.ts, checked once per process on first use, shared by concurrent calls):
  • macOS: Cua Driver needs Accessibility (read and operate apps) and Screen Recording (screenshots), authorized to CuaDriver.app (com.trycua.driver, Cua's developer signature), not Ta. Check with cua-driver call check_permissions (~20 ms). When permissions are missing, CuaDriver.app shows a two-step guide; Ta posts a notification explaining which two toggles to enable, then checks every second. Once enabled, the call continues without retry. If the guide is dismissed, Ta opens the system settings page. Waits up to 3 minutes; the user can press "Stop" to cancel immediately. Without Accessibility, the call fails, and the model is told to ask the user to enable it; /computer-use can guide them again. Without only Screen Recording, the call continues (reading and operating work), with a one-time notification. The daemon returns permissions_pending for a few hundred ms after startup, so Ta silently waits up to 3 seconds to avoid false alarms.
  • Linux: no permissions needed, but requires a graphical session (X11 or Wayland with XWayland), AT-SPI accessibility bus, and libX11 / libXi / libxkbcommon. Check with cua-driver doctor --json (~0.3 s). Without a graphical session (e.g., SSH) or missing libraries, the call fails and explains why plus install commands; AT-SPI warnings notify the user once and continue.
  • Windows: no permissions needed; use doctor to check for desktop session. Windows running as administrator requires Ta to also run as administrator (UIPI), noted in the hint.
  • When the system can't tell (your daemon hasn't started yet), the call is not blocked; it checks next time.
• Bundled Cua Driver on macOS (src/cua-driver-daemon.ts): the MCP process only forwards; the actual daemon must launch as CuaDriver.app via LaunchServices for authorization to belong to it. Ta starts a daemon on its own socket (~/Library/Caches/ta/cua-driver.sock), and the MCP server is cua-driver mcp --embedded --socket <it>. Here's how:
  • An app nested inside another app bundle is treated by macOS as part of the outer app (permissions attach to Ta and are lost on each build). So on first use, Ta clones it by version to ~/Library/Application Support/Ta/cua-driver/<version>/ (APFS clone, ~30 ms, no extra disk), launches from there, and deletes older version clones.
  • LaunchServices picks variants by bundle id and might pick CuaDriver.app from another app (different version, forwarding reports incompatibility). So before launch, Ta registers this copy with lsregister -f, then verifies the actual running program's path after launch. If it doesn't match, Ta stops it and suggests cua-driver doctor --fix. An existing daemon on the socket (from another Ta process or left over) is checked the same way.
  • The bundled copy disables Cua's product telemetry and update checks (CUA_DRIVER_RS_TELEMETRY_ENABLED=0, CUA_DRIVER_RS_UPDATE_CHECK=0); Ta's version updates handle upgrades. Your own installation keeps your settings.
  • Before each use, the daemon is confirmed alive (one socket connect); if it crashed, it restarts. After 10 minutes idle, it exits (~60 MB), and the forwarder exits too; pi auto-reconnects on the next call. When the process that started it exits, it stops the daemon too.

Measured (M4 Pro, fake model running ta -p, bundled Cua Driver 0.47.5, from Ta.app bundle): the first call is ~0.3 s slower (clone + daemon start). Each subsequent first call in a new process is within measurement error compared to not calling (~1.18 s vs ~1.20 s). Daemon startup ~150–250 ms, permission check ~20 ms, idle ~33–60 MB.

Limitations

• If mcp.json already has a server named cua-driver, the file takes precedence. Cua Driver configured under another name (e.g., from cua-driver connect as cua-computer-use) won't be recognized, and tools appear twice (you can disable this extension).
• When remote execution (ta serve --exec) is used, the MCP server runs in the execution environment, but the program is found on the ta serve machine. When paths differ on each side, it won't connect.
• On macOS, if the user clicks "Quit and Reopen" in the permission guide (Screen Recording requirement), macOS restarts CuaDriver.app without Ta's arguments; Ta then restarts its daemon and stops the extra instance after permissions are granted.
• Windows and Linux installer packages have not been verified on real machines; Linux's doctor output has been tested in containers.