The guide

A command line
for your desktop.

Install the tool, give your agent the context, and try one action on a page you can inspect.

Install and check

Native desktop control requires macOS 13 or later, on Apple Silicon or Intel. Run this in Terminal:

curl -fsSL https://raw.githubusercontent.com/ensomniac/codex-ui/main/install.sh | sh

The installer supplies uv if needed and creates an isolated Python environment. Open a new terminal if codex-ui is not yet on your path, then check desktop access:

codex-ui doctor

doctor reports Accessibility and screen-capture availability, the current displays, and the running Python interpreter. An unhealthy check returns a nonzero exit code.

macOS may ask for Accessibility, Screen Recording or Automation access for your terminal or agent host. Grant the access the system requests, then run the check again. codex-ui adds no per-action permission prompt.

Prefer a package manager? Homebrew, uv and npm are shown on the main page. The repository has all installation paths, including pipx, pip, pnpm, Yarn, Bun and portable Windows installation.

Connect an agent

The bundled skill teaches your agent the commands, target selectors, screenshot geometry and verification workflow.

codex-ui skill install

This installs the skill for Codex. For Claude, run codex-ui skill install --agent claude. For another host, use --path /your/skills/control-local-ui, or provide the agent with the command reference.

Any agent that can run a local command and inspect an image can use codex-ui. codex-ui schema describes the CLI; codex-ui capabilities reports platform support. Desktop work runs in the account and session that launched the command.

Your first action

Open the codex-ui home page in Chrome. Its example has a real button named Mark reviewed. Ask your agent to run:

codex-ui press \
  --chrome-url 'ensomniac.github.io/codex-ui/' \
  --role button \
  --name 'Mark reviewed' \
  --capture-after

The example should change from “Not reviewed” to “Reviewed,” and its activation count should increase by one. The command returns a capture path; have your agent open that image and inspect the changed state.

ok: true means the command completed. The page state and captured image establish whether the intended result happened.

The example changes only this page. For a local test surface, clone the repository and open examples/playground.html in Chrome.

Choose a target

Use a distinctive selector. If several windows or controls match, narrow it before acting.

codex-ui windows --app 'Google Chrome'
codex-ui windows --app TextEdit

Selector	Use it for
`--chrome-url`	A literal URL substring, including a local `file://` page.
`--window-id`	A window ID returned by `windows`.
`--app` + `--title`	A native application and a distinctive window title.
`--display`	A screen capture using a display index from `doctor`.

Chrome targeting prefers matching active tabs, then searches background tabs. Use a URL specific to the page you intend. Verify the returned target when multiple tabs share a path.

Observe, then act

Capture the target and inspect its controls before choosing an action.

codex-ui capture --chrome-url 'your-project/index.html' --label before
codex-ui inspect --chrome-url 'your-project/index.html' --search Save --all
codex-ui press --chrome-url 'your-project/index.html' \
  --role button --name 'Save changes' --capture-after

Prefer a named control when available. Coordinates can be given as desktop points, window-relative points or screenshot pixels with the capture metadata. The full reference covers clicking, dragging, scrolling, typing and window layouts.

Chrome inspection

codex-ui chrome-dom --chrome-url 'your-project/index.html' \
  --selector '#app' --depth 2
codex-ui chrome-console --chrome-url 'your-project/index.html' --level error
codex-ui chrome-eval --chrome-url 'your-project/index.html' \
  '({title: document.title, ready: document.readyState})'

These commands use DevTools in the Chrome profile already open. They preserve the clipboard and normally restore the panel layout. Console results depend on the visible entries and current filters; unreadable rows do not establish that a page has no errors.

Run related actions together

A JSON plan keeps related actions in one visible control session. Save this as review.json with the home page open in Chrome:

{
  "schema_version": 1,
  "target": {"chrome_url": "ensomniac.github.io/codex-ui/"},
  "actions": [
    {"type": "press", "role": "button", "name": "Mark reviewed"},
    {"type": "capture", "label": "reviewed"}
  ]
}

codex-ui run --plan-file review.json

Re-observe when the next action depends on an uncertain result. Keep separate desktop commands sequential; they share your mouse, keyboard and focus. The plan schema lists the supported actions.

How control works

codex-ui uses macOS Accessibility, Quartz and Cocoa to work with the actual desktop. It has no separate browser profile, remote daemon, account or telemetry service.

Visible activity: blue while observing, red while controlling, amber when yielding to physical input, and green on completion.
Shared input: the tool pauses when you use the mouse or keyboard and resumes after three quiet seconds.
Pointer restoration: control sessions restore the pointer, including after failure.
Correct geometry: screenshot sidecars record the screen rectangle and scale. Retina pixels and global desktop points are different units.

The tool uses the authority of your local account. Your instructions and your agent host determine the scope of the work. It does not bypass operating-system protections.

Update or remove

codex-ui upgrade --check
codex-ui upgrade
codex-ui skill install

The updater checks the latest stable GitHub release, verifies its wheel digest, and updates the environment that owns the command. Homebrew installations use the tap’s pinned formula. The skill is refreshed separately so local customizations can be backed up.

Direct release URLs are pinned. A generic package-manager update may keep an old URL. Use codex-ui upgrade, or install a newer release URL. JavaScript users can update the bridge itself by reinstalling its new release archive.

To remove an installation made by the recommended installer, run uv tool uninstall ensomniac-codex-ui. Screenshots and installed skills are kept. See uninstall and recovery for other managers.

Troubleshooting

“No window matched” or more than one match

Run codex-ui windows --app 'Google Chrome' and inspect the list. Use a more specific URL or exact window ID. Restore a minimized window before capturing it.

Desktop access is unavailable

Run doctor and check macOS Privacy & Security settings for the host running the command. A changed Python runtime can have different permission attribution. Restart the host if macOS requests it.

The command is waiting

An amber indicator means it detected physical input. Leave the mouse and keyboard quiet for three seconds. Use the reported state to diagnose a failure; do not treat an observation timeout as proof that the process stopped.

The command ran, but the page did not change

Inspect the capture and returned target. Confirm the control’s name, role and enabled state with inspect. Report the command, version, actual result and relevant environment in a bug report.

Windows or Linux reports no desktop backend

That is the current support boundary. Installation, upgrades, schema and skills are available; native desktop control is planned.

A command linefor your desktop.