Skip to main content
PortfolioBlogAboutContact
Hero image

cua-driver vs Computer Use: Why Your Accessibility Bugs Are Now Agent Bugs

Published: July 31, 2026

Last week I gave an AI agent Accessibility and Screen Recording permission on my actual working Mac. The machine I bill hours on, rather than a throwaway VM or the demo container everyone runs once and deletes.

I went in to answer a narrow question: how trycua's cua-driver compares to Anthropic's computer use and which one I should be building on. I came out with a different answer than the one I went looking for. The comparison turned out to be shaped wrong and underneath it was a finding I did not expect: the thing that lets an agent operate an application is the same thing that lets a screen reader operate it. When it is missing, both fail and now only one of those failures shows up on an invoice.

This post covers what I installed, what the permission prompt told me about who really holds that grant and why the accessibility defects I already get paid to find have quietly acquired a second cost.

What I Installed and What the Permission Prompt Told Me

cua-driver 0.10.0, running natively on Apple silicon. It exposes 49 tools over MCP or a plain CLI, covering window enumeration, accessibility snapshots, clicks, typing, scrolling, screen capture and trajectory recording. cua-driver doctor came back clean on every check, including two that look for legacy install artefacts the project has since moved away from.

Then I ran the permission check. This is where it got interesting:

Accessibility:    ✅ granted
Screen Recording: ✅ granted
Source: driver-daemon

That third line is doing real work. The driver answers the question through its running daemon, so the result carries the identity of the signed CuaDriver.app bundle rather than the identity of my terminal. With no daemon running it reports unknown instead of quietly reporting my terminal's grants and pretending they belong to the driver.

The reason that matters is what the lazy version of this tool looks like. The obvious way to build a CLI that clicks things is to have the CLI itself request Accessibility. It works immediately and it permanently hands input synthesis and screen reading rights over the entire machine to Ghostty or iTerm or Terminal. Every script you ever run in that terminal inherits them. Every npx you fire off inherits them. You wanted to automate one app and you granted a blanket capability to a shell.

cua-driver routes the grant through LaunchServices specifically so the macOS dialog attributes to the app bundle instead. It is a small detail that tells you the people who built it thought about blast radius before they thought about convenience.

These Two Things Sit at Different Layers

Here is the reframe that makes the rest of this post make sense.

Anthropic's computer use is a model capability. It is a tool definition inside the Messages API, currently computer_20250124 and the newer computer_20251124. It governs how the model perceives and decides. A screenshot goes in, an action with pixel coordinates comes out. It is client-executed, which means Anthropic never hosts your desktop or performs the action. You supply the environment and you supply the executor. The canonical reference implementation is a Linux desktop inside a Docker container, deliberately not your machine.

cua-driver is a host driver. It governs how an action actually reaches the operating system: which process receives the event, which window it lands in and whether the target app steals your focus while it happens. There is no model anywhere inside it.

They compose rather than compete. You can run cua-driver as the executor behind a computer use loop and have both. The comparison worth having sits one layer below the product names, in how each approach perceives a screen.

Two Ways to See a Screen

The pixel path. Computer use takes a screenshot, the model reads the image and returns coordinates. On current models images run up to 2576 pixels on the long edge with returned coordinates mapping one to one onto actual pixels, so there is no scale factor arithmetic to get wrong. Anthropic's own guidance points at 1080p rather than maximum resolution as the sensible balance, which is a hint about cost that I will come back to.

The tree path. cua-driver's get_window_state walks the application's accessibility tree and hands back a structured array of elements, a readable rendering of the same tree and a screenshot, all from one call. Actions then dispatch either by element handle or by pixel, chosen per action, same tool, mutually exclusive parameters.

I snapshotted a Finder window to see what that actually returns. 377 nodes. Each one carries a role, a label, a frame, a parent index and a stable handle. A real entry looks like this:

{
  "element_index": 170,
  "element_token": "s000b:170",
  "role": "AXButton",
  "label": "Customize Folder…",
  "frame": { "x": 1.0, "y": 1090.0, "w": 118.0, "h": 18.0 }
}

An agent holding that does not need to look at a picture to know what the control is or what happens when it activates it. It also does not need the window to be visible. The handle path works on occluded, backgrounded and minimised windows, because it never depended on the pixels in the first place.

The Accessibility Tree Is the Agent's API

Now the part I did not go looking for.

Of those 377 Finder nodes, 321 were actionable controls with real labels. The 27 unlabeled ones carried no label and no frame at all, which is the signature of a menu separator rather than a control. So Apple's own file manager hands an agent 321 addressable, self-describing handles and almost no ambiguity. That is what a healthy tree looks like.

Now consider the applications I get called in to audit.

An icon-only button with no accessible name is an AXButton with an empty label. A screen reader announces "button" and stops there. An agent snapshotting that tree sees a node it cannot identify and cannot safely activate, because nothing in the node says whether it saves the record or deletes it. Same node, same missing attribute, two entirely different consumers left equally stuck.

Run the rest of the audit list and it holds up item for item. A div wired up with a click handler and no role does not appear as a control to either consumer. A form input with no associated label is an unnamed text field to both. A canvas-rendered interface with no shadow tree is an empty box to both. Every one of those is simultaneously a WCAG failure and an automation failure, for the identical underlying reason: the interface never described itself in a way software could read.

This is the same argument I made about HTTP APIs in Your API Isn't Ready for Agent Customers, except one layer up the stack. Your interface acquired a new category of consumer and that consumer does not have eyes, patience or the ability to guess what an unlabeled icon does.

What changes with agents in the picture is the business framing. Accessibility has spent a decade being sold as compliance and ethics. Plenty of owners nod along and then defer it to next quarter. Agent readiness is a different conversation, because it has a number attached.

What the Fallback Costs You

When the tree is unusable, the agent does not give up. It falls back to pixels. That is the expensive path.

High resolution images run up to roughly 4784 tokens each on current models, about three times the previous ceiling. That is per perception step and a GUI loop takes a perception step before every single action. A forty step task at that rate burns somewhere near 190,000 image tokens before the model has produced one token of reasoning or output.

The same task driven off the accessibility tree sends structured text. That is the entire difference between reading a label and looking at a photograph of a label.

You can watch the tool authors reach the same conclusion. cua-driver's snapshot call takes include_screenshot: false as an explicit option for when you only need to re-index the tree and do not need fresh pixels. They built a switch specifically to skip the image, which tells you which half of the payload they consider costly.

The second cost is reliability. That one does not show up on a bill. Pixel grounding breaks when a layout shifts, a theme changes, a font renders differently or a window opens at another size. An element handle survives all four. Every defect that pushes an agent off the tree and onto the screen makes the automation both more expensive and more brittle at the same time.

When Pixels Genuinely Win

I am not going to pretend the tree wins everywhere, because it does not.

Canvas-rendered applications, games, remote desktop sessions and anything drawing its own widgets have no meaningful accessibility layer to read. Chromium and Electron content regularly sits beyond what the native accessibility path can reach and cua-driver ships a documented workaround for exactly that case: pixel-click to focus the renderer, then type into it. That workaround exists because the problem is real and common.

Computer use also wins outright on portability. One capability, no per-host install, works against arbitrary environments you do not control and have never seen. If your agent needs to operate a machine you cannot provision in advance, screenshots are the only honest option.

cua-driver's own design concedes this. Its escalation ladder goes element handle first, then pixel, then foreground delivery, then full desktop scope, with a rule I keep thinking about: foreground is a reaction to an observed failure and never a prediction. You are required to watch something fail before you are allowed to escalate. The tradeoff is not hidden anywhere, it is encoded in the order of operations.

Two Answers to the Same Security Problem

Both vendors are solving for the same threat. A screenshot or a page can carry text that the model reads as an instruction, which means anything an agent looks at can try to steer it.

Anthropic's answer is isolation. Run the whole thing in a disposable container, keep sensitive data out of it, require human confirmation on consequential actions and accept that the agent is operating in a sandbox rather than on your desktop.

cua-driver's answer is to constrain the driver and make every action attributable. It ships a bounded permission mode locked to an immutable tool manifest that a tool call cannot modify at runtime. Attaching to an authenticated browser profile requires an explicit approval token that is single use and expires in five minutes. Each run gets a colour-coded on-screen cursor so you can see what is being touched while it happens and trajectory recording captures before and after state with screenshots as an audit log.

Neither answer is wrong. They fit different situations. Which situation you are in is the question worth asking. If the agent is working in a throwaway environment, isolate it. If it is operating the machine you actually work on, you want attribution, scoping and a record.

Conclusion

I set out to pick a winner between two tools and found out they were never in the same category. Computer use is the half that decides what to do, while cua-driver is what makes the action land on the right window. The interesting question was never which one wins, it was what an agent does when it reads your interface and cannot understand it.

The answer is that it stops reading and starts looking, which costs more per step and breaks more often. Finder gave an agent 321 self-describing handles. Plenty of production software I have audited would give it a few dozen and a lot of anonymous boxes.

Those defects were already worth fixing. The original reasons have not changed. Users who rely on screen readers still cannot operate your product. The EAA and ADA exposure is still sitting there. What has changed is that the same missing attributes now degrade a second class of consumer and that one is metered per turn.

If you want to know how your product reads to software rather than to people, that is what an accessibility audit answers. I go through the interface control by control, find the elements that describe themselves to nothing and give you the fixes in priority order. It is the same pass that got CeHDI to WCAG compliance and it now covers both consumers at once. Get in touch if you want your product looked at properly.

If the agent workflow half of this is what caught your attention, I run a community called the Agentic Architect AI Lab where I share the tooling I actually use in production, including this one.

Sources

Building with AI beyond this article?

I run The Agentic Architect Lab, live builds, agent workflows, and a playbook for technical founders shipping solo. No toy demos.

Join the Lab
Next Post →