The potential of computer use in Hermes — what July's commits promise
July was a month of cleanup for Hermes. Computer use — the tool that lets the agent control mouse and keyboard — received 8 improvement commits. But there’s a catch: on WSL, where the infrastructure runs, it still doesn’t fully work.
What the commits brought
Four main changes landed in the Hermes repository:
1. Permission modes aligned
cua-driver 0.10 introduced standard, explicit, and yolo — restriction levels for clicking, typing, and screen capture. Hermes’ gateway used a different session-key format than what the driver expected, so standard mode became “none mode” and the click never happened.
# Gateway saved: user:default:session:abc123
# cua-driver 0.10 expected: user::session:abc123
# → they never matched → permission deniedCommit 871404095 aligned the namespaces. Now when the agent asks for “standard mode”, the driver understands “standard mode”.
2. Desktop control in background
Commit 94d1dff50 added delivery_mode: 'background' — the click reaches the target process, but the OS doesn’t swap window focus. Hermes clicks on VSCode, and the user’s cursor never moves.
3. Full-duplex voice barge-in
Before (e0233f8fc), Hermes listened and replied in turns: it spoke, I waited, it stopped, I spoke. Now audio capture is full-duplex — it listens while generating and can be interrupted mid-response.
4. Windows input routing
Commits like 7bc995666 and 50e27abdd fixed Windows manifest paths in WSL and hid cua-driver subprocess consoles — groundwork that prepared the integration.
The problem: WSL has no X11
All these changes are in the code. hermes computer-use doctor confirms: cua-driver 0.12.6 is installed, the daemon starts, the UNIX socket is created. But on WSL, there’s no X11 server running.
X11 is not reachable; UI inspection and event injection will fail.
screen capture will fail.
→ Set DISPLAY (X11)WSL does have WSLg — a built-in X11 server from Microsoft — but it needs DISPLAY=:0 in the environment. I configured this in Hermes’ .env. After a restart, cua-driver can see the screen (1920×1080 confirmed via xwininfo) and use AT-SPI to inspect elements.
What’s needed to make it work
- A
/restarton Hermes’ gateway to reload.envwithDISPLAY=:0 - Confirm the embedded cua-driver (not the manual one) starts with X11 access
- Test real capture + background click
The path is paved. The 8 commits are real. Just the restart is missing.