forked from Blendman974/faceclaw
423 lines
22 KiB
Markdown
423 lines
22 KiB
Markdown
# Voice Assistant Architecture
|
||
|
||
Status: design draft, 2026-07-19; revised 2026-07-24 (tool availability
|
||
ontology, misc fixes). **Phase 1 is implemented** (see the Implementation plan
|
||
below); phases 2-4 are still design-only.
|
||
|
||
## Goal
|
||
|
||
A default voice assistant reachable from anywhere in the shell: hold the
|
||
button, speak, and the utterance goes to an agent that can answer in text on
|
||
the lenses and act on the glasses through tools. The agent is pluggable:
|
||
|
||
- **Direct mode**: the phone runs the agent loop itself against the Anthropic
|
||
Messages API (tool-use loop, conversation state on-phone).
|
||
- **External mode**: the utterance is routed to the user's own long-running
|
||
agent (OpenClaw, Hermes, or anything similar running on their machine), and
|
||
that agent gets access to the glasses' tools while it works.
|
||
|
||
Tools come from two sources, with four availability tiers (see "Tool
|
||
availability ontology" below):
|
||
|
||
- **Glasses/system tools**: provided by the shell, always available (show
|
||
text, notifications, screen state, open app, ...).
|
||
- **App tools**: contributed by apps. Each app tool declares how much of the
|
||
app needs to exist for the tool to be live: merely **installed** ("start a
|
||
5 minute timer" must not require the Timer app to be open), **open**
|
||
("next track" needs a running Music window, foreground or not), or
|
||
**foreground** ("scroll down" only makes sense aimed at the foreground app).
|
||
|
||
## What we already have
|
||
|
||
- **PTT capture UI**: `VoiceInputLayer` (app/ui/apps/voice-input.ts) handles
|
||
push-to-talk, live transcripts, and a Send/Continue/Discard menu; `Shell`
|
||
opens it on long-press and delivers Send to the foreground window's
|
||
`receiveTextInput` (app/ui/shell/shell.ts).
|
||
- **Wakeword**: On-device classifier listens for the wakeword "Hey Even",
|
||
which sends a message to the phone which wakes the screen and starts voice
|
||
input.
|
||
- **STT**: `voiceControlBridge` with onboard Moonshine, Whisper, or ElevenLabs
|
||
cloud.
|
||
- **LLM client**: `streamAnthropicMessage` (app/native/anthropic.ts) — raw
|
||
SSE streaming over `FaceclawSseRequest`. Text-only today; no tool use.
|
||
- **Worker app protocol**: `WorkerAppMessage`/`WorkerAppReply`
|
||
(app/ui/shell/worker-window.ts) — small-JSON postMessage protocol between
|
||
shell and app workers; the natural place to add tool declarations.
|
||
- **Dial-out websocket precedent**: `G2MirrorClient` — JSON over a websocket
|
||
to a server on the user's machine (tailscale host/port/token settings).
|
||
The external-agent bridge should copy this shape.
|
||
|
||
## Architecture overview
|
||
|
||
```
|
||
main isolate (shell)
|
||
┌───────────────────────────────────────────────────────────┐
|
||
│ Shell ──(PTT)──> VoiceInputLayer ──> AssistantSession │
|
||
│ │ │
|
||
│ AssistantLayer (overlay UI) <──streaming──┤ │
|
||
│ │ │
|
||
│ AssistantBackend │
|
||
│ ┌────────┴─────────┐ │
|
||
│ │ Direct │ External│ │
|
||
│ └────┬────┴────┬────┘ │
|
||
│ │ │ │
|
||
│ ToolRegistry <────────────────────────┴─────────┘ │
|
||
│ ├─ system tools (in-process handlers) │
|
||
│ ├─ app "installed" tools (shell-side or launch-on-call) │
|
||
│ └─ app "open"/"foreground" tools (proxied to app worker) │
|
||
└──────────┬──────────────────────────────────────┬─────────┘
|
||
│ postMessage │
|
||
app worker (e.g. terminal) Anthropic API (SSE)
|
||
set-tools / tool-call / or agent bridge (WS)
|
||
tool-result on the user's machine
|
||
```
|
||
|
||
Everything assistant-side lives on the main isolate. Rationale: tool handlers
|
||
need shell state (window list, focus, notifications, settings), the agent
|
||
loop is I/O-bound not CPU-bound, and the overlay UI is a shell layer anyway.
|
||
If conversation rendering ever gets expensive it can move to a worker later
|
||
without changing the tool protocol, which is already async message passing.
|
||
|
||
### Core pieces
|
||
|
||
1. **`ToolRegistry`** (new, `app/assistant/tool-registry.ts`)
|
||
- Holds `ToolSpec`s and handlers; answers `listTools(scopeFilter)` and
|
||
`callTool(name, args) -> Promise<ToolResult>`.
|
||
- Emits `onToolsChanged` (needed for MCP `tools/list_changed` in external
|
||
mode, and consulted at each turn boundary in direct mode).
|
||
|
||
2. **`AssistantBackend`** (interface) with two implementations:
|
||
- `DirectAnthropicBackend`: tool-use loop over the Messages API.
|
||
- `ExternalAgentBackend`: websocket bridge to the user's agent.
|
||
|
||
3. **`AssistantSession`**: one conversation. Owns history/turn state,
|
||
exposes `sendUtterance(text, ctx)`, `cancel()`, and streaming callbacks
|
||
(`onTextDelta`, `onToolActivity`, `onTurnDone`, `onError`). Both backends
|
||
implement the same session surface so the UI is backend-agnostic.
|
||
|
||
4. **`AssistantLayer`**: shell overlay showing streamed reply text, a status
|
||
line for tool activity ("→ calendar.list_events"), and a small menu
|
||
(Follow-up / Done). Double-click cancels the in-flight turn.
|
||
|
||
## Entry points and routing
|
||
|
||
- **Wakeword** Saying "Hey Even" wakes the screen if necessary and opens the
|
||
dictation dialog. For one-off voice commands, this will be the main entry
|
||
point.
|
||
- **Context menu** The sidebar context menu and app context menu already have
|
||
a "Voice input" option, which opens the dictation dialog whose Send goes to
|
||
the foreground window. We'll split the Send option in this dialog into two
|
||
options, "Send to Assistant" and "Type Into App". If the entry point was via
|
||
context menu, "Type Into App" is highlighted by default; if the entry point
|
||
was the wakeword, "Send to Assistant" is highlighted by default instead.
|
||
(The capture UI is the existing `VoiceInputLayer` flow, with Send handing off
|
||
to `AssistantSession` and morphing into `AssistantLayer`).
|
||
|
||
## Tool model
|
||
|
||
```ts
|
||
type ToolAvailability =
|
||
| "always" // system tools; no app involved
|
||
| "installed" // app tool, live whenever the app is installed —
|
||
// the app need not have any window open
|
||
| "open" // app tool, live while the app has a window open,
|
||
// foreground or backgrounded
|
||
| "foreground"; // app tool, live only while the app owns the
|
||
// foreground window
|
||
|
||
type ToolSpec = {
|
||
name: string; // namespaced: "glasses.show_notification",
|
||
// "app.terminal.send_input"
|
||
description: string;
|
||
inputSchema: object; // JSON Schema
|
||
availability: ToolAvailability;
|
||
/** May be invoked by the external agent outside an active voice turn. */
|
||
proactive?: boolean;
|
||
timeoutMs?: number; // default 10s; registry enforces
|
||
};
|
||
|
||
type ToolResult =
|
||
| { ok: true; content: string } // text payload back to the model
|
||
| { ok: false; error: string };
|
||
```
|
||
|
||
### Tool availability ontology
|
||
|
||
Every tool sits in exactly one tier, and the tier answers two questions: *when
|
||
does the tool appear in `listTools()`*, and *who handles the call*.
|
||
|
||
| Tier | Live when | Handler | Examples |
|
||
|---|---|---|---|
|
||
| `always` | always | shell, in-process | `glasses.get_state`, `notifications.list` |
|
||
| `installed` | app installed (launcher entry exists) | shell-side handler registered with the app entry, or launch-the-app-then-proxy | `timer.set` ("start a 5 minute timer" must work with no Timer window), `music.play` (cold-start playback) |
|
||
| `open` | app has ≥1 open window, foreground or not | app worker (or in-process window) via tool-call proxy | `music.next_track`, `terminal.send_input` |
|
||
| `foreground` | app owns the foreground window | app worker via tool-call proxy | `teleprompt.scroll_down`, anything meaning "act on what I'm looking at" |
|
||
|
||
Mechanics per tier:
|
||
|
||
- **`installed`** tools cannot be declared by the worker (it may not be
|
||
running), so they are declared statically alongside the app's launcher
|
||
registration. Two handler flavors: (a) a shell-side handler function
|
||
registered with the entry — right for tools that don't actually need the
|
||
app UI (timers, starting playback); (b) *launch-on-call*: the shell opens
|
||
the app (as if launched from the launcher), waits for its `set-tools`
|
||
declaration, then proxies the call. Start with (a) only; (b) adds a
|
||
launch-latency/timeout dance that no phase-1/2 tool needs.
|
||
- **`open`** and **`foreground`** tools are declared by the running app via
|
||
`set-tools` (worker apps) or direct registration (in-process windows).
|
||
The registry filters `foreground` tools by whether the declaring app owns
|
||
the current foreground window at list time and again at call time —
|
||
a `foreground` tool called after a foreground swap fails with a normal tool
|
||
error rather than acting on a window the user is no longer looking at.
|
||
- A tier is a *floor*, not a routing rule: an `installed` tool stays listed
|
||
while the app is open or foreground. If an app wants different behavior when
|
||
its window exists (e.g. `timer.set` also shows a countdown in the open
|
||
Timer window), the shell-side handler checks for the window itself.
|
||
|
||
> **Flag:** with multiple windows per app (the worker protocol is
|
||
> per-window), tool names are namespaced per *app*, so two windows of one
|
||
> app declaring the same tool name would collide. Rule: `set-tools`
|
||
> replaces the app's toolset per window, and a call routes to the declaring
|
||
> window; if two windows declare the same name, last declaration wins and
|
||
> earlier ones are dropped with a logged warning. Revisit only if a real
|
||
> app needs per-window tools with the same name (none of the planned ones
|
||
> do).
|
||
|
||
### System tools (initial set)
|
||
|
||
| Tool | Notes |
|
||
|---|---|
|
||
| `glasses.show_alert` | short text popup on the lenses; proactive-capable |
|
||
| `glasses.list_windows` / `glasses.focus_window` / `glasses.open_app` | shell control |
|
||
| `glasses.get_state` | screen on/off, foreground app, battery levels, time |
|
||
| `glasses.read_screen` | text/description of the current foreground surface (start with window title + app-reported summary; pixel OCR is out of scope) |
|
||
| `calendar.list_events` | wraps FaceclawCalendarProvider |
|
||
| `media.play_pause` / `media.next` / `media.now_playing` | wraps FaceclawMediaController |
|
||
| `notifications.list` / `notifications.dismiss` | wraps notification listener |
|
||
| `timer.set` / `timer.cancel` | new, but trivial and high-value for an assistant |
|
||
|
||
Keep the first release to ~8–12 tools; every tool costs prompt tokens and
|
||
model attention in direct mode.
|
||
|
||
Note that `calendar.*`, `media.*`, and `timer.*` are conceptually
|
||
`installed`-tier app tools that happen to be implemented by in-process shell
|
||
code; listing them here just means the shell registers their handlers
|
||
directly. If any of them later becomes a real app, its tools move to the
|
||
app's launcher registration without the model-visible surface changing.
|
||
|
||
### App tools
|
||
|
||
Running apps declare `open`/`foreground` tools per window over the existing
|
||
worker protocol (`installed` tools are declared statically with the launcher
|
||
entry, per the ontology above):
|
||
|
||
```ts
|
||
// WorkerAppReply additions (worker -> shell)
|
||
| { type: "set-tools"; windowId: string; tools: ToolSpec[] } // replaces the set
|
||
| { type: "tool-result"; callId: string; result: ToolResult }
|
||
|
||
// WorkerAppMessage additions (shell -> worker)
|
||
| { type: "tool-call"; callId: string; windowId: string; name: string; args: unknown }
|
||
```
|
||
|
||
Registry rules:
|
||
|
||
- Liveness follows the availability tier: `open` tools are removed when the
|
||
app's last window closes, `foreground` tools additionally drop out of
|
||
`listTools()` when the app loses the foreground. Either transition fires
|
||
`onToolsChanged`; in-flight calls to a closed window resolve as errors,
|
||
and calls to a `foreground` tool are re-checked at dispatch time.
|
||
- Names are auto-prefixed `app.<appId>.` by the registry so apps can't
|
||
shadow system tools or each other.
|
||
- The shell enforces `timeoutMs` on the postMessage round-trip; a hung
|
||
worker yields a tool error, not a hung turn (same philosophy as input
|
||
handling — the shell must survive a stuck app).
|
||
- In-process shell apps (calendar, music, ...) register handlers directly
|
||
with the registry; the worker protocol is just the remote flavor of the
|
||
same registration.
|
||
- **Mid-turn availability changes** (foreground swap, window close): direct
|
||
mode re-lists tools at each loop iteration (tool set is per-API-call
|
||
anyway); external mode sends MCP `notifications/tools/list_changed`. A
|
||
call to a tool that just vanished returns a normal tool error the model
|
||
can react to.
|
||
|
||
First app to wire up: the terminal —
|
||
`app.terminal.send_input` (type into the attached session; `open`,
|
||
proactive: false — "rerun the build" should work while the terminal is
|
||
backgrounded), `app.terminal.read_screen` (current grid contents; `open`),
|
||
`app.terminal.list_sessions` (`open`). This immediately enables "tell the
|
||
terminal to rerun the build" style commands and doubles as the reference
|
||
implementation — deliberately all `open`-tier; the first `foreground` tool
|
||
should come from an app where focus genuinely matters (teleprompt scrolling).
|
||
|
||
## Direct mode: agent loop on the phone
|
||
|
||
Extend `app/native/anthropic.ts`:
|
||
|
||
1. **Streaming tool-use support**: handle `content_block_start` with
|
||
`tool_use`, accumulate `input_json_delta`, surface complete tool_use
|
||
blocks alongside text deltas. (Today the client only reads `text_delta`.)
|
||
2. **`runAgentTurn` loop** (new, `app/assistant/direct-backend.ts`):
|
||
messages + `tools` from the registry → stream → if `stop_reason ==
|
||
"tool_use"`, execute each call via the registry, append `tool_result`
|
||
blocks, continue; until `end_turn`/`refusal` or a safety cap
|
||
(max 8 loop iterations, max ~2 min per turn).
|
||
3. **Session state**: history kept in memory per `AssistantSession`,
|
||
trimmed from the head past a token budget. Sessions idle-expire
|
||
(~10 min) so a new PTT starts fresh; "Follow-up" continues the session.
|
||
4. **System prompt**: glasses context — output renders on a 576×288
|
||
monochrome HUD, so answer in 1–3 short sentences, no markdown, no lists
|
||
unless asked; prefer acting via tools over describing; current time,
|
||
foreground app, and screen state are injected per turn.
|
||
5. Model `claude-sonnet-5`, `effort: "low"` (same latency posture as the
|
||
dictation refiner). Model/effort become settings later if needed.
|
||
|
||
## External mode: bridge to the user's agent
|
||
|
||
### Topology
|
||
|
||
The phone **dials out** (g2mirror pattern — phones don't accept inbound
|
||
connections reliably; tailscale host + port + token settings) to a small
|
||
**bridge server** co-located with the user's agent. One websocket, JSON
|
||
frames, three multiplexed channels:
|
||
|
||
```
|
||
{ v: 1, chan: "ctl" | "chat" | "mcp", ... }
|
||
|
||
ctl: hello {deviceName, token, capabilities}, hello-ack, ping/pong, error
|
||
chat: utterance {turnId, text, ctx} (phone -> agent)
|
||
text-delta {turnId, text} (agent -> phone)
|
||
tool-activity {turnId, label} (agent -> phone, optional status)
|
||
turn-done {turnId, stopReason} / turn-error {turnId, message}
|
||
cancel {turnId} (phone -> agent)
|
||
mcp: { msg: <raw MCP JSON-RPC frame> } (bidirectional)
|
||
```
|
||
|
||
On the `mcp` channel the **phone is an MCP server** (MCP roles come from the
|
||
initialize handshake, not from who dialed). It serves `tools/list` from the
|
||
ToolRegistry, `tools/call` into it, and emits `tools/list_changed` on
|
||
foreground changes. This means the agent-side integration needs zero
|
||
Faceclaw-specific tool code — the glasses appear as a normal MCP server.
|
||
|
||
### Agent-side adapters
|
||
|
||
The bridge server is a small standalone package (`faceclaw-agent-bridge`,
|
||
sibling of g2mirror) with two faces:
|
||
|
||
1. **MCP face**: re-exposes the phone's MCP channel as a localhost
|
||
streamable-HTTP MCP endpoint. The user registers
|
||
`http://localhost:<port>/mcp` with their agent (OpenClaw, Hermes, Claude
|
||
Code, anything MCP-capable) exactly like any other MCP server. When the
|
||
phone is disconnected, `tools/list` returns empty rather than erroring.
|
||
2. **Chat face**: pluggable per-agent adapter that forwards `utterance`
|
||
frames into the agent and streams replies back:
|
||
- **OpenClaw**: a plugin that injects the utterance as an inbound message
|
||
on a dedicated "glasses" channel/session and relays the streamed reply.
|
||
- **Hermes**: equivalent module against its session API.
|
||
- **Fallback adapter**: spawn a configured CLI (e.g. `claude -p --resume`)
|
||
per turn — lowest-fidelity but makes the bridge useful with no plugin.
|
||
|
||
Utterance `ctx` carries `{foregroundApp, screenOn, localTime}` so the remote
|
||
agent has the same situational grounding the direct loop gets in its system
|
||
prompt.
|
||
|
||
### Proactive agent actions
|
||
|
||
Because the MCP channel is live whenever the bridge is connected, the
|
||
external agent can call glasses tools **outside a voice turn** — e.g. push a
|
||
notification to the lenses when a long job finishes. Gating:
|
||
|
||
- Only tools marked `proactive: true` are callable outside an active turn
|
||
(registry-enforced; enforced on the phone, never trusted to the bridge).
|
||
Availability tiers apply to proactive calls unchanged: a proactive
|
||
`installed` tool (e.g. `timer.set`) works anytime, while a proactive call
|
||
to an `open`/`foreground` tool whose app isn't in the right state gets the
|
||
same tool error a mid-turn call would.
|
||
- Master setting `assistant.allowProactive` (default on — it's a marquee
|
||
feature — but visible and easy to turn off).
|
||
- Proactive display actions are rate-limited (e.g. 6/min) and never turn
|
||
the screen on unless the user enabled that specifically.
|
||
|
||
## Settings
|
||
|
||
New "Assistant" section in dashboard settings:
|
||
|
||
- `assistant.backend`: `anthropic` | `external` (default `anthropic` if an
|
||
API key is set, else `external` if a bridge host is set).
|
||
- `assistant.bridgeHost` / `assistant.bridgePort` / `assistant.bridgeToken`
|
||
(mirrors the terminal.* trio).
|
||
- `assistant.allowProactive`: boolean.
|
||
- `voice.skipConfirmationAfterWakeword`: Wakeword-triggered input sends to the
|
||
assistant immediately at the end of an utterance, rather than waiting for a
|
||
menu selection.
|
||
- Reuses existing `llm.anthropicApiKey` for direct mode.
|
||
|
||
## Security notes
|
||
|
||
- Bridge auth: bearer token in the `hello` frame, TLS optional because the
|
||
expected transport is tailscale (same stance as g2mirror); document that
|
||
plainly.
|
||
- The phone enforces all tool gating (proactive flags, rate limits,
|
||
timeouts). The bridge and agent are treated as honest-but-fallible.
|
||
- Tool results may contain private data (calendar, notifications); external
|
||
mode ships them to the user's own machine only — no third-party hop. In
|
||
direct mode they go to the Anthropic API like any other request content.
|
||
- App tools execute inside the app's worker with the app's existing
|
||
privileges; the registry adds no new capability beyond what the app could
|
||
already do.
|
||
|
||
## Implementation plan
|
||
|
||
**Phase 1 — direct assistant, system tools only.** ✅ Landed 2026-07-24
|
||
(builds + typechecks; not yet hardware-tested).
|
||
ToolRegistry (`app/assistant/tool-registry.ts`) with the full availability-tier
|
||
model but only `always` tools wired; 8 system tools (`glasses.get_state`,
|
||
`glasses.show_alert`, `calendar.list_events`, `media.now_playing`/`play_pause`/
|
||
`next`, `notifications.list`/`dismiss`) in `app/assistant/system-tools.ts`;
|
||
tool-use streaming in `anthropic.ts` (content-block assembly, `tools` param);
|
||
`DirectAnthropicBackend` + `AssistantSession`; `AssistantLayer` overlay
|
||
(`app/ui/apps/assistant.ts`); `VoiceInputLayer` reworked to route via a
|
||
`sendTargets` list ("Send to Assistant" / "Type Into App") with entry-point
|
||
default highlight and wakeword skip-confirmation auto-send;
|
||
`shell.showAlert` popup; `assistant.skipConfirmationAfterWakeword` setting.
|
||
*Deliverable: speak wakeword, ask "what's on my calendar", get an answer
|
||
on-lens.*
|
||
|
||
**Phase 2 — app tools.**
|
||
Worker protocol extension (set-tools / tool-call / tool-result),
|
||
availability-tier filtering in the registry (`open` on window open/close,
|
||
`foreground` on foreground change), `installed`-tier registration on launcher
|
||
entries (shell-side handlers only; launch-on-call deferred), terminal app
|
||
tools as the reference. *Deliverable: "run the build again" typed into the
|
||
terminal even while it's backgrounded.*
|
||
|
||
**Phase 3 — external bridge.**
|
||
Bridge wire protocol; ExternalAgentBackend (phone side, reusing the
|
||
FaceclawWebSocket plumbing); `faceclaw-agent-bridge` server with the MCP
|
||
face + fallback CLI chat adapter; settings; reconnect/offline behavior.
|
||
*Deliverable: utterances answered by the user's own agent, which can also
|
||
call glasses tools mid-task.*
|
||
|
||
**Phase 4 — OpenClaw/Hermes adapters + proactive polish.**
|
||
Native chat adapters for both agents; proactive gating + rate limits;
|
||
`tools/list_changed` on foreground swap; session persistence across
|
||
reconnects.
|
||
|
||
## Open questions (deferred, not blockers):
|
||
|
||
- Continuous conversation: after a turn ends, listen for a follow-up
|
||
utterance without requiring the wakeword/PTT again? There are some UI
|
||
design questions here; maybe multi-turn conversations take the form of an
|
||
Assistant app.
|
||
- Launch-on-call for `installed`-tier tools that genuinely need the app UI
|
||
(open the app, await `set-tools`, proxy the call) — deferred until a tool
|
||
needs it.
|
||
- `glasses.read_screen` fidelity beyond app-reported summaries? We probably
|
||
don't want to do OCR, but we might do something like extending the draw-text
|
||
draw call with an on-by-default option that also appends the drawn text to
|
||
a string that gets submitted along with the frame.
|
||
- Whether assistant conversations should be reviewable in the phone UI
|
||
(transcript log) — probably yes, cheap once sessions are objects. The
|
||
natural UI home for this would be inside an Assistant app window.
|
||
|