diff --git a/notes/apps.txt b/notes/apps.txt deleted file mode 100644 index c6d66831..00000000 --- a/notes/apps.txt +++ /dev/null @@ -1,87 +0,0 @@ -# Even-Analogous Apps - -Ideally we'd like full feature parity with the stock Even app (or at least, -close enough that not-using the stock app doesn't feel like a sacrifice). -A rough list of things the stock app has that we currently don't: - -## Quicklist - -Quicklist is Even's version of a todo app, in the form of a dashboard card -that you add to and remove from with assistant voice commands. It's pretty -simple to build once a general voice assistant is in place, but I don't -actually like their design: a todo list really wants to be synchronized -with something external like Roam or Notion, so I think it'd be better to -build one of those integrations instead. - -## Stocks, News - -Even has a dashboard card which is a stock ticker. I think this is probably -misguided; while there do exist people who would want and benefit from a stock -tracker on their glasses, I do'm not sure there exist people who would want and -benefit from a stock tracker that isn't a professional-tier tool connected -to carefully chosen low-latency data sources, and neither I (nor Even) can -build that tool. - -The News dashboard card is more broadly useful, but has a similar issue: a -generic news ticker isn't really useful without a much higher degree of -customization. Probably the right thing to build instead is more like an RSS -reader. - -## Ring Connection Management - -The stock Even app is responsible for pairing the glasses with the ring (ie, -sending the glasses a configuration messages telling it to connect to the -ring's MAC address). If you pair them in the stock app and then switch to a -custom app, they remain paired indefinitely (this configuration is stored in -flash), and our custom app can make use of the fact that they're paired. - -We don't yet have support for connecting or disconnecting the glasses and ring -to/from each other. For latency reasons, it would be nice if we could unpair -the glasses from the ring and connect the phone to it directly; we don't yet -support that either. - -## Ring Health Metrics - -The R1 ring contains an oximeter-style LED sensor and an accelerometer, which -can be used for a step count, heart rate measurement, etc. I'm not sure -whether the sensor interpretation is done within the firmware (in which case -it would make sense to display them in the UI) or in the Even phone app (in -which case we'd be reimplementing all the interpretation algorithms, which -isn't worth it. - -## Navigate - -Even has a Navigate app, which is basically Google Maps for the specific use -case of "name a destination, get turn by turn directions to there". I believe -this is using Mapbox, which takes care of most of the heavy lifting (if you -have an API key). I don't think Even's version is very good (they ran into -trouble with image-upload performance preventing a good map display size), so -it's likely not too hard to do better. One caveat is the location permission; -Android normally only wants to prompt for location permission "while using the -app", but Android's idea of in use (phone screen unlocked and window focused) -doesn't match our idea of in use (G2 glasses on, phone could be locked). This -app also gets a lot better if we figure out how to use the device's -magnetometer (if it indeed has a working magnetometer). - -## Distance/Viewport Configuration - -During glasses onboarding, the stock firmware has you select a vertical offset -for the screen image. This accounts for glasses resting at different positions -on your face (the optics don't let you see the whole screen area), but cuts -the usable screen height from 480px to 288px. There is also a "distance" -config option in the stock Android app, which adjusts the horizontal position -of the image on the two lenses separately to create the impression of the -image being closer or farther; this cuts the usable screen width from 640px to -576px. Unlike the vertical size reduction, this 64px reduction is not -optically necessary. - -## Even AI -## Translate and Conversate - -## Other Configuration - -Brightness, head-up trigger - - -# Other Apps - diff --git a/notes/hermes-assistant.md b/notes/hermes-assistant.md deleted file mode 100644 index 5aab3885..00000000 --- a/notes/hermes-assistant.md +++ /dev/null @@ -1,41 +0,0 @@ -# Hermes agent as the assistant - -The "Hermes agent" assistant model sends what you say to a -[Hermes Agent](https://github.com/NousResearch/hermes-agent) gateway through its -OpenAI-compatible API server and streams the answer back to the glasses. Hermes -runs its own tools and keeps the conversation: each Faceclaw assistant -conversation maps to one Hermes session (`X-Hermes-Session-Id`), and only the -latest message is sent. Faceclaw's own glasses tools are not offered to Hermes. -The session ID is saved with the conversation, so a conversation restored after -an app restart continues the same Hermes session. - -A Hermes turn may run for up to 20 minutes (other models get 2), because Hermes -runs whole multi-tool tasks in one request and closing the stream interrupts the -run on the server. A run that fails or is interrupted on the server shows as an -error on the glasses, with any partial answer kept. - -## Enable the API server in Hermes - -In the Hermes config, enable the `api_server` platform and bind it where the -phone can reach it, then set a key in Hermes's `.env` and restart the gateway: - - hermes config set platforms.api_server.enabled true - hermes config set platforms.api_server.extra.host
- hermes config set platforms.api_server.extra.port 8642 - echo "API_SERVER_KEY=$(openssl rand -hex 24)" >> ~/.hermes/.env - -Check it with `curl -H "Authorization: Bearer " http://:8642/v1/models`. - -## Settings (Settings > Assistant) - -- **Assistant model**: Hermes agent. Disabled until the URL is set. -- **Hermes URL**: e.g. `http://host:8642/v1` (a bare host gets `/v1` added). -- **Hermes key**: the `API_SERVER_KEY` value. - -On iOS, App Transport Security blocks plain `http://` to IP addresses unless -`App_Resources/iOS/Info.plist` has an `NSExceptionDomains` entry for that host -(`NSExceptionAllowsInsecureHTTPLoads`); otherwise serve the API over https, for -example with `tailscale serve`. - -Actions that Hermes holds for approval (`approvals.mode manual`) cannot be -approved from the glasses; approve them from Hermes itself. diff --git a/notes/ios-protocol-sync.md b/notes/ios-protocol-sync.md deleted file mode 100644 index ec419a95..00000000 --- a/notes/ios-protocol-sync.md +++ /dev/null @@ -1,66 +0,0 @@ -# iOS protocol synchronization — 2026-09-17 - -Merged `main` at `1727474` into `ios-port`. The iOS session now uses the -same transport contract as Android's `CfwTransport`, `CfwMessageWindow`, -`OutboundMessage`, and `BleImageOptimizer`. - -- The EvenHub page contains only the input text container. Display commands - and cleanup use SID `0xf0`, without image-container or image-update headers. -- Mode 6 full frames and mode 3 bounding boxes contain uncompressed RLE. - Persistent zlib with SYNC_FLUSH runs in wire order at the transport layer. - Record lengths and checksums describe the same data as Android; packet - sizes respect CoreBluetooth's maximum write value length (ATT MTU minus 3). -- Commands larger than the uint16 record limit become independent 64-row - mode 3 bands. Frame metrics advance only when all bands complete. -- The three-command window requires ACKs for both processing lenses on the - ingress BLE link, with matching stream ID, ordinal, decoded size, and CRC. - Primary and redundant ACKs are validated before application; completion - remains ordered even when replies arrive out of order. -- NACKs or 500 ms ACK timeouts replay the entire unresolved window, using - fresh stream IDs and reset compression contexts. Three retries are allowed - before reconnecting and invalidating the delta base. Deadlines begin after - writes finish. Old queued attempts and late ACKs cannot complete replays. -- Firmware acceptance uses shared numeric revision checks (currently - Faceclaw/13 or newer), replacing the old EVENCFW feature-token check. - Legacy or stock firmware is rejected before display startup. - -Merge conflicts retained iOS voice restrictions while preserving main's -asynchronous transcription finalization. The iOS voice-state bridge supplies -the new listening/detail fields. Test harnesses account for platform globals -and the shared SVG rasterizer. - -Validation: TypeScript checking and signed physical-device iOS build pass. -The simulator build launches and renders the main screen and shared launcher. -The final signed app is installed on the development iPhone. The user renewed -developer-certificate trust after iOS initially blocked launch. A subsequent -NativeScript deployment launched successfully, connected to the glasses, and -received both-lens ACKs for a display frame. -The transport parity test compiles Android's production Java encoder and -compares complete packets, including persistent history, reset, MTU limits, -and raw fallback for incompressible maximum-size records. Session tests cover -ACK identity, two-lens completion, ordered pipelining, band reconstruction, -NACK replay, timeout/retry limits, microphone/input responsiveness, and cleanup. - -The full suite has 420 passing tests and 21 failures. All 21 reproduce on a -clean archive of main at 1727474: compass, pinball/touch, and terminal test -harness/behavior checks. There are no additional failing tests from this merge. - -Further hardware acceptance: scroll and animate the display, visually check -both lenses, exercise mic and ring input, lock the phone, and disconnect/reconnect. -The initial successful frame does not establish sustained radio reliability. - -## Development deployment and stale LiveSync code - -Installing the IPA with `devicectl` alone left an old September 8 JavaScript -bundle in `Library/Application Support/LiveSync/app` in the app's data container. -NativeScript debug startup used that bundle instead of the newly installed -one, causing the obsolete `img640, fbguard, wearnotify` firmware rejection. -Refreshing through NativeScript fixed the issue without clearing app settings: - -```sh -npm run run:ios -- --device 00008150-000C383921A3C01C --no-hmr --no-watch -``` - -Use this deployment path when updating an existing development installation. -The refreshed on-device `bundle.mjs` was copied back and verified against the -prepared bundle; it contains the shared revision-13 check and SID `0xf0`. diff --git a/notes/media-resume-via-media-key.md b/notes/media-resume-via-media-key.md deleted file mode 100644 index 0b052996..00000000 --- a/notes/media-resume-via-media-key.md +++ /dev/null @@ -1,13 +0,0 @@ -# Music: "Resume" action via media-key dispatch - -Follow-up to the MediaBrowserService browsing support. When no media session -is active, `AudioManager.dispatchMediaKeyEvent(KEYCODE_MEDIA_PLAY)` (down+up -pair) is routed by the system to the *last* media app's manifest-declared -`MEDIA_BUTTON` receiver, even if that app's process is dead — the same -mechanism a Bluetooth headset play button uses. Most players respond by -resuming the last queue. - -Idea: add a "Resume last" row to the Music app's no-active-session screen -(next to the browse entry). ~5 lines in `FaceclawMediaController` plus one UI -row; no per-app negotiation needed. Caveat: it's fire-and-forget — no feedback -if nothing handles it, so the UI should just wait for a session to appear. diff --git a/notes/mentraos-compat-assessment.md b/notes/mentraos-compat-assessment.md deleted file mode 100644 index 274d7d84..00000000 --- a/notes/mentraos-compat-assessment.md +++ /dev/null @@ -1,133 +0,0 @@ -# MentraOS app compatibility for Faceclaw — feasibility assessment (2026-07-23) - -MentraOS: https://github.com/Mentra-Community/MentraOS (MIT), docs at https://docs.mentraglass.com. - -## TL;DR - -Technically feasible with moderate effort, but the timing is awkward: MentraOS is -**deprecating its entire current app API**. MentraOS 3.0 ships **Aug 3, 2026** and -cloud-SDK apps stop working on it (a "MentraOS Legacy" mode preserves them through -**October 2026**). The replacement — a **Miniapp SDK** where apps are phone-local JS -bundles running in a WebView — gets a public spec in **September 2026**, and that model -is a *much* better fit for Faceclaw than the cloud model. Recommendation: wait for the -miniapp spec and target that; only build a minimal cloud-protocol emulator if access to -the existing app corpus is wanted in the interim. - -## MentraOS app model (current, "Cloud SDK" / TPA era) - -- Apps are developer-hosted server processes (Node/Bun, `@mentra/sdk`). Chain: - glasses → phone (BLE) → MentraOS Cloud (WS) → app server (WS). -- Start flow: cloud POSTs a `session_request` webhook `{sessionId, userId, websocketUrl}` - to the app's registered URL; the app dials back to `websocketUrl` and sends - `tpa_connection_init {packageName, sessionId, apiKey}`; cloud acks with settings + - device capabilities. **The webhook carries the WS URL, so an unmodified `@mentra/sdk` - app can be pointed at any server** — that is the compat hook. -- Protocol: JSON over WebSocket (binary frames for audio). 28 app→cloud + 28 cloud→app - message types, defined in `cloud/packages/sdk/src/types/` (`message-types.ts`, - `streams.ts`, `layouts.ts`, `webhooks.ts`, `capabilities.ts`). The TS types are the spec. -- Subscriptions: `subscription_update` with a full list of stream types, some - parameterized (`transcription:en-US`, `touch_event:forward_swipe`). Events arrive in a - `data_stream` envelope. -- Display: 7 layout types — `text_wall`, `double_text_wall`, `dashboard_card`, - `reference_card`, `bitmap_view` (base64), `bitmap_animation`, `clear_view` — sent as - `display_event {view: 'main'|'dashboard', layout, durationMs?, forceDisplay?}`. - Cloud throttles displays to 1 per 300 ms. Dashboard = shared look-up surface. -- Streams: `transcription` (`{text, isFinal, ...}`), `translation`, `VAD`, `audio_chunk` - (PCM), `button_press`, `touch_event`, `head_position` (up/down), `glasses_battery_update`, - `phone_battery_update`, `location_update`, `phone_notification`(+dismissed), - `calendar_event`, plus camera/RTMP/LED families irrelevant to G2. -- Other: ElevenLabs TTS (`audio.speak`), per-app settings (schema in dev console, 9 - setting types, live `settings_update`), webview UI with signed-token auth, cloud KV - storage, app-to-app messaging. -- Hardware abstraction: `Capabilities` flags per device. **Even Realities G2 is a - first-class device** (`capabilities/even-realities-g2.ts`): display 640×200 (their - rendering canvas, not the 576×288 EvenHub raster), maxTextLines 5, bitmap-capable, - capacitive swipe bar with TAP/DOUBLE_TAP/TRIPLE_TAP/PRESS_HOLD/SWIPE_UP/SWIPE_DOWN, - IMU, mic, no camera/speaker/LED. -- Licensing: MIT throughout; cloud is self-hostable in principle (Bun + MongoDB + S3 + - third-party STT/TTS keys) but heavy. - -## Coming model (Miniapp SDK, MentraOS 3.0) - -Miniapps are JS bundles (`index.html` + `miniapp.json`) running **on the phone** inside -the MentraOS host app's WebView ("Mentra Runtime"), packages `@mentra/miniapp` + -`@mentra/miniapp-cli`, sideloaded via QR. API is analogous but smaller: -`session.transcription`, `session.display.showTextWall`, `session.storage`. Example: -https://github.com/Mentra-Community/LiveCaptionsLocalSdkMiniapp. Public spec Sept 2026. - -## Fit with Faceclaw - -Faceclaw's worker-per-app architecture is unusually favorable: apps are already isolated -contexts speaking a small typed JSON vocabulary (`WorkerAppMessage`/`WorkerAppReply` in -`app/ui/shell/worker-window.ts`), with pixels out-of-band. The compat layer would be one -compiled-in "MentraOS host" app (a peer of `WorkerAppHost`) that presents each external -app as a `ShellWindow` and: - -- **Rasterizes layouts** → trivial. All 7 layout types map onto `GrayImage` + - `drawTextWrapped`/`bitBlt` into the 540×260 `APP_VIEWPORT`. Bitmaps need scaling from - their 640×200/576×136 assumptions; monochrome already matches. -- **Maps input** → good. `DashboardInputEvent` (`click`/`double-click`/`long-press`/ - `scroll-up`/`scroll-down`) covers their G2 button-event list almost 1:1 (TRIPLE_TAP - missing). Work item: input is currently routed only to the focused window; MentraOS - wants multi-subscriber fan-out (background apps), so a broadcast layer is needed. -- **Transcription** → the load-bearing stream, and Faceclaw already has it: - `voiceControlBridge.onTranscript` (`{text, isFinal}`, replace semantics — matches - MentraOS's `TranscriptionData` model well). Onboard Moonshine is English-only; - parameterized languages would fall back to ElevenLabs/Whisper cloud providers. - Needs multi-consumer fan-out and a policy for when the mic is open (MentraOS keeps it - effectively continuous; Faceclaw is PTT/wakeword-oriented — battery implications). -- **VAD** → sherpa endpointing exists. **audio_chunk** → PCM already surfaced on the - ElevenLabs path (`onPcm`). -- **Battery / phone notifications / calendar** → data already flows in Faceclaw - (top-bar battery, notification listener, calendar app); just needs re-emission as - streams. -- **head_position** → gap. Faceclaw exposes raw IMU only (no gesture classification); - the look-up dashboard gesture would need a small accelerometer classifier (cmd 19 - stream exists, see imu-accelerometer memory). -- **TTS / audio out** → G2 has no speaker; MentraOS plays through the phone. Faceclaw - would need phone-side audio playback (new but simple Android work). -- **Settings** → MentraOS per-app settings map onto the shared `FaceclawSettings` store - with a `mentra..*` prefix; the 9 setting types map onto - `ConfigSetting` subclasses, mostly. -- **Not applicable on G2**: camera/photo/RTMP/LiveKit, LED, WiFi setup — can be - rejected via capabilities, which the protocol explicitly supports. - -## The hard parts - -1. **Who plays "cloud"?** Cloud-SDK apps dial out to a WS URL given in a webhook, so - Faceclaw must (a) run a WS **server** — none exists anywhere in the codebase today, - all networking is outbound okhttp client (`FaceclawWebSocket`, g2mirror pattern) — - reachable from wherever the app server runs (fine on LAN; NAT pain otherwise), and - (b) send the session webhook itself (simple outbound HTTP POST). A phone-hosted - mini-cloud with Java-WebSocket/NanoWSD listening on LAN is the sane MVP; self-hosting - their actual cloud (Bun+Mongo+Docker) is overkill. -2. **Session lifecycle**: connection init/ack, `subscription_update` bookkeeping, - `reconnect`/`reconnect_ack/rejected/deferred`, `app_stopped`, per-app permission - gating. This is bookkeeping-heavy but not deep. -3. **Display arbitration**: foreground vs `background`+`forceDisplay` apps, `durationMs` - auto-revert, the shared dashboard view. Faceclaw's shell has focus/foreground but no - equivalent of background-app display preemption or a look-up dashboard. -4. **App discovery/registration UX**: no dev console; needs a simple "add app by URL + - API key" settings flow (API key check can be a no-op or shared secret). - -## Effort estimate - -MVP covering the bulk of display-glasses apps (live captions, teleprompters, dashboards, -notification relays): ~10–15 message types + ~8 stream types + webhook sender + LAN WS -server + layout rasterizer + settings mapping. Roughly a few weeks of focused work, with -transcription-stream policy and the WS server being the riskiest pieces. Full-surface -emulation (webview auth, app messaging rooms, TTS, location tiers) is substantially more -and mostly not worth it. - -## Recommendation - -- **Don't invest in the cloud-TPA protocol as the long-term target** — the vendor is - killing it within ~3 months of now, and the ecosystem's apps will migrate to miniapps. -- **Target the Miniapp SDK when its spec lands (Sept 2026).** It's strictly easier for - Faceclaw: JS bundle in an Android WebView with an injected `session` bridge — no - server, no webhooks, no reconnect protocol — and Faceclaw already has every underlying - facility the bridge needs. The rasterizer/input/transcription plumbing built for it - would be shared with any interim TPA emulator anyway. -- If interim access to existing cloud-SDK apps matters, build the minimal LAN emulator - described above, scoped to text layouts + transcription + button streams, and treat it - as disposable. diff --git a/notes/voice-assistant-design.md b/notes/voice-assistant-design.md deleted file mode 100644 index 675d1c8d..00000000 --- a/notes/voice-assistant-design.md +++ /dev/null @@ -1,466 +0,0 @@ -# Voice Assistant Architecture - -Status: design draft, 2026-07-19; revised 2026-07-24 (tool availability -ontology, misc fixes); revised 2026-08-01 (external bridge landed). -**Phases 1 and 2 are implemented**, and **phase 3 + the OpenClaw part of -phase 4 landed 2026-08-01** with one architecture change: the bridge server -is not a standalone package — it *is* the OpenClaw plugin -(`~/repositories/faceclaw-agent-bridge`, installed on the OpenClaw host). -The plugin hosts the websocket endpoint the phone dials into, runs -utterances as embedded OpenClaw agent turns (`runEmbeddedAgent` with a -dedicated session, default key `faceclaw:glasses`), and exposes the -glasses' MCP-served tools to the agent via two fixed meta-tools -(`glasses_list_tools` / `glasses_call` — fixed names because OpenClaw's -manifest `contracts.tools` wants static declarations, while the glasses -toolset is dynamic). The generic "MCP face for any agent" from the original -design is unbuilt; the wire protocol is unchanged, so a standalone bridge -for other agents can be added later without phone changes. - -## Goal - -A default voice assistant reachable from anywhere in the shell: hold the -button, speak, and the utterance goes to an agent that can answer in text on -the lenses and act on the glasses through tools. The agent is pluggable: - -- **Direct mode**: the phone runs the agent loop itself against the Anthropic - Messages API (tool-use loop, conversation state on-phone). -- **External mode**: the utterance is routed to the user's own long-running - agent (OpenClaw, Hermes, or anything similar running on their machine), and - that agent gets access to the glasses' tools while it works. - -Tools come from two sources, with four availability tiers (see "Tool -availability ontology" below): - -- **Glasses/system tools**: provided by the shell, always available (show - text, notifications, screen state, open app, ...). -- **App tools**: contributed by apps. Each app tool declares how much of the - app needs to exist for the tool to be live: merely **installed** ("start a - 5 minute timer" must not require the Timer app to be open), **open** - ("next track" needs a running Music window, foreground or not), or - **foreground** ("scroll down" only makes sense aimed at the foreground app). - -## What we already have - -- **PTT capture UI**: `VoiceInputLayer` (app/ui/apps/voice-input.ts) handles - push-to-talk, live transcripts, and a Send/Continue/Discard menu; `Shell` - opens it on long-press and delivers Send to the foreground window's - `receiveTextInput` (app/ui/shell/shell.ts). -- **Wakeword**: On-device classifier listens for the wakeword "Hey Even", - which sends a message to the phone which wakes the screen and starts voice - input. -- **STT**: `voiceControlBridge` with onboard Moonshine, Whisper, or ElevenLabs - cloud. -- **LLM client**: `streamAnthropicMessage` (app/native/anthropic.ts) — raw - SSE streaming over `FaceclawSseRequest`. Text-only today; no tool use. -- **Worker app protocol**: `WorkerAppMessage`/`WorkerAppReply` - (app/ui/shell/worker-window.ts) — small-JSON postMessage protocol between - shell and app workers; the natural place to add tool declarations. -- **Dial-out websocket precedent**: `G2MirrorClient` — JSON over a websocket - to a server on the user's machine (tailscale host/port/token settings). - The external-agent bridge should copy this shape. - -## Architecture overview - -``` - main isolate (shell) - ┌───────────────────────────────────────────────────────────┐ - │ Shell ──(PTT)──> VoiceInputLayer ──> AssistantSession │ - │ │ │ - │ AssistantLayer (overlay UI) <──streaming──┤ │ - │ │ │ - │ AssistantBackend │ - │ ┌────────┴─────────┐ │ - │ │ Direct │ External│ │ - │ └────┬────┴────┬────┘ │ - │ │ │ │ - │ ToolRegistry <────────────────────────┴─────────┘ │ - │ ├─ system tools (in-process handlers) │ - │ ├─ app "installed" tools (shell-side or launch-on-call) │ - │ └─ app "open"/"foreground" tools (proxied to app worker) │ - └──────────┬──────────────────────────────────────┬─────────┘ - │ postMessage │ - app worker (e.g. terminal) Anthropic API (SSE) - set-tools / tool-call / or agent bridge (WS) - tool-result on the user's machine -``` - -Everything assistant-side lives on the main isolate. Rationale: tool handlers -need shell state (window list, focus, notifications, settings), the agent -loop is I/O-bound not CPU-bound, and the overlay UI is a shell layer anyway. -If conversation rendering ever gets expensive it can move to a worker later -without changing the tool protocol, which is already async message passing. - -### Core pieces - -1. **`ToolRegistry`** (new, `app/assistant/tool-registry.ts`) - - Holds `ToolSpec`s and handlers; answers `listTools(scopeFilter)` and - `callTool(name, args) -> Promise`. - - Emits `onToolsChanged` (needed for MCP `tools/list_changed` in external - mode, and consulted at each turn boundary in direct mode). - -2. **`AssistantBackend`** (interface) with two implementations: - - `DirectAnthropicBackend`: tool-use loop over the Messages API. - - `ExternalAgentBackend`: websocket bridge to the user's agent. - -3. **`AssistantSession`**: one conversation. Owns history/turn state, - exposes `sendUtterance(text, ctx)`, `cancel()`, and streaming callbacks - (`onTextDelta`, `onToolActivity`, `onTurnDone`, `onError`). Both backends - implement the same session surface so the UI is backend-agnostic. - -4. **`AssistantLayer`**: shell overlay showing streamed reply text, a status - line for tool activity ("→ calendar.list_events"), and a small menu - (Follow-up / Done). Double-click cancels the in-flight turn. - -## Entry points and routing - -- **Wakeword** Saying "Hey Even" wakes the screen if necessary and opens the - dictation dialog. For one-off voice commands, this will be the main entry - point. -- **Context menu** The sidebar context menu and app context menu already have - a "Voice input" option, which opens the dictation dialog whose Send goes to - the foreground window. We'll split the Send option in this dialog into two - options, "Send to Assistant" and "Type Into App". If the entry point was via - context menu, "Type Into App" is highlighted by default; if the entry point - was the wakeword, "Send to Assistant" is highlighted by default instead. - (The capture UI is the existing `VoiceInputLayer` flow, with Send handing off - to `AssistantSession` and morphing into `AssistantLayer`). - -## Tool model - -```ts -type ToolAvailability = - | "always" // system tools; no app involved - | "installed" // app tool, live whenever the app is installed — - // the app need not have any window open - | "open" // app tool, live while the app has a window open, - // foreground or backgrounded - | "foreground"; // app tool, live only while the app owns the - // foreground window - -type ToolSpec = { - name: string; // namespaced: "glasses.show_notification", - // "app.terminal.send_input" - description: string; - inputSchema: object; // JSON Schema - availability: ToolAvailability; - /** May be invoked by the external agent outside an active voice turn. */ - proactive?: boolean; - timeoutMs?: number; // default 10s; registry enforces -}; - -type ToolResult = - | { ok: true; content: string } // text payload back to the model - | { ok: false; error: string }; -``` - -### Tool availability ontology - -Every tool sits in exactly one tier, and the tier answers two questions: *when -does the tool appear in `listTools()`*, and *who handles the call*. - -| Tier | Live when | Handler | Examples | -|---|---|---|---| -| `always` | always | shell, in-process | `glasses.get_state`, `notifications.list` | -| `installed` | app installed (launcher entry exists) | shell-side handler registered with the app entry, or launch-the-app-then-proxy | `timer.set` ("start a 5 minute timer" must work with no Timer window), `music.play` (cold-start playback) | -| `open` | app has ≥1 open window, foreground or not | app worker (or in-process window) via tool-call proxy | `music.next_track`, `terminal.send_input` | -| `foreground` | app owns the foreground window | app worker via tool-call proxy | `teleprompt.scroll_down`, anything meaning "act on what I'm looking at" | - -Mechanics per tier: - -- **`installed`** tools cannot be declared by the worker (it may not be - running), so they are declared statically alongside the app's launcher - registration. Two handler flavors: (a) a shell-side handler function - registered with the entry — right for tools that don't actually need the - app UI (timers, starting playback); (b) *launch-on-call*: the shell opens - the app (as if launched from the launcher), waits for its `set-tools` - declaration, then proxies the call. Start with (a) only; (b) adds a - launch-latency/timeout dance that no phase-1/2 tool needs. -- **`open`** and **`foreground`** tools are declared by the running app via - `set-tools` (worker apps) or direct registration (in-process windows). - The registry filters `foreground` tools by whether the declaring app owns - the current foreground window at list time and again at call time — - a `foreground` tool called after a foreground swap fails with a normal tool - error rather than acting on a window the user is no longer looking at. -- A tier is a *floor*, not a routing rule: an `installed` tool stays listed - while the app is open or foreground. If an app wants different behavior when - its window exists (e.g. `timer.set` also shows a countdown in the open - Timer window), the shell-side handler checks for the window itself. - -> **Flag:** with multiple windows per app (the worker protocol is -> per-window), tool names are namespaced per *app*, so two windows of one -> app declaring the same tool name would collide. Rule: `set-tools` -> replaces the app's toolset per window, and a call routes to the declaring -> window; if two windows declare the same name, last declaration wins and -> earlier ones are dropped with a logged warning. Revisit only if a real -> app needs per-window tools with the same name (none of the planned ones -> do). - -### System tools (initial set) - -| Tool | Notes | -|---|---| -| `glasses.show_alert` | short text popup on the lenses; proactive-capable | -| `glasses.list_windows` / `glasses.focus_window` / `glasses.open_app` | shell control | -| `glasses.get_state` | screen on/off, foreground app, battery levels, time | -| `glasses.read_screen` | text/description of the current foreground surface (start with window title + app-reported summary; pixel OCR is out of scope) | -| `calendar.list_events` | wraps FaceclawCalendarProvider | -| `media.play_pause` / `media.next` / `media.now_playing` | wraps FaceclawMediaController | -| `notifications.list` / `notifications.dismiss` | wraps notification listener | -| `timer.set` / `timer.cancel` | new, but trivial and high-value for an assistant | - -Keep the first release to ~8–12 tools; every tool costs prompt tokens and -model attention in direct mode. - -Note that `calendar.*`, `media.*`, and `timer.*` are conceptually -`installed`-tier app tools that happen to be implemented by in-process shell -code; listing them here just means the shell registers their handlers -directly. If any of them later becomes a real app, its tools move to the -app's launcher registration without the model-visible surface changing. - -### App tools - -Running apps declare `open`/`foreground` tools per window over the existing -worker protocol (`installed` tools are declared statically with the launcher -entry, per the ontology above): - -```ts -// WorkerAppReply additions (worker -> shell) -| { type: "set-tools"; windowId: string; tools: ToolSpec[] } // replaces the set -| { type: "tool-result"; callId: string; result: ToolResult } - -// WorkerAppMessage additions (shell -> worker) -| { type: "tool-call"; callId: string; windowId: string; name: string; args: unknown } -``` - -Registry rules: - -- Liveness follows the availability tier: `open` tools are removed when the - app's last window closes, `foreground` tools additionally drop out of - `listTools()` when the app loses the foreground. Either transition fires - `onToolsChanged`; in-flight calls to a closed window resolve as errors, - and calls to a `foreground` tool are re-checked at dispatch time. -- Names are auto-prefixed `app..` by the registry so apps can't - shadow system tools or each other. -- The shell enforces `timeoutMs` on the postMessage round-trip; a hung - worker yields a tool error, not a hung turn (same philosophy as input - handling — the shell must survive a stuck app). -- In-process shell apps (calendar, music, ...) register handlers directly - with the registry; the worker protocol is just the remote flavor of the - same registration. -- **Mid-turn availability changes** (foreground swap, window close): direct - mode re-lists tools at each loop iteration (tool set is per-API-call - anyway); external mode sends MCP `notifications/tools/list_changed`. A - call to a tool that just vanished returns a normal tool error the model - can react to. - -First app to wire up: the terminal — -`app.terminal.send_input` (type into the attached session; `open`, -proactive: false — "rerun the build" should work while the terminal is -backgrounded), `app.terminal.read_screen` (current grid contents; `open`), -`app.terminal.list_sessions` (`open`). This immediately enables "tell the -terminal to rerun the build" style commands and doubles as the reference -implementation — deliberately all `open`-tier; the first `foreground` tool -should come from an app where focus genuinely matters (teleprompt scrolling). - -## Direct mode: agent loop on the phone - -Extend `app/native/anthropic.ts`: - -1. **Streaming tool-use support**: handle `content_block_start` with - `tool_use`, accumulate `input_json_delta`, surface complete tool_use - blocks alongside text deltas. (Today the client only reads `text_delta`.) -2. **`runAgentTurn` loop** (new, `app/assistant/direct-backend.ts`): - messages + `tools` from the registry → stream → if `stop_reason == - "tool_use"`, execute each call via the registry, append `tool_result` - blocks, continue; until `end_turn`/`refusal` or a safety cap - (max 8 loop iterations, max ~2 min per turn). -3. **Session state**: history kept in memory per `AssistantSession`, - trimmed from the head past a token budget. Sessions idle-expire - (~10 min) so a new PTT starts fresh; "Follow-up" continues the session. -4. **System prompt**: glasses context — output renders on a 576×288 - monochrome HUD, so answer in 1–3 short sentences, no markdown, no lists - unless asked; prefer acting via tools over describing; current time, - foreground app, and screen state are injected per turn. -5. Model `claude-sonnet-5`, `effort: "low"` (same latency posture as the - dictation refiner). Model/effort become settings later if needed. - -## External mode: bridge to the user's agent - -### Topology - -The phone **dials out** (g2mirror pattern — phones don't accept inbound -connections reliably; tailscale host + port + token settings) to a small -**bridge server** co-located with the user's agent. One websocket, JSON -frames, three multiplexed channels: - -``` -{ v: 1, chan: "ctl" | "chat" | "mcp", ... } - -ctl: hello {deviceName, token, capabilities}, hello-ack, ping/pong, error -chat: utterance {turnId, text, ctx} (phone -> agent) - text-delta {turnId, text} (agent -> phone) - tool-activity {turnId, label} (agent -> phone, optional status) - turn-done {turnId, stopReason} / turn-error {turnId, message} - cancel {turnId} (phone -> agent) -mcp: { msg: } (bidirectional) -``` - -On the `mcp` channel the **phone is an MCP server** (MCP roles come from the -initialize handshake, not from who dialed). It serves `tools/list` from the -ToolRegistry, `tools/call` into it, and emits `tools/list_changed` on -foreground changes. This means the agent-side integration needs zero -Faceclaw-specific tool code — the glasses appear as a normal MCP server. - -### Agent-side adapters - -The bridge server is a small standalone package (`faceclaw-agent-bridge`, -sibling of g2mirror) with two faces: - -1. **MCP face**: re-exposes the phone's MCP channel as a localhost - streamable-HTTP MCP endpoint. The user registers - `http://localhost:/mcp` with their agent (OpenClaw, Hermes, Claude - Code, anything MCP-capable) exactly like any other MCP server. When the - phone is disconnected, `tools/list` returns empty rather than erroring. -2. **Chat face**: pluggable per-agent adapter that forwards `utterance` - frames into the agent and streams replies back: - - **OpenClaw**: a plugin that injects the utterance as an inbound message - on a dedicated "glasses" channel/session and relays the streamed reply. - - **Hermes**: equivalent module against its session API. - - **Fallback adapter**: spawn a configured CLI (e.g. `claude -p --resume`) - per turn — lowest-fidelity but makes the bridge useful with no plugin. - -Utterance `ctx` carries `{foregroundApp, screenOn, localTime}` so the remote -agent has the same situational grounding the direct loop gets in its system -prompt. - -### Proactive agent actions - -Because the MCP channel is live whenever the bridge is connected, the -external agent can call glasses tools **outside a voice turn** — e.g. push a -notification to the lenses when a long job finishes. Gating: - -- Only tools marked `proactive: true` are callable outside an active turn - (registry-enforced; enforced on the phone, never trusted to the bridge). - Availability tiers apply to proactive calls unchanged: a proactive - `installed` tool (e.g. `timer.set`) works anytime, while a proactive call - to an `open`/`foreground` tool whose app isn't in the right state gets the - same tool error a mid-turn call would. -- Master setting `assistant.allowProactive` (default on — it's a marquee - feature — but visible and easy to turn off). -- Proactive display actions are rate-limited (e.g. 6/min) and never turn - the screen on unless the user enabled that specifically. - -## Settings - -New "Assistant" section in dashboard settings: - -- `assistant.backend`: `anthropic` | `external` (default `anthropic` if an - API key is set, else `external` if a bridge host is set). -- `assistant.bridgeHost` / `assistant.bridgePort` / `assistant.bridgeToken` - (mirrors the terminal.* trio). -- `assistant.allowProactive`: boolean. -- `voice.skipConfirmationAfterWakeword`: Wakeword-triggered input sends to the - assistant immediately at the end of an utterance, rather than waiting for a - menu selection. -- Reuses existing `llm.anthropicApiKey` for direct mode. - -## Security notes - -- Bridge auth: bearer token in the `hello` frame, TLS optional because the - expected transport is tailscale (same stance as g2mirror); document that - plainly. -- The phone enforces all tool gating (proactive flags, rate limits, - timeouts). The bridge and agent are treated as honest-but-fallible. -- Tool results may contain private data (calendar, notifications); external - mode ships them to the user's own machine only — no third-party hop. In - direct mode they go to the Anthropic API like any other request content. -- App tools execute inside the app's worker with the app's existing - privileges; the registry adds no new capability beyond what the app could - already do. - -## Implementation plan - -**Phase 1 — direct assistant, system tools only.** ✅ Landed 2026-07-24 -(builds + typechecks; not yet hardware-tested). -ToolRegistry (`app/assistant/tool-registry.ts`) with the full availability-tier -model but only `always` tools wired; 8 system tools (`glasses.get_state`, -`glasses.show_alert`, `calendar.list_events`, `media.now_playing`/`play_pause`/ -`next`, `notifications.list`/`dismiss`) in `app/assistant/system-tools.ts`; -tool-use streaming in `anthropic.ts` (content-block assembly, `tools` param); -`DirectAnthropicBackend` + `AssistantSession`; `AssistantLayer` overlay -(`app/ui/apps/assistant.ts`); `VoiceInputLayer` reworked to route via a -`sendTargets` list ("Send to Assistant" / "Type Into App") with entry-point -default highlight and wakeword skip-confirmation auto-send; -`shell.showAlert` popup; `assistant.skipConfirmationAfterWakeword` setting. -*Deliverable: speak wakeword, ask "what's on my calendar", get an answer -on-lens.* - -**Phase 2 — app tools.** ✅ Landed 2026-07-25 (builds + typechecks; not yet -hardware-tested). -Worker protocol extended with `set-tools` / `tool-result` (WorkerAppReply) and -`tool-call` (WorkerAppMessage). `ToolRegistry.setAppTools`/`removeAppTools` -key tools by windowId, prefix names `app..`, and gate `foreground` -tools on an `isForeground()` predicate (checked at list and dispatch time). -`WorkerAppHost` proxies calls to its worker (callId round-trip, 15s host -timeout backstop on top of the registry's per-tool timeout), registers a -window's tools on `set-tools`, and withdraws them + fails in-flight calls on -window close. Direct mode now re-lists tools each loop iteration. -Terminal is the reference: the hub window declares `send_input` / -`read_screen` / `list_sessions` (all `open`-tier); send_input/read_screen act -on the active view (foregrounded → last-active → sole view). *Deliverable: -"run the build again" typed into the terminal even while it's backgrounded.* - -*Not yet done from the Phase 2 design:* `installed`-tier registration on -launcher entries (no `installed` app tool exists yet — timer stays deferred); -in-process-window tool registration (only worker apps wired); the per-window -same-name collision handling (the terminal sidesteps it by declaring all tools -on the single hub window). - -**Phase 3 — external bridge.** ✅ Landed 2026-08-01 (see the status note at -the top for the architecture change: bridge server == OpenClaw plugin). -Phone side: `app/assistant/bridge-client.ts` (`assistantBridge` singleton — -dial-out FaceclawWebSocket, hello/token auth, auto-reconnect with 1s→60s -backoff, started at boot by the dashboard controller when configured), -`app/assistant/mcp-server.ts` (MCP over the `mcp` channel: initialize / -tools/list / tools/call from the ToolRegistry, list_changed on -onToolsChanged, proactive gating + 6/min rate limit enforced on-phone), -external mode in `AssistantSession` (`AssistantBackendConfig` union selects -direct vs external; external history lives on the agent's machine), settings -(`assistant.backend`, `assistant.bridgeHost/Port/Token`, -`assistant.allowProactive`) in a new Assistant settings section. -Server side: `~/repositories/faceclaw-agent-bridge` (own repo/package). -*Verified against the live OpenClaw on serac with a simulated phone -(`test/fake-phone.js`): auth, streamed turns, and both tool directions -work; a real model turn needs the OpenClaw instance to have provider auth.* - -**Phase 4 — OpenClaw/Hermes adapters + proactive polish.** Partially landed -2026-08-01: the OpenClaw adapter is the plugin itself (no separate chat -adapter needed); proactive gating + rate limits and `tools/list_changed` -are done. Remaining: Hermes (or generic-agent) support — likely a -standalone bridge speaking the same wire protocol with an MCP face + -CLI chat adapter; session persistence questions across reconnects -(currently: OpenClaw session is persistent, phone session is UI-only). - -**Hardware-test TODO (external mode):** end-to-end on the real phone + -glasses against serac — connect status, a streamed turn, cancel, follow-up, -proactive `glasses.show_alert` from another OpenClaw channel, and behavior -across phone network transitions (the reconnect backoff is untested on -device). - -## Open questions (deferred, not blockers): - -- Continuous conversation: after a turn ends, listen for a follow-up - utterance without requiring the wakeword/PTT again? There are some UI - design questions here; maybe multi-turn conversations take the form of an - Assistant app. -- Launch-on-call for `installed`-tier tools that genuinely need the app UI - (open the app, await `set-tools`, proxy the call) — deferred until a tool - needs it. -- `glasses.read_screen` fidelity beyond app-reported summaries? We probably - don't want to do OCR, but we might do something like extending the draw-text - draw call with an on-by-default option that also appends the drawn text to - a string that gets submitted along with the frame. -- Whether assistant conversations should be reviewable in the phone UI - (transcript log) — probably yes, cheap once sessions are objects. The - natural UI home for this would be inside an Assistant app window. - diff --git a/notes/whisper-server.md b/notes/whisper-server.md deleted file mode 100644 index a17357c8..00000000 --- a/notes/whisper-server.md +++ /dev/null @@ -1,53 +0,0 @@ -# Self-hosted Whisper transcription - -The "Whisper server (self-hosted)" transcription provider sends voice audio to -any OpenAI-compatible Whisper server instead of a cloud service: vLLM, speaches -(faster-whisper-server), LocalAI and similar. It uses the batch endpoint -`POST {url}/audio/transcriptions` with a 16 kHz mono WAV. Push-to-talk -utterances are one request each; in Transcribe, each span ended by a pause is -one request. While a span is open its audio so far is re-sent every 1.5 s, so -text appears while you are still talking. A span that runs 25 s without a pause -(nonstop speech, music) is cut at its quietest moment in the last 5 s, so no -request outgrows Whisper's 30 s window; the pieces still arrive as one final. -On Android the WAV goes out through `FaceclawBinaryPost` (okhttp), since -NativeScript's http module copies a binary body into Java a byte at a time. - -## Server - -Any server exposing `/v1/models` and `/v1/audio/transcriptions` works, e.g. - - vllm serve openai/whisper-large-v3 --port 8004 - -## Settings (Settings > Voice) - -- **Transcription Provider**: Whisper server (self-hosted). Disabled until the - URL is set. -- **Whisper server URL**: the API root, e.g. `http://host:8004/v1`. A bare - `http://host:8004` gets `/v1` added. -- **Whisper server model**: empty uses the first model the server lists. -- **Whisper language**: an ISO code such as `en`, or empty to auto-detect. - A wrong fixed language makes Whisper translate rather than transcribe. -- **Whisper server key**: optional bearer token. - -## iOS: plain http - -iOS App Transport Security blocks plain `http://` to IP addresses (including -Tailscale's 100.x addresses) unless the app's Info.plist has an exception for -that host. Either serve the API over https (for example with `tailscale serve`) -or add your server to `NSExceptionDomains` in `App_Resources/iOS/Info.plist` -before building: - - 192.0.2.10 - NSExceptionAllowsInsecureHTTPLoads - -## Transcribe on iOS - -Transcribe needs a cloud or Whisper-server provider on iOS (Apple's recognizer -only handles short utterances). Its capture follows the glasses: a disconnect, -a glasses lock or backgrounding (for phone-mic capture) transcribes what was -already heard, and listening resumes once the glasses reconnect, unlock or the -app returns to the foreground. Without a glasses session the phone mic stands in -while Faceclaw is in the foreground. With "Save Transcribe audio" on (Voice -settings), each session's audio is kept as `transcribe-