# MentraOS app compatibility for Faceclaw — feasibility assessment (2026-07-23) MentraOS: https://github.com/Mentra-Community/MentraOS (MIT), docs at https://docs.mentraglass.com. ## TL;DR Technically feasible with moderate effort, but the timing is awkward: MentraOS is **deprecating its entire current app API**. MentraOS 3.0 ships **Aug 3, 2026** and cloud-SDK apps stop working on it (a "MentraOS Legacy" mode preserves them through **October 2026**). The replacement — a **Miniapp SDK** where apps are phone-local JS bundles running in a WebView — gets a public spec in **September 2026**, and that model is a *much* better fit for Faceclaw than the cloud model. Recommendation: wait for the miniapp spec and target that; only build a minimal cloud-protocol emulator if access to the existing app corpus is wanted in the interim. ## MentraOS app model (current, "Cloud SDK" / TPA era) - Apps are developer-hosted server processes (Node/Bun, `@mentra/sdk`). Chain: glasses → phone (BLE) → MentraOS Cloud (WS) → app server (WS). - Start flow: cloud POSTs a `session_request` webhook `{sessionId, userId, websocketUrl}` to the app's registered URL; the app dials back to `websocketUrl` and sends `tpa_connection_init {packageName, sessionId, apiKey}`; cloud acks with settings + device capabilities. **The webhook carries the WS URL, so an unmodified `@mentra/sdk` app can be pointed at any server** — that is the compat hook. - Protocol: JSON over WebSocket (binary frames for audio). 28 app→cloud + 28 cloud→app message types, defined in `cloud/packages/sdk/src/types/` (`message-types.ts`, `streams.ts`, `layouts.ts`, `webhooks.ts`, `capabilities.ts`). The TS types are the spec. - Subscriptions: `subscription_update` with a full list of stream types, some parameterized (`transcription:en-US`, `touch_event:forward_swipe`). Events arrive in a `data_stream` envelope. - Display: 7 layout types — `text_wall`, `double_text_wall`, `dashboard_card`, `reference_card`, `bitmap_view` (base64), `bitmap_animation`, `clear_view` — sent as `display_event {view: 'main'|'dashboard', layout, durationMs?, forceDisplay?}`. Cloud throttles displays to 1 per 300 ms. Dashboard = shared look-up surface. - Streams: `transcription` (`{text, isFinal, ...}`), `translation`, `VAD`, `audio_chunk` (PCM), `button_press`, `touch_event`, `head_position` (up/down), `glasses_battery_update`, `phone_battery_update`, `location_update`, `phone_notification`(+dismissed), `calendar_event`, plus camera/RTMP/LED families irrelevant to G2. - Other: ElevenLabs TTS (`audio.speak`), per-app settings (schema in dev console, 9 setting types, live `settings_update`), webview UI with signed-token auth, cloud KV storage, app-to-app messaging. - Hardware abstraction: `Capabilities` flags per device. **Even Realities G2 is a first-class device** (`capabilities/even-realities-g2.ts`): display 640×200 (their rendering canvas, not the 576×288 EvenHub raster), maxTextLines 5, bitmap-capable, capacitive swipe bar with TAP/DOUBLE_TAP/TRIPLE_TAP/PRESS_HOLD/SWIPE_UP/SWIPE_DOWN, IMU, mic, no camera/speaker/LED. - Licensing: MIT throughout; cloud is self-hostable in principle (Bun + MongoDB + S3 + third-party STT/TTS keys) but heavy. ## Coming model (Miniapp SDK, MentraOS 3.0) Miniapps are JS bundles (`index.html` + `miniapp.json`) running **on the phone** inside the MentraOS host app's WebView ("Mentra Runtime"), packages `@mentra/miniapp` + `@mentra/miniapp-cli`, sideloaded via QR. API is analogous but smaller: `session.transcription`, `session.display.showTextWall`, `session.storage`. Example: https://github.com/Mentra-Community/LiveCaptionsLocalSdkMiniapp. Public spec Sept 2026. ## Fit with Faceclaw Faceclaw's worker-per-app architecture is unusually favorable: apps are already isolated contexts speaking a small typed JSON vocabulary (`WorkerAppMessage`/`WorkerAppReply` in `app/ui/shell/worker-window.ts`), with pixels out-of-band. The compat layer would be one compiled-in "MentraOS host" app (a peer of `WorkerAppHost`) that presents each external app as a `ShellWindow` and: - **Rasterizes layouts** → trivial. All 7 layout types map onto `GrayImage` + `drawTextWrapped`/`bitBlt` into the 540×260 `APP_VIEWPORT`. Bitmaps need scaling from their 640×200/576×136 assumptions; monochrome already matches. - **Maps input** → good. `DashboardInputEvent` (`click`/`double-click`/`long-press`/ `scroll-up`/`scroll-down`) covers their G2 button-event list almost 1:1 (TRIPLE_TAP missing). Work item: input is currently routed only to the focused window; MentraOS wants multi-subscriber fan-out (background apps), so a broadcast layer is needed. - **Transcription** → the load-bearing stream, and Faceclaw already has it: `voiceControlBridge.onTranscript` (`{text, isFinal}`, replace semantics — matches MentraOS's `TranscriptionData` model well). Onboard Moonshine is English-only; parameterized languages would fall back to ElevenLabs/Whisper cloud providers. Needs multi-consumer fan-out and a policy for when the mic is open (MentraOS keeps it effectively continuous; Faceclaw is PTT/wakeword-oriented — battery implications). - **VAD** → sherpa endpointing exists. **audio_chunk** → PCM already surfaced on the ElevenLabs path (`onPcm`). - **Battery / phone notifications / calendar** → data already flows in Faceclaw (top-bar battery, notification listener, calendar app); just needs re-emission as streams. - **head_position** → gap. Faceclaw exposes raw IMU only (no gesture classification); the look-up dashboard gesture would need a small accelerometer classifier (cmd 19 stream exists, see imu-accelerometer memory). - **TTS / audio out** → G2 has no speaker; MentraOS plays through the phone. Faceclaw would need phone-side audio playback (new but simple Android work). - **Settings** → MentraOS per-app settings map onto the shared `FaceclawSettings` store with a `mentra..*` prefix; the 9 setting types map onto `ConfigSetting` subclasses, mostly. - **Not applicable on G2**: camera/photo/RTMP/LiveKit, LED, WiFi setup — can be rejected via capabilities, which the protocol explicitly supports. ## The hard parts 1. **Who plays "cloud"?** Cloud-SDK apps dial out to a WS URL given in a webhook, so Faceclaw must (a) run a WS **server** — none exists anywhere in the codebase today, all networking is outbound okhttp client (`FaceclawWebSocket`, g2mirror pattern) — reachable from wherever the app server runs (fine on LAN; NAT pain otherwise), and (b) send the session webhook itself (simple outbound HTTP POST). A phone-hosted mini-cloud with Java-WebSocket/NanoWSD listening on LAN is the sane MVP; self-hosting their actual cloud (Bun+Mongo+Docker) is overkill. 2. **Session lifecycle**: connection init/ack, `subscription_update` bookkeeping, `reconnect`/`reconnect_ack/rejected/deferred`, `app_stopped`, per-app permission gating. This is bookkeeping-heavy but not deep. 3. **Display arbitration**: foreground vs `background`+`forceDisplay` apps, `durationMs` auto-revert, the shared dashboard view. Faceclaw's shell has focus/foreground but no equivalent of background-app display preemption or a look-up dashboard. 4. **App discovery/registration UX**: no dev console; needs a simple "add app by URL + API key" settings flow (API key check can be a no-op or shared secret). ## Effort estimate MVP covering the bulk of display-glasses apps (live captions, teleprompters, dashboards, notification relays): ~10–15 message types + ~8 stream types + webhook sender + LAN WS server + layout rasterizer + settings mapping. Roughly a few weeks of focused work, with transcription-stream policy and the WS server being the riskiest pieces. Full-surface emulation (webview auth, app messaging rooms, TTS, location tiers) is substantially more and mostly not worth it. ## Recommendation - **Don't invest in the cloud-TPA protocol as the long-term target** — the vendor is killing it within ~3 months of now, and the ecosystem's apps will migrate to miniapps. - **Target the Miniapp SDK when its spec lands (Sept 2026).** It's strictly easier for Faceclaw: JS bundle in an Android WebView with an injected `session` bridge — no server, no webhooks, no reconnect protocol — and Faceclaw already has every underlying facility the bridge needs. The rasterizer/input/transcription plumbing built for it would be shared with any interim TPA emulator anyway. - If interim access to existing cloud-SDK apps matters, build the minimal LAN emulator described above, scoped to text layouts + transcription + button streams, and treat it as disposable.