8.6 KiB
MentraOS app compatibility for Faceclaw — feasibility assessment (2026-07-23)
MentraOS: https://github.com/Mentra-Community/MentraOS (MIT), docs at https://docs.mentraglass.com.
TL;DR
Technically feasible with moderate effort, but the timing is awkward: MentraOS is deprecating its entire current app API. MentraOS 3.0 ships Aug 3, 2026 and cloud-SDK apps stop working on it (a "MentraOS Legacy" mode preserves them through October 2026). The replacement — a Miniapp SDK where apps are phone-local JS bundles running in a WebView — gets a public spec in September 2026, and that model is a much better fit for Faceclaw than the cloud model. Recommendation: wait for the miniapp spec and target that; only build a minimal cloud-protocol emulator if access to the existing app corpus is wanted in the interim.
MentraOS app model (current, "Cloud SDK" / TPA era)
- Apps are developer-hosted server processes (Node/Bun,
@mentra/sdk). Chain: glasses → phone (BLE) → MentraOS Cloud (WS) → app server (WS). - Start flow: cloud POSTs a
session_requestwebhook{sessionId, userId, websocketUrl}to the app's registered URL; the app dials back towebsocketUrland sendstpa_connection_init {packageName, sessionId, apiKey}; cloud acks with settings + device capabilities. The webhook carries the WS URL, so an unmodified@mentra/sdkapp can be pointed at any server — that is the compat hook. - Protocol: JSON over WebSocket (binary frames for audio). 28 app→cloud + 28 cloud→app
message types, defined in
cloud/packages/sdk/src/types/(message-types.ts,streams.ts,layouts.ts,webhooks.ts,capabilities.ts). The TS types are the spec. - Subscriptions:
subscription_updatewith a full list of stream types, some parameterized (transcription:en-US,touch_event:forward_swipe). Events arrive in adata_streamenvelope. - Display: 7 layout types —
text_wall,double_text_wall,dashboard_card,reference_card,bitmap_view(base64),bitmap_animation,clear_view— sent asdisplay_event {view: 'main'|'dashboard', layout, durationMs?, forceDisplay?}. Cloud throttles displays to 1 per 300 ms. Dashboard = shared look-up surface. - Streams:
transcription({text, isFinal, ...}),translation,VAD,audio_chunk(PCM),button_press,touch_event,head_position(up/down),glasses_battery_update,phone_battery_update,location_update,phone_notification(+dismissed),calendar_event, plus camera/RTMP/LED families irrelevant to G2. - Other: ElevenLabs TTS (
audio.speak), per-app settings (schema in dev console, 9 setting types, livesettings_update), webview UI with signed-token auth, cloud KV storage, app-to-app messaging. - Hardware abstraction:
Capabilitiesflags per device. Even Realities G2 is a first-class device (capabilities/even-realities-g2.ts): display 640×200 (their rendering canvas, not the 576×288 EvenHub raster), maxTextLines 5, bitmap-capable, capacitive swipe bar with TAP/DOUBLE_TAP/TRIPLE_TAP/PRESS_HOLD/SWIPE_UP/SWIPE_DOWN, IMU, mic, no camera/speaker/LED. - Licensing: MIT throughout; cloud is self-hostable in principle (Bun + MongoDB + S3 + third-party STT/TTS keys) but heavy.
Coming model (Miniapp SDK, MentraOS 3.0)
Miniapps are JS bundles (index.html + miniapp.json) running on the phone inside
the MentraOS host app's WebView ("Mentra Runtime"), packages @mentra/miniapp +
@mentra/miniapp-cli, sideloaded via QR. API is analogous but smaller:
session.transcription, session.display.showTextWall, session.storage. Example:
https://github.com/Mentra-Community/LiveCaptionsLocalSdkMiniapp. Public spec Sept 2026.
Fit with Faceclaw
Faceclaw's worker-per-app architecture is unusually favorable: apps are already isolated
contexts speaking a small typed JSON vocabulary (WorkerAppMessage/WorkerAppReply in
app/ui/shell/worker-window.ts), with pixels out-of-band. The compat layer would be one
compiled-in "MentraOS host" app (a peer of WorkerAppHost) that presents each external
app as a ShellWindow and:
- Rasterizes layouts → trivial. All 7 layout types map onto
GrayImage+drawTextWrapped/bitBltinto the 540×260APP_VIEWPORT. Bitmaps need scaling from their 640×200/576×136 assumptions; monochrome already matches. - Maps input → good.
DashboardInputEvent(click/double-click/long-press/scroll-up/scroll-down) covers their G2 button-event list almost 1:1 (TRIPLE_TAP missing). Work item: input is currently routed only to the focused window; MentraOS wants multi-subscriber fan-out (background apps), so a broadcast layer is needed. - Transcription → the load-bearing stream, and Faceclaw already has it:
voiceControlBridge.onTranscript({text, isFinal}, replace semantics — matches MentraOS'sTranscriptionDatamodel well). Onboard Moonshine is English-only; parameterized languages would fall back to ElevenLabs/Whisper cloud providers. Needs multi-consumer fan-out and a policy for when the mic is open (MentraOS keeps it effectively continuous; Faceclaw is PTT/wakeword-oriented — battery implications). - VAD → sherpa endpointing exists. audio_chunk → PCM already surfaced on the
ElevenLabs path (
onPcm). - Battery / phone notifications / calendar → data already flows in Faceclaw (top-bar battery, notification listener, calendar app); just needs re-emission as streams.
- head_position → gap. Faceclaw exposes raw IMU only (no gesture classification); the look-up dashboard gesture would need a small accelerometer classifier (cmd 19 stream exists, see imu-accelerometer memory).
- TTS / audio out → G2 has no speaker; MentraOS plays through the phone. Faceclaw would need phone-side audio playback (new but simple Android work).
- Settings → MentraOS per-app settings map onto the shared
FaceclawSettingsstore with amentra.<packageName>.*prefix; the 9 setting types map ontoConfigSettingsubclasses, mostly. - Not applicable on G2: camera/photo/RTMP/LiveKit, LED, WiFi setup — can be rejected via capabilities, which the protocol explicitly supports.
The hard parts
- Who plays "cloud"? Cloud-SDK apps dial out to a WS URL given in a webhook, so
Faceclaw must (a) run a WS server — none exists anywhere in the codebase today,
all networking is outbound okhttp client (
FaceclawWebSocket, g2mirror pattern) — reachable from wherever the app server runs (fine on LAN; NAT pain otherwise), and (b) send the session webhook itself (simple outbound HTTP POST). A phone-hosted mini-cloud with Java-WebSocket/NanoWSD listening on LAN is the sane MVP; self-hosting their actual cloud (Bun+Mongo+Docker) is overkill. - Session lifecycle: connection init/ack,
subscription_updatebookkeeping,reconnect/reconnect_ack/rejected/deferred,app_stopped, per-app permission gating. This is bookkeeping-heavy but not deep. - Display arbitration: foreground vs
background+forceDisplayapps,durationMsauto-revert, the shared dashboard view. Faceclaw's shell has focus/foreground but no equivalent of background-app display preemption or a look-up dashboard. - App discovery/registration UX: no dev console; needs a simple "add app by URL + API key" settings flow (API key check can be a no-op or shared secret).
Effort estimate
MVP covering the bulk of display-glasses apps (live captions, teleprompters, dashboards, notification relays): ~10–15 message types + ~8 stream types + webhook sender + LAN WS server + layout rasterizer + settings mapping. Roughly a few weeks of focused work, with transcription-stream policy and the WS server being the riskiest pieces. Full-surface emulation (webview auth, app messaging rooms, TTS, location tiers) is substantially more and mostly not worth it.
Recommendation
- Don't invest in the cloud-TPA protocol as the long-term target — the vendor is killing it within ~3 months of now, and the ecosystem's apps will migrate to miniapps.
- Target the Miniapp SDK when its spec lands (Sept 2026). It's strictly easier for
Faceclaw: JS bundle in an Android WebView with an injected
sessionbridge — no server, no webhooks, no reconnect protocol — and Faceclaw already has every underlying facility the bridge needs. The rasterizer/input/transcription plumbing built for it would be shared with any interim TPA emulator anyway. - If interim access to existing cloud-SDK apps matters, build the minimal LAN emulator described above, scoped to text layouts + transcription + button streams, and treat it as disposable.