Commit Graph
66 Commits
Author SHA1 Message Date
MasterControl 6150a1ac57 Merge upstream main preserving custom feature integrations 2026-11-10 17:08:15 +00:00
James Babcock b866224909 Don't show EvenHub store, and show compatibility warning on sideloaded apps, when using RayNeo iO 2026-11-07 22:53:28 -08:00
James Babcock 43738b181d Merge branch 'main' of https://github.com/jimrandomh/faceclaw 2026-11-07 22:29:39 -08:00
James Babcock 10a9c85f27 EvenHub compativility fixes: Z-order changes, different text color distribution, text containers scroll 2026-11-07 22:28:49 -08:00
James Babcock c713815694 EvenHub: Fix an issue where indentation was incorrectly dropped in text containers
This caused text to overlap with icons in Iris Pro and Ocuclaw
2026-11-07 20:57:55 -08:00
James Babcock 3a0908fde2 EvenHub UI improvements
Handle left/right swipes on the tab bar, not just up/down
Update All button on the Updates page main list
Scroll by pages rather than lines on the permissions list
2026-11-07 20:31:31 -08:00
Blendman974 55bd5f891c feat: add application close, full-screen and keep-awake controls 2026-11-06 04:26:16 +01:00
James Babcock 53812609ed Fix EvenHub apps on iOS access to the mic 2026-10-25 14:19:45 -07:00
James Babcock 247459bf99 Fix an issue that could cause corrupted installs of EvenHub apps, and redirect corrupted installs into the update/reinstall 2026-10-25 14:05:36 -07:00
James Babcock 07859c95ad Fixes to evenhub extensions library 2026-10-24 01:51:34 -07:00
James Babcock 60595d7321 Fixes to EvenHub webview handling 2026-10-16 18:22:35 -07:00
James Babcock f4751ad480 Fix a bug where searching EvenHub for a string with spaces would fail 2026-10-16 16:54:54 -07:00
James Babcock 60c87521b0 Fix a bunch of microphones app reliability issues
<details>
<summary>Prompts Used</summary>
User: Review the Microphones app, focusing on error handling and reliability.

Opus 5.5:
> I've finished reading the Microphones code and its native pieces. The 2026-10-14 commit `b50898e` fixed a lot, and the main problem left is the same in most places: the app reports "Listening" or "captions on" but never checks that audio is actually arriving. When something fails partway, the session usually stops working with no visible sign.
>
> ## High: silent loss of function or data
>
> **1. In stock mode, other mic users can take the mic away for good.**
> - `stopRawCapture()` tears down the single shared audio tap no matter who started it (`app/native/voice-control.ts:257`).
> - The EvenHub mic router calls it whenever a mic-using EvenHub app goes to the background or closes (`app/apps/evenhub/mic-router.ts:88`). That kills Microphones' audio.
> - Transcribe takes the mic without setting `voiceActivity` (`voice-control.ts:269`), so Microphones is never told to get it back.
> - If capture starts while the session isn't ready, the native controller only records the request, so no audio ever flows.
> - `checkExtendedCapture` (`mic-session.ts:456`) only covers 4-mic mode. Its comment says the stock path "is resumed by the voice-control bridge", but that only covers the session-dropped-while-active case.
> - In 4-mic mode the problem runs the other way: Transcribe takes over the shared Kotlin listener (`GlassesSessionCore.kt:470`) while the glasses keep sending 4-mic frames, so Transcribe gets audio in the wrong format.
> - **Fix:** track who holds the raw tap (like `captureHolders`), add a stock-mode branch to the 3 s check, and have Transcribe set the same "mic in use" signal as the assistant.
>
> **2. One bad utterance ends captions for the rest of the session.**
> - The caption worker has no per-utterance error handling. Any exception from `recognize()` or `embed()` stops the worker for good (`CaptionEngineCore.kt:334`).
> - `started` stays true, so TypeScript still thinks captions are running and recording carries on. The only sign is a "Captions failed:" status string.
> - **Fix:** catch errors per utterance in `onUtterance` (`:367`). Report a fatal worker exit through its own callback so the session can restart the engine or mark captions inactive.
>
> **3. A crash or low-memory kill during recording leaves files that are never shown or deleted.**
> - The WAV header is written with a data size of 0 (`FaceclawWavRecorder.kt:26`) and only corrected in `finish()`.
> - The file is only linked to its conversation in `stopRecording` (`mic-session.ts:1200`), and the conversation row is never closed.
> - Nothing cleans up `mic-recordings/`, so after a crash you get the same never-expired orphan files the last commit fixed for the captions-off case.
> - **Fix:** link the file to its conversation when recording starts. In the boot-time cleanup: close conversations left open (`ended_at IS NULL`), repair the WAV header from the file length, and delete files no conversation points to.
>
> **4. When the transcoder's success callback fails, it deletes both copies of the recording.**
> - `FaceclawAudioTranscoder.kt:66-83` deletes the WAV before calling `onDone`. If `onDone` throws, the catch block deletes the M4A too, and the database still points at the deleted WAV.
> - Separately, the JS listener is referenced only from Java (`mic-session.ts:1208`). Under `markingMode: 'none'` that's the situation `mic-control.ts:74` warns about.
> - I haven't confirmed the garbage-collection risk. `asr-model.ts` does the same thing and its downloads seem to work. But the delete order is wrong either way.
> - `rediarize.ts` and `mic-models.ts` hold their listeners the same way.
>
> ## Medium
>
> 5. **The last utterance is lost when you close the app or turn captions off.** When the caption engine stops, it flushes the utterance in progress and posts it to the main thread. By the time it arrives, `stop()` has already run `finishSessionRow()` (`mic-session.ts:328`), so the line is never saved and is missing from the session insights. If decoding takes longer than the 2 s wait, the line can land in the *next* session with wrong times. Also, `stop()` can block the UI for up to 2 s.
> 6. **The beam filter can mute 4-mic captions permanently.** The direction and signal strength only update on frames with a non-zero angle or strength (`:685`), and they never go stale. If you turn off "Direction detection", the last direction stays frozen. If it was outside the beam, `deliverExtendedAudio` drops all audio from then on (`:751-757`). Stock mode avoids this because it updates the strength on every frame. **Fix:** timestamp the fused direction and treat it as unknown after `TEMPLE_STALE_MS`.
> 7. **The 4-mic health check looks at a phone-side flag, not at whether audio is arriving.**
>    - The check only re-arms when `isAudioCaptureActive()` is false.
>    - A configure command can be skipped while the session isn't ready even though `send()` returns true (`mic-control.ts:155`, `GlassesSessionCore.kt:890`).
>    - A temple can reboot on its own, or `apply()` can refuse to arm when the developer setting is turned off mid-session.
>    - Each of these leaves forwarding "active" with no frames arriving.
>    - **Fix:** you already track `templeFrameAtMs`. Re-send the config when an enabled temple has gone quiet, and show that in the status instead of "Listening (4-mic array)".
> 8. **A failed start is permanent.** `start()` returns before installing any timers (`:280`). A failed mode switch mid-session (`:376`) leaves `forwardingProxy` null, so the health check exits at `:457` every time. **Fix:** stay in a "waiting for glasses" state and let the timer retry.
> 9. **The caption feed can flip between left and right on every frame.** The choice at `:744` doesn't take the current side into account, and `strongerChanged` is just a `!==` check. Near the 1.5× threshold, each flip splices about 25 ms from the other temple (which runs on its own clock) into captions and the recording, either repeating or skipping audio.
>
> ## Lower
> - **Unguarded speaker call (`:1034`):** the speaker rename for a pending introduction runs before the line is added. A database error drops the line and leaves the introduction pending, so the next 2 minutes of lines from unnamed speakers fail the same way. Add the line first, then do the extra steps each with its own error handling.
> - **Speaker list grows forever:** every unknown voice adds a permanent "Speaker N" with a voiceprint. Retention never prunes them. `assign()` reloads and decodes every profile on every utterance, on the main thread.
> - **Translation:** there's no timeout, so lines can wait indefinitely while a model download is stuck offline. The detected language is only saved after translation finishes (`:1132`). The "hysteresis" comment at `:1120` describes logic that doesn't exist: `conversationLang` is written but never read.
> - **Frame fields ignored:** the sequence number (but `TempleAudioPipeline` assumes consecutive frames), the truncated flag, and `sampleRateHz`. The UI offers 8–48 kHz while the code assumes 16 kHz.
> - **Duplicated model path:** `moonshineModelDir()` (`:1337`) copies `asr-model.ts`'s directory name. A model version bump there would quietly drop captions to "no ASR model".
> - **Silent recording gaps:** `WavRecorder.append` swallows each write error, so a full disk means log spam and silent gaps in the recording.
>
> The startup order with cleanup on failure, the 90 s lease that recovers on its own from a lost STOP, and keeping the last good speaker cache all hold up well.
>
> I worked from the code only; nothing was tested on hardware. #6 relies on the protocol comment saying the angle is 0 when direction detection is off.
>
> I'd fix #1, #2, #5 and #6 first, since those are the ones you'd hit in normal use. Want me to start on them?

<hr>

User: Thanks. Let's fix all of the high and medium issues you found now.

Opus 5.5:
> All nine high and medium issues are fixed. TS typecheck, lint, 813 TS tests, the Kotlin host tests (4 new), the iOS Kotlin compile and `build.sh` all pass, and I confirmed the new code is in the APK. Nothing has been tested on the glasses yet, and nothing is committed.
>
> **High**
> 1. **Mic taken by other apps:**
>    - The shared audio tap now tracks who's using it, so an EvenHub app backgrounding no longer cuts off Microphones.
>    - Transcribe now marks the mic as in use, like the assistant does, so Microphones and EvenHub apps step aside while it runs.
>    - A check every 3 s now covers both modes. It restarts a dead stock tap and switches back to stock mode if the developer setting is turned off mid-session.
> 2. **One bad utterance ending captions:** a decode error now skips that utterance instead of stopping the caption engine (`CaptionEngineCore.kt`). If the engine dies anyway, it's restarted on the same conversation up to 3 times. After that, the Captions toggle shows off and the view says why; turning it on retries.
> 3. **Recordings orphaned by a crash:** the recording is linked to its conversation as soon as it starts. At boot, the app closes conversations a crash left open, repairs their WAV headers (`FaceclawWavRecorder.repairHeader`), transcodes them, and deletes files no conversation points to. That cleanup matches by file name, so moving the app to other storage can't delete live recordings.
> 4. **Transcoder deleting both copies:** a failure while recording the new file no longer deletes both files. The app now deletes the WAV itself, only after the conversation points at the M4A. The transcoder, re-diarize and model-download callbacks are now held by the app until they fire.
>
> **Medium**
> 5. **Last utterance lost:** stopping captions no longer blocks the UI. The engine finishes the audio still queued, then sends a new `onStopped` callback, and the conversation stays open until that arrives (20 s fallback). A late line from a previous session won't show up in a newly reopened one.
> 6. **Beam filter muting captions:** the direction of arrival only filters audio while it's under 1 s old, and it resets when "Direction detection" changes.
> 7. **4-mic health check:** a temple that sends no audio for 5 s gets its config re-sent, to that temple only. Mic-control sends now report when they weren't actually queued. The status line says when no audio is arriving, but not after routine restarts like the voice dialog closing.
> 8. **Start failures being permanent:** the session now waits ("Waiting for the glasses microphone...") and the 3 s check retries. Captions start only once the mic is up, so waiting doesn't create empty conversations.
> 9. **Left/right flapping:** the caption feed only switches temples when the other one is clearly louder for 500 ms.
>
> Behaviour changes you'll notice:
> - The watch shows "Listening…" during Transcribe.
> - EvenHub mic apps get no audio while Transcribe runs.
> - In 4-mic mode, an EvenHub mic app still takes over the single native audio listener. Fixing that needs a change on the Kotlin side, so I left it.
>
> On hardware, I'd test:
> - The assistant, Transcribe and an EvenHub mic app taking the mic while Microphones is open, then giving it back.
> - Reconnecting in both stock and 4-mic modes.
> - Killing the app mid-recording, then checking the conversation is recovered at the next start.
>
> I saved these notes to memory as `microphones-reliability.md`.
</details>
2026-10-14 02:17:05 -07:00
James Babcock ccbf523e9a Add EvenHub icon to phone preview screen 2026-10-12 02:33:22 -07:00
James Babcock 5a9ca08921 Unify vertical position and display mode settings; add to onboarding 2026-10-10 22:48:26 -07:00
James Babcock 84baee9c76 EvenHub extensions: Fonts, window icons, touch-start events 2026-10-07 01:22:42 -07:00
James Babcock 4ee0af86cc EvenHub API extensions: Set package name, fix some bugs 2026-10-06 23:02:08 -07:00
James Babcock 295922effd Various UI tweaks
Fix missing submenu indicators and change their appearance. Clean up
styling in the Terminal app and move Settings to the top-level menu.
Remove right padding from the bottom bar.
2026-10-02 12:19:25 -07:00
James Babcock 06146c2b08 Introduce oxlint, and fix lint warnings 2026-09-29 11:54:22 -07:00
James Babcock 45d2ece0d2 Optimize phone CPU usage 2026-09-27 23:16:41 -07:00
James Babcock e879c58d9c EvenHub phone UI: add top bar, add open icon 2026-09-27 21:38:04 -07:00
James Babcock 77bda86ede Fix line-wrapping on EvenHub permissions dialog 2026-09-27 18:22:49 -07:00
James Babcock 3601a998a3 CPU usage optimizations 2026-09-27 16:33:43 -07:00
James Babcock aeeea5ead3 Animations in EvenHub menus; closer style matching 2026-09-26 22:46:30 -07:00
James BabcockandClaude Opus 5.5 1812818ae2 Stop leaking the remaining JS ArrayBuffers passed to Java
The same NativeScript conversion leak as the frame path (see
java-direct-buffer.ts) applied to every other ArrayBuffer handed to a
Java ByteBuffer parameter:

- Shell scenes (submitShellScene on the bridge and the preview display),
  submitted on every shell render, which the frame-path fix predates.
  These now go through a reused JavaDirectBuffer like frames do.
- Glyph, firmware-glyph and image atlas registration, Mapbox image
  decode, the preview placeholder bitmap, firmware hash/write, binary
  file writes and the EvenHub privacy-policy download, plus the Kotlin
  protocol smoke test. These are rare or large, so they cross through a
  new copyToJavaByteBuffer: a fresh Java direct buffer per call that is
  freed normally, rather than one pinned for the life of the isolate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 22:28:45 -07:00
James Babcock a6291cf937 Fix missing rerender when EvenHub apps finish downloading 2026-09-25 02:02:16 -07:00
James Babcock 09a59b9821 Fix the EvenHub login form sizing and missing cancel button 2026-09-25 01:56:52 -07:00
James Babcock 71c1971992 Use animated menus in a bunch more places 2026-09-23 23:57:50 -07:00
JMO 662cbd75dc Fix the EvenHub sign-in form reopening every launch and never closing
Open EvenHub once without finishing sign in and the phone side form comes back
on every app start, with no way to close it. Back exits the app rather than
closing the form.

Three things combine:

- EvenHubStoreLayer.paint() opened the credential editor on its first paint
  when no credentials were stored. First paint includes the restore of a
  still-open EvenHub at startup, so an unfinished sign in put the form back
  every time. handleInput already opens it on a click, so the auto open only
  removed the choice of when.

- LayerActions.startTextSettingsEdit declares a fifth onCancel parameter, but
  the DashboardController wiring forwarded four and the method accepted four.
  Nothing failed to compile, because a 4-parameter function is assignable to a
  5-parameter function type. That callback resets openCredentialEditor's
  phoneEditor guard, so with it dropped the guard latched at "credentials":
  the screen timeout hid the panel without resetting the state behind it, and
  the form could then be neither closed nor reopened. ios-preview-controller.ts
  already forwards all five, so this was the G2 path only.

- The panel has no Cancel control, so the keyboard's Done key on the password
  field is the only way to end the edit. On an untouched form that reached
  submitCredentials, which called enterLogin("Email and password are
  required."), which reopens the editor. A loop with no exit.

The auto open is removed; onCancel is forwarded, stored and fired from
endTextSettingEdit, with finishTextSettingEdit clearing it first so a completed
edit does not also report a cancel; and a wholly empty submission is treated as
a cancel. A half-filled form is a real mistake and still reopens with the
message.

tests/evenhub-signin.test.cjs covers the open and close contract through the
existing load-typescript.cjs loader. Two of its five tests fail against the
current store-layer.ts and pass with this change.
2026-09-22 19:07:22 -07:00
James Babcock e50da605e0 Add support for buzzer sound effects on iOS 2026-09-19 12:03:57 -07:00
James Babcock 7e58c7c05f EvenHub compatibility for iOS 2026-09-19 00:48:26 -07:00
James Babcock 44bd1d8a2d iOS porting: EvenHub app compatibility first pass 2026-09-08 14:47:58 -07:00
James Babcock b69fdc220a EvenHub app can search and check for updates 2026-09-04 11:41:51 -07:00
James Babcock 83c3732ba7 Swap app and system context menu gestures 2026-09-02 21:17:15 -07:00
James Babcock 76825f68a7 Flashing uses 2.2.9 as base version 2026-09-02 00:30:50 -07:00
James Babcock a72e29a04f Compass has higher sample rate, and distinguishes map north from true north 2026-09-01 14:37:09 -07:00
James Babcock 641e8e322e Ehpk app permission prompts scroll if too long 2026-08-31 23:59:00 -07:00
James Babcock 052398d401 Update EvenHub compatibility layer to support new functionality (context menus and some other things) from 0.0.14 2026-08-31 23:42:22 -07:00
James Babcock 80d688dd4e Move and rename DashboardInputEvent->InputEvent 2026-08-30 13:02:44 -07:00
James Babcock d542acb891 Merge branch 'main' of https://github.com/jimrandomh/faceclaw 2026-08-30 00:42:27 -07:00
James Babcock 6c46f2f7cc EvenHub app stores login token instead of password 2026-08-30 00:35:55 -07:00
Kalani Helekunihi 8fa04d1b46 Add Wear OS companion app and remote controls
Add a Wearable Data Layer bridge for syncing dashboard state and commands.

Introduce a standalone watch app with assistant, app launcher, touchpad remote, settings, and connectivity UI.
2026-08-23 15:02:35 -05:00
James Babcock 35e6977c6c UI refinement pass
Remove gesture-help bottom-bars from many places in the UI, adjust
layouts to be able to handle larger fonts, and other small UI fixes.
2026-08-22 18:48:20 -07:00
James Babcock 92cd05b6ca Add options to load an app from a URL or a QR code
(The QR code reading does not work yet)
2026-08-21 12:03:24 -07:00
James Babcock 8cdee2a701 HTTP requests identify Faceclaw in their user-agent string 2026-08-20 11:32:50 -07:00
James Babcock 9e447b70e2 Fix a bug in the EvenHub emulator causing stray text at the end of containers 2026-08-20 09:07:38 -07:00
James Babcock eef9fbaab4 Adjust UI to account for user-configured variable font size 2026-08-19 20:48:18 -07:00
James Babcock 96842ceb23 Use TTF fonts throughout the UI; also add some more functions to app extension API 2026-08-19 19:35:24 -07:00
James Babcock 5abbe2a405 Fix an EvenHub compatibility issue when a text container is updated with only an ID and not a container name 2026-08-19 12:40:57 -07:00
James Babcock 90603b5aa4 Use the texture cache and builtin-font drawing firmware extensions 2026-08-19 10:36:16 -07:00