Commit Graph
13 Commits
Author SHA1 Message Date
James Babcock b50898ea02 Fix a bunch of microphone-related bugs
> Most serious fixes
> - Recordings kept forever: a recording made with captions off, or with "Save captions"
>   off, was transcoded to .m4a and kept indefinitely. No conversation linked to it, so
>   the phone UI never showed it and the retention settings never deleted it. Recording
>   now only happens alongside a saved caption session, and turning captions off ends
>   that session's recording properly.
> - 4-mic mode went silent after a reconnect: a disconnect clears the phone's audio
>   forwarding, and nothing turned it back on, so the app kept showing "Listening (4-mic
>   array)" while hearing nothing. A check every 3 s now restores forwarding and re-sends
>   the config when it has been lost. The normal single-stream mode already recovered on
>   its own.
> - Turning off "4-mic capture" mid-session went silent: it stopped the glasses' stream
>   but never switched to the normal stream. Setup changes now take effect live,
>   including switching between the two modes.
> - Setup changes were undone: after the assistant closed, the app re-sent the config it
>   loaded at launch to the glasses, reverting any changes you'd made since.
> - Startup failures: if the caption engine failed to start, the session got stuck
>   half-running. If the glasses weren't connected, listeners piled up and an empty
>   conversation could be left open. Startup now brings up the microphone first and
>   cleans up properly on failure.
>
> Other fixes
> - Audio quality: the 4-mic beamformer put a small dropout at the start of every 25 ms
>   frame (an audible 40 Hz artifact in the audio feeding captions). Fixed, with a test
>   that fails on the old code.
> - Stale direction data: an arm that stopped sending kept steering the direction
>   estimate and could hold the caption feed on a silent arm.
> - Compass: closing Microphones switched off the Compass app's compass, because both
>   used the same owner name.
> - Speaker database failures:
>   - A failed speaker reload wiped the in-memory list, so the next voice became a
>     duplicate "Speaker N".
>   - A failed speaker insert crashed caption handling; in re-diarize it reassigned
>     transcript lines to a nonexistent speaker.
> - Translation:
>   - A crash path inside the model-download callback is now caught.
>   - Phones set to Hebrew, Indonesian or Yiddish never got translations, because of
>     old-style language codes.
> - Ask questions: once the speaker list grew past the size limit (every unknown voice
>   adds a "Speaker N"), it pushed all transcript lines out of the context sent to the
>   model. When trimming is needed, the oldest lines now go first.
> - Smaller fixes:
>   - The captions view stopped jumping back to live speech after 60 lines.
>   - "This is Dana calling…" was treated as introducing someone else.
>   - A model download that failed to start was stuck at "downloading" until restart.
>   - The developer mic monitor grew without limit while hidden.
2026-10-14 00:56:54 -07:00
James Babcock 295922effd Various UI tweaks
Fix missing submenu indicators and change their appearance. Clean up
styling in the Terminal app and move Settings to the top-level menu.
Remove right padding from the bottom bar.
2026-10-02 12:19:25 -07:00
James Babcock 566592c074 Add Dense style for battery indicators 2026-10-01 22:56:09 -07:00
James Babcock f7fa4addc0 Merge branch 'microphone-fixes' 2026-09-29 20:34:09 -07:00
James Babcock 06146c2b08 Introduce oxlint, and fix lint warnings 2026-09-29 11:54:22 -07:00
James Babcock 7d0becfaa7 Fix some microphone-related bugs, add a debug panel 2026-09-26 14:12:29 -07:00
James BabcockandKalani Helekunihi e8d2f5ade9 Fix argument order, sizing, and interleaving bugs in microphone handling
Adapted from commit dd43d476b023acfcaab4bdee942c98fd30d83f50 in
https://github.com/jimrandomh/g2flash/pull/4.

Co-Authored-By: Kalani Helekunihi <i@am.guru>
2026-09-26 11:57:40 -07:00
Chris LeeandClaude Sonnet 5 2d717a9cdb Add on-device Whisper as a second offline transcription option
sherpa-onnx's native library already reads an offline Whisper config
(verified against sherpa-onnx/jni/offline-recognizer.cc at v1.13.0);
only the Java wrapper was a package-private stub. This adds a public
OfflineWhisperModelConfig builder and wires base.en (int8) in as a
second on-device model next to Moonshine.

- FaceclawVoiceController: setOnboardModelKind() selects the model
  before start("onboard"); defaults to Moonshine. Whisper skips the
  700ms live-partial redecode (each call re-encodes the whole utterance
  plus ~10s of tail padding) and gates near-silent segments before
  decode.
- asr-model.ts: ASR_MODELS registry with per-model download state.
- Settings > Voice: "On-device (Moonshine)", "On-device (Whisper)",
  and "OpenAI (Whisper)" for the existing cloud provider (stored
  value unchanged); one model row per on-device model.
- Microphones live captions pass "moonshine" explicitly (unchanged).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 22:46:22 -04:00
James Babcock b0cf1e4f67 Switch from capability strings to numeric versioning for firmwares 2026-09-11 00:22:36 -07:00
James Babcock e98f19d7e4 Add "Use microphone control" option to developer settings 2026-08-30 21:18:32 -07:00
James Babcock 80d688dd4e Move and rename DashboardInputEvent->InputEvent 2026-08-30 13:02:44 -07:00
Kalani Helekunihi 20026c918a Replace Unicode property escapes, which crash the glasses' V8
The NativeScript Android runtime embeds a V8 built without ICU, so a regex
literal containing `\p{...}` cannot be parsed there. Because the app ships as a
single ES module, the three such literals added with the calculator and
microphone apps made the whole of bundle.mjs fail to compile:

  com.tns.NativeScriptException: Cannot compile ES module .../app/bundle.mjs
  SyntaxError: Invalid regular expression: /\p{L}/: Invalid property name

That throws inside Application.onCreate, so the app died on launch before any
of its own code ran — every screen, not just the two new ones. Neither tsc nor
the test suite caught it: the syntax is valid ES2018 and node has ICU.

app/util/unicode-class.ts replaces them with explicit ranges. The two uses pull
in opposite directions, so they are stated differently:

  - isLetter (maths lexer) is a letter table. The lexer's alphabet is small and
    known, and an unrecognised letter just becomes an identifier matching no
    known function, which is already a handled error.

  - NON_WORD_RUN (conversation search, sentiment) is a *separator* table rather
    than the complement of that letter table. Complementing it would make every
    letter the table omits — minority scripts, later Unicode additions — act as
    a separator and silently shred that text; listing separators instead keeps
    an unlisted character as word content. It also holds decomposed graphemes
    together, which `\p{L}` did not.

Verified against the real properties under node's ICU: identical output on a
prose corpus except where the new split is better (a Lao vowel sign and a
combining acute now stay attached).

The new test's last case scans app/ for `\p{` outside comments — the check that
would have caught this before it reached a device.
2026-08-25 00:41:43 -05:00
Kalani Helekunihi fe85846e6f Add calculator and microphone conversation apps
Add symbolic math and graphing tools, dual-microphone audio processing, live captions, translation, speaker diarization, sentiment analysis, and searchable conversation history.

Extend Android voice and BLE support for microphone control, recording, transcoding, and on-device speaker identification, with coverage for calculator and microphone workflows.
2026-08-25 00:22:19 -05:00