From 60c87521b0667ead63409e529f9d57c93847211e Mon Sep 17 00:00:00 2001 From: James Babcock Date: Wed, 14 Oct 2026 02:17:05 -0700 Subject: [PATCH] Fix a bunch of microphones app reliability issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit
Prompts Used User: Review the Microphones app, focusing on error handling and reliability. Opus 5.5: > I've finished reading the Microphones code and its native pieces. The 2026-10-14 commit `b50898e` fixed a lot, and the main problem left is the same in most places: the app reports "Listening" or "captions on" but never checks that audio is actually arriving. When something fails partway, the session usually stops working with no visible sign. > > ## High: silent loss of function or data > > **1. In stock mode, other mic users can take the mic away for good.** > - `stopRawCapture()` tears down the single shared audio tap no matter who started it (`app/native/voice-control.ts:257`). > - The EvenHub mic router calls it whenever a mic-using EvenHub app goes to the background or closes (`app/apps/evenhub/mic-router.ts:88`). That kills Microphones' audio. > - Transcribe takes the mic without setting `voiceActivity` (`voice-control.ts:269`), so Microphones is never told to get it back. > - If capture starts while the session isn't ready, the native controller only records the request, so no audio ever flows. > - `checkExtendedCapture` (`mic-session.ts:456`) only covers 4-mic mode. Its comment says the stock path "is resumed by the voice-control bridge", but that only covers the session-dropped-while-active case. > - In 4-mic mode the problem runs the other way: Transcribe takes over the shared Kotlin listener (`GlassesSessionCore.kt:470`) while the glasses keep sending 4-mic frames, so Transcribe gets audio in the wrong format. > - **Fix:** track who holds the raw tap (like `captureHolders`), add a stock-mode branch to the 3 s check, and have Transcribe set the same "mic in use" signal as the assistant. > > **2. One bad utterance ends captions for the rest of the session.** > - The caption worker has no per-utterance error handling. Any exception from `recognize()` or `embed()` stops the worker for good (`CaptionEngineCore.kt:334`). > - `started` stays true, so TypeScript still thinks captions are running and recording carries on. The only sign is a "Captions failed:" status string. > - **Fix:** catch errors per utterance in `onUtterance` (`:367`). Report a fatal worker exit through its own callback so the session can restart the engine or mark captions inactive. > > **3. A crash or low-memory kill during recording leaves files that are never shown or deleted.** > - The WAV header is written with a data size of 0 (`FaceclawWavRecorder.kt:26`) and only corrected in `finish()`. > - The file is only linked to its conversation in `stopRecording` (`mic-session.ts:1200`), and the conversation row is never closed. > - Nothing cleans up `mic-recordings/`, so after a crash you get the same never-expired orphan files the last commit fixed for the captions-off case. > - **Fix:** link the file to its conversation when recording starts. In the boot-time cleanup: close conversations left open (`ended_at IS NULL`), repair the WAV header from the file length, and delete files no conversation points to. > > **4. When the transcoder's success callback fails, it deletes both copies of the recording.** > - `FaceclawAudioTranscoder.kt:66-83` deletes the WAV before calling `onDone`. If `onDone` throws, the catch block deletes the M4A too, and the database still points at the deleted WAV. > - Separately, the JS listener is referenced only from Java (`mic-session.ts:1208`). Under `markingMode: 'none'` that's the situation `mic-control.ts:74` warns about. > - I haven't confirmed the garbage-collection risk. `asr-model.ts` does the same thing and its downloads seem to work. But the delete order is wrong either way. > - `rediarize.ts` and `mic-models.ts` hold their listeners the same way. > > ## Medium > > 5. **The last utterance is lost when you close the app or turn captions off.** When the caption engine stops, it flushes the utterance in progress and posts it to the main thread. By the time it arrives, `stop()` has already run `finishSessionRow()` (`mic-session.ts:328`), so the line is never saved and is missing from the session insights. If decoding takes longer than the 2 s wait, the line can land in the *next* session with wrong times. Also, `stop()` can block the UI for up to 2 s. > 6. **The beam filter can mute 4-mic captions permanently.** The direction and signal strength only update on frames with a non-zero angle or strength (`:685`), and they never go stale. If you turn off "Direction detection", the last direction stays frozen. If it was outside the beam, `deliverExtendedAudio` drops all audio from then on (`:751-757`). Stock mode avoids this because it updates the strength on every frame. **Fix:** timestamp the fused direction and treat it as unknown after `TEMPLE_STALE_MS`. > 7. **The 4-mic health check looks at a phone-side flag, not at whether audio is arriving.** > - The check only re-arms when `isAudioCaptureActive()` is false. > - A configure command can be skipped while the session isn't ready even though `send()` returns true (`mic-control.ts:155`, `GlassesSessionCore.kt:890`). > - A temple can reboot on its own, or `apply()` can refuse to arm when the developer setting is turned off mid-session. > - Each of these leaves forwarding "active" with no frames arriving. > - **Fix:** you already track `templeFrameAtMs`. Re-send the config when an enabled temple has gone quiet, and show that in the status instead of "Listening (4-mic array)". > 8. **A failed start is permanent.** `start()` returns before installing any timers (`:280`). A failed mode switch mid-session (`:376`) leaves `forwardingProxy` null, so the health check exits at `:457` every time. **Fix:** stay in a "waiting for glasses" state and let the timer retry. > 9. **The caption feed can flip between left and right on every frame.** The choice at `:744` doesn't take the current side into account, and `strongerChanged` is just a `!==` check. Near the 1.5× threshold, each flip splices about 25 ms from the other temple (which runs on its own clock) into captions and the recording, either repeating or skipping audio. > > ## Lower > - **Unguarded speaker call (`:1034`):** the speaker rename for a pending introduction runs before the line is added. A database error drops the line and leaves the introduction pending, so the next 2 minutes of lines from unnamed speakers fail the same way. Add the line first, then do the extra steps each with its own error handling. > - **Speaker list grows forever:** every unknown voice adds a permanent "Speaker N" with a voiceprint. Retention never prunes them. `assign()` reloads and decodes every profile on every utterance, on the main thread. > - **Translation:** there's no timeout, so lines can wait indefinitely while a model download is stuck offline. The detected language is only saved after translation finishes (`:1132`). The "hysteresis" comment at `:1120` describes logic that doesn't exist: `conversationLang` is written but never read. > - **Frame fields ignored:** the sequence number (but `TempleAudioPipeline` assumes consecutive frames), the truncated flag, and `sampleRateHz`. The UI offers 8–48 kHz while the code assumes 16 kHz. > - **Duplicated model path:** `moonshineModelDir()` (`:1337`) copies `asr-model.ts`'s directory name. A model version bump there would quietly drop captions to "no ASR model". > - **Silent recording gaps:** `WavRecorder.append` swallows each write error, so a full disk means log spam and silent gaps in the recording. > > The startup order with cleanup on failure, the 90 s lease that recovers on its own from a lost STOP, and keeping the last good speaker cache all hold up well. > > I worked from the code only; nothing was tested on hardware. #6 relies on the protocol comment saying the angle is 0 when direction detection is off. > > I'd fix #1, #2, #5 and #6 first, since those are the ones you'd hit in normal use. Want me to start on them?
User: Thanks. Let's fix all of the high and medium issues you found now. Opus 5.5: > All nine high and medium issues are fixed. TS typecheck, lint, 813 TS tests, the Kotlin host tests (4 new), the iOS Kotlin compile and `build.sh` all pass, and I confirmed the new code is in the APK. Nothing has been tested on the glasses yet, and nothing is committed. > > **High** > 1. **Mic taken by other apps:** > - The shared audio tap now tracks who's using it, so an EvenHub app backgrounding no longer cuts off Microphones. > - Transcribe now marks the mic as in use, like the assistant does, so Microphones and EvenHub apps step aside while it runs. > - A check every 3 s now covers both modes. It restarts a dead stock tap and switches back to stock mode if the developer setting is turned off mid-session. > 2. **One bad utterance ending captions:** a decode error now skips that utterance instead of stopping the caption engine (`CaptionEngineCore.kt`). If the engine dies anyway, it's restarted on the same conversation up to 3 times. After that, the Captions toggle shows off and the view says why; turning it on retries. > 3. **Recordings orphaned by a crash:** the recording is linked to its conversation as soon as it starts. At boot, the app closes conversations a crash left open, repairs their WAV headers (`FaceclawWavRecorder.repairHeader`), transcodes them, and deletes files no conversation points to. That cleanup matches by file name, so moving the app to other storage can't delete live recordings. > 4. **Transcoder deleting both copies:** a failure while recording the new file no longer deletes both files. The app now deletes the WAV itself, only after the conversation points at the M4A. The transcoder, re-diarize and model-download callbacks are now held by the app until they fire. > > **Medium** > 5. **Last utterance lost:** stopping captions no longer blocks the UI. The engine finishes the audio still queued, then sends a new `onStopped` callback, and the conversation stays open until that arrives (20 s fallback). A late line from a previous session won't show up in a newly reopened one. > 6. **Beam filter muting captions:** the direction of arrival only filters audio while it's under 1 s old, and it resets when "Direction detection" changes. > 7. **4-mic health check:** a temple that sends no audio for 5 s gets its config re-sent, to that temple only. Mic-control sends now report when they weren't actually queued. The status line says when no audio is arriving, but not after routine restarts like the voice dialog closing. > 8. **Start failures being permanent:** the session now waits ("Waiting for the glasses microphone...") and the 3 s check retries. Captions start only once the mic is up, so waiting doesn't create empty conversations. > 9. **Left/right flapping:** the caption feed only switches temples when the other one is clearly louder for 500 ms. > > Behaviour changes you'll notice: > - The watch shows "Listening…" during Transcribe. > - EvenHub mic apps get no audio while Transcribe runs. > - In 4-mic mode, an EvenHub mic app still takes over the single native audio listener. Fixing that needs a change on the Kotlin side, so I left it. > > On hardware, I'd test: > - The assistant, Transcribe and an EvenHub mic app taking the mic while Microphones is open, then giving it back. > - Reconnecting in both stock and 4-mic modes. > - Killing the app mid-recording, then checking the conversation is recovered at the next start. > > I saved these notes to memory as `microphones-reliability.md`.
--- .../faceclaw/app/FaceclawAudioTranscoder.kt | 30 +- .../faceclaw/app/FaceclawBleCommunicator.kt | 2 +- .../com/faceclaw/app/FaceclawCaptionEngine.kt | 5 + .../faceclaw/app/FaceclawConversationStore.kt | 66 ++ app/apps/ai-chat/voice-draft.ts | 4 +- app/apps/evenhub/mic-router.ts | 7 +- app/apps/microphones/captions-layer.ts | 2 +- app/apps/microphones/index.ts | 4 +- app/apps/microphones/mic-control.ts | 15 +- app/apps/microphones/mic-models.ts | 8 + app/apps/microphones/mic-session.ts | 909 +++++++++++++----- app/apps/microphones/microphones-app.ts | 5 +- app/apps/microphones/rediarize.ts | 34 +- app/g2/dashboard-controller.ts | 8 + app/native/voice-control.ts | 52 +- app/ui/shell/voice-activity.ts | 26 +- .../com/faceclaw/app/FaceclawWavRecorder.kt | 24 + .../faceclaw/app/audio/CaptionEngineCore.kt | 64 +- .../FaceclawCaptionEngineListener.kt | 6 + .../g2protocol/session/GlassesSessionCore.kt | 10 +- .../com/faceclaw/app/ble/IosGlassesSession.kt | 2 +- .../com/faceclaw/app/WavRecorderRepairTest.kt | 24 + .../com/faceclaw/app/CaptionEngineCoreTest.kt | 101 ++ 23 files changed, 1101 insertions(+), 307 deletions(-) create mode 100644 tests/kotlin/src/androidHostTest/kotlin/com/faceclaw/app/WavRecorderRepairTest.kt diff --git a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawAudioTranscoder.kt b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawAudioTranscoder.kt index 58e016cd..2bb9ca77 100644 --- a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawAudioTranscoder.kt +++ b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawAudioTranscoder.kt @@ -13,7 +13,8 @@ import java.io.IOException /** * Transcodes a 16-bit PCM WAV conversation recording into AAC-LC in an M4A * container, cutting speech recordings to roughly a tenth of their WAV size. - * Runs on its own thread; the source WAV is deleted on success when requested. + * Runs on its own thread; when requested, the source WAV is deleted once the + * listener's onDone has returned normally. */ class FaceclawAudioTranscoder( private val wavPath: String?, @@ -66,19 +67,28 @@ class FaceclawAudioTranscoder( private fun run() { try { transcode() - val size = File(outputPath).length() - if (deleteSourceOnSuccess) { - File(wavPath).delete() - } - if (listener != null) { - listener.onDone(outputPath, size) - } } catch (t: Throwable) { Log.w(TAG, "transcode failed for $wavPath", t) File(outputPath).delete() - if (listener != null) { - listener.onError(if (t.message == null) t.toString() else t.message) + try { + listener?.onError(if (t.message == null) t.toString() else t.message) + } catch (listenerError: Throwable) { + Log.w(TAG, "transcode error listener failed", listenerError) } + return + } + val size = File(outputPath).length() + try { + listener?.onDone(outputPath, size) + } catch (t: Throwable) { + // The caller never took up the new file, so whatever it recorded still names the + // source: keep that and drop the output, rather than lose both. + Log.w(TAG, "transcode listener failed for $wavPath", t) + File(outputPath).delete() + return + } + if (deleteSourceOnSuccess) { + File(wavPath).delete() } } diff --git a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawBleCommunicator.kt b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawBleCommunicator.kt index cdff794d..cdc7d5a5 100644 --- a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawBleCommunicator.kt +++ b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawBleCommunicator.kt @@ -207,7 +207,7 @@ class FaceclawBleCommunicator(context: Context, rightAddress: String?, leftAddre fun removeMicStatusListener(listener: FaceclawMicStatusListener?) = core.removeMicStatusListener(listener) - fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean) = + fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean = core.sendFaceclawMicControl(record, label, rightTemple, leftTemple) fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean = core.startG2AudioForwarding(listener) diff --git a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawCaptionEngine.kt b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawCaptionEngine.kt index d985766e..c76bd02d 100644 --- a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawCaptionEngine.kt +++ b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawCaptionEngine.kt @@ -68,6 +68,11 @@ class FaceclawCaptionEngine { session.stop() } + /** Stop without blocking; the listener's onStopped follows the trailing utterance. */ + fun requestStop() { + session.requestStop() + } + /** Push decoded 16 kHz mono S16LE PCM (any chunking). */ fun acceptPcm(pcm16le: ByteArray?) { session.acceptPcm(pcm16le) diff --git a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawConversationStore.kt b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawConversationStore.kt index 2d9f316f..dfce1746 100644 --- a/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawConversationStore.kt +++ b/App_Resources/Android/src/main/java/com/faceclaw/app/FaceclawConversationStore.kt @@ -718,6 +718,72 @@ class FaceclawConversationStore private constructor(context: Context) : SQLiteOp } } + // ---- crash recovery ---- + + /** + * Close out sessions a dead process left open (ended_at NULL): the end becomes the last + * stored segment's end (or the start), and the tone average comes from the stored segments. + * excludeIdsJson is a JSON array of session ids that are legitimately still open. Returns + * the sessions it closed as a JSON array of {id, audioPath, audioCodec}. + */ + fun closeInterruptedSessions(excludeIdsJson: String?): String { + val exclude = HashSet() + if (excludeIdsJson != null && !excludeIdsJson.isEmpty()) { + val ids = JSONArray(excludeIdsJson) + for (i in 0 until ids.length()) { + exclude.add(ids.getLong(i)) + } + } + val db = writableDatabase + val closed = JSONArray() + val closedIds = ArrayList() + db.rawQuery("SELECT id, audio_path, audio_codec FROM sessions WHERE ended_at IS NULL", null).use { c -> + while (c.moveToNext()) { + val id = c.getLong(0) + if (id in exclude) continue + closedIds.add(id) + val row = JSONObject() + row.put("id", id) + row.put("audioPath", if (c.isNull(1)) JSONObject.NULL else c.getString(1)) + row.put("audioCodec", if (c.isNull(2)) JSONObject.NULL else c.getString(2)) + closed.put(row) + } + } + for (id in closedIds) { + db.execSQL( + "UPDATE sessions SET" + + " ended_at = COALESCE((SELECT MAX(ended_at) FROM segments WHERE session_id = ?), started_at)," + + " avg_sentiment = COALESCE((SELECT AVG(sentiment) FROM segments WHERE session_id = ?), 0)" + + " WHERE id = ?", + arrayOf(id, id, id)) + } + return closed.toString() + } + + /** + * Delete recordings in dirPath that no session's audio_path names: files a session row never + * learned about, or partial transcoder output. Matched by file name, so a storage move that + * changed the directory's absolute path can't make live recordings look orphaned. Only call + * while no recording or transcode is in flight. Returns how many files were deleted. + */ + fun deleteUnreferencedRecordings(dirPath: String?): Int { + if (dirPath == null) return 0 + val files = File(dirPath).listFiles() ?: return 0 + val referenced = HashSet() + readableDatabase.rawQuery("SELECT audio_path FROM sessions WHERE audio_path IS NOT NULL", null).use { c -> + while (c.moveToNext()) { + referenced.add(File(c.getString(0)).name) + } + } + var deleted = 0 + for (file in files) { + if (file.isFile && file.name !in referenced && file.delete()) { + deleted++ + } + } + return deleted + } + fun deleteSession(sessionId: Long) { val db = writableDatabase val path = getSessionAudioPath(sessionId) diff --git a/app/apps/ai-chat/voice-draft.ts b/app/apps/ai-chat/voice-draft.ts index f91093fc..d07c4f0b 100644 --- a/app/apps/ai-chat/voice-draft.ts +++ b/app/apps/ai-chat/voice-draft.ts @@ -39,7 +39,7 @@ export class VoiceDraft { this.phase = "listening"; this.status = "Listening... release to send"; this.ownsVoiceActivity = true; - voiceActivity.setActive(true); + voiceActivity.setActive(true, "voice-draft"); this.unsubscribe.push(voiceControlBridge.onTranscript((event) => { if (!this.active || generation !== this.generation) return; if (event.isFinal) { @@ -129,7 +129,7 @@ export class VoiceDraft { } if (this.ownsVoiceActivity) { this.ownsVoiceActivity = false; - voiceActivity.setActive(false); + voiceActivity.setActive(false, "voice-draft"); } this.changed(); } diff --git a/app/apps/evenhub/mic-router.ts b/app/apps/evenhub/mic-router.ts index fca3e374..76f7fe15 100644 --- a/app/apps/evenhub/mic-router.ts +++ b/app/apps/evenhub/mic-router.ts @@ -17,6 +17,9 @@ import { voiceActivity } from "../../ui/shell/voice-activity"; declare const com: any; +// The router's claim on the shared raw-PCM tap (the Microphones app holds its own). +const RAW_CAPTURE_OWNER = "evenhub-mic-router"; + export type EvenHubMicClient = { readonly windowId: string; isForeground(): boolean; @@ -83,9 +86,9 @@ class EvenHubMicRouter { // eligibility change so it self-heals after an STT preemption. Returns // false if STT currently owns the mic; a later trigger (assistant modal // close, foreground/screen change) re-tries. - voiceControlBridge.startRawCapture({ communicator: activeCommunicator() }); + voiceControlBridge.startRawCapture({ communicator: activeCommunicator(), owner: RAW_CAPTURE_OWNER }); } else { - voiceControlBridge.stopRawCapture(); + voiceControlBridge.stopRawCapture(RAW_CAPTURE_OWNER); } } } diff --git a/app/apps/microphones/captions-layer.ts b/app/apps/microphones/captions-layer.ts index 9cfc6dec..01d87f54 100644 --- a/app/apps/microphones/captions-layer.ts +++ b/app/apps/microphones/captions-layer.ts @@ -39,7 +39,7 @@ export class CaptionsLayer implements Layer { image.drawText(font, 12, 4, "Captions", 220); if (!state.captionsActive) { - image.drawText(font, 12, 26, "Captions are off (enable in the Microphones menu).", 130); + image.drawText(font, 12, 26, state.captionsNote, 130); } const rows = this.buildRows(font, width - 36, state.captionLines); diff --git a/app/apps/microphones/index.ts b/app/apps/microphones/index.ts index f17d5c30..c75ef469 100644 --- a/app/apps/microphones/index.ts +++ b/app/apps/microphones/index.ts @@ -17,7 +17,9 @@ const microphonesApp: AppDefinition = { title: "Microphones", icon: "mic", boot: () => { - // Prune expired captions/recordings per the retention settings. + // Close out conversations and recordings a dead process left open, then + // prune expired captions/recordings per the retention settings. + micSession.recoverInterruptedConversations(); void micSession.applyRetentionSweep(); }, launch: async (ctx) => { diff --git a/app/apps/microphones/mic-control.ts b/app/apps/microphones/mic-control.ts index 54a80d5d..3ac8ddca 100644 --- a/app/apps/microphones/mic-control.ts +++ b/app/apps/microphones/mic-control.ts @@ -125,6 +125,17 @@ class MicArrayController { return true; } + /** + * Re-send CONFIGURE to one temple whose stream went quiet (a lost write, or + * a temple that restarted on its own), leaving the other temple's stream + * untouched. + */ + reconfigureTemple(config: MicConfig, side: MicSide): boolean { + if (!templeActive(config, side)) return false; + if (config.armHardware && !micControlSupported()) return false; + return this.send("configure", config, `configure-${side}`, side === "right", side === "left"); + } + query(): boolean { return this.send("query", undefined, "query"); } @@ -152,8 +163,8 @@ class MicArrayController { try { this.ensureStatusProxy(communicator); const record = encodeMicControl(op, config); - communicator.sendFaceclawMicControl(toJavaBytes(record), label, rightTemple, leftTemple); - return true; + // False when the session wasn't ready and nothing was queued. + return communicator.sendFaceclawMicControl(toJavaBytes(record), label, rightTemple, leftTemple) !== false; } catch (error) { console.warn(`mic control ${label} failed: ${error}`); return false; diff --git a/app/apps/microphones/mic-models.ts b/app/apps/microphones/mic-models.ts index 010cd2ff..75db76d7 100644 --- a/app/apps/microphones/mic-models.ts +++ b/app/apps/microphones/mic-models.ts @@ -75,6 +75,9 @@ export type MicModelState = { }; const downloaders = new Map(); +// Each download's listener is referenced only from Java while it runs; hold +// it here so it isn't GC'd before its callbacks. +const downloadListeners = new Map(); const progress = new Map(); const stateListeners = new Set<(id: MicModelId, state: MicModelState) => void>(); @@ -152,6 +155,7 @@ function downloadNextFile(id: MicModelId): void { const nextFile = definition.files.find((file) => !isFilePresent(id, file)); if (!nextFile) { downloaders.delete(id); + downloadListeners.delete(id); notifyStateChanged(id); return; } @@ -169,9 +173,11 @@ function downloadNextFile(id: MicModelId): void { onError: (message: string) => { console.error(`Mic model download failed (${id}/${nextFile.name}): ${message}`); downloaders.delete(id); + downloadListeners.delete(id); notifyStateChanged(id); }, }); + downloadListeners.set(id, listener); try { const downloader = new com.faceclaw.app.FaceclawModelDownloader( `${definition.baseUrl}${nextFile.remoteName}`, @@ -186,6 +192,7 @@ function downloadNextFile(id: MicModelId): void { // Without this, a dead entry would pin the model at "downloading". console.error(`Mic model download failed to start (${id}/${nextFile.name}): ${String(error)}`); downloaders.delete(id); + downloadListeners.delete(id); notifyStateChanged(id); } } @@ -195,6 +202,7 @@ export function cancelMicModelDownload(id: MicModelId): void { if (!downloader) return; downloader.cancel(); downloaders.delete(id); + downloadListeners.delete(id); notifyStateChanged(id); } diff --git a/app/apps/microphones/mic-session.ts b/app/apps/microphones/mic-session.ts index f8849dda..3b3d7ef1 100644 --- a/app/apps/microphones/mic-session.ts +++ b/app/apps/microphones/mic-session.ts @@ -67,6 +67,7 @@ import { formatRelativeTime } from "../../util/date-util"; import { deviceLanguage, identifyLanguage, translateText } from "./translate"; declare const com: any; +declare const java: any; /** * The Microphones session: owns mic capture while the app window is open, @@ -83,6 +84,11 @@ declare const com: any; * mics as raw PCM 'SM' frames; per-temple delay-and-sum beamforming plus * NLMS noise cancellation run on the phone, and the two temple bearings * are fused for the radar. + * + * Neither path can assume an earlier start still holds: the mic enable + * lives in the glasses session (an arm drop or EvenHub resume clears it), + * and other mic users preempt the shared tap. A periodic check brings + * capture up when it isn't and re-acquires it when audio stops arriving. */ const SAMPLE_RATE = 16000; @@ -98,10 +104,34 @@ const ENCOUNTER_CARD_MAX_ACTION_ITEMS = 2; // A temple that has sent no frame (or no bearing) for this long no longer // counts: its stream stopped, its mics were toggled off, or its arm dropped. const TEMPLE_STALE_MS = 1000; -// How often the extended path checks that its forwarding survived the -// glasses session (an arm drop clears it). +// An enabled temple silent this long while forwarding holds gets its config +// re-sent; it is also the time a temple gets to start streaming after each +// CONFIGURE. +const TEMPLE_SILENT_MS = 5000; +// The fused direction of arrival only gates audio while it is this fresh: a +// direction nobody reports any more (Direction detection turned off, the +// temples gone quiet) must not keep muting the captions. +const DOA_STALE_MS = TEMPLE_STALE_MS; +// The caption feed moves to the other temple only once that temple has +// clearly out-heard the current one for a while. Each switch splices in +// audio on another temple's clock, so flapping near the threshold glitches it. +const FEED_SWITCH_RATIO = 1.5; +const FEED_SWITCH_HOLD_MS = 500; +// How often capture is checked: brought up if it isn't, re-acquired if +// audio stopped arriving. const CAPTURE_CHECK_MS = 3000; +// A caption engine that dies on its own is restarted on the same +// conversation, with growing delays, this many times before captions give up. +const CAPTION_MAX_RESTARTS = 3; +const CAPTION_RESTART_BASE_MS = 2000; +// How long a stopping caption engine gets to deliver its trailing utterance +// before its conversation is closed without it. +const CAPTION_STOP_TIMEOUT_MS = 20_000; const COMPASS_OWNER = "microphones"; +// This app's claim on the shared raw-PCM tap (stock path). +const RAW_CAPTURE_OWNER = "microphones"; +const WAITING_TEXT = "Waiting for the glasses microphone..."; +const STALLED_TEXT = "No audio from the glasses; reconnecting the mic..."; export type CaptionLine = { speakerId: number | null; @@ -140,22 +170,59 @@ export type MicSessionState = { /** Human-readable description of the active noise-cancellation engine. */ ancEngine: string; captionsActive: boolean; + /** Why captions aren't running, for the captions view ("" while they are). */ + captionsNote: string; captionLines: CaptionLine[]; speakerDots: SpeakerDot[]; }; type StateListener = (state: MicSessionState) => void; +/** One saved caption conversation: its store row and its running tone average. */ +type ConversationRecord = { rowId: number; sentimentSum: number; sentimentCount: number }; + +/** + * One caption-engine instance. Its utterance times are on its own sample + * clock, converted with the wall time and recording offset taken at its + * start; an engine restarted after a failure is a new run on the same + * conversation. + */ +type CaptionRun = { + engine: any; + // The Java listener proxy must stay referenced or it gets GC'd. + listener: any; + startWallMs: number; + audioBaseMs: number; + conversation: ConversationRecord | null; + /** The app session (start() count) it belongs to. */ + generation: number; + stopTimer: ReturnType | null; + /** Runs once the stopped engine has delivered its trailing utterance. */ + afterStop: (() => void) | null; +}; + +// Boot-time recovery runs once per process. +let recoveryDone = false; + class MicSession { private readonly stateListeners = new Set(); private running = false; private mode: "off" | "stock" | "extended" = "off"; private statusText = "Idle"; + // Whether the current mode's capture path is up; until it is, the capture + // check keeps retrying. + private captureUp = false; + // Capture is up but audio isn't arriving (the check is re-acquiring it). + private captureStalled = false; + // Consecutive checks that found the stock raw tap dead and restarted it. + private stockRestarts = 0; // Radar state. private levels: number[] = []; private levelLabels: string[] = []; private doaDeviceDeg: number | null = null; + // When doaDeviceDeg was last updated from a reported bearing. + private doaAtMs = 0; private ssr = 0; private headingDeg: number | null = null; private beamWorldDeg = 0; @@ -165,13 +232,20 @@ class MicSession { // Captions state. private captionLines: CaptionLine[] = []; - private captionEngine: any = null; - private captionListenerProxy: any = null; + // Captions are on for this app session: an engine is running, or being + // restarted after a failure. private captionsActive = false; - private engineStartWallMs = 0; - private sessionRowId = -1; - private sentimentSum = 0; - private sentimentCount = 0; + private captionRun: CaptionRun | null = null; + // Runs told to stop that haven't reported back; each holds its + // conversation open until its trailing utterance is in. + private readonly stoppingRuns = new Set(); + private conversation: ConversationRecord | null = null; + private captionFailures = 0; + private captionRestartTimer: ReturnType | null = null; + // Why captions gave up after repeated engine failures ("" when they haven't). + private captionsFailure = ""; + // Bumped per start(), so a stopping run's late line isn't shown in a later session. + private sessionGeneration = 0; private conversationLang = "und"; // Recording state. @@ -190,7 +264,11 @@ class MicSession { private templeSsr: { left: number; right: number } = { left: 0, right: 0 }; private templeAngleAtMs: Record = { left: 0, right: 0 }; private templeFrameAtMs: Record = { left: 0, right: 0 }; + // When each temple was last sent CONFIGURE, for its start-up grace period. + private templeArmedAtMs: Record = { left: 0, right: 0 }; private captionFeedSide: MicSide = "left"; + // When the other temple first clearly out-heard the caption feed's (0 = it hasn't). + private feedSwitchSinceMs = 0; // Cached at capture start so the ~80 Hz packet path never re-reads settings. private activeConfig: MicConfig = loadMicConfig(); // Hot-path flags mirrored from settings so the packet path avoids a JNI @@ -242,6 +320,7 @@ class MicSession { ancOn: ancEnabledSetting.get(), ancEngine: this.describeAncEngine(), captionsActive: this.captionsActive, + captionsNote: this.captionsNote(), captionLines: [...this.captionLines], speakerDots: [...(this.speakerDots = pruneDots(this.speakerDots, Date.now()))], }; @@ -254,6 +333,16 @@ class MicSession { : this.suppressorEngine; } + private captionsNote(): string { + if (this.captionsActive) return ""; + if (this.captionsFailure) { + return `Captions stopped: ${this.captionsFailure}. Turn Captions back on in the menu to retry.`; + } + // Wanted, but waiting for the microphone to come up. + if (this.running && captionsEnabledSetting.get()) return this.statusText; + return "Captions are off (enable in the Microphones menu)."; + } + private notify(): void { const state = this.getState(); this.stateListeners.forEach((listener) => listener(state)); @@ -264,41 +353,26 @@ class MicSession { start(): void { if (this.running || !global.isAndroid) return; this.running = true; + this.sessionGeneration += 1; this.captionLines = []; this.speakerDots = []; - this.sentimentSum = 0; - this.sentimentCount = 0; + this.captionsFailure = ""; this.conversationLang = "und"; this.activeConfig = loadMicConfig(); this.refreshCachedFlags(); - const extended = micControlSupported() && this.activeConfig.armHardware; - this.mode = extended ? "extended" : "stock"; - this.running = true; - // Capture first, so a failure has no captions session, recording, or - // subscriptions to unwind. - if (!this.startCapture()) { - this.statusText = "Microphone unavailable (glasses not connected?)"; - this.running = false; - this.mode = "off"; - this.notify(); - return; - } - this.statusText = this.listeningText(); - this.offSettings = onSettingsStoreChanged(() => this.refreshCachedFlags()); + this.mode = this.desiredMode(); + this.offSettings = onSettingsStoreChanged(() => this.handleSettingsChanged()); this.offSpeakerChanges = speakerRegistry.onChanged(() => this.refreshSpeakerIdentities()); this.offVoiceActivity = voiceActivity.subscribe((active) => this.handleVoiceActivity(active)); this.startSuppressor(); this.startCompass(); - if (captionsEnabledSetting.get()) { - this.startCaptions(); - } - if (this.shouldRecord()) { - this.startRecording(); - } + // Captions start once capture is up (bringUpCapture), so a session + // waiting on the glasses doesn't open an empty conversation. + this.bringUpCapture(); // Levels/DOA update at packet rate; coalesce UI renders to 5 Hz. this.renderTimer = setInterval(() => this.notify(), 200); - this.captureCheckTimer = setInterval(() => this.checkExtendedCapture(), CAPTURE_CHECK_MS); + this.captureCheckTimer = setInterval(() => this.checkCapture(), CAPTURE_CHECK_MS); this.notify(); } @@ -320,16 +394,17 @@ class MicSession { this.offSpeakerChanges = null; this.offVoiceActivity?.(); this.offVoiceActivity = null; + // Closes the recording now; the conversation closes once the engine's + // trailing utterance is in. this.stopCaptions(); - this.stopRecording(true); this.stopCompass(); this.suppressor = null; this.speechActive = false; - this.finishSessionRow(); this.mode = "off"; this.statusText = "Idle"; + this.captureStalled = false; this.levels = []; - this.doaDeviceDeg = null; + this.clearDirection(); this.notify(); void this.applyRetentionSweep(); } @@ -338,6 +413,30 @@ class MicSession { return this.mode === "extended" ? "Listening (4-mic array)" : "Listening"; } + private desiredMode(): "stock" | "extended" { + return micControlSupported() && this.activeConfig.armHardware ? "extended" : "stock"; + } + + /** + * Start the current mode's capture path. Until it is up the session stays + * running and the capture check retries; captions begin once it is. + */ + private bringUpCapture(): boolean { + // The voice dialog or Transcribe owns the mic; take it back when it ends. + if (voiceActivity.isActive() || !this.startCapture()) { + this.captureUp = false; + this.statusText = WAITING_TEXT; + return false; + } + this.captureUp = true; + this.captureStalled = false; + this.statusText = this.listeningText(); + if (captionsEnabledSetting.get() && !this.captionsActive && !this.captionsFailure) { + this.startCaptions(); + } + return true; + } + /** Start the current mode's capture path, undoing any partial start on failure. */ private startCapture(): boolean { try { @@ -350,6 +449,7 @@ class MicSession { } private stopCapture(): void { + this.captureUp = false; if (this.mode === "extended") { this.stopExtended(); } else { @@ -364,25 +464,43 @@ class MicSession { */ updateConfig(mutate: (config: MicConfig) => MicConfig): void { const config = mutate(loadMicConfig()); - saveMicConfig(config); + const directionChanged = config.beamform !== this.activeConfig.beamform; + const modeBefore = this.mode; + // Before saving: the save's settings event re-checks the capture mode. this.activeConfig = config; + saveMicConfig(config); if (this.running) { - const extended = micControlSupported() && config.armHardware; - if (extended !== (this.mode === "extended")) { - this.stopCapture(); - this.mode = extended ? "extended" : "stock"; - this.doaDeviceDeg = null; - this.ssr = 0; - this.statusText = this.startCapture() - ? this.listeningText() - : "Microphone unavailable (glasses not connected?)"; - } else if (extended && !voiceActivity.isActive()) { - micArrayController.apply(wireConfig(config)); + if (directionChanged) this.clearDirection(); + this.syncCaptureMode(); + if ( + this.mode === modeBefore && + this.mode === "extended" && + this.captureUp && + !voiceActivity.isActive() + ) { + this.armExtended(); } } this.notify(); } + private handleSettingsChanged(): void { + this.refreshCachedFlags(); + // The developer gate on mic control can change under a running session. + this.syncCaptureMode(); + } + + /** Swap capture paths when 4-mic capture (or its developer gate) changed. */ + private syncCaptureMode(): void { + if (!this.running) return; + const mode = this.desiredMode(); + if (mode === this.mode) return; + this.stopCapture(); + this.mode = mode; + this.clearDirection(); + this.bringUpCapture(); + } + private refreshCachedFlags(): void { this.beamFilterOn = beamFilterSetting.get(); const wasOn = this.ancOn; @@ -417,15 +535,16 @@ class MicSession { } /** - * The assistant modal owns the single G2 mic while open. Pause our capture - * for its duration and re-acquire when it closes — the stock raw tap is - * preempted by the bridge and must be re-requested; the extended SM stream - * must stop so the assistant's stock-format session can run at all. + * Speech capture (the assistant modal, a dictation draft, Transcribe) owns + * the single G2 mic while active. Pause our capture for its duration and + * re-acquire when it ends — the stock raw tap is preempted by the bridge + * and must be re-requested; the extended SM stream must stop so the + * stock-format capture can run at all. */ private handleVoiceActivity(active: boolean): void { if (!this.running) return; if (active) { - if (this.mode === "extended") { + if (this.mode === "extended" && this.captureUp) { micArrayController.stopStreaming(); try { this.nativeCommunicator()?.stopG2AudioForwarding(); @@ -435,36 +554,88 @@ class MicSession { } return; } - const communicator = this.nativeCommunicator(); - if (!communicator) return; - if (this.mode === "extended") { - if (this.forwardingProxy && communicator.startG2AudioForwarding(this.forwardingProxy)) { - micArrayController.apply(wireConfig(this.activeConfig)); + // Take the mic back now rather than at the next check. + this.checkCapture(); + } + + /** + * Bring capture up if it isn't, and re-acquire it if audio stopped + * arriving. Runs every CAPTURE_CHECK_MS and when speech capture ends. + */ + private checkCapture(): void { + if (!this.running || voiceActivity.isActive()) return; + try { + if (!this.captureUp) { + // Retrying while disconnected would only log skipped sends. + if (this.nativeCommunicator()?.isSessionReady()) this.bringUpCapture(); + return; } - } else { - voiceControlBridge.startRawCapture({ communicator }); + const healthy = this.mode === "extended" ? this.checkExtendedCapture() : this.checkStockCapture(); + this.setCaptureStalled(!healthy); + } catch (error) { + console.warn(`mic capture check failed: ${error}`); } } + private setCaptureStalled(stalled: boolean): void { + if (stalled === this.captureStalled) return; + this.captureStalled = stalled; + this.statusText = stalled ? STALLED_TEXT : this.listeningText(); + } + + /** + * The stock raw tap dies with the glasses session, and an STT capture + * preempts it. Restart it whenever it isn't live. Returns false (stalled) + * while disconnected or once restarts keep failing. + */ + private checkStockCapture(): boolean { + if (voiceControlBridge.isRawCaptureLive()) { + this.stockRestarts = 0; + return true; + } + const communicator = this.nativeCommunicator(); + if (!communicator?.isSessionReady()) return false; + voiceControlBridge.startRawCapture({ communicator, owner: RAW_CAPTURE_OWNER }); + // One restart is routine (after the voice dialog, or a reconnect); only + // report a stall once restarts keep failing. + this.stockRestarts += 1; + return this.stockRestarts < 2; + } + /** * The extended stream's forwarding lives in the glasses session: an arm - * drop clears it and nothing else re-arms it on the next session, so the - * app would sit at "Listening" hearing nothing. Re-establish forwarding - * and re-send the config whenever a ready session has lost it. (The stock - * path is resumed by the voice-control bridge.) + * drop or EvenHub resume clears it, so re-establish it and re-send the + * config. With forwarding in place, each enabled temple must actually be + * streaming: a lost CONFIGURE or a temple that restarted on its own looks + * like silence, so a temple quiet past its grace period is reconfigured. + * Returns whether audio is arriving from every enabled temple. */ - private checkExtendedCapture(): void { - if (!this.running || this.mode !== "extended" || !this.forwardingProxy) return; - if (voiceActivity.isActive()) return; - try { - const communicator = this.nativeCommunicator(); - if (!communicator?.isSessionReady() || communicator.isAudioCaptureActive()) return; - if (communicator.startG2AudioForwarding(this.forwardingProxy)) { - micArrayController.apply(wireConfig(this.activeConfig)); - } - } catch (error) { - console.warn(`mic capture re-arm failed: ${error}`); + private checkExtendedCapture(): boolean { + const communicator = this.nativeCommunicator(); + if (!communicator?.isSessionReady()) return false; + if (!communicator.isAudioCaptureActive()) { + if (!this.forwardingProxy || !communicator.startG2AudioForwarding(this.forwardingProxy)) return false; + // Routine after the voice dialog or a reconnect; the frame check below + // reports a stall if the temples don't start streaming. + return this.armExtended() && !this.captureStalled; } + const now = Date.now(); + const wire = wireConfig(this.activeConfig); + let healthy = true; + for (const side of ["left", "right"] as const) { + if (!templeActive(this.activeConfig, side)) continue; + if (now - this.templeFrameAtMs[side] < TEMPLE_SILENT_MS) continue; + if (now - this.templeArmedAtMs[side] < TEMPLE_SILENT_MS) { + // Just (re)configured: give it time to start streaming, without + // calling it healthy if it was already stalled. + if (this.captureStalled) healthy = false; + continue; + } + healthy = false; + this.templeArmedAtMs[side] = now; + micArrayController.reconfigureTemple(wire, side); + } + return healthy; } private startSuppressor(): void { @@ -535,6 +706,34 @@ class MicSession { ); } + /** Record a newly reported direction of arrival (device frame). */ + private updateDirection(deviceDeg: number, nowMs: number): void { + this.doaDeviceDeg = smoothAngleDeg(this.doaDeviceDeg, deviceDeg, DOA_SMOOTHING); + this.doaAtMs = nowMs; + if (this.beamLocked && this.doaDeviceDeg !== null) { + this.beamWorldDeg = this.toWorldDeg(this.doaDeviceDeg); + } + } + + private clearDirection(): void { + this.doaDeviceDeg = null; + this.doaAtMs = 0; + this.ssr = 0; + this.templeAngles = { left: null, right: null }; + this.templeSsr = { left: 0, right: 0 }; + this.templeAngleAtMs = { left: 0, right: 0 }; + } + + /** + * Beam filter: whether to drop audio because it arrives from outside the + * beam. Only a current direction can gate; with none, audio passes. + */ + private outsideBeam(): boolean { + if (!this.beamFilterOn || this.ssr <= 0 || this.doaDeviceDeg === null) return false; + if (Date.now() - this.doaAtMs >= DOA_STALE_MS) return false; + return !this.inBeam(this.doaDeviceDeg); + } + // ---- compass (world-frame radar + beam-follow while head turns) ---- private startCompass(): void { @@ -564,19 +763,18 @@ class MicSession { private startStock(): boolean { const communicator = this.nativeCommunicator(); - if (!communicator) return false; + // The tap only records intent without a session; wait for one instead. + if (!communicator?.isSessionReady()) return false; this.levels = [0]; this.levelLabels = ["Glasses mic"]; - const started = voiceControlBridge.startRawCapture({ communicator }); + this.stockRestarts = 0; + const started = voiceControlBridge.startRawCapture({ communicator, owner: RAW_CAPTURE_OWNER }); if (!started) return false; this.offRawPcm = voiceControlBridge.onRawPcm((pcm) => this.handleStockPcm(pcm)); this.offFrameMeta = voiceControlBridge.onFrameMeta((meta) => { this.ssr = meta.ssr; if (meta.angleDegrees !== 0 || meta.ssr !== 0) { - this.doaDeviceDeg = smoothAngleDeg(this.doaDeviceDeg, meta.angleDegrees, DOA_SMOOTHING); - if (this.beamLocked && this.doaDeviceDeg !== null) { - this.beamWorldDeg = this.toWorldDeg(this.doaDeviceDeg); - } + this.updateDirection(meta.angleDegrees, Date.now()); } this.trackLiveSpeaker(); }); @@ -588,18 +786,13 @@ class MicSession { this.offRawPcm = null; this.offFrameMeta?.(); this.offFrameMeta = null; - voiceControlBridge.stopRawCapture(); + voiceControlBridge.stopRawCapture(RAW_CAPTURE_OWNER); } private handleStockPcm(pcm: Uint8Array): void { const level = pcmLevel(pcm); this.levels[0] = Math.max(level, (this.levels[0] ?? 0) * LEVEL_DECAY); - const gated = - this.beamFilterOn && - this.doaDeviceDeg !== null && - this.ssr > 0 && - !this.inBeam(this.doaDeviceDeg); - if (gated) return; + if (this.outsideBeam()) return; this.feedPcm(pcm); } @@ -607,7 +800,7 @@ class MicSession { private startExtended(): boolean { const communicator = this.nativeCommunicator(); - if (!communicator) return false; + if (!communicator?.isSessionReady()) return false; // Each active temple streams its full front+rear stereo pair; the four // mics are split, metered, and selected here on the host. this.levelLabels = MIC_CHANNEL_KEYS.map(micChannelLabel); @@ -622,6 +815,13 @@ class MicSession { }, }); if (!communicator.startG2AudioForwarding(this.forwardingProxy)) return false; + return this.armExtended(); + } + + /** Send the config to the temples, starting each one's grace period to begin streaming. */ + private armExtended(): boolean { + const now = Date.now(); + this.templeArmedAtMs = { left: now, right: now }; return micArrayController.apply(wireConfig(this.activeConfig)); } @@ -648,6 +848,8 @@ class MicSession { this.templeSsr = { left: 0, right: 0 }; this.templeAngleAtMs = { left: 0, right: 0 }; this.templeFrameAtMs = { left: 0, right: 0 }; + this.templeArmedAtMs = { left: 0, right: 0 }; + this.feedSwitchSinceMs = 0; } /** A temple's last bearing, or null once it has gone quiet. */ @@ -701,11 +903,8 @@ class MicSession { } else { fusedDeg = (left ?? right)!; } - this.doaDeviceDeg = smoothAngleDeg(this.doaDeviceDeg, fusedDeg, DOA_SMOOTHING); this.ssr = Math.max(left === null ? 0 : this.templeSsr.left, right === null ? 0 : this.templeSsr.right); - if (this.beamLocked && this.doaDeviceDeg !== null) { - this.beamWorldDeg = this.toWorldDeg(this.doaDeviceDeg); - } + this.updateDirection(fusedDeg, now); this.trackLiveSpeaker(); } @@ -731,38 +930,51 @@ class MicSession { } private deliverExtendedAudio(side: MicSide, samples: Float32Array): void { - // Feed captions from one temple only; prefer the one hearing more - // signal, among the temples that still have an enabled microphone and - // are still streaming. - const now = Date.now(); - const leftUsable = - templeActive(this.activeConfig, "left") && now - this.templeFrameAtMs.left < TEMPLE_STALE_MS; - const rightUsable = - templeActive(this.activeConfig, "right") && now - this.templeFrameAtMs.right < TEMPLE_STALE_MS; - const leftSsr = this.freshTempleAngle("left", now) === null ? 0 : this.templeSsr.left; - const rightSsr = this.freshTempleAngle("right", now) === null ? 0 : this.templeSsr.right; - let stronger: MicSide = rightSsr > leftSsr * 1.5 ? "right" : "left"; - if (stronger === "right" && !rightUsable) stronger = "left"; - if (stronger === "left" && !leftUsable) stronger = "right"; - if (strongerChanged(this.captionFeedSide, stronger)) { - this.captionFeedSide = stronger; - } + // Feed captions from one temple only. + this.updateCaptionFeedSide(Date.now()); if (side !== this.captionFeedSide) return; - if ( - this.beamFilterOn && - this.doaDeviceDeg !== null && - this.ssr > 0 && - !this.inBeam(this.doaDeviceDeg) - ) { + if (this.outsideBeam()) return; + this.feedPcm(floatToPcm16(samples)); + } + + /** + * Pick the temple that feeds captions: the one hearing more signal, among + * the temples with an enabled microphone that are still streaming. A + * switch needs the other temple ahead by FEED_SWITCH_RATIO for + * FEED_SWITCH_HOLD_MS; losing the current temple switches at once. + */ + private updateCaptionFeedSide(nowMs: number): void { + const usable = (side: MicSide) => + templeActive(this.activeConfig, side) && nowMs - this.templeFrameAtMs[side] < TEMPLE_STALE_MS; + const heard = (side: MicSide) => (this.freshTempleAngle(side, nowMs) === null ? 0 : this.templeSsr[side]); + const current = this.captionFeedSide; + const other: MicSide = current === "left" ? "right" : "left"; + if (!usable(other)) { + this.feedSwitchSinceMs = 0; return; } - this.feedPcm(floatToPcm16(samples)); + if (!usable(current)) { + this.captionFeedSide = other; + this.feedSwitchSinceMs = 0; + return; + } + if (heard(other) <= heard(current) * FEED_SWITCH_RATIO) { + this.feedSwitchSinceMs = 0; + return; + } + if (this.feedSwitchSinceMs === 0) { + this.feedSwitchSinceMs = nowMs; + } else if (nowMs - this.feedSwitchSinceMs >= FEED_SWITCH_HOLD_MS) { + this.captionFeedSide = other; + this.feedSwitchSinceMs = 0; + } } // ---- shared audio consumers ---- private feedPcm(pcm: Uint8Array): void { - if (!this.captionEngine && !this.recorder) return; + const engine = this.captionRun?.engine; + if (!engine && !this.recorder) return; let javaBytes = toJavaBytes(pcm); if (this.ancOn && this.suppressor) { try { @@ -771,9 +983,9 @@ class MicSession { console.warn(`noise suppression failed: ${error}`); } } - if (this.captionEngine) { + if (engine) { try { - this.captionEngine.acceptPcm(javaBytes); + engine.acceptPcm(javaBytes); } catch (error) { console.warn(`caption feed failed: ${error}`); } @@ -789,50 +1001,93 @@ class MicSession { // ---- captions ---- + /** Whether captions are on, as the menu toggle shows it (off after they gave up). */ + captionsEnabled(): boolean { + return captionsEnabledSetting.get() && !this.captionsFailure; + } + setCaptionsEnabled(enabled: boolean): void { captionsEnabledSetting.set(enabled); - if (!this.running) return; - if (enabled && !this.captionsActive) { - this.startCaptions(); - if (!this.recorder && this.shouldRecord()) this.startRecording(); - } else if (!enabled && this.captionsActive) { - this.stopCaptions(); - // The recording belongs to this caption session (its segment offsets - // are relative to this engine's clock); close it out with the row. - this.stopRecording(true); - this.finishSessionRow(); + this.captionsFailure = ""; + if (this.running) { + if (enabled && !this.captionsActive) { + // While capture is still coming up, bringUpCapture starts them. + if (this.captureUp) this.startCaptions(); + } else if (!enabled && this.captionsActive) { + this.stopCaptions(); + } } this.notify(); } + /** Begin a captioned conversation: the engine, then its store row and recording. */ private startCaptions(): void { if (this.captionsActive) return; + const run = this.createCaptionRun(); + if (!run) { + this.captionsFailure = "the caption engine could not start"; + return; + } + this.captionsActive = true; + this.captionsFailure = ""; + this.captionFailures = 0; + this.encounterShown.clear(); + this.conversation = this.openConversation(); + run.conversation = this.conversation; + this.captionRun = run; + if (this.conversation && this.shouldRecord()) this.startRecording(this.conversation); + } + + /** Start a caption engine on the current conversation, or null if it can't start. */ + private createCaptionRun(): CaptionRun | null { let engine: any = null; try { engine = new com.faceclaw.app.FaceclawCaptionEngine(); - this.captionListenerProxy = new com.faceclaw.app.FaceclawCaptionEngineListener({ - onUtterance: (text: string, embedding: any, startMs: number, endMs: number, peakRms: number) => { + const run: CaptionRun = { + engine, + listener: null, + startWallMs: Date.now(), + // A restarted engine's clock starts at zero partway into the recording. + audioBaseMs: this.recorder ? Number(this.recorder.getDurationMs()) : 0, + conversation: this.conversation, + generation: this.sessionGeneration, + stopTimer: null, + afterStop: null, + }; + run.listener = new com.faceclaw.app.FaceclawCaptionEngineListener({ + onUtterance: (text: string, embedding: any, startMs: number, endMs: number, _peakRms: number) => { // Never let an exception escape into the engine's Java callback. try { - this.handleUtterance(String(text), embedding, Number(startMs), Number(endMs), Number(peakRms)); + this.handleUtterance(run, String(text), embedding, Number(startMs), Number(endMs)); } catch (error) { console.warn(`caption utterance failed: ${error}`); } }, onSpeechStart: (_startMs: number) => { - this.speechActive = true; + if (run === this.captionRun) this.speechActive = true; }, onStatus: (status: string) => { - this.statusText = String(status); + // Capture problems outrank the engine's own progress messages. + if (run === this.captionRun && this.captureUp && !this.captureStalled) { + this.statusText = String(status); + } + }, + onStopped: (error: string | null) => { + try { + this.handleRunStopped(run, error === null || error === undefined ? null : String(error)); + } catch (stopError) { + console.warn(`caption engine stop handling failed: ${stopError}`); + } }, }); - engine.setListener(this.captionListenerProxy); + engine.setListener(run.listener); // Live captions are a separate sherpa-onnx pipeline (FaceclawCaptionEngine) // that only ever uses Moonshine; unaffected by the two-model on-device // Whisper addition in native/voice-control.ts's push-to-talk path. engine.setAsrModelDir(isAsrModelReady("moonshine") ? moonshineModelDir() : null); engine.setSpeakerModelPath(micModelPath("speaker-embedding")); engine.start(); + return run; } catch (error) { console.warn(`caption engine start failed: ${error}`); try { @@ -840,36 +1095,125 @@ class MicSession { } catch { // It never started. } - this.captionListenerProxy = null; - this.statusText = "Captions unavailable"; - return; - } - this.captionEngine = engine; - this.captionsActive = true; - this.engineStartWallMs = Date.now(); - this.encounterShown.clear(); - if (saveCaptionsSetting.get() && captionsRetentionSetting.get() !== "none") { - try { - this.sessionRowId = Number( - this.store().startSession(Date.now(), `Conversation ${new Date().toLocaleString()}`), - ); - } catch (error) { - console.warn(`caption session start failed: ${error}`); - this.sessionRowId = -1; - } + return null; } } + private handleRunStopped(run: CaptionRun, error: string | null): void { + if (run === this.captionRun) { + // The live engine ended on its own (its models failed to load, or the + // worker died): restart it on the same conversation. + this.captionRun = null; + this.speechActive = false; + this.scheduleCaptionRestart(error ?? "the caption engine stopped"); + return; + } + this.settleRun(run); + } + + private scheduleCaptionRestart(reason: string): void { + this.captionFailures += 1; + if (this.captionFailures > CAPTION_MAX_RESTARTS) { + console.warn(`captions giving up after ${this.captionFailures} failures: ${reason}`); + this.stopCaptions(); + this.captionsFailure = reason; + this.notify(); + return; + } + this.statusText = `Captions failed (${reason}); restarting...`; + this.captionRestartTimer = setTimeout(() => { + this.captionRestartTimer = null; + if (!this.running || !this.captionsActive || this.captionRun) return; + const run = this.createCaptionRun(); + if (run) { + this.captionRun = run; + } else { + this.scheduleCaptionRestart("the caption engine could not restart"); + } + }, CAPTION_RESTART_BASE_MS * this.captionFailures); + this.notify(); + } + + private clearCaptionRestart(): void { + if (this.captionRestartTimer) { + clearTimeout(this.captionRestartTimer); + this.captionRestartTimer = null; + } + } + + /** + * End the captioned conversation. The recording holds every byte the + * engine was fed, so it closes now; the store row stays open until the + * engine has delivered its trailing utterance (or a timeout passes). + */ private stopCaptions(): void { if (!this.captionsActive) return; + this.captionsActive = false; + this.speechActive = false; + this.clearCaptionRestart(); + const endedAtMs = Date.now(); + const run = this.captionRun; + this.captionRun = null; + const conversation = this.conversation; + this.conversation = null; + this.stopRecording(true, conversation); + if (run) { + this.stopRun(run, () => this.finishConversation(conversation, endedAtMs)); + } else { + this.finishConversation(conversation, endedAtMs); + } + } + + private stopRun(run: CaptionRun, afterStop: () => void): void { + run.afterStop = afterStop; + this.stoppingRuns.add(run); + run.stopTimer = setTimeout(() => { + console.warn("caption engine did not report stopping; closing its conversation anyway"); + this.settleRun(run); + }, CAPTION_STOP_TIMEOUT_MS); try { - this.captionEngine?.stop(); + run.engine.requestStop(); } catch (error) { console.warn(`caption engine stop failed: ${error}`); + this.settleRun(run); } - this.captionEngine = null; - this.captionListenerProxy = null; - this.captionsActive = false; + } + + /** A stopping run is done (reported back, or timed out): close what it held open. */ + private settleRun(run: CaptionRun): void { + if (!this.stoppingRuns.delete(run)) return; + if (run.stopTimer) clearTimeout(run.stopTimer); + run.stopTimer = null; + const afterStop = run.afterStop; + run.afterStop = null; + afterStop?.(); + } + + /** The store row for a new conversation, when captions are being saved. */ + private openConversation(): ConversationRecord | null { + if (!saveCaptionsSetting.get() || captionsRetentionSetting.get() === "none") return null; + try { + const rowId = Number( + this.store().startSession(Date.now(), `Conversation ${new Date().toLocaleString()}`), + ); + return rowId >= 0 ? { rowId, sentimentSum: 0, sentimentCount: 0 } : null; + } catch (error) { + console.warn(`caption session start failed: ${error}`); + return null; + } + } + + private finishConversation(conversation: ConversationRecord | null, endedAtMs: number): void { + if (!conversation) return; + const avg = conversation.sentimentCount > 0 ? conversation.sentimentSum / conversation.sentimentCount : 0; + try { + this.store().endSession(conversation.rowId, endedAtMs, "", "", avg); + } catch (error) { + console.warn(`session finish failed: ${error}`); + } + // Recap/action-items/facts per speaker, generated in the background with + // the on-phone model (no-op when it isn't downloaded). + generateSessionInsights(conversation.rowId); } /** The next captioned utterance enrolls (or refreshes) the wearer's voice. */ @@ -995,15 +1339,20 @@ class MicSession { } private handleUtterance( + run: CaptionRun, text: string, embedding: any, startMs: number, endMs: number, - _peakRms: number, ): void { if (!text.trim() && !embedding) return; + const live = run === this.captionRun; + // A stopping run's trailing line still shows if it belongs to this app + // session; it is saved to its own conversation either way. + const shown = this.running && run.generation === this.sessionGeneration; + if (live) this.captionFailures = 0; const embeddingFloats = embedding ? floatArrayToFloat32(embedding) : null; - if (this.wearerEnrollmentArmed && embeddingFloats) { + if (live && this.wearerEnrollmentArmed && embeddingFloats) { this.wearerEnrollmentArmed = false; try { const wearer = speakerRegistry.enrollWearer(embeddingFloats); @@ -1022,11 +1371,11 @@ class MicSession { console.warn(`speaker assignment failed: ${error}`); } } - if (match) this.maybeShowEncounterCard(match); + if (match && shown) this.maybeShowEncounterCard(match); // Spoken introductions name the auto-numbered speakers: "I'm Alice" // renames the speaker of the line; "this is Bob" pends the name for the // next different still-generic voice. User-set names are never touched. - if (match && text.trim()) { + if (match && shown && text.trim()) { const now = Date.now(); const pending = this.pendingIntroduction; if (pending && pendingApplies(pending, match.profile.id, match.profile.name, now)) { @@ -1044,11 +1393,12 @@ class MicSession { if (refreshed) match = { ...match, profile: refreshed }; } const isWearer = match?.profile.isWearer ?? false; - const doaAtUtterance = this.doaDeviceDeg; + const doaAtUtterance = shown ? this.doaDeviceDeg : null; const analyzed = text.trim() ? analyzeLine(text) : null; - if (analyzed) { - this.sentimentSum += analyzed.score; - this.sentimentCount += 1; + const conversation = run.conversation; + if (analyzed && conversation) { + conversation.sentimentSum += analyzed.score; + conversation.sentimentCount += 1; } const line: CaptionLine = { @@ -1061,37 +1411,39 @@ class MicSession { lang: "", emotion: analyzed?.emotion ?? "", sentiment: analyzed?.score ?? 0, - atMs: this.engineStartWallMs + startMs, + atMs: run.startWallMs + startMs, }; - this.captionLines.push(line); - if (this.captionLines.length > CAPTION_LINE_LIMIT) { - this.captionLines.splice(0, this.captionLines.length - CAPTION_LINE_LIMIT); - } + if (shown) { + this.captionLines.push(line); + if (this.captionLines.length > CAPTION_LINE_LIMIT) { + this.captionLines.splice(0, this.captionLines.length - CAPTION_LINE_LIMIT); + } - // Radar dot at the utterance's direction (world frame, so it stays put - // as the head turns and follows the person when they move). - this.speechActive = false; - clearSpeaking(this.speakerDots); - if (match && doaAtUtterance !== null) { - this.speakerDots = placeDotForUtterance( - this.speakerDots, - { speakerId: match.profile.id, name: match.profile.name, color: match.profile.color }, - this.toWorldDeg(doaAtUtterance), - Date.now(), - ); + // Radar dot at the utterance's direction (world frame, so it stays put + // as the head turns and follows the person when they move). + this.speechActive = false; + clearSpeaking(this.speakerDots); + if (match && doaAtUtterance !== null) { + this.speakerDots = placeDotForUtterance( + this.speakerDots, + { speakerId: match.profile.id, name: match.profile.name, color: match.profile.color }, + this.toWorldDeg(doaAtUtterance), + Date.now(), + ); + } } let segmentId = -1; - if (this.sessionRowId >= 0 && line.text) { + if (conversation && line.text) { try { segmentId = Number( this.store().insertSegment( JSON.stringify({ - sessionId: this.sessionRowId, + sessionId: conversation.rowId, speakerId: match?.profile.id, - startedAt: this.engineStartWallMs + startMs, - endedAt: this.engineStartWallMs + endMs, - audioOffsetMs: startMs, + startedAt: run.startWallMs + startMs, + endedAt: run.startWallMs + endMs, + audioOffsetMs: run.audioBaseMs + startMs, text: line.text, sentiment: line.sentiment, emotion: line.emotion, @@ -1105,7 +1457,7 @@ class MicSession { console.warn(`caption segment save failed: ${error}`); } } - this.notify(); + if (shown) this.notify(); // Language identification + translation are async; the UI line and the // stored segment update when they resolve. @@ -1155,70 +1507,51 @@ class MicSession { // ---- recording ---- /** - * Recordings belong to a saved caption session: retention and the phone UI - * find audio only through its session row, so a file recorded without one - * would never be shown or expired. + * Recordings belong to a saved caption conversation: retention and the + * phone UI find audio only through its session row, so a file recorded + * without one would never be shown or expired. */ private shouldRecord(): boolean { - return ( - this.sessionRowId >= 0 && - saveRecordingsSetting.get() && - recordingsRetentionSetting.get() !== "none" - ); + return saveRecordingsSetting.get() && recordingsRetentionSetting.get() !== "none"; } - private startRecording(): void { + private startRecording(conversation: ConversationRecord): void { try { - const context = Utils.android.getApplicationContext(); - const dir = `${context.getExternalFilesDir(null).getAbsolutePath()}/mic-recordings`; const stamp = new Date().toISOString().replace(/[:.]/g, "-"); - this.recordingPath = `${dir}/conversation-${stamp}.wav`; + this.recordingPath = `${recordingsDir()}/conversation-${stamp}.wav`; this.recorder = new com.faceclaw.app.FaceclawWavRecorder(this.recordingPath, SAMPLE_RATE, 1); } catch (error) { console.warn(`recording start failed: ${error}`); this.recorder = null; this.recordingPath = ""; + return; + } + // Link the file at once: if the process dies mid-recording, the row + // still names it, and boot recovery repairs and keeps it. + try { + this.store().setSessionAudio(conversation.rowId, this.recordingPath, "wav"); + } catch (error) { + console.warn(`recording link failed: ${error}`); } } - private stopRecording(keep: boolean): void { + private stopRecording(keep: boolean, conversation: ConversationRecord | null): void { const recorder = this.recorder; this.recorder = null; if (!recorder) return; + const wavPath = this.recordingPath; try { const bytes = Number(recorder.finish()); - const sessionRowId = this.sessionRowId; - if (!keep || bytes < SAMPLE_RATE || sessionRowId < 0) { + if (!keep || bytes < SAMPLE_RATE || !conversation) { // Under half a second of audio isn't worth keeping, and audio with no // session row to own it would be unreachable and never expire. recorder.discard(); + if (conversation) this.store().setSessionAudio(conversation.rowId, null, null); return; } - const wavPath = this.recordingPath; - const m4aPath = wavPath.replace(/\.wav$/, ".m4a"); const store = this.store(); - store.setSessionAudio(sessionRowId, wavPath, "wav"); - // Transcode in the background; swap the session's audio to the M4A on - // success (the transcoder deletes the WAV). - const transcoder = new com.faceclaw.app.FaceclawAudioTranscoder( - wavPath, - m4aPath, - RECORDING_AAC_BITRATE, - true, - new com.faceclaw.app.FaceclawAudioTranscoder.Listener({ - onDone: (outputPath: string, _outputBytes: number) => { - try { - store.setSessionAudio(sessionRowId, String(outputPath), "aac"); - } catch (error) { - console.warn(`recording audio update failed: ${error}`); - } - }, - onError: (message: string) => { - console.warn(`recording transcode failed: ${message}`); - }, - }), - ); - transcoder.start(); + store.setSessionAudio(conversation.rowId, wavPath, "wav"); + transcodeRecording(store, conversation.rowId, wavPath); } catch (error) { console.warn(`recording stop failed: ${error}`); } @@ -1231,21 +1564,6 @@ class MicSession { return com.faceclaw.app.FaceclawConversationStore.getInstance(context); } - private finishSessionRow(): void { - if (this.sessionRowId < 0) return; - const avg = this.sentimentCount > 0 ? this.sentimentSum / this.sentimentCount : 0; - const finishedSessionId = this.sessionRowId; - try { - this.store().endSession(finishedSessionId, Date.now(), "", "", avg); - } catch (error) { - console.warn(`session finish failed: ${error}`); - } - this.sessionRowId = -1; - // Recap/action-items/facts per speaker, generated in the background with - // the on-phone model (no-op when it isn't downloaded). - generateSessionInsights(finishedSessionId); - } - async applyRetentionSweep(): Promise { if (!global.isAndroid || this.running) return; try { @@ -1262,6 +1580,47 @@ class MicSession { } } + /** + * Repair what a dead process left behind, once per process at boot: + * conversations it never closed, their unfinished recordings (header + * patched to the audio on disk, then transcoded), and recording files no + * session row knows about. + */ + recoverInterruptedConversations(): void { + if (!global.isAndroid || recoveryDone || this.running) return; + recoveryDone = true; + try { + const store = this.store(); + const closed = JSON.parse( + String(store.closeInterruptedSessions(JSON.stringify(this.openConversationRows()))), + ) as Array<{ id: number; audioPath: string | null; audioCodec: string | null }>; + // Before any transcode below starts writing, so its output isn't swept. + const removed = Number(store.deleteUnreferencedRecordings(recordingsDir())); + for (const session of closed) { + const rowId = Number(session.id); + if (session.audioPath && session.audioCodec === "wav") { + recoverRecording(store, rowId, String(session.audioPath)); + } + generateSessionInsights(rowId); + } + if (closed.length > 0 || removed > 0) { + console.log(`mic recovery: closed ${closed.length} conversation(s), removed ${removed} orphaned recording(s)`); + } + } catch (error) { + console.warn(`mic recovery failed: ${error}`); + } + } + + /** Session rows legitimately still open: the live conversation and ones awaiting a trailing line. */ + private openConversationRows(): number[] { + const rows = new Set(); + if (this.conversation) rows.add(this.conversation.rowId); + for (const run of this.stoppingRuns) { + if (run.conversation) rows.add(run.conversation.rowId); + } + return Array.from(rows); + } + private nativeCommunicator(): any { try { return com.faceclaw.app.FaceclawBleCommunicator.getActive(); @@ -1330,13 +1689,79 @@ function encodeEmbeddingBase64(embedding: Float32Array): string { return encodeBase64(new Uint8Array(embedding.buffer, embedding.byteOffset, embedding.byteLength)); } -function strongerChanged(current: MicSide, candidate: MicSide): boolean { - return current !== candidate; -} - function moonshineModelDir(): string { const context = Utils.android.getApplicationContext(); return `${context.getFilesDir().getAbsolutePath()}/faceclaw-voice-asr/sherpa-onnx-moonshine-base-en-quantized-2026-02-27`; } +function recordingsDir(): string { + const context = Utils.android.getApplicationContext(); + return `${context.getExternalFilesDir(null).getAbsolutePath()}/mic-recordings`; +} + +function deleteFile(path: string): void { + try { + new java.io.File(path).delete(); + } catch (error) { + console.warn(`could not delete ${path}: ${error}`); + } +} + +// Transcoder listeners are referenced only from Java while a transcode runs; +// hold them here so they aren't GC'd before their callback. +const pendingTranscodes = new Set(); + +/** + * Transcode a finished WAV recording to M4A in the background and move its + * session onto it. The WAV is deleted only once the row names the M4A, so a + * failure at any step leaves the session pointing at a file that exists. + */ +function transcodeRecording(store: any, rowId: number, wavPath: string): void { + const m4aPath = wavPath.replace(/\.wav$/, ".m4a"); + const listener = new com.faceclaw.app.FaceclawAudioTranscoder.Listener({ + onDone: (outputPath: string, _outputBytes: number) => { + pendingTranscodes.delete(listener); + try { + store.setSessionAudio(rowId, String(outputPath), "aac"); + } catch (error) { + console.warn(`recording audio update failed: ${error}`); + deleteFile(String(outputPath)); + return; + } + deleteFile(wavPath); + }, + onError: (message: string) => { + pendingTranscodes.delete(listener); + console.warn(`recording transcode failed: ${message}`); + }, + }); + pendingTranscodes.add(listener); + try { + // deleteSourceOnSuccess false: the WAV goes only after the row update above. + new com.faceclaw.app.FaceclawAudioTranscoder(wavPath, m4aPath, RECORDING_AAC_BITRATE, false, listener).start(); + } catch (error) { + pendingTranscodes.delete(listener); + console.warn(`recording transcode start failed: ${error}`); + } +} + +/** + * Keep a recording its process died writing: patch the WAV header to the + * audio on disk, then transcode it as a normal stop would. Too short (or + * gone) and it is dropped from the session instead. + */ +function recoverRecording(store: any, rowId: number, wavPath: string): void { + try { + const bytes = Number(com.faceclaw.app.FaceclawWavRecorder.repairHeader(wavPath, SAMPLE_RATE, 1)); + if (bytes < SAMPLE_RATE) { + deleteFile(wavPath); + store.setSessionAudio(rowId, null, null); + return; + } + transcodeRecording(store, rowId, wavPath); + } catch (error) { + console.warn(`recording recovery failed (${wavPath}): ${error}`); + } +} + export const micSession = new MicSession(); diff --git a/app/apps/microphones/microphones-app.ts b/app/apps/microphones/microphones-app.ts index c7da0c86..a520da1d 100644 --- a/app/apps/microphones/microphones-app.ts +++ b/app/apps/microphones/microphones-app.ts @@ -453,8 +453,9 @@ export function createMicrophonesAppWindow(options: InProcessAppOptions): InProc }), configToggleItem( "Captions", - () => captionsEnabledSetting.get(), - () => micSession.setCaptionsEnabled(!captionsEnabledSetting.get()), + // Shows off after repeated engine failures, so turning it on retries. + () => micSession.captionsEnabled(), + () => micSession.setCaptionsEnabled(!micSession.captionsEnabled()), undefined, captionsEnabledSetting.description, ), diff --git a/app/apps/microphones/rediarize.ts b/app/apps/microphones/rediarize.ts index d5be9456..3a2bfc4e 100644 --- a/app/apps/microphones/rediarize.ts +++ b/app/apps/microphones/rediarize.ts @@ -35,6 +35,10 @@ function store(): any { return com.faceclaw.app.FaceclawConversationStore.getInstance(context); } +// Each run's listener is referenced only from Java while the diarizer works; +// hold it here so it isn't GC'd before its callbacks. +const activeListeners = new Set(); + export function rediarizeSession( sessionId: number, options: { numSpeakers?: number; threshold?: number }, @@ -54,6 +58,7 @@ export function rediarizeSession( const listener = new com.faceclaw.app.FaceclawDiarizer.Listener({ onProgress: (progress: number) => callbacks.onProgress(Number(progress)), onDone: (turnsJson: string) => { + activeListeners.delete(listener); try { const turns = JSON.parse(String(turnsJson)) as Turn[]; callbacks.onDone(applyTurns(sessionId, turns)); @@ -61,17 +66,26 @@ export function rediarizeSession( callbacks.onError(`Applying diarization failed: ${String(error)}`); } }, - onError: (message: string) => callbacks.onError(String(message)), + onError: (message: string) => { + activeListeners.delete(listener); + callbacks.onError(String(message)); + }, }); - const diarizer = new com.faceclaw.app.FaceclawDiarizer( - segmentationModel, - embeddingModel, - audioPath, - options.numSpeakers ?? -1, - options.threshold ?? 0.5, - listener, - ); - diarizer.start(); + activeListeners.add(listener); + try { + const diarizer = new com.faceclaw.app.FaceclawDiarizer( + segmentationModel, + embeddingModel, + audioPath, + options.numSpeakers ?? -1, + options.threshold ?? 0.5, + listener, + ); + diarizer.start(); + } catch (error) { + activeListeners.delete(listener); + callbacks.onError(`Re-diarization failed to start: ${String(error)}`); + } } type StoredSegment = { diff --git a/app/g2/dashboard-controller.ts b/app/g2/dashboard-controller.ts index 49e0ac8d..a763ac68 100644 --- a/app/g2/dashboard-controller.ts +++ b/app/g2/dashboard-controller.ts @@ -52,6 +52,7 @@ import { isPreviewOnlyMode } from "../phone-ui/onboarding-state"; import { claimGlassesOnboardingAutoLaunch, ONBOARDING_APP_ID } from "../apps/onboarding/onboarding-progress"; import { beginRenderPass, endRenderPass } from "../util/render-freshness"; import { voiceControlBridge } from "../native/voice-control"; +import { voiceActivity } from "../ui/shell/voice-activity"; import { G2_LENS_HEIGHT, G2_LENS_WIDTH, GrayImage } from "../graphics/image"; import { flattenPlanesWithDraws, planesFingerprint, type Plane } from "../graphics/plane"; import { prepareFrameDraws } from "../graphics/glyph-wire"; @@ -163,6 +164,8 @@ const SHELL_REFRESH_INTERVAL_MS = 60_000; // schedules a refresh), so this just covers anything that slips through. const PREVIEW_INTERVAL_MS = 1_000; const SCREEN_TIMEOUT_CHECK_MS = 1_000; +// voiceActivity holder for Transcribe's continuous capture. +const CONTINUOUS_CAPTURE_HOLDER = "continuous-capture"; // The lock screen is a short notice, not something to read at leisure; it // ignores the screen-timeout setting (including "never"). const LOCK_SCREEN_TIMEOUT_MS = 15_000; @@ -2058,11 +2061,16 @@ class DashboardController { } private startContinuousVoiceCapture(): void { + // Continuous capture (Transcribe) owns the glasses mic for as long as it + // runs. Claim it before the capture starts so the Microphones app and + // EvenHub mic apps stand down first, as they do for the voice dialog. + voiceActivity.setActive(true, CONTINUOUS_CAPTURE_HOLDER); void this.beginVoiceCapture("continuous"); } private stopContinuousVoiceCapture(): void { voiceControlBridge.stopContinuousCapture(); + voiceActivity.setActive(false, CONTINUOUS_CAPTURE_HOLDER); } /** diff --git a/app/native/voice-control.ts b/app/native/voice-control.ts index 7462b22c..91db6624 100644 --- a/app/native/voice-control.ts +++ b/app/native/voice-control.ts @@ -122,13 +122,17 @@ export class FaceclawVoiceControlBridge { private captureEndpointing = false; // Non-null while a cloud provider owns the transcript; Java only decodes PCM. private cloudClient: CloudSttClient | null = null; - // Raw-PCM tap (EvenHub mic apps): when active, the controller runs in the - // decode-only "cloud" mode and every decoded PCM frame is broadcast to these - // listeners. STT capture preempts it — a live assistant/transcribe session - // owns the single G2 mic and the raw tap is stopped for its duration. + // Raw-PCM tap (EvenHub mic apps, the Microphones app): when active, the + // controller runs in the decode-only "cloud" mode and every decoded PCM + // frame is broadcast to these listeners. STT capture preempts it — a live + // assistant/transcribe session owns the single G2 mic and the raw tap is + // stopped for its duration. private readonly rawPcmListeners = new Set(); private readonly frameMetaListeners = new Set(); private rawActive = false; + // Who wants the raw tap. It runs while any owner does, so one owner letting + // go (an EvenHub app backgrounding) doesn't silence another. + private readonly rawOwners = new Set(); // Set when the current capture session runs "my voice only" verification; // a non-wearer verdict arrives just before the final transcript and makes // the bridge swallow it. @@ -220,11 +224,33 @@ export class FaceclawVoiceControlBridge { } /** - * Start the decode-only raw-PCM tap for an EvenHub mic app. Fails (returns - * false) if an STT capture already owns the mic; the caller re-tries when its - * eligibility is re-evaluated. Idempotent while already running. + * Start the decode-only raw-PCM tap on behalf of `owner`. Fails (returns + * false) if an STT capture already owns the mic; the owner stays registered + * and re-tries later (the EvenHub router when eligibility is re-evaluated, + * the Microphones app from its capture check). Idempotent while already + * running. */ - startRawCapture(options: { communicator: any }): boolean { + startRawCapture(options: { communicator: any; owner: string }): boolean { + if (!global.isAndroid) return false; + this.rawOwners.add(options.owner); + return this.startRawTap(options.communicator); + } + + /** `owner` no longer wants the raw tap; it stops once no owner does. */ + stopRawCapture(owner: string): void { + this.rawOwners.delete(owner); + if (this.rawOwners.size === 0) this.endRawTap(); + } + + /** + * Whether the raw tap is running and its glasses mic enable still holds + * (or is still coming up). False after an STT preemption or a session drop. + */ + isRawCaptureLive(): boolean { + return this.rawActive && this.micIsLive(); + } + + private startRawTap(communicator: any): boolean { if (!global.isAndroid) return false; if (this.rawActive && this.micIsLive()) return true; // An STT holder owns the mic even while its stream waits on a new glasses @@ -234,7 +260,7 @@ export class FaceclawVoiceControlBridge { if (this.started || this.rawActive) this.teardownCapture(); this.suspendedRaw = false; this.ensureController(); - this.controller?.setCommunicator(options.communicator); + this.controller?.setCommunicator(communicator); // The raw tap is strictly the G2 stream; clear any phone-mic flag a // preview-mode capture may have left on the shared controller. this.controller?.setUsePhoneMic(false); @@ -254,7 +280,7 @@ export class FaceclawVoiceControlBridge { return true; } - stopRawCapture(): void { + private endRawTap(): void { this.suspendedRaw = false; if (!this.rawActive) return; this.rawActive = false; @@ -266,7 +292,7 @@ export class FaceclawVoiceControlBridge { if (!global.isAndroid) return; // STT preempts the raw-PCM tap: a live transcription session owns the mic, // and an EvenHub app gets no audio for its duration. - if (this.rawActive) this.stopRawCapture(); + if (this.rawActive) this.endRawTap(); // Whether the mic is already running is a question about the stream, not // about this set: a holder whose capture died with the glasses session // (or whose mic enable was refused) is still in it. Trusting the set here @@ -428,8 +454,8 @@ export class FaceclawVoiceControlBridge { for (const holder of holders) { this.acquireCapture(holder, { ...options, endpointing: this.captureEndpointing }); } - if (resumeRaw && !holders.length) { - this.startRawCapture({ communicator: options.communicator }); + if (resumeRaw && !holders.length && this.rawOwners.size > 0) { + this.startRawTap(options.communicator); } } diff --git a/app/ui/shell/voice-activity.ts b/app/ui/shell/voice-activity.ts index e38acf82..b46df694 100644 --- a/app/ui/shell/voice-activity.ts +++ b/app/ui/shell/voice-activity.ts @@ -1,8 +1,13 @@ /** - * A tiny global signal for whether a voice-input modal (the assistant dialog) - * is currently open. The shell sets it; the EvenHub mic router reads it to - * decide whether an app may hold the glasses microphone — an app receives no - * audio while the assistant is active, since both share the one G2 mic. + * A tiny global signal for whether speech capture owns the glasses + * microphone: the assistant's voice dialog, a dictation draft, or + * Transcribe's continuous capture. The EvenHub mic router and the + * Microphones app read it to yield the one G2 mic while it is active — an + * app receives no audio meanwhile, since they all share it. + * + * Each user holds it under its own name, so one finishing doesn't clear + * another's hold. Set it before starting the capture and clear it after + * releasing, so other mic users stand down first and resume after. * * Kept as a dependency-free leaf so both the shell and the EvenHub layer can * import it without a cycle. @@ -10,16 +15,19 @@ type Listener = (active: boolean) => void; -let active = false; +const holders = new Set(); const listeners = new Set(); export const voiceActivity = { isActive(): boolean { - return active; + return holders.size > 0; }, - setActive(next: boolean): void { - if (active === next) return; - active = next; + setActive(next: boolean, holder = "voice-input"): void { + const was = holders.size > 0; + if (next) holders.add(holder); + else holders.delete(holder); + const active = holders.size > 0; + if (active === was) return; listeners.forEach((listener) => listener(active)); }, subscribe(listener: Listener): () => void { diff --git a/native/kotlin/shared/src/androidMain/kotlin/com/faceclaw/app/FaceclawWavRecorder.kt b/native/kotlin/shared/src/androidMain/kotlin/com/faceclaw/app/FaceclawWavRecorder.kt index 8e0d8482..ed002424 100644 --- a/native/kotlin/shared/src/androidMain/kotlin/com/faceclaw/app/FaceclawWavRecorder.kt +++ b/native/kotlin/shared/src/androidMain/kotlin/com/faceclaw/app/FaceclawWavRecorder.kt @@ -13,6 +13,30 @@ constructor( private val sampleRate: Int, private val channels: Int, ) { + companion object { + /** + * Rewrite the header of a recording whose writer never finished (the process died + * mid-recording, leaving the empty header written at start) to cover the PCM actually on + * disk, trimming any partial sample frame. Returns the PCM byte count, or -1 when the + * file doesn't exist. + */ + @JvmStatic + @Throws(IOException::class) + fun repairHeader(path: String, sampleRate: Int, channels: Int): Long { + val target = File(path) + if (!target.exists()) return -1 + val headerBytes = BinaryEncoding.wavHeader(0, sampleRate, channels).size.toLong() + val frameBytes = channels * 2L + val pcmBytes = maxOf(0L, target.length() - headerBytes) / frameBytes * frameBytes + RandomAccessFile(target, "rw").use { + it.setLength(headerBytes + pcmBytes) + it.seek(0) + it.write(BinaryEncoding.wavHeader(minOf(pcmBytes, Int.MAX_VALUE.toLong()).toInt(), sampleRate, channels)) + } + return pcmBytes + } + } + private var file: RandomAccessFile? private var dataBytes = 0L diff --git a/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/audio/CaptionEngineCore.kt b/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/audio/CaptionEngineCore.kt index 6d18e7e6..c3142aeb 100644 --- a/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/audio/CaptionEngineCore.kt +++ b/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/audio/CaptionEngineCore.kt @@ -312,6 +312,21 @@ class CaptionSession( } } + /** + * Stops without waiting. The worker still flushes the trailing utterance and releases the + * models, then reports onStopped, so a caller can close out after the last line arrives + * instead of blocking its thread on the decode. + */ + fun requestStop() { + lock.withLock { + if (!started) { + return + } + started = false + } + queue.wake() + } + /** Push decoded 16 kHz mono S16LE PCM (any chunking). */ fun acceptPcm(pcm16le: ByteArray?) { if (!started || pcm16le == null || pcm16le.size < 2) { @@ -322,6 +337,7 @@ class CaptionSession( private fun runLoop(finished: Latch) { workerThreadName = currentThreadName() + var failure: String? = null try { loadModels() segmenter.reset() @@ -329,18 +345,35 @@ class CaptionSession( val chunk = queue.take(250) { started } ?: continue segmenter.processChunk(chunk) } - // Flush a trailing utterance so its text isn't lost on stop. + // Finish the audio already queued when the stop landed, then flush the trailing + // utterance so its text isn't lost on stop. + while (true) { + val chunk = queue.take(0) { false } ?: break + segmenter.processChunk(chunk) + } segmenter.flush() } catch (t: Throwable) { logError("caption engine failed", t) - emitStatus("Captions failed: " + t.message) + failure = t.message ?: t.toString() + emitStatus("Captions failed: $failure") } finally { - decoder?.release() + try { + decoder?.release() + } catch (t: Throwable) { + logError("caption model release failed", t) + } decoder = null lock.withLock { - workerFinished = null - workerThreadName = null + // A worker that died on its own leaves the session stopped, so the owner can + // tell (and acceptPcm stops queueing for nobody). + if (workerFinished === finished) { + started = false + workerFinished = null + workerThreadName = null + } } + // Posted before the latch opens, so a blocking stop() returns after it. + emitStopped(failure) finished.countDown() } } @@ -366,8 +399,20 @@ class CaptionSession( override fun onUtterance(samples: ShortArray, length: Int, startMs: Long, endMs: Long, peakRms: Double) { val currentDecoder = decoder ?: return - val text = currentDecoder.recognize(samples, length) - val embedding = currentDecoder.embed(samples, length) + // One utterance the models choke on must not end captioning for the session: deliver + // what did decode and keep listening. + val text = try { + currentDecoder.recognize(samples, length) + } catch (t: Throwable) { + logError("caption recognition failed", t) + "" + } + val embedding = try { + currentDecoder.embed(samples, length) + } catch (t: Throwable) { + logError("caption voice-print failed", t) + null + } val currentListener = listener ?: return val embeddingCopy = embedding?.copyOf() host.dispatcher.post { currentListener.onUtterance(text, embeddingCopy, startMs, endMs, peakRms) } @@ -377,4 +422,9 @@ class CaptionSession( val currentListener = listener ?: return host.dispatcher.post { currentListener.onStatus(status) } } + + private fun emitStopped(error: String?) { + val currentListener = listener ?: return + host.dispatcher.post { currentListener.onStopped(error) } + } } diff --git a/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/callbacks/FaceclawCaptionEngineListener.kt b/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/callbacks/FaceclawCaptionEngineListener.kt index 857492de..66cc929a 100644 --- a/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/callbacks/FaceclawCaptionEngineListener.kt +++ b/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/callbacks/FaceclawCaptionEngineListener.kt @@ -19,4 +19,10 @@ interface FaceclawCaptionEngineListener { fun onSpeechStart(startMs: Long): Unit fun onStatus(status: String?): Unit + + /** + * The worker has exited, after delivering any trailing utterance. error is null after + * stop()/requestStop(), or the failure that ended captioning on its own. + */ + fun onStopped(error: String?): Unit } diff --git a/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/g2protocol/session/GlassesSessionCore.kt b/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/g2protocol/session/GlassesSessionCore.kt index 9a966edc..86542be5 100644 --- a/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/g2protocol/session/GlassesSessionCore.kt +++ b/native/kotlin/shared/src/commonMain/kotlin/com/faceclaw/app/g2protocol/session/GlassesSessionCore.kt @@ -885,16 +885,17 @@ class GlassesSessionCore( * Queue a CFW mic_control record (['M','C',ver,op,...]) as a settings * field-103 write to both temples, or to a single one. Fire-and-forget: * the firmware answers with a field-104 status notify per temple, which - * arrives through addMicStatusListener. + * arrives through addMicStatusListener. Returns false when nothing was + * queued (no session, or a malformed record). */ - fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean) { + fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean { if (record == null || record.size < 4) { - return + return false } monitor.withLock { if (!running || !sessionReady) { logLine("skip mic control ($label); session not ready") - return + return false } if (rightTemple) { pendingMessages.addLast(messageBuilder.faceclawMicControl(record, label, false)) @@ -905,6 +906,7 @@ class GlassesSessionCore( logLine("queue mic control $label") } interruptibleSleep.interrupt() + return true } /** diff --git a/native/kotlin/shared/src/iosMain/kotlin/com/faceclaw/app/ble/IosGlassesSession.kt b/native/kotlin/shared/src/iosMain/kotlin/com/faceclaw/app/ble/IosGlassesSession.kt index 548c483f..3084b297 100644 --- a/native/kotlin/shared/src/iosMain/kotlin/com/faceclaw/app/ble/IosGlassesSession.kt +++ b/native/kotlin/shared/src/iosMain/kotlin/com/faceclaw/app/ble/IosGlassesSession.kt @@ -181,7 +181,7 @@ class IosGlassesSession internal constructor( fun removeMicStatusListener(listener: FaceclawMicStatusListener?) = core.removeMicStatusListener(listener) - fun sendFaceclawMicControl(record: NSData?, label: String, rightTemple: Boolean, leftTemple: Boolean) = + fun sendFaceclawMicControl(record: NSData?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean = core.sendFaceclawMicControl(record?.byteArray(), label, rightTemple, leftTemple) fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean = diff --git a/tests/kotlin/src/androidHostTest/kotlin/com/faceclaw/app/WavRecorderRepairTest.kt b/tests/kotlin/src/androidHostTest/kotlin/com/faceclaw/app/WavRecorderRepairTest.kt new file mode 100644 index 00000000..f35e7013 --- /dev/null +++ b/tests/kotlin/src/androidHostTest/kotlin/com/faceclaw/app/WavRecorderRepairTest.kt @@ -0,0 +1,24 @@ +package com.faceclaw.app + +import java.io.File +import kotlin.test.Test +import kotlin.test.assertEquals + +class WavRecorderRepairTest { + @Test + fun repairsTheHeaderOfARecordingThatWasNeverFinished() { + val file = File.createTempFile("unfinished", ".wav") + try { + // What a process death mid-recording leaves: the empty header from the start, the PCM + // written since, and a torn half-sample at the end. + file.writeBytes(BinaryEncoding.wavHeader(0, 16000, 1) + ByteArray(3201) { 1 }) + assertEquals(3200L, FaceclawWavRecorder.repairHeader(file.path, 16000, 1)) + val bytes = file.readBytes() + assertEquals(44 + 3200, bytes.size) + assertEquals(BinaryEncoding.wavHeader(3200, 16000, 1).toList(), bytes.copyOf(44).toList()) + assertEquals(-1L, FaceclawWavRecorder.repairHeader(file.path + ".missing", 16000, 1)) + } finally { + file.delete() + } + } +} diff --git a/tests/kotlin/src/commonTest/kotlin/com/faceclaw/app/CaptionEngineCoreTest.kt b/tests/kotlin/src/commonTest/kotlin/com/faceclaw/app/CaptionEngineCoreTest.kt index 486d4130..32dd74d7 100644 --- a/tests/kotlin/src/commonTest/kotlin/com/faceclaw/app/CaptionEngineCoreTest.kt +++ b/tests/kotlin/src/commonTest/kotlin/com/faceclaw/app/CaptionEngineCoreTest.kt @@ -90,6 +90,7 @@ class CaptionEngineCoreTest { } override fun onSpeechStart(startMs: Long) { events.add("start:$startMs"); startLatch.countDown() } override fun onStatus(status: String?) { events.add("status:$status") } + override fun onStopped(error: String?) { events.add("stopped:$error") } } val host = object : CaptionHost { override val dispatcher = CallbackDispatcher { it() } @@ -123,5 +124,105 @@ class CaptionEngineCoreTest { assertEquals("status:Captions listening...", events[1]) assertEquals("start:0", events[2]) assertEquals("utterance:hi:2:0", events[3]) + assertEquals("stopped:null", events.last()) + } + + /** Records listener events; `stopped` opens once the worker reports onStopped. */ + private class RecordingListener : FaceclawCaptionEngineListener { + val events = mutableListOf() + val stopped = Latch(1, testPlatform()) + var utteranceLatch = Latch(1, testPlatform()) + override fun onUtterance(text: String?, embedding: FloatArray?, startMs: Long, endMs: Long, peakRms: Double) { + events.add("utterance:$text") + utteranceLatch.countDown() + } + override fun onSpeechStart(startMs: Long) {} + override fun onStatus(status: String?) {} + override fun onStopped(error: String?) { + events.add("stopped:$error") + stopped.countDown() + } + // Read only after a latch: callbacks run on the worker (the test dispatcher is inline). + fun snapshot(): List = events.toList() + } + + private fun captionHost(transcriber: () -> OfflineTranscriber?) = object : CaptionHost { + override val dispatcher = CallbackDispatcher { it() } + override fun loadTranscriber(modelDir: String): OfflineTranscriber? = transcriber() + override fun loadEmbedder(modelPath: String): SpeakerEmbedder? = null + } + + private fun speak(session: CaptionSession, loudChunks: Int = 10, quietChunks: Int = 8) { + val loud = AudioSegmentation.pcm16ToLittleEndian(chunk(3000), 0, 800) + val quiet = AudioSegmentation.pcm16ToLittleEndian(chunk(0), 0, 800) + val idle = AudioSegmentation.pcm16ToLittleEndian(chunk(10), 0, 800) + repeat(2) { session.acceptPcm(idle) } + repeat(loudChunks) { session.acceptPcm(loud) } + repeat(quietChunks) { session.acceptPcm(quiet) } + } + + @Test + fun captionSessionSurvivesAnUtteranceTheModelRejects() { + var calls = 0 + val transcriber = object : OfflineTranscriber { + override fun recognize(samples: FloatArray, count: Int): String { + calls++ + if (calls == 1) throw IllegalStateException("bad input") + return "second" + } + override fun release() {} + } + val listener = RecordingListener() + val errors = mutableListOf() + val session = CaptionSession(captionHost { transcriber }, testPlatform(), "caption-reject-test") { m, _ -> errors.add(m) } + session.setListener(listener) + session.setAsrModelDir("models") + session.setSilenceMs(300) + session.start() + speak(session) + assertTrue(listener.utteranceLatch.await(5000)) + listener.utteranceLatch = Latch(1, testPlatform()) + speak(session) + assertTrue(listener.utteranceLatch.await(5000)) + session.stop() + // The rejected utterance still arrives (without text), and the worker kept going. + assertEquals(listOf("utterance:", "utterance:second", "stopped:null"), listener.snapshot()) + assertEquals(listOf("caption recognition failed"), errors.toList()) + } + + @Test + fun requestStopDeliversTheTrailingUtteranceBeforeStopped() { + val transcriber = object : OfflineTranscriber { + override fun recognize(samples: FloatArray, count: Int): String = "last words" + override fun release() {} + } + val listener = RecordingListener() + val session = CaptionSession(captionHost { transcriber }, testPlatform(), "caption-flush-test") { m, e -> error("$m: $e") } + session.setListener(listener) + session.setAsrModelDir("models") + session.start() + // Speech with no closing silence: the utterance is still open when the stop lands. + speak(session, loudChunks = 10, quietChunks = 0) + session.requestStop() + assertTrue(listener.stopped.await(5000)) + assertEquals(listOf("utterance:last words", "stopped:null"), listener.snapshot()) + } + + @Test + fun aModelLoadFailureReportsStoppedWithTheError() { + val listener = RecordingListener() + val session = CaptionSession( + captionHost { throw IllegalStateException("model corrupt") }, + testPlatform(), + "caption-load-test", + ) { _, _ -> } + session.setListener(listener) + session.setAsrModelDir("models") + session.start() + assertTrue(listener.stopped.await(5000)) + assertEquals(listOf("stopped:model corrupt"), listener.snapshot()) + // The dead worker left the session stopped, so a later stop is a no-op. + session.stop() + session.requestStop() } }