Fix a bunch of microphones app reliability issues

<details>
<summary>Prompts Used</summary>
User: Review the Microphones app, focusing on error handling and reliability.

Opus 5.5:
> I've finished reading the Microphones code and its native pieces. The 2026-10-14 commit `b50898e` fixed a lot, and the main problem left is the same in most places: the app reports "Listening" or "captions on" but never checks that audio is actually arriving. When something fails partway, the session usually stops working with no visible sign.
>
> ## High: silent loss of function or data
>
> **1. In stock mode, other mic users can take the mic away for good.**
> - `stopRawCapture()` tears down the single shared audio tap no matter who started it (`app/native/voice-control.ts:257`).
> - The EvenHub mic router calls it whenever a mic-using EvenHub app goes to the background or closes (`app/apps/evenhub/mic-router.ts:88`). That kills Microphones' audio.
> - Transcribe takes the mic without setting `voiceActivity` (`voice-control.ts:269`), so Microphones is never told to get it back.
> - If capture starts while the session isn't ready, the native controller only records the request, so no audio ever flows.
> - `checkExtendedCapture` (`mic-session.ts:456`) only covers 4-mic mode. Its comment says the stock path "is resumed by the voice-control bridge", but that only covers the session-dropped-while-active case.
> - In 4-mic mode the problem runs the other way: Transcribe takes over the shared Kotlin listener (`GlassesSessionCore.kt:470`) while the glasses keep sending 4-mic frames, so Transcribe gets audio in the wrong format.
> - **Fix:** track who holds the raw tap (like `captureHolders`), add a stock-mode branch to the 3 s check, and have Transcribe set the same "mic in use" signal as the assistant.
>
> **2. One bad utterance ends captions for the rest of the session.**
> - The caption worker has no per-utterance error handling. Any exception from `recognize()` or `embed()` stops the worker for good (`CaptionEngineCore.kt:334`).
> - `started` stays true, so TypeScript still thinks captions are running and recording carries on. The only sign is a "Captions failed:" status string.
> - **Fix:** catch errors per utterance in `onUtterance` (`:367`). Report a fatal worker exit through its own callback so the session can restart the engine or mark captions inactive.
>
> **3. A crash or low-memory kill during recording leaves files that are never shown or deleted.**
> - The WAV header is written with a data size of 0 (`FaceclawWavRecorder.kt:26`) and only corrected in `finish()`.
> - The file is only linked to its conversation in `stopRecording` (`mic-session.ts:1200`), and the conversation row is never closed.
> - Nothing cleans up `mic-recordings/`, so after a crash you get the same never-expired orphan files the last commit fixed for the captions-off case.
> - **Fix:** link the file to its conversation when recording starts. In the boot-time cleanup: close conversations left open (`ended_at IS NULL`), repair the WAV header from the file length, and delete files no conversation points to.
>
> **4. When the transcoder's success callback fails, it deletes both copies of the recording.**
> - `FaceclawAudioTranscoder.kt:66-83` deletes the WAV before calling `onDone`. If `onDone` throws, the catch block deletes the M4A too, and the database still points at the deleted WAV.
> - Separately, the JS listener is referenced only from Java (`mic-session.ts:1208`). Under `markingMode: 'none'` that's the situation `mic-control.ts:74` warns about.
> - I haven't confirmed the garbage-collection risk. `asr-model.ts` does the same thing and its downloads seem to work. But the delete order is wrong either way.
> - `rediarize.ts` and `mic-models.ts` hold their listeners the same way.
>
> ## Medium
>
> 5. **The last utterance is lost when you close the app or turn captions off.** When the caption engine stops, it flushes the utterance in progress and posts it to the main thread. By the time it arrives, `stop()` has already run `finishSessionRow()` (`mic-session.ts:328`), so the line is never saved and is missing from the session insights. If decoding takes longer than the 2 s wait, the line can land in the *next* session with wrong times. Also, `stop()` can block the UI for up to 2 s.
> 6. **The beam filter can mute 4-mic captions permanently.** The direction and signal strength only update on frames with a non-zero angle or strength (`:685`), and they never go stale. If you turn off "Direction detection", the last direction stays frozen. If it was outside the beam, `deliverExtendedAudio` drops all audio from then on (`:751-757`). Stock mode avoids this because it updates the strength on every frame. **Fix:** timestamp the fused direction and treat it as unknown after `TEMPLE_STALE_MS`.
> 7. **The 4-mic health check looks at a phone-side flag, not at whether audio is arriving.**
>    - The check only re-arms when `isAudioCaptureActive()` is false.
>    - A configure command can be skipped while the session isn't ready even though `send()` returns true (`mic-control.ts:155`, `GlassesSessionCore.kt:890`).
>    - A temple can reboot on its own, or `apply()` can refuse to arm when the developer setting is turned off mid-session.
>    - Each of these leaves forwarding "active" with no frames arriving.
>    - **Fix:** you already track `templeFrameAtMs`. Re-send the config when an enabled temple has gone quiet, and show that in the status instead of "Listening (4-mic array)".
> 8. **A failed start is permanent.** `start()` returns before installing any timers (`:280`). A failed mode switch mid-session (`:376`) leaves `forwardingProxy` null, so the health check exits at `:457` every time. **Fix:** stay in a "waiting for glasses" state and let the timer retry.
> 9. **The caption feed can flip between left and right on every frame.** The choice at `:744` doesn't take the current side into account, and `strongerChanged` is just a `!==` check. Near the 1.5× threshold, each flip splices about 25 ms from the other temple (which runs on its own clock) into captions and the recording, either repeating or skipping audio.
>
> ## Lower
> - **Unguarded speaker call (`:1034`):** the speaker rename for a pending introduction runs before the line is added. A database error drops the line and leaves the introduction pending, so the next 2 minutes of lines from unnamed speakers fail the same way. Add the line first, then do the extra steps each with its own error handling.
> - **Speaker list grows forever:** every unknown voice adds a permanent "Speaker N" with a voiceprint. Retention never prunes them. `assign()` reloads and decodes every profile on every utterance, on the main thread.
> - **Translation:** there's no timeout, so lines can wait indefinitely while a model download is stuck offline. The detected language is only saved after translation finishes (`:1132`). The "hysteresis" comment at `:1120` describes logic that doesn't exist: `conversationLang` is written but never read.
> - **Frame fields ignored:** the sequence number (but `TempleAudioPipeline` assumes consecutive frames), the truncated flag, and `sampleRateHz`. The UI offers 8–48 kHz while the code assumes 16 kHz.
> - **Duplicated model path:** `moonshineModelDir()` (`:1337`) copies `asr-model.ts`'s directory name. A model version bump there would quietly drop captions to "no ASR model".
> - **Silent recording gaps:** `WavRecorder.append` swallows each write error, so a full disk means log spam and silent gaps in the recording.
>
> The startup order with cleanup on failure, the 90 s lease that recovers on its own from a lost STOP, and keeping the last good speaker cache all hold up well.
>
> I worked from the code only; nothing was tested on hardware. #6 relies on the protocol comment saying the angle is 0 when direction detection is off.
>
> I'd fix #1, #2, #5 and #6 first, since those are the ones you'd hit in normal use. Want me to start on them?

<hr>

User: Thanks. Let's fix all of the high and medium issues you found now.

Opus 5.5:
> All nine high and medium issues are fixed. TS typecheck, lint, 813 TS tests, the Kotlin host tests (4 new), the iOS Kotlin compile and `build.sh` all pass, and I confirmed the new code is in the APK. Nothing has been tested on the glasses yet, and nothing is committed.
>
> **High**
> 1. **Mic taken by other apps:**
>    - The shared audio tap now tracks who's using it, so an EvenHub app backgrounding no longer cuts off Microphones.
>    - Transcribe now marks the mic as in use, like the assistant does, so Microphones and EvenHub apps step aside while it runs.
>    - A check every 3 s now covers both modes. It restarts a dead stock tap and switches back to stock mode if the developer setting is turned off mid-session.
> 2. **One bad utterance ending captions:** a decode error now skips that utterance instead of stopping the caption engine (`CaptionEngineCore.kt`). If the engine dies anyway, it's restarted on the same conversation up to 3 times. After that, the Captions toggle shows off and the view says why; turning it on retries.
> 3. **Recordings orphaned by a crash:** the recording is linked to its conversation as soon as it starts. At boot, the app closes conversations a crash left open, repairs their WAV headers (`FaceclawWavRecorder.repairHeader`), transcodes them, and deletes files no conversation points to. That cleanup matches by file name, so moving the app to other storage can't delete live recordings.
> 4. **Transcoder deleting both copies:** a failure while recording the new file no longer deletes both files. The app now deletes the WAV itself, only after the conversation points at the M4A. The transcoder, re-diarize and model-download callbacks are now held by the app until they fire.
>
> **Medium**
> 5. **Last utterance lost:** stopping captions no longer blocks the UI. The engine finishes the audio still queued, then sends a new `onStopped` callback, and the conversation stays open until that arrives (20 s fallback). A late line from a previous session won't show up in a newly reopened one.
> 6. **Beam filter muting captions:** the direction of arrival only filters audio while it's under 1 s old, and it resets when "Direction detection" changes.
> 7. **4-mic health check:** a temple that sends no audio for 5 s gets its config re-sent, to that temple only. Mic-control sends now report when they weren't actually queued. The status line says when no audio is arriving, but not after routine restarts like the voice dialog closing.
> 8. **Start failures being permanent:** the session now waits ("Waiting for the glasses microphone...") and the 3 s check retries. Captions start only once the mic is up, so waiting doesn't create empty conversations.
> 9. **Left/right flapping:** the caption feed only switches temples when the other one is clearly louder for 500 ms.
>
> Behaviour changes you'll notice:
> - The watch shows "Listening…" during Transcribe.
> - EvenHub mic apps get no audio while Transcribe runs.
> - In 4-mic mode, an EvenHub mic app still takes over the single native audio listener. Fixing that needs a change on the Kotlin side, so I left it.
>
> On hardware, I'd test:
> - The assistant, Transcribe and an EvenHub mic app taking the mic while Microphones is open, then giving it back.
> - Reconnecting in both stock and 4-mic modes.
> - Killing the app mid-recording, then checking the conversation is recovered at the next start.
>
> I saved these notes to memory as `microphones-reliability.md`.
</details>
This commit is contained in:
James Babcock committed 2026-10-14 02:17:05 -07:00
1 parent 42c2d5cd55
commit 60c87521b0
23 files changed
+1101 -307

No files matched your search

@@ -13,7 +13,8 @@ import java.io.IOException
/** /**
* Transcodes a 16-bit PCM WAV conversation recording into AAC-LC in an M4A * Transcodes a 16-bit PCM WAV conversation recording into AAC-LC in an M4A
* container, cutting speech recordings to roughly a tenth of their WAV size. * container, cutting speech recordings to roughly a tenth of their WAV size.
* Runs on its own thread; the source WAV is deleted on success when requested. * Runs on its own thread; when requested, the source WAV is deleted once the
* listener's onDone has returned normally.
*/ */
class FaceclawAudioTranscoder( class FaceclawAudioTranscoder(
private val wavPath: String?, private val wavPath: String?,
@@ -66,19 +67,28 @@ class FaceclawAudioTranscoder(
private fun run() { private fun run() {
try { try {
transcode() transcode()
val size = File(outputPath).length()
if (deleteSourceOnSuccess) {
File(wavPath).delete()
}
if (listener != null) {
listener.onDone(outputPath, size)
}
} catch (t: Throwable) { } catch (t: Throwable) {
Log.w(TAG, "transcode failed for $wavPath", t) Log.w(TAG, "transcode failed for $wavPath", t)
File(outputPath).delete() File(outputPath).delete()
if (listener != null) { try {
listener.onError(if (t.message == null) t.toString() else t.message) listener?.onError(if (t.message == null) t.toString() else t.message)
} catch (listenerError: Throwable) {
Log.w(TAG, "transcode error listener failed", listenerError)
} }
return
}
val size = File(outputPath).length()
try {
listener?.onDone(outputPath, size)
} catch (t: Throwable) {
// The caller never took up the new file, so whatever it recorded still names the
// source: keep that and drop the output, rather than lose both.
Log.w(TAG, "transcode listener failed for $wavPath", t)
File(outputPath).delete()
return
}
if (deleteSourceOnSuccess) {
File(wavPath).delete()
} }
} }
@@ -207,7 +207,7 @@ class FaceclawBleCommunicator(context: Context, rightAddress: String?, leftAddre
fun removeMicStatusListener(listener: FaceclawMicStatusListener?) = core.removeMicStatusListener(listener) fun removeMicStatusListener(listener: FaceclawMicStatusListener?) = core.removeMicStatusListener(listener)
fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean) = fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean =
core.sendFaceclawMicControl(record, label, rightTemple, leftTemple) core.sendFaceclawMicControl(record, label, rightTemple, leftTemple)
fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean = core.startG2AudioForwarding(listener) fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean = core.startG2AudioForwarding(listener)
@@ -68,6 +68,11 @@ class FaceclawCaptionEngine {
session.stop() session.stop()
} }
/** Stop without blocking; the listener's onStopped follows the trailing utterance. */
fun requestStop() {
session.requestStop()
}
/** Push decoded 16 kHz mono S16LE PCM (any chunking). */ /** Push decoded 16 kHz mono S16LE PCM (any chunking). */
fun acceptPcm(pcm16le: ByteArray?) { fun acceptPcm(pcm16le: ByteArray?) {
session.acceptPcm(pcm16le) session.acceptPcm(pcm16le)
@@ -718,6 +718,72 @@ class FaceclawConversationStore private constructor(context: Context) : SQLiteOp
} }
} }
// ---- crash recovery ----
/**
* Close out sessions a dead process left open (ended_at NULL): the end becomes the last
* stored segment's end (or the start), and the tone average comes from the stored segments.
* excludeIdsJson is a JSON array of session ids that are legitimately still open. Returns
* the sessions it closed as a JSON array of {id, audioPath, audioCodec}.
*/
fun closeInterruptedSessions(excludeIdsJson: String?): String {
val exclude = HashSet<Long>()
if (excludeIdsJson != null && !excludeIdsJson.isEmpty()) {
val ids = JSONArray(excludeIdsJson)
for (i in 0 until ids.length()) {
exclude.add(ids.getLong(i))
}
}
val db = writableDatabase
val closed = JSONArray()
val closedIds = ArrayList<Long>()
db.rawQuery("SELECT id, audio_path, audio_codec FROM sessions WHERE ended_at IS NULL", null).use { c ->
while (c.moveToNext()) {
val id = c.getLong(0)
if (id in exclude) continue
closedIds.add(id)
val row = JSONObject()
row.put("id", id)
row.put("audioPath", if (c.isNull(1)) JSONObject.NULL else c.getString(1))
row.put("audioCodec", if (c.isNull(2)) JSONObject.NULL else c.getString(2))
closed.put(row)
}
}
for (id in closedIds) {
db.execSQL(
"UPDATE sessions SET"
+ " ended_at = COALESCE((SELECT MAX(ended_at) FROM segments WHERE session_id = ?), started_at),"
+ " avg_sentiment = COALESCE((SELECT AVG(sentiment) FROM segments WHERE session_id = ?), 0)"
+ " WHERE id = ?",
arrayOf<Any?>(id, id, id))
}
return closed.toString()
}
/**
* Delete recordings in dirPath that no session's audio_path names: files a session row never
* learned about, or partial transcoder output. Matched by file name, so a storage move that
* changed the directory's absolute path can't make live recordings look orphaned. Only call
* while no recording or transcode is in flight. Returns how many files were deleted.
*/
fun deleteUnreferencedRecordings(dirPath: String?): Int {
if (dirPath == null) return 0
val files = File(dirPath).listFiles() ?: return 0
val referenced = HashSet<String>()
readableDatabase.rawQuery("SELECT audio_path FROM sessions WHERE audio_path IS NOT NULL", null).use { c ->
while (c.moveToNext()) {
referenced.add(File(c.getString(0)).name)
}
}
var deleted = 0
for (file in files) {
if (file.isFile && file.name !in referenced && file.delete()) {
deleted++
}
}
return deleted
}
fun deleteSession(sessionId: Long) { fun deleteSession(sessionId: Long) {
val db = writableDatabase val db = writableDatabase
val path = getSessionAudioPath(sessionId) val path = getSessionAudioPath(sessionId)
+2 -2
View File
@@ -39,7 +39,7 @@ export class VoiceDraft {
this.phase = "listening"; this.phase = "listening";
this.status = "Listening... release to send"; this.status = "Listening... release to send";
this.ownsVoiceActivity = true; this.ownsVoiceActivity = true;
voiceActivity.setActive(true); voiceActivity.setActive(true, "voice-draft");
this.unsubscribe.push(voiceControlBridge.onTranscript((event) => { this.unsubscribe.push(voiceControlBridge.onTranscript((event) => {
if (!this.active || generation !== this.generation) return; if (!this.active || generation !== this.generation) return;
if (event.isFinal) { if (event.isFinal) {
@@ -129,7 +129,7 @@ export class VoiceDraft {
} }
if (this.ownsVoiceActivity) { if (this.ownsVoiceActivity) {
this.ownsVoiceActivity = false; this.ownsVoiceActivity = false;
voiceActivity.setActive(false); voiceActivity.setActive(false, "voice-draft");
} }
this.changed(); this.changed();
} }
+5 -2
View File
@@ -17,6 +17,9 @@ import { voiceActivity } from "../../ui/shell/voice-activity";
declare const com: any; declare const com: any;
// The router's claim on the shared raw-PCM tap (the Microphones app holds its own).
const RAW_CAPTURE_OWNER = "evenhub-mic-router";
export type EvenHubMicClient = { export type EvenHubMicClient = {
readonly windowId: string; readonly windowId: string;
isForeground(): boolean; isForeground(): boolean;
@@ -83,9 +86,9 @@ class EvenHubMicRouter {
// eligibility change so it self-heals after an STT preemption. Returns // eligibility change so it self-heals after an STT preemption. Returns
// false if STT currently owns the mic; a later trigger (assistant modal // false if STT currently owns the mic; a later trigger (assistant modal
// close, foreground/screen change) re-tries. // close, foreground/screen change) re-tries.
voiceControlBridge.startRawCapture({ communicator: activeCommunicator() }); voiceControlBridge.startRawCapture({ communicator: activeCommunicator(), owner: RAW_CAPTURE_OWNER });
} else { } else {
voiceControlBridge.stopRawCapture(); voiceControlBridge.stopRawCapture(RAW_CAPTURE_OWNER);
} }
} }
} }
+1 -1
View File
@@ -39,7 +39,7 @@ export class CaptionsLayer implements Layer {
image.drawText(font, 12, 4, "Captions", 220); image.drawText(font, 12, 4, "Captions", 220);
if (!state.captionsActive) { if (!state.captionsActive) {
image.drawText(font, 12, 26, "Captions are off (enable in the Microphones menu).", 130); image.drawText(font, 12, 26, state.captionsNote, 130);
} }
const rows = this.buildRows(font, width - 36, state.captionLines); const rows = this.buildRows(font, width - 36, state.captionLines);
+3 -1
View File
@@ -17,7 +17,9 @@ const microphonesApp: AppDefinition = {
title: "Microphones", title: "Microphones",
icon: "mic", icon: "mic",
boot: () => { boot: () => {
// Prune expired captions/recordings per the retention settings. // Close out conversations and recordings a dead process left open, then
// prune expired captions/recordings per the retention settings.
micSession.recoverInterruptedConversations();
void micSession.applyRetentionSweep(); void micSession.applyRetentionSweep();
}, },
launch: async (ctx) => { launch: async (ctx) => {
+13 -2
View File
@@ -125,6 +125,17 @@ class MicArrayController {
return true; return true;
} }
/**
* Re-send CONFIGURE to one temple whose stream went quiet (a lost write, or
* a temple that restarted on its own), leaving the other temple's stream
* untouched.
*/
reconfigureTemple(config: MicConfig, side: MicSide): boolean {
if (!templeActive(config, side)) return false;
if (config.armHardware && !micControlSupported()) return false;
return this.send("configure", config, `configure-${side}`, side === "right", side === "left");
}
query(): boolean { query(): boolean {
return this.send("query", undefined, "query"); return this.send("query", undefined, "query");
} }
@@ -152,8 +163,8 @@ class MicArrayController {
try { try {
this.ensureStatusProxy(communicator); this.ensureStatusProxy(communicator);
const record = encodeMicControl(op, config); const record = encodeMicControl(op, config);
communicator.sendFaceclawMicControl(toJavaBytes(record), label, rightTemple, leftTemple); // False when the session wasn't ready and nothing was queued.
return true; return communicator.sendFaceclawMicControl(toJavaBytes(record), label, rightTemple, leftTemple) !== false;
} catch (error) { } catch (error) {
console.warn(`mic control ${label} failed: ${error}`); console.warn(`mic control ${label} failed: ${error}`);
return false; return false;
+8
View File
@@ -75,6 +75,9 @@ export type MicModelState = {
}; };
const downloaders = new Map<MicModelId, any>(); const downloaders = new Map<MicModelId, any>();
// Each download's listener is referenced only from Java while it runs; hold
// it here so it isn't GC'd before its callbacks.
const downloadListeners = new Map<MicModelId, any>();
const progress = new Map<MicModelId, number>(); const progress = new Map<MicModelId, number>();
const stateListeners = new Set<(id: MicModelId, state: MicModelState) => void>(); const stateListeners = new Set<(id: MicModelId, state: MicModelState) => void>();
@@ -152,6 +155,7 @@ function downloadNextFile(id: MicModelId): void {
const nextFile = definition.files.find((file) => !isFilePresent(id, file)); const nextFile = definition.files.find((file) => !isFilePresent(id, file));
if (!nextFile) { if (!nextFile) {
downloaders.delete(id); downloaders.delete(id);
downloadListeners.delete(id);
notifyStateChanged(id); notifyStateChanged(id);
return; return;
} }
@@ -169,9 +173,11 @@ function downloadNextFile(id: MicModelId): void {
onError: (message: string) => { onError: (message: string) => {
console.error(`Mic model download failed (${id}/${nextFile.name}): ${message}`); console.error(`Mic model download failed (${id}/${nextFile.name}): ${message}`);
downloaders.delete(id); downloaders.delete(id);
downloadListeners.delete(id);
notifyStateChanged(id); notifyStateChanged(id);
}, },
}); });
downloadListeners.set(id, listener);
try { try {
const downloader = new com.faceclaw.app.FaceclawModelDownloader( const downloader = new com.faceclaw.app.FaceclawModelDownloader(
`${definition.baseUrl}${nextFile.remoteName}`, `${definition.baseUrl}${nextFile.remoteName}`,
@@ -186,6 +192,7 @@ function downloadNextFile(id: MicModelId): void {
// Without this, a dead entry would pin the model at "downloading". // Without this, a dead entry would pin the model at "downloading".
console.error(`Mic model download failed to start (${id}/${nextFile.name}): ${String(error)}`); console.error(`Mic model download failed to start (${id}/${nextFile.name}): ${String(error)}`);
downloaders.delete(id); downloaders.delete(id);
downloadListeners.delete(id);
notifyStateChanged(id); notifyStateChanged(id);
} }
} }
@@ -195,6 +202,7 @@ export function cancelMicModelDownload(id: MicModelId): void {
if (!downloader) return; if (!downloader) return;
downloader.cancel(); downloader.cancel();
downloaders.delete(id); downloaders.delete(id);
downloadListeners.delete(id);
notifyStateChanged(id); notifyStateChanged(id);
} }
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -453,8 +453,9 @@ export function createMicrophonesAppWindow(options: InProcessAppOptions): InProc
}), }),
configToggleItem( configToggleItem(
"Captions", "Captions",
() => captionsEnabledSetting.get(), // Shows off after repeated engine failures, so turning it on retries.
() => micSession.setCaptionsEnabled(!captionsEnabledSetting.get()), () => micSession.captionsEnabled(),
() => micSession.setCaptionsEnabled(!micSession.captionsEnabled()),
undefined, undefined,
captionsEnabledSetting.description, captionsEnabledSetting.description,
), ),
+24 -10
View File
@@ -35,6 +35,10 @@ function store(): any {
return com.faceclaw.app.FaceclawConversationStore.getInstance(context); return com.faceclaw.app.FaceclawConversationStore.getInstance(context);
} }
// Each run's listener is referenced only from Java while the diarizer works;
// hold it here so it isn't GC'd before its callbacks.
const activeListeners = new Set<any>();
export function rediarizeSession( export function rediarizeSession(
sessionId: number, sessionId: number,
options: { numSpeakers?: number; threshold?: number }, options: { numSpeakers?: number; threshold?: number },
@@ -54,6 +58,7 @@ export function rediarizeSession(
const listener = new com.faceclaw.app.FaceclawDiarizer.Listener({ const listener = new com.faceclaw.app.FaceclawDiarizer.Listener({
onProgress: (progress: number) => callbacks.onProgress(Number(progress)), onProgress: (progress: number) => callbacks.onProgress(Number(progress)),
onDone: (turnsJson: string) => { onDone: (turnsJson: string) => {
activeListeners.delete(listener);
try { try {
const turns = JSON.parse(String(turnsJson)) as Turn[]; const turns = JSON.parse(String(turnsJson)) as Turn[];
callbacks.onDone(applyTurns(sessionId, turns)); callbacks.onDone(applyTurns(sessionId, turns));
@@ -61,17 +66,26 @@ export function rediarizeSession(
callbacks.onError(`Applying diarization failed: ${String(error)}`); callbacks.onError(`Applying diarization failed: ${String(error)}`);
} }
}, },
onError: (message: string) => callbacks.onError(String(message)), onError: (message: string) => {
activeListeners.delete(listener);
callbacks.onError(String(message));
},
}); });
const diarizer = new com.faceclaw.app.FaceclawDiarizer( activeListeners.add(listener);
segmentationModel, try {
embeddingModel, const diarizer = new com.faceclaw.app.FaceclawDiarizer(
audioPath, segmentationModel,
options.numSpeakers ?? -1, embeddingModel,
options.threshold ?? 0.5, audioPath,
listener, options.numSpeakers ?? -1,
); options.threshold ?? 0.5,
diarizer.start(); listener,
);
diarizer.start();
} catch (error) {
activeListeners.delete(listener);
callbacks.onError(`Re-diarization failed to start: ${String(error)}`);
}
} }
type StoredSegment = { type StoredSegment = {
+8
View File
@@ -52,6 +52,7 @@ import { isPreviewOnlyMode } from "../phone-ui/onboarding-state";
import { claimGlassesOnboardingAutoLaunch, ONBOARDING_APP_ID } from "../apps/onboarding/onboarding-progress"; import { claimGlassesOnboardingAutoLaunch, ONBOARDING_APP_ID } from "../apps/onboarding/onboarding-progress";
import { beginRenderPass, endRenderPass } from "../util/render-freshness"; import { beginRenderPass, endRenderPass } from "../util/render-freshness";
import { voiceControlBridge } from "../native/voice-control"; import { voiceControlBridge } from "../native/voice-control";
import { voiceActivity } from "../ui/shell/voice-activity";
import { G2_LENS_HEIGHT, G2_LENS_WIDTH, GrayImage } from "../graphics/image"; import { G2_LENS_HEIGHT, G2_LENS_WIDTH, GrayImage } from "../graphics/image";
import { flattenPlanesWithDraws, planesFingerprint, type Plane } from "../graphics/plane"; import { flattenPlanesWithDraws, planesFingerprint, type Plane } from "../graphics/plane";
import { prepareFrameDraws } from "../graphics/glyph-wire"; import { prepareFrameDraws } from "../graphics/glyph-wire";
@@ -163,6 +164,8 @@ const SHELL_REFRESH_INTERVAL_MS = 60_000;
// schedules a refresh), so this just covers anything that slips through. // schedules a refresh), so this just covers anything that slips through.
const PREVIEW_INTERVAL_MS = 1_000; const PREVIEW_INTERVAL_MS = 1_000;
const SCREEN_TIMEOUT_CHECK_MS = 1_000; const SCREEN_TIMEOUT_CHECK_MS = 1_000;
// voiceActivity holder for Transcribe's continuous capture.
const CONTINUOUS_CAPTURE_HOLDER = "continuous-capture";
// The lock screen is a short notice, not something to read at leisure; it // The lock screen is a short notice, not something to read at leisure; it
// ignores the screen-timeout setting (including "never"). // ignores the screen-timeout setting (including "never").
const LOCK_SCREEN_TIMEOUT_MS = 15_000; const LOCK_SCREEN_TIMEOUT_MS = 15_000;
@@ -2058,11 +2061,16 @@ class DashboardController {
} }
private startContinuousVoiceCapture(): void { private startContinuousVoiceCapture(): void {
// Continuous capture (Transcribe) owns the glasses mic for as long as it
// runs. Claim it before the capture starts so the Microphones app and
// EvenHub mic apps stand down first, as they do for the voice dialog.
voiceActivity.setActive(true, CONTINUOUS_CAPTURE_HOLDER);
void this.beginVoiceCapture("continuous"); void this.beginVoiceCapture("continuous");
} }
private stopContinuousVoiceCapture(): void { private stopContinuousVoiceCapture(): void {
voiceControlBridge.stopContinuousCapture(); voiceControlBridge.stopContinuousCapture();
voiceActivity.setActive(false, CONTINUOUS_CAPTURE_HOLDER);
} }
/** /**
+39 -13
View File
@@ -122,13 +122,17 @@ export class FaceclawVoiceControlBridge {
private captureEndpointing = false; private captureEndpointing = false;
// Non-null while a cloud provider owns the transcript; Java only decodes PCM. // Non-null while a cloud provider owns the transcript; Java only decodes PCM.
private cloudClient: CloudSttClient | null = null; private cloudClient: CloudSttClient | null = null;
// Raw-PCM tap (EvenHub mic apps): when active, the controller runs in the // Raw-PCM tap (EvenHub mic apps, the Microphones app): when active, the
// decode-only "cloud" mode and every decoded PCM frame is broadcast to these // controller runs in the decode-only "cloud" mode and every decoded PCM
// listeners. STT capture preempts it — a live assistant/transcribe session // frame is broadcast to these listeners. STT capture preempts it — a live
// owns the single G2 mic and the raw tap is stopped for its duration. // assistant/transcribe session owns the single G2 mic and the raw tap is
// stopped for its duration.
private readonly rawPcmListeners = new Set<RawPcmListener>(); private readonly rawPcmListeners = new Set<RawPcmListener>();
private readonly frameMetaListeners = new Set<FrameMetaListener>(); private readonly frameMetaListeners = new Set<FrameMetaListener>();
private rawActive = false; private rawActive = false;
// Who wants the raw tap. It runs while any owner does, so one owner letting
// go (an EvenHub app backgrounding) doesn't silence another.
private readonly rawOwners = new Set<string>();
// Set when the current capture session runs "my voice only" verification; // Set when the current capture session runs "my voice only" verification;
// a non-wearer verdict arrives just before the final transcript and makes // a non-wearer verdict arrives just before the final transcript and makes
// the bridge swallow it. // the bridge swallow it.
@@ -220,11 +224,33 @@ export class FaceclawVoiceControlBridge {
} }
/** /**
* Start the decode-only raw-PCM tap for an EvenHub mic app. Fails (returns * Start the decode-only raw-PCM tap on behalf of `owner`. Fails (returns
* false) if an STT capture already owns the mic; the caller re-tries when its * false) if an STT capture already owns the mic; the owner stays registered
* eligibility is re-evaluated. Idempotent while already running. * and re-tries later (the EvenHub router when eligibility is re-evaluated,
* the Microphones app from its capture check). Idempotent while already
* running.
*/ */
startRawCapture(options: { communicator: any }): boolean { startRawCapture(options: { communicator: any; owner: string }): boolean {
if (!global.isAndroid) return false;
this.rawOwners.add(options.owner);
return this.startRawTap(options.communicator);
}
/** `owner` no longer wants the raw tap; it stops once no owner does. */
stopRawCapture(owner: string): void {
this.rawOwners.delete(owner);
if (this.rawOwners.size === 0) this.endRawTap();
}
/**
* Whether the raw tap is running and its glasses mic enable still holds
* (or is still coming up). False after an STT preemption or a session drop.
*/
isRawCaptureLive(): boolean {
return this.rawActive && this.micIsLive();
}
private startRawTap(communicator: any): boolean {
if (!global.isAndroid) return false; if (!global.isAndroid) return false;
if (this.rawActive && this.micIsLive()) return true; if (this.rawActive && this.micIsLive()) return true;
// An STT holder owns the mic even while its stream waits on a new glasses // An STT holder owns the mic even while its stream waits on a new glasses
@@ -234,7 +260,7 @@ export class FaceclawVoiceControlBridge {
if (this.started || this.rawActive) this.teardownCapture(); if (this.started || this.rawActive) this.teardownCapture();
this.suspendedRaw = false; this.suspendedRaw = false;
this.ensureController(); this.ensureController();
this.controller?.setCommunicator(options.communicator); this.controller?.setCommunicator(communicator);
// The raw tap is strictly the G2 stream; clear any phone-mic flag a // The raw tap is strictly the G2 stream; clear any phone-mic flag a
// preview-mode capture may have left on the shared controller. // preview-mode capture may have left on the shared controller.
this.controller?.setUsePhoneMic(false); this.controller?.setUsePhoneMic(false);
@@ -254,7 +280,7 @@ export class FaceclawVoiceControlBridge {
return true; return true;
} }
stopRawCapture(): void { private endRawTap(): void {
this.suspendedRaw = false; this.suspendedRaw = false;
if (!this.rawActive) return; if (!this.rawActive) return;
this.rawActive = false; this.rawActive = false;
@@ -266,7 +292,7 @@ export class FaceclawVoiceControlBridge {
if (!global.isAndroid) return; if (!global.isAndroid) return;
// STT preempts the raw-PCM tap: a live transcription session owns the mic, // STT preempts the raw-PCM tap: a live transcription session owns the mic,
// and an EvenHub app gets no audio for its duration. // and an EvenHub app gets no audio for its duration.
if (this.rawActive) this.stopRawCapture(); if (this.rawActive) this.endRawTap();
// Whether the mic is already running is a question about the stream, not // Whether the mic is already running is a question about the stream, not
// about this set: a holder whose capture died with the glasses session // about this set: a holder whose capture died with the glasses session
// (or whose mic enable was refused) is still in it. Trusting the set here // (or whose mic enable was refused) is still in it. Trusting the set here
@@ -428,8 +454,8 @@ export class FaceclawVoiceControlBridge {
for (const holder of holders) { for (const holder of holders) {
this.acquireCapture(holder, { ...options, endpointing: this.captureEndpointing }); this.acquireCapture(holder, { ...options, endpointing: this.captureEndpointing });
} }
if (resumeRaw && !holders.length) { if (resumeRaw && !holders.length && this.rawOwners.size > 0) {
this.startRawCapture({ communicator: options.communicator }); this.startRawTap(options.communicator);
} }
} }
+17 -9
View File
@@ -1,8 +1,13 @@
/** /**
* A tiny global signal for whether a voice-input modal (the assistant dialog) * A tiny global signal for whether speech capture owns the glasses
* is currently open. The shell sets it; the EvenHub mic router reads it to * microphone: the assistant's voice dialog, a dictation draft, or
* decide whether an app may hold the glasses microphone — an app receives no * Transcribe's continuous capture. The EvenHub mic router and the
* audio while the assistant is active, since both share the one G2 mic. * Microphones app read it to yield the one G2 mic while it is active — an
* app receives no audio meanwhile, since they all share it.
*
* Each user holds it under its own name, so one finishing doesn't clear
* another's hold. Set it before starting the capture and clear it after
* releasing, so other mic users stand down first and resume after.
* *
* Kept as a dependency-free leaf so both the shell and the EvenHub layer can * Kept as a dependency-free leaf so both the shell and the EvenHub layer can
* import it without a cycle. * import it without a cycle.
@@ -10,16 +15,19 @@
type Listener = (active: boolean) => void; type Listener = (active: boolean) => void;
let active = false; const holders = new Set<string>();
const listeners = new Set<Listener>(); const listeners = new Set<Listener>();
export const voiceActivity = { export const voiceActivity = {
isActive(): boolean { isActive(): boolean {
return active; return holders.size > 0;
}, },
setActive(next: boolean): void { setActive(next: boolean, holder = "voice-input"): void {
if (active === next) return; const was = holders.size > 0;
active = next; if (next) holders.add(holder);
else holders.delete(holder);
const active = holders.size > 0;
if (active === was) return;
listeners.forEach((listener) => listener(active)); listeners.forEach((listener) => listener(active));
}, },
subscribe(listener: Listener): () => void { subscribe(listener: Listener): () => void {
@@ -13,6 +13,30 @@ constructor(
private val sampleRate: Int, private val sampleRate: Int,
private val channels: Int, private val channels: Int,
) { ) {
companion object {
/**
* Rewrite the header of a recording whose writer never finished (the process died
* mid-recording, leaving the empty header written at start) to cover the PCM actually on
* disk, trimming any partial sample frame. Returns the PCM byte count, or -1 when the
* file doesn't exist.
*/
@JvmStatic
@Throws(IOException::class)
fun repairHeader(path: String, sampleRate: Int, channels: Int): Long {
val target = File(path)
if (!target.exists()) return -1
val headerBytes = BinaryEncoding.wavHeader(0, sampleRate, channels).size.toLong()
val frameBytes = channels * 2L
val pcmBytes = maxOf(0L, target.length() - headerBytes) / frameBytes * frameBytes
RandomAccessFile(target, "rw").use {
it.setLength(headerBytes + pcmBytes)
it.seek(0)
it.write(BinaryEncoding.wavHeader(minOf(pcmBytes, Int.MAX_VALUE.toLong()).toInt(), sampleRate, channels))
}
return pcmBytes
}
}
private var file: RandomAccessFile? private var file: RandomAccessFile?
private var dataBytes = 0L private var dataBytes = 0L
@@ -312,6 +312,21 @@ class CaptionSession(
} }
} }
/**
* Stops without waiting. The worker still flushes the trailing utterance and releases the
* models, then reports onStopped, so a caller can close out after the last line arrives
* instead of blocking its thread on the decode.
*/
fun requestStop() {
lock.withLock {
if (!started) {
return
}
started = false
}
queue.wake()
}
/** Push decoded 16 kHz mono S16LE PCM (any chunking). */ /** Push decoded 16 kHz mono S16LE PCM (any chunking). */
fun acceptPcm(pcm16le: ByteArray?) { fun acceptPcm(pcm16le: ByteArray?) {
if (!started || pcm16le == null || pcm16le.size < 2) { if (!started || pcm16le == null || pcm16le.size < 2) {
@@ -322,6 +337,7 @@ class CaptionSession(
private fun runLoop(finished: Latch) { private fun runLoop(finished: Latch) {
workerThreadName = currentThreadName() workerThreadName = currentThreadName()
var failure: String? = null
try { try {
loadModels() loadModels()
segmenter.reset() segmenter.reset()
@@ -329,18 +345,35 @@ class CaptionSession(
val chunk = queue.take(250) { started } ?: continue val chunk = queue.take(250) { started } ?: continue
segmenter.processChunk(chunk) segmenter.processChunk(chunk)
} }
// Flush a trailing utterance so its text isn't lost on stop. // Finish the audio already queued when the stop landed, then flush the trailing
// utterance so its text isn't lost on stop.
while (true) {
val chunk = queue.take(0) { false } ?: break
segmenter.processChunk(chunk)
}
segmenter.flush() segmenter.flush()
} catch (t: Throwable) { } catch (t: Throwable) {
logError("caption engine failed", t) logError("caption engine failed", t)
emitStatus("Captions failed: " + t.message) failure = t.message ?: t.toString()
emitStatus("Captions failed: $failure")
} finally { } finally {
decoder?.release() try {
decoder?.release()
} catch (t: Throwable) {
logError("caption model release failed", t)
}
decoder = null decoder = null
lock.withLock { lock.withLock {
workerFinished = null // A worker that died on its own leaves the session stopped, so the owner can
workerThreadName = null // tell (and acceptPcm stops queueing for nobody).
if (workerFinished === finished) {
started = false
workerFinished = null
workerThreadName = null
}
} }
// Posted before the latch opens, so a blocking stop() returns after it.
emitStopped(failure)
finished.countDown() finished.countDown()
} }
} }
@@ -366,8 +399,20 @@ class CaptionSession(
override fun onUtterance(samples: ShortArray, length: Int, startMs: Long, endMs: Long, peakRms: Double) { override fun onUtterance(samples: ShortArray, length: Int, startMs: Long, endMs: Long, peakRms: Double) {
val currentDecoder = decoder ?: return val currentDecoder = decoder ?: return
val text = currentDecoder.recognize(samples, length) // One utterance the models choke on must not end captioning for the session: deliver
val embedding = currentDecoder.embed(samples, length) // what did decode and keep listening.
val text = try {
currentDecoder.recognize(samples, length)
} catch (t: Throwable) {
logError("caption recognition failed", t)
""
}
val embedding = try {
currentDecoder.embed(samples, length)
} catch (t: Throwable) {
logError("caption voice-print failed", t)
null
}
val currentListener = listener ?: return val currentListener = listener ?: return
val embeddingCopy = embedding?.copyOf() val embeddingCopy = embedding?.copyOf()
host.dispatcher.post { currentListener.onUtterance(text, embeddingCopy, startMs, endMs, peakRms) } host.dispatcher.post { currentListener.onUtterance(text, embeddingCopy, startMs, endMs, peakRms) }
@@ -377,4 +422,9 @@ class CaptionSession(
val currentListener = listener ?: return val currentListener = listener ?: return
host.dispatcher.post { currentListener.onStatus(status) } host.dispatcher.post { currentListener.onStatus(status) }
} }
private fun emitStopped(error: String?) {
val currentListener = listener ?: return
host.dispatcher.post { currentListener.onStopped(error) }
}
} }
@@ -19,4 +19,10 @@ interface FaceclawCaptionEngineListener {
fun onSpeechStart(startMs: Long): Unit fun onSpeechStart(startMs: Long): Unit
fun onStatus(status: String?): Unit fun onStatus(status: String?): Unit
/**
* The worker has exited, after delivering any trailing utterance. error is null after
* stop()/requestStop(), or the failure that ended captioning on its own.
*/
fun onStopped(error: String?): Unit
} }
@@ -885,16 +885,17 @@ class GlassesSessionCore(
* Queue a CFW mic_control record (['M','C',ver,op,...]) as a settings * Queue a CFW mic_control record (['M','C',ver,op,...]) as a settings
* field-103 write to both temples, or to a single one. Fire-and-forget: * field-103 write to both temples, or to a single one. Fire-and-forget:
* the firmware answers with a field-104 status notify per temple, which * the firmware answers with a field-104 status notify per temple, which
* arrives through addMicStatusListener. * arrives through addMicStatusListener. Returns false when nothing was
* queued (no session, or a malformed record).
*/ */
fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean) { fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean {
if (record == null || record.size < 4) { if (record == null || record.size < 4) {
return return false
} }
monitor.withLock { monitor.withLock {
if (!running || !sessionReady) { if (!running || !sessionReady) {
logLine("skip mic control ($label); session not ready") logLine("skip mic control ($label); session not ready")
return return false
} }
if (rightTemple) { if (rightTemple) {
pendingMessages.addLast(messageBuilder.faceclawMicControl(record, label, false)) pendingMessages.addLast(messageBuilder.faceclawMicControl(record, label, false))
@@ -905,6 +906,7 @@ class GlassesSessionCore(
logLine("queue mic control $label") logLine("queue mic control $label")
} }
interruptibleSleep.interrupt() interruptibleSleep.interrupt()
return true
} }
/** /**
@@ -181,7 +181,7 @@ class IosGlassesSession internal constructor(
fun removeMicStatusListener(listener: FaceclawMicStatusListener?) = core.removeMicStatusListener(listener) fun removeMicStatusListener(listener: FaceclawMicStatusListener?) = core.removeMicStatusListener(listener)
fun sendFaceclawMicControl(record: NSData?, label: String, rightTemple: Boolean, leftTemple: Boolean) = fun sendFaceclawMicControl(record: NSData?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean =
core.sendFaceclawMicControl(record?.byteArray(), label, rightTemple, leftTemple) core.sendFaceclawMicControl(record?.byteArray(), label, rightTemple, leftTemple)
fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean = fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean =
@@ -0,0 +1,24 @@
package com.faceclaw.app
import java.io.File
import kotlin.test.Test
import kotlin.test.assertEquals
class WavRecorderRepairTest {
@Test
fun repairsTheHeaderOfARecordingThatWasNeverFinished() {
val file = File.createTempFile("unfinished", ".wav")
try {
// What a process death mid-recording leaves: the empty header from the start, the PCM
// written since, and a torn half-sample at the end.
file.writeBytes(BinaryEncoding.wavHeader(0, 16000, 1) + ByteArray(3201) { 1 })
assertEquals(3200L, FaceclawWavRecorder.repairHeader(file.path, 16000, 1))
val bytes = file.readBytes()
assertEquals(44 + 3200, bytes.size)
assertEquals(BinaryEncoding.wavHeader(3200, 16000, 1).toList(), bytes.copyOf(44).toList())
assertEquals(-1L, FaceclawWavRecorder.repairHeader(file.path + ".missing", 16000, 1))
} finally {
file.delete()
}
}
}
@@ -90,6 +90,7 @@ class CaptionEngineCoreTest {
} }
override fun onSpeechStart(startMs: Long) { events.add("start:$startMs"); startLatch.countDown() } override fun onSpeechStart(startMs: Long) { events.add("start:$startMs"); startLatch.countDown() }
override fun onStatus(status: String?) { events.add("status:$status") } override fun onStatus(status: String?) { events.add("status:$status") }
override fun onStopped(error: String?) { events.add("stopped:$error") }
} }
val host = object : CaptionHost { val host = object : CaptionHost {
override val dispatcher = CallbackDispatcher { it() } override val dispatcher = CallbackDispatcher { it() }
@@ -123,5 +124,105 @@ class CaptionEngineCoreTest {
assertEquals("status:Captions listening...", events[1]) assertEquals("status:Captions listening...", events[1])
assertEquals("start:0", events[2]) assertEquals("start:0", events[2])
assertEquals("utterance:hi:2:0", events[3]) assertEquals("utterance:hi:2:0", events[3])
assertEquals("stopped:null", events.last())
}
/** Records listener events; `stopped` opens once the worker reports onStopped. */
private class RecordingListener : FaceclawCaptionEngineListener {
val events = mutableListOf<String>()
val stopped = Latch(1, testPlatform())
var utteranceLatch = Latch(1, testPlatform())
override fun onUtterance(text: String?, embedding: FloatArray?, startMs: Long, endMs: Long, peakRms: Double) {
events.add("utterance:$text")
utteranceLatch.countDown()
}
override fun onSpeechStart(startMs: Long) {}
override fun onStatus(status: String?) {}
override fun onStopped(error: String?) {
events.add("stopped:$error")
stopped.countDown()
}
// Read only after a latch: callbacks run on the worker (the test dispatcher is inline).
fun snapshot(): List<String> = events.toList()
}
private fun captionHost(transcriber: () -> OfflineTranscriber?) = object : CaptionHost {
override val dispatcher = CallbackDispatcher { it() }
override fun loadTranscriber(modelDir: String): OfflineTranscriber? = transcriber()
override fun loadEmbedder(modelPath: String): SpeakerEmbedder? = null
}
private fun speak(session: CaptionSession, loudChunks: Int = 10, quietChunks: Int = 8) {
val loud = AudioSegmentation.pcm16ToLittleEndian(chunk(3000), 0, 800)
val quiet = AudioSegmentation.pcm16ToLittleEndian(chunk(0), 0, 800)
val idle = AudioSegmentation.pcm16ToLittleEndian(chunk(10), 0, 800)
repeat(2) { session.acceptPcm(idle) }
repeat(loudChunks) { session.acceptPcm(loud) }
repeat(quietChunks) { session.acceptPcm(quiet) }
}
@Test
fun captionSessionSurvivesAnUtteranceTheModelRejects() {
var calls = 0
val transcriber = object : OfflineTranscriber {
override fun recognize(samples: FloatArray, count: Int): String {
calls++
if (calls == 1) throw IllegalStateException("bad input")
return "second"
}
override fun release() {}
}
val listener = RecordingListener()
val errors = mutableListOf<String>()
val session = CaptionSession(captionHost { transcriber }, testPlatform(), "caption-reject-test") { m, _ -> errors.add(m) }
session.setListener(listener)
session.setAsrModelDir("models")
session.setSilenceMs(300)
session.start()
speak(session)
assertTrue(listener.utteranceLatch.await(5000))
listener.utteranceLatch = Latch(1, testPlatform())
speak(session)
assertTrue(listener.utteranceLatch.await(5000))
session.stop()
// The rejected utterance still arrives (without text), and the worker kept going.
assertEquals(listOf("utterance:", "utterance:second", "stopped:null"), listener.snapshot())
assertEquals(listOf("caption recognition failed"), errors.toList())
}
@Test
fun requestStopDeliversTheTrailingUtteranceBeforeStopped() {
val transcriber = object : OfflineTranscriber {
override fun recognize(samples: FloatArray, count: Int): String = "last words"
override fun release() {}
}
val listener = RecordingListener()
val session = CaptionSession(captionHost { transcriber }, testPlatform(), "caption-flush-test") { m, e -> error("$m: $e") }
session.setListener(listener)
session.setAsrModelDir("models")
session.start()
// Speech with no closing silence: the utterance is still open when the stop lands.
speak(session, loudChunks = 10, quietChunks = 0)
session.requestStop()
assertTrue(listener.stopped.await(5000))
assertEquals(listOf("utterance:last words", "stopped:null"), listener.snapshot())
}
@Test
fun aModelLoadFailureReportsStoppedWithTheError() {
val listener = RecordingListener()
val session = CaptionSession(
captionHost { throw IllegalStateException("model corrupt") },
testPlatform(),
"caption-load-test",
) { _, _ -> }
session.setListener(listener)
session.setAsrModelDir("models")
session.start()
assertTrue(listener.stopped.await(5000))
assertEquals(listOf("stopped:model corrupt"), listener.snapshot())
// The dead worker left the session stopped, so a later stop is a no-op.
session.stop()
session.requestStop()
} }
} }