forked from Blendman974/faceclaw
<details> <summary>Prompts Used</summary> User: Review the Microphones app, focusing on error handling and reliability. Opus 5.5: > I've finished reading the Microphones code and its native pieces. The 2026-10-14 commit `b50898e` fixed a lot, and the main problem left is the same in most places: the app reports "Listening" or "captions on" but never checks that audio is actually arriving. When something fails partway, the session usually stops working with no visible sign. > > ## High: silent loss of function or data > > **1. In stock mode, other mic users can take the mic away for good.** > - `stopRawCapture()` tears down the single shared audio tap no matter who started it (`app/native/voice-control.ts:257`). > - The EvenHub mic router calls it whenever a mic-using EvenHub app goes to the background or closes (`app/apps/evenhub/mic-router.ts:88`). That kills Microphones' audio. > - Transcribe takes the mic without setting `voiceActivity` (`voice-control.ts:269`), so Microphones is never told to get it back. > - If capture starts while the session isn't ready, the native controller only records the request, so no audio ever flows. > - `checkExtendedCapture` (`mic-session.ts:456`) only covers 4-mic mode. Its comment says the stock path "is resumed by the voice-control bridge", but that only covers the session-dropped-while-active case. > - In 4-mic mode the problem runs the other way: Transcribe takes over the shared Kotlin listener (`GlassesSessionCore.kt:470`) while the glasses keep sending 4-mic frames, so Transcribe gets audio in the wrong format. > - **Fix:** track who holds the raw tap (like `captureHolders`), add a stock-mode branch to the 3 s check, and have Transcribe set the same "mic in use" signal as the assistant. > > **2. One bad utterance ends captions for the rest of the session.** > - The caption worker has no per-utterance error handling. Any exception from `recognize()` or `embed()` stops the worker for good (`CaptionEngineCore.kt:334`). > - `started` stays true, so TypeScript still thinks captions are running and recording carries on. The only sign is a "Captions failed:" status string. > - **Fix:** catch errors per utterance in `onUtterance` (`:367`). Report a fatal worker exit through its own callback so the session can restart the engine or mark captions inactive. > > **3. A crash or low-memory kill during recording leaves files that are never shown or deleted.** > - The WAV header is written with a data size of 0 (`FaceclawWavRecorder.kt:26`) and only corrected in `finish()`. > - The file is only linked to its conversation in `stopRecording` (`mic-session.ts:1200`), and the conversation row is never closed. > - Nothing cleans up `mic-recordings/`, so after a crash you get the same never-expired orphan files the last commit fixed for the captions-off case. > - **Fix:** link the file to its conversation when recording starts. In the boot-time cleanup: close conversations left open (`ended_at IS NULL`), repair the WAV header from the file length, and delete files no conversation points to. > > **4. When the transcoder's success callback fails, it deletes both copies of the recording.** > - `FaceclawAudioTranscoder.kt:66-83` deletes the WAV before calling `onDone`. If `onDone` throws, the catch block deletes the M4A too, and the database still points at the deleted WAV. > - Separately, the JS listener is referenced only from Java (`mic-session.ts:1208`). Under `markingMode: 'none'` that's the situation `mic-control.ts:74` warns about. > - I haven't confirmed the garbage-collection risk. `asr-model.ts` does the same thing and its downloads seem to work. But the delete order is wrong either way. > - `rediarize.ts` and `mic-models.ts` hold their listeners the same way. > > ## Medium > > 5. **The last utterance is lost when you close the app or turn captions off.** When the caption engine stops, it flushes the utterance in progress and posts it to the main thread. By the time it arrives, `stop()` has already run `finishSessionRow()` (`mic-session.ts:328`), so the line is never saved and is missing from the session insights. If decoding takes longer than the 2 s wait, the line can land in the *next* session with wrong times. Also, `stop()` can block the UI for up to 2 s. > 6. **The beam filter can mute 4-mic captions permanently.** The direction and signal strength only update on frames with a non-zero angle or strength (`:685`), and they never go stale. If you turn off "Direction detection", the last direction stays frozen. If it was outside the beam, `deliverExtendedAudio` drops all audio from then on (`:751-757`). Stock mode avoids this because it updates the strength on every frame. **Fix:** timestamp the fused direction and treat it as unknown after `TEMPLE_STALE_MS`. > 7. **The 4-mic health check looks at a phone-side flag, not at whether audio is arriving.** > - The check only re-arms when `isAudioCaptureActive()` is false. > - A configure command can be skipped while the session isn't ready even though `send()` returns true (`mic-control.ts:155`, `GlassesSessionCore.kt:890`). > - A temple can reboot on its own, or `apply()` can refuse to arm when the developer setting is turned off mid-session. > - Each of these leaves forwarding "active" with no frames arriving. > - **Fix:** you already track `templeFrameAtMs`. Re-send the config when an enabled temple has gone quiet, and show that in the status instead of "Listening (4-mic array)". > 8. **A failed start is permanent.** `start()` returns before installing any timers (`:280`). A failed mode switch mid-session (`:376`) leaves `forwardingProxy` null, so the health check exits at `:457` every time. **Fix:** stay in a "waiting for glasses" state and let the timer retry. > 9. **The caption feed can flip between left and right on every frame.** The choice at `:744` doesn't take the current side into account, and `strongerChanged` is just a `!==` check. Near the 1.5× threshold, each flip splices about 25 ms from the other temple (which runs on its own clock) into captions and the recording, either repeating or skipping audio. > > ## Lower > - **Unguarded speaker call (`:1034`):** the speaker rename for a pending introduction runs before the line is added. A database error drops the line and leaves the introduction pending, so the next 2 minutes of lines from unnamed speakers fail the same way. Add the line first, then do the extra steps each with its own error handling. > - **Speaker list grows forever:** every unknown voice adds a permanent "Speaker N" with a voiceprint. Retention never prunes them. `assign()` reloads and decodes every profile on every utterance, on the main thread. > - **Translation:** there's no timeout, so lines can wait indefinitely while a model download is stuck offline. The detected language is only saved after translation finishes (`:1132`). The "hysteresis" comment at `:1120` describes logic that doesn't exist: `conversationLang` is written but never read. > - **Frame fields ignored:** the sequence number (but `TempleAudioPipeline` assumes consecutive frames), the truncated flag, and `sampleRateHz`. The UI offers 8–48 kHz while the code assumes 16 kHz. > - **Duplicated model path:** `moonshineModelDir()` (`:1337`) copies `asr-model.ts`'s directory name. A model version bump there would quietly drop captions to "no ASR model". > - **Silent recording gaps:** `WavRecorder.append` swallows each write error, so a full disk means log spam and silent gaps in the recording. > > The startup order with cleanup on failure, the 90 s lease that recovers on its own from a lost STOP, and keeping the last good speaker cache all hold up well. > > I worked from the code only; nothing was tested on hardware. #6 relies on the protocol comment saying the angle is 0 when direction detection is off. > > I'd fix #1, #2, #5 and #6 first, since those are the ones you'd hit in normal use. Want me to start on them? <hr> User: Thanks. Let's fix all of the high and medium issues you found now. Opus 5.5: > All nine high and medium issues are fixed. TS typecheck, lint, 813 TS tests, the Kotlin host tests (4 new), the iOS Kotlin compile and `build.sh` all pass, and I confirmed the new code is in the APK. Nothing has been tested on the glasses yet, and nothing is committed. > > **High** > 1. **Mic taken by other apps:** > - The shared audio tap now tracks who's using it, so an EvenHub app backgrounding no longer cuts off Microphones. > - Transcribe now marks the mic as in use, like the assistant does, so Microphones and EvenHub apps step aside while it runs. > - A check every 3 s now covers both modes. It restarts a dead stock tap and switches back to stock mode if the developer setting is turned off mid-session. > 2. **One bad utterance ending captions:** a decode error now skips that utterance instead of stopping the caption engine (`CaptionEngineCore.kt`). If the engine dies anyway, it's restarted on the same conversation up to 3 times. After that, the Captions toggle shows off and the view says why; turning it on retries. > 3. **Recordings orphaned by a crash:** the recording is linked to its conversation as soon as it starts. At boot, the app closes conversations a crash left open, repairs their WAV headers (`FaceclawWavRecorder.repairHeader`), transcodes them, and deletes files no conversation points to. That cleanup matches by file name, so moving the app to other storage can't delete live recordings. > 4. **Transcoder deleting both copies:** a failure while recording the new file no longer deletes both files. The app now deletes the WAV itself, only after the conversation points at the M4A. The transcoder, re-diarize and model-download callbacks are now held by the app until they fire. > > **Medium** > 5. **Last utterance lost:** stopping captions no longer blocks the UI. The engine finishes the audio still queued, then sends a new `onStopped` callback, and the conversation stays open until that arrives (20 s fallback). A late line from a previous session won't show up in a newly reopened one. > 6. **Beam filter muting captions:** the direction of arrival only filters audio while it's under 1 s old, and it resets when "Direction detection" changes. > 7. **4-mic health check:** a temple that sends no audio for 5 s gets its config re-sent, to that temple only. Mic-control sends now report when they weren't actually queued. The status line says when no audio is arriving, but not after routine restarts like the voice dialog closing. > 8. **Start failures being permanent:** the session now waits ("Waiting for the glasses microphone...") and the 3 s check retries. Captions start only once the mic is up, so waiting doesn't create empty conversations. > 9. **Left/right flapping:** the caption feed only switches temples when the other one is clearly louder for 500 ms. > > Behaviour changes you'll notice: > - The watch shows "Listening…" during Transcribe. > - EvenHub mic apps get no audio while Transcribe runs. > - In 4-mic mode, an EvenHub mic app still takes over the single native audio listener. Fixing that needs a change on the Kotlin side, so I left it. > > On hardware, I'd test: > - The assistant, Transcribe and an EvenHub mic app taking the mic while Microphones is open, then giving it back. > - Reconnecting in both stock and 4-mic modes. > - Killing the app mid-recording, then checking the conversation is recovered at the next start. > > I saved these notes to memory as `microphones-reliability.md`. </details>
499 lines
17 KiB
TypeScript
499 lines
17 KiB
TypeScript
import { imageFromAsciiArt } from "../../graphics/image";
|
|
import { getDefaultSmallFont } from "../../graphics/ui-fonts";
|
|
import { appViewportSize } from "../../ui/shell/geometry";
|
|
import {
|
|
MenuLayer,
|
|
TextPageLayer,
|
|
drawRightValueMenuItem,
|
|
drawToggleMenuItem,
|
|
openModalMenu,
|
|
submenuItem,
|
|
type MenuItem,
|
|
} from "../../ui/menu";
|
|
import { enumSettingMenuItem, toggleSettingMenuItem } from "../../ui/dashboard-settings";
|
|
import {
|
|
createInProcessWindow,
|
|
YieldAtRootLayer,
|
|
type InProcessAppOptions,
|
|
type InProcessWindow,
|
|
} from "../../ui/shell/in-process-window";
|
|
import { shell } from "../../ui/shell/shell";
|
|
import { isAsrModelReady, startAsrModelDownload, asrModelState } from "../../native/asr-model";
|
|
import { CaptionsLayer } from "./captions-layer";
|
|
import { LevelsLayer } from "./levels-layer";
|
|
import { RadarLayer } from "./radar-layer";
|
|
import { loadMicConfig, micArrayController, micControlSupported } from "./mic-control";
|
|
import { isMicModelReady, micModelState, startMicModelDownload } from "./mic-models";
|
|
import { micChannelLabel, type MicChannelKey, type MicConfig } from "./mic-protocol";
|
|
import { micSession } from "./mic-session";
|
|
import { speakerRegistry, type SpeakerProfile } from "./speakers";
|
|
import { formatRelativeTime } from "../../util/date-util";
|
|
import {
|
|
ancEnabledSetting,
|
|
beamFilterSetting,
|
|
captionsEnabledSetting,
|
|
captionsRetentionSetting,
|
|
recordingsRetentionSetting,
|
|
saveCaptionsSetting,
|
|
saveRecordingsSetting,
|
|
translateEnabledSetting,
|
|
wearerCommandsOnlySetting,
|
|
} from "./mic-settings";
|
|
|
|
export const MICROPHONES_WINDOW_ID = "microphones";
|
|
export const MICROPHONES_SURFACE_ID = "window:microphones";
|
|
|
|
const TRAY_ICON_ID = "microphones";
|
|
|
|
// 14x15 dual-mic glyph for the top-bar tray.
|
|
const TRAY_ICON = imageFromAsciiArt(
|
|
[
|
|
" ## ## ",
|
|
" # # # # ",
|
|
" # # # # ",
|
|
" # # # # ",
|
|
" # # # # ",
|
|
"# ## # # ## #",
|
|
" # # # # ",
|
|
" ## ## ",
|
|
" # # ",
|
|
" ### ### ",
|
|
],
|
|
{ "#": 220 },
|
|
);
|
|
|
|
const MENU_LAYOUT = {
|
|
x: 8,
|
|
y: 8,
|
|
width: 292,
|
|
showBorder: false,
|
|
minHeight: 0,
|
|
maxHeight: appViewportSize("min").height - 16,
|
|
opaque: true,
|
|
};
|
|
|
|
/** Persist a config change; the running session applies it live. */
|
|
function updateMicConfig(mutate: (config: MicConfig) => MicConfig): void {
|
|
micSession.updateConfig(mutate);
|
|
}
|
|
|
|
function micChannelToggleItem(
|
|
key: MicChannelKey,
|
|
description: string,
|
|
disabled: () => boolean,
|
|
): MenuItem {
|
|
return configToggleItem(
|
|
`${micChannelLabel(key)} mic`,
|
|
() => loadMicConfig().hostMics[key],
|
|
() => micSession.setMicChannelEnabled(key, !loadMicConfig().hostMics[key]),
|
|
disabled,
|
|
description,
|
|
);
|
|
}
|
|
|
|
function configToggleItem(
|
|
label: string,
|
|
isOn: () => boolean,
|
|
toggle: () => void,
|
|
disabled?: () => boolean,
|
|
description?: string,
|
|
): MenuItem {
|
|
return {
|
|
label,
|
|
description,
|
|
disabled,
|
|
onSelect: () => toggle(),
|
|
render: ({ image, x, y, width, selected }) => {
|
|
drawToggleMenuItem(image, getDefaultSmallFont(), x, y, width, label, isOn(), selected);
|
|
},
|
|
};
|
|
}
|
|
|
|
function configPickerItem<T extends string | number>(
|
|
label: string,
|
|
values: readonly T[],
|
|
format: (value: T) => string,
|
|
current: () => T,
|
|
apply: (value: T) => void,
|
|
disabled?: () => boolean,
|
|
description?: string,
|
|
): MenuItem {
|
|
return {
|
|
label,
|
|
description,
|
|
disabled,
|
|
onSelect: (ctx) => {
|
|
const items = values.map(
|
|
(value): MenuItem => ({
|
|
label: format(value) + (current() === value ? " *" : ""),
|
|
onSelect: (innerCtx) => {
|
|
apply(value);
|
|
innerCtx.stack.pop();
|
|
},
|
|
}),
|
|
);
|
|
openModalMenu(ctx, label, items, Math.max(0, values.indexOf(current())));
|
|
},
|
|
render: ({ image, x, y, width }) => {
|
|
drawRightValueMenuItem(image, getDefaultSmallFont(), x, y, width, label, format(current()));
|
|
},
|
|
};
|
|
}
|
|
|
|
function micSetupMenu(): MenuLayer {
|
|
const unsupported = () => !micControlSupported();
|
|
return new MenuLayer(
|
|
"Microphone setup",
|
|
[
|
|
{
|
|
label: "Array status",
|
|
description:
|
|
"Per-temple mic-control status. Requires the Use microphone control developer setting.",
|
|
onSelect: (ctx) => {
|
|
micArrayController.query();
|
|
const { left, right } = micArrayController.getStatuses();
|
|
const describe = (side: string, status: typeof left) =>
|
|
status
|
|
? `${side}: ${status.active ? "active" : "idle"}, mics ${status.channelMask & 1 ? "front" : ""}${
|
|
status.channelMask & 2 ? "+rear" : ""
|
|
}, ${status.effectiveRateHz / 1000} kHz, ${status.hardwareArmed ? "armed" : "not armed"}, ${
|
|
status.framesEmitted
|
|
} frames`
|
|
: `${side}: no status yet`;
|
|
const consistency = micArrayController.arrayConsistent()
|
|
? "Array consistent (L/R matched)."
|
|
: "Array not yet consistent.";
|
|
ctx.stack.push(
|
|
new TextPageLayer(
|
|
"Array status",
|
|
micControlSupported()
|
|
? `${describe("Left", left)}\n${describe("Right", right)}\n${consistency}`
|
|
: "Mic-control is disabled. Turn on Settings > Developer > Use microphone control to control the four mics individually; while it is off the glasses stream one mixed channel with a firmware-computed direction of arrival.",
|
|
),
|
|
);
|
|
},
|
|
},
|
|
// Each temple streams its full front+rear pair; these four toggles are
|
|
// host-side selection over the split channels, so any combination of
|
|
// the four microphones can be used independently.
|
|
micChannelToggleItem("leftFront", "Enable the left temple's front microphone (near the hinge).", unsupported),
|
|
micChannelToggleItem("leftRear", "Enable the left temple's rear microphone (by the touchpad).", unsupported),
|
|
micChannelToggleItem("rightFront", "Enable the right temple's front microphone (near the hinge).", unsupported),
|
|
micChannelToggleItem("rightRear", "Enable the right temple's rear microphone (by the touchpad).", unsupported),
|
|
configPickerItem(
|
|
"Source",
|
|
["pdm", "codec"] as const,
|
|
(value) => (value === "pdm" ? "PDM mics" : "Codec DMIC"),
|
|
() => loadMicConfig().source,
|
|
(value) => updateMicConfig((config) => ({ ...config, source: value })),
|
|
unsupported,
|
|
"PDM streams one channel; the codec front end streams both mics of a temple.",
|
|
),
|
|
configPickerItem(
|
|
"Codec",
|
|
["raw", "lc3"] as const,
|
|
(value) => (value === "raw" ? "Raw PCM" : "LC3"),
|
|
() => loadMicConfig().codec,
|
|
(value) => updateMicConfig((config) => ({ ...config, codec: value })),
|
|
unsupported,
|
|
"Raw PCM preserves inter-mic phase for beamforming. (Current firmware always streams raw.)",
|
|
),
|
|
configPickerItem(
|
|
"Sample rate",
|
|
[8000, 16000, 24000, 32000, 48000] as const,
|
|
(value) => `${value / 1000} kHz`,
|
|
() => loadMicConfig().sampleRateHz,
|
|
(value) => updateMicConfig((config) => ({ ...config, sampleRateHz: value })),
|
|
unsupported,
|
|
"Requested capture rate per mic pair. Current firmware pins the effective rate at 16 kHz and echoes the request in the status.",
|
|
),
|
|
configPickerItem(
|
|
"LC3 bitrate",
|
|
[0, 16000, 32000, 48000, 64000] as const,
|
|
(value) => (value === 0 ? "Default" : `${value / 1000} kbps`),
|
|
() => loadMicConfig().lc3BitrateBps,
|
|
(value) => updateMicConfig((config) => ({ ...config, lc3BitrateBps: value })),
|
|
unsupported,
|
|
"Requested LC3 bitrate per mic pair (takes effect once the firmware's LC3 path lands).",
|
|
),
|
|
configToggleItem(
|
|
"Direction detection",
|
|
() => loadMicConfig().beamform,
|
|
() => updateMicConfig((config) => ({ ...config, beamform: !config.beamform })),
|
|
unsupported,
|
|
"Have each temple compute and stream its own sound direction (TDOA angle + signal strength).",
|
|
),
|
|
{
|
|
label: "4-mic capture (advanced)",
|
|
description:
|
|
"Arms the microphone hardware and multi-channel streaming on the glasses. Leave off until the firmware validation checklist has passed on this hardware.",
|
|
disabled: unsupported,
|
|
onSelect: (ctx) => {
|
|
const config = loadMicConfig();
|
|
if (config.armHardware) {
|
|
updateMicConfig((current) => ({ ...current, armHardware: false }));
|
|
micArrayController.stopStreaming();
|
|
return;
|
|
}
|
|
openModalMenu(ctx, "Arm mic hardware?", [
|
|
{
|
|
label: "Cancel",
|
|
onSelect: (innerCtx) => innerCtx.stack.pop(),
|
|
},
|
|
{
|
|
label: "Enable (validated hardware only)",
|
|
onSelect: (innerCtx) => {
|
|
updateMicConfig((current) => ({ ...current, armHardware: true }));
|
|
innerCtx.stack.pop();
|
|
},
|
|
},
|
|
]);
|
|
},
|
|
render: ({ image, x, y, width, selected }) => {
|
|
drawToggleMenuItem(
|
|
image,
|
|
getDefaultSmallFont(),
|
|
x,
|
|
y,
|
|
width,
|
|
"4-mic capture (advanced)",
|
|
loadMicConfig().armHardware,
|
|
selected,
|
|
);
|
|
},
|
|
},
|
|
],
|
|
MENU_LAYOUT,
|
|
);
|
|
}
|
|
|
|
function modelItem(
|
|
label: string,
|
|
ready: () => boolean,
|
|
stateText: () => string,
|
|
download: () => void,
|
|
): MenuItem {
|
|
return {
|
|
label,
|
|
onSelect: () => {
|
|
if (!ready()) download();
|
|
},
|
|
render: ({ image, x, y, width }) => {
|
|
drawRightValueMenuItem(image, getDefaultSmallFont(), x, y, width, label, stateText());
|
|
},
|
|
};
|
|
}
|
|
|
|
function voiceSpeakersMenu(): MenuLayer {
|
|
return new MenuLayer(
|
|
"Voice & speakers",
|
|
[
|
|
{
|
|
label: "Enroll my voice",
|
|
description:
|
|
"Speak a sentence after selecting; the next utterance becomes (or refreshes) your voice-print, used to label you in captions and to restrict commands to your voice.",
|
|
disabled: () => !micSession.getState().captionsActive,
|
|
onSelect: () => micSession.armWearerEnrollment(),
|
|
},
|
|
toggleSettingMenuItem(wearerCommandsOnlySetting),
|
|
modelItem(
|
|
"Speaker voice model",
|
|
() => isMicModelReady("speaker-embedding"),
|
|
() => {
|
|
const state = micModelState("speaker-embedding");
|
|
if (state.status === "ready") return "ready";
|
|
if (state.status === "downloading") {
|
|
return `${Math.round((state.bytesDownloaded / state.totalBytes) * 100)}%`;
|
|
}
|
|
return "download (28 MB)";
|
|
},
|
|
() => startMicModelDownload("speaker-embedding"),
|
|
),
|
|
modelItem(
|
|
// Live captions always use Moonshine (see the matching note in
|
|
// mic-session.ts); the second on-device model (Whisper) is a
|
|
// push-to-talk dictation option only, managed in Settings > Voice.
|
|
"Caption ASR model",
|
|
() => isAsrModelReady("moonshine"),
|
|
() => {
|
|
const state = asrModelState("moonshine");
|
|
if (state.status === "ready") return "ready";
|
|
if (state.status === "downloading") {
|
|
return `${Math.round((state.bytesDownloaded / state.totalBytes) * 100)}%`;
|
|
}
|
|
return "download (141 MB)";
|
|
},
|
|
() => startAsrModelDownload("moonshine"),
|
|
),
|
|
modelItem(
|
|
"Re-diarization model",
|
|
() => isMicModelReady("diarization-segmentation"),
|
|
() => {
|
|
const state = micModelState("diarization-segmentation");
|
|
if (state.status === "ready") return "ready";
|
|
if (state.status === "downloading") {
|
|
return `${Math.round((state.bytesDownloaded / state.totalBytes) * 100)}%`;
|
|
}
|
|
return "download (6 MB)";
|
|
},
|
|
() => startMicModelDownload("diarization-segmentation"),
|
|
),
|
|
],
|
|
MENU_LAYOUT,
|
|
);
|
|
}
|
|
|
|
/** The full contact card behind a People row (and behind the encounter popup). */
|
|
function speakerDetailBody(profile: SpeakerProfile): string {
|
|
const parts: string[] = [
|
|
`Last heard ${profile.lastHeardAt ? `${formatRelativeTime(profile.lastHeardAt)} ago` : "never"}.`,
|
|
];
|
|
if (profile.tag) parts.push(`Note: ${profile.tag}`);
|
|
if (profile.lastRecap) parts.push(`Last conversation: ${profile.lastRecap}`);
|
|
if (profile.actionItems.length) {
|
|
parts.push(`To do:\n${profile.actionItems.map((item) => `- ${item}`).join("\n")}`);
|
|
}
|
|
if (profile.facts.length) {
|
|
parts.push(`Facts:\n${profile.facts.map((fact) => `- ${fact}`).join("\n")}`);
|
|
}
|
|
if (parts.length === 1) {
|
|
parts.push(
|
|
"No conversation insights yet. A recap, action items, and remembered facts are " +
|
|
"generated when a captioned conversation with this person ends (requires the " +
|
|
"on-phone assistant model).",
|
|
);
|
|
}
|
|
return parts.join("\n\n");
|
|
}
|
|
|
|
/**
|
|
* People: the speaker-recognition contacts, most recently heard first — the
|
|
* review surface behind the encounter popup. Selecting a person shows when
|
|
* they were last heard, their note, the recap and action items of the last
|
|
* conversation, and the facts remembered about them.
|
|
*/
|
|
function peopleMenu(): MenuLayer {
|
|
const profiles = speakerRegistry.all().filter((profile) => !profile.isWearer);
|
|
const items: MenuItem[] = profiles.length
|
|
? profiles.map((profile) => ({
|
|
label: profile.name,
|
|
onSelect: (ctx) => {
|
|
ctx.stack.push(new TextPageLayer(profile.name, speakerDetailBody(profile)));
|
|
},
|
|
render: ({ image, x, y, width }) => {
|
|
drawRightValueMenuItem(
|
|
image,
|
|
getDefaultSmallFont(),
|
|
x,
|
|
y,
|
|
width,
|
|
profile.name,
|
|
profile.lastHeardAt ? `${formatRelativeTime(profile.lastHeardAt)} ago` : "",
|
|
);
|
|
},
|
|
}))
|
|
: [
|
|
{
|
|
label: "No people heard yet",
|
|
disabled: true,
|
|
onSelect: () => {},
|
|
},
|
|
];
|
|
return new MenuLayer("People", items, MENU_LAYOUT);
|
|
}
|
|
|
|
function storageMenu(): MenuLayer {
|
|
return new MenuLayer(
|
|
"Storage",
|
|
[
|
|
toggleSettingMenuItem(saveCaptionsSetting),
|
|
toggleSettingMenuItem(saveRecordingsSetting),
|
|
enumSettingMenuItem(captionsRetentionSetting),
|
|
enumSettingMenuItem(recordingsRetentionSetting),
|
|
{
|
|
label: "Review on phone",
|
|
description:
|
|
"Saved conversations, transcripts, speakers, and recordings are reviewed in the Faceclaw phone app: Conversations in the top-right menu.",
|
|
onSelect: (ctx) => {
|
|
ctx.stack.push(
|
|
new TextPageLayer(
|
|
"Review on phone",
|
|
"Open the Faceclaw app on your phone and choose Conversations from the top-right menu to search captions, filter by speaker, play linked recordings, rename/merge speakers, re-diarize sessions, and see the tone/sentiment graph of each conversation.",
|
|
),
|
|
);
|
|
},
|
|
},
|
|
],
|
|
MENU_LAYOUT,
|
|
);
|
|
}
|
|
|
|
export function createMicrophonesAppWindow(options: InProcessAppOptions): InProcessWindow {
|
|
const menu = new MenuLayer(
|
|
"Microphones",
|
|
[
|
|
submenuItem("Sonic Radar", (ctx) => {
|
|
const layer = new RadarLayer();
|
|
ctx.stack.push(layer);
|
|
layer.start(ctx.actions.requestRender);
|
|
}),
|
|
submenuItem("Levels", (ctx) => {
|
|
const layer = new LevelsLayer();
|
|
ctx.stack.push(layer);
|
|
layer.start(ctx.actions.requestRender);
|
|
}),
|
|
submenuItem("Captions view", (ctx) => {
|
|
const layer = new CaptionsLayer();
|
|
ctx.stack.push(layer);
|
|
layer.start(ctx.actions.requestRender);
|
|
}),
|
|
submenuItem("People", (ctx) => {
|
|
speakerRegistry.reload();
|
|
ctx.stack.push(peopleMenu());
|
|
}),
|
|
configToggleItem(
|
|
"Captions",
|
|
// Shows off after repeated engine failures, so turning it on retries.
|
|
() => micSession.captionsEnabled(),
|
|
() => micSession.setCaptionsEnabled(!micSession.captionsEnabled()),
|
|
undefined,
|
|
captionsEnabledSetting.description,
|
|
),
|
|
toggleSettingMenuItem(translateEnabledSetting),
|
|
toggleSettingMenuItem(beamFilterSetting),
|
|
toggleSettingMenuItem(ancEnabledSetting),
|
|
submenuItem("Microphone setup", (ctx) => {
|
|
ctx.stack.push(micSetupMenu());
|
|
}),
|
|
submenuItem("Voice & speakers", (ctx) => {
|
|
ctx.stack.push(voiceSpeakersMenu());
|
|
}),
|
|
submenuItem("Storage", (ctx) => {
|
|
ctx.stack.push(storageMenu());
|
|
}),
|
|
],
|
|
MENU_LAYOUT,
|
|
);
|
|
const app = createInProcessWindow({
|
|
appId: "microphones",
|
|
windowId: MICROPHONES_WINDOW_ID,
|
|
title: "Microphones",
|
|
iconLetter: "Mi",
|
|
icon: "mic",
|
|
closeable: true,
|
|
actions: options.actions,
|
|
baseLayer: new YieldAtRootLayer(menu),
|
|
submitFrame: options.submitFrame,
|
|
setSurfaceVisible: options.setSurfaceVisible,
|
|
removeSurface: options.removeSurface,
|
|
onClosed: () => {
|
|
micSession.stop();
|
|
shell.setTrayIcon(TRAY_ICON_ID, null);
|
|
options.onClosed();
|
|
},
|
|
});
|
|
micSession.start();
|
|
shell.setTrayIcon(TRAY_ICON_ID, TRAY_ICON);
|
|
return app;
|
|
}
|