Compare commits

..
Author SHA1 Message Date
MasterControl 4b4ab1b02c fix(ble): drop CONNECTION_PRIORITY_HIGH for G2 arms
HIGH negotiates a ~5s supervision timeout that kills the link on any
>5s BLE gap and can strand an arm in a half-open state where it stops
advertising (recoverable only by a 5-tap reset). Mirrors the ring fix
in connectRing, which removed HIGH for the same reason. BALANCED keeps
a longer supervision timeout; the firmware's own fast-profile request
still negotiates speed for transfers.
2026-11-16 06:51:41 +00:00
Blendman974 4d9c2bac76 Merge pull request 'merge: intégrer upstream c162c8e6 dans le fork all-features' (#21) from MasterControl/faceclaw:sync/upstream-all-features-20261114 into build/all-features
Reviewed-on: #21
2026-11-14 23:30:50 +00:00
MasterControl 5790e98d27 [verified] merge upstream c162c8e6 into all-features 2026-11-14 23:24:08 +00:00
Blendman974 10648aec4a Merge pull request 'fix(ble): fall back to direct connect when autoConnect can't reach an arm' (#19) from MasterControl/faceclaw:fix/ble-direct-connect-fallback into build/all-features
Reviewed-on: #19
2026-11-14 22:20:32 +00:00
MasterControl 52ead474e2 fix(ble): fall back to direct connect when autoConnect can't reach an arm
autoConnect only connects on a fresh advertisement, so an arm stuck in a
half-open state (stopped advertising after a dropped link) is never reached.
Retry with a direct connect: its aggressive scan catches a briefly-advertising
arm and the cached bond can recover the half-open link. Adds a
g2.connect-fallback-direct counter for diagnostics.
2026-11-14 21:46:58 +00:00
Blendman974 e4d3286b3d Merge pull request 'fix(ble): connect G2 arms with autoConnect=true' (#18) from MasterControl/faceclaw:fix/ble-autoconnect-arms into build/all-features
Reviewed-on: #18
2026-11-14 20:06:31 +00:00
MasterControl a9df3de59b fix(ble): connect G2 arms with autoConnect=true
Direct connect (autoConnect=false) is supervised less patiently by Android's
BLE stack; the ring already requires autoConnect=true for this reason, and the
glasses arms showed the same short-lived direct-connect drops. Switch both
arms' beginConnect and connect to autoConnect=true.
2026-11-14 20:05:15 +00:00
Blendman974 3eb9b09e3d Hide phone UI action in EvenAnchor menu 2026-11-14 20:55:08 +01:00
Blendman974 3dd307483d Merge pull request 'feat(notifications): suivre le mode Ne pas déranger du téléphone' (#17) from MasterControl/faceclaw:feat/follow-phone-dnd into build/all-features
Reviewed-on: #17
2026-11-11 18:03:07 +00:00
MasterControl 9ca3574ae7 [verified] feat: follow phone Do Not Disturb for notification wake
Tests / test (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
2026-11-11 17:43:35 +00:00
Blendman974 54ba6b4f10 Merge dedicated offline French Whisper model 2026-11-10 23:22:24 +01:00
Blendman974 b4b073412f feat(voice): add dedicated offline French Whisper model
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
2026-11-10 23:20:31 +01:00
Blendman974 ceeadea9bf Merge pull request 'fix: masquer le fond des notifications uniquement après un réveil écran' (#16) from MasterControl/faceclaw:fix/notification-wake-backdrop into build/all-features
Reviewed-on: #16
2026-11-10 20:42:42 +00:00
MasterControl a263a055d7 Merge wake-only notification backdrop correction
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
2026-11-10 20:33:25 +00:00
MasterControl 9ddc864b0d [verified] fix: hide notification background only on screen wake
Secret scan / gitleaks (pull_request) Failing after 1m9s
Tests / test (pull_request) Failing after 1m24s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
2026-11-10 20:33:16 +00:00
Blendman974 35b02aef52 Merge pull request 'build: intégrer upstream et la lisibilité des notifications/menus' (#15) from MasterControl/faceclaw:sync/upstream-main-all-features into build/all-features
Reviewed-on: #15
2026-11-10 17:36:23 +00:00
MasterControl c3d1b5f3d3 docs: record notification readability integration and validation
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-10 17:34:44 +00:00
MasterControl 309f05547b Merge notification focus and selected menu-label scrolling 2026-11-10 17:31:25 +00:00
MasterControl 22df4d76d9 feat: isolate notification popups and scroll selected menu labels
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-10 17:31:13 +00:00
MasterControl 6150a1ac57 Merge upstream main preserving custom feature integrations 2026-11-10 17:08:15 +00:00
Blendman974 16fe517b34 docs: record detailed heart-rate integration validation 2026-11-08 01:15:40 +01:00
Blendman974 d2f846e2ba Merge detailed heart-rate collection and export
# Conflicts:
#	docs/validation-health-export.md
2026-11-08 01:14:35 +01:00
Blendman974 a575db8e27 feat: retain and export latest heart-rate measurements
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-08 01:13:55 +01:00
Blendman974 86b78d45a5 Merge selectable translation output language 2026-11-08 00:56:37 +01:00
Blendman974 26e570f0e1 feat: choose translation output language independently
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
2026-11-08 00:56:36 +01:00
Blendman974 dbcdb1e3f6 Merge independent translation model setting 2026-11-08 00:49:11 +01:00
Blendman974 69bb92da7b feat: configure an independent OpenAI translation model
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
2026-11-08 00:48:54 +01:00
Blendman974 2a607b32d0 Merge configurable OpenAI transcription model 2026-11-08 00:37:07 +01:00
Blendman974 a80d6697b5 Allow choosing an OpenAI transcription model available to the project
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-08 00:35:51 +01:00
Blendman974 ca05030857 Merge health API repository documentation 2026-11-07 23:43:33 +01:00
Blendman974 8d57b7c1d0 Document the separate health API repository
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
2026-11-07 23:43:32 +01:00
Blendman974 d01fe585bc Keep feature proposal links in one branch table 2026-11-07 23:18:40 +01:00
Blendman974 693c442faf Record health export proposals and combined validation 2026-11-07 23:18:03 +01:00
Blendman974 f0bb727a80 Add opt-in authenticated HTTPS export of R1 health history
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-07 23:15:29 +01:00
Blendman974 3677a77a52 Merge optional R1 health export 2026-11-07 23:15:29 +01:00
Blendman974 1eee926cbe Merge notification display filtering 2026-11-07 21:13:02 +01:00
Blendman974 0f62573b1c Filter notification lists and status icons by source preferences
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-07 21:12:40 +01:00
Blendman974 f9d3edf58c Merge silent-exit display recovery
# Conflicts:
#	docs/validation-silent-mode.md
2026-11-06 22:28:09 +01:00
Blendman974 799bdc153f fix: restore the display before silent-exit confirmation
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 22:23:26 +01:00
Blendman974 1997258b25 Remove tilt-down hide and experimental angle measurement
# Conflicts:
#	app/g2/dashboard-controller.ts
#	app/g2/ios-preview-controller.ts
#	docs/validation-head-up.md
2026-11-06 22:21:13 +01:00
Blendman974 fe87e75fc2 Revert "feat: add shared tilt-down hide and current-angle measurement"
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
This reverts commit f029e3ce91.
2026-11-06 22:20:24 +01:00
Blendman974 fe0cf5b777 Merge shared tilt-down hide and angle measurement
# Conflicts:
#	app/g2/dashboard-controller.ts
#	app/g2/ios-preview-controller.ts
2026-11-06 22:10:41 +01:00
Blendman974 f029e3ce91 feat: add shared tilt-down hide and current-angle measurement
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 22:01:21 +01:00
Blendman974 3aeed72f5d Merge silent-mode feedback
# Conflicts:
#	tests/kotlin/src/commonTest/kotlin/com/faceclaw/app/GlassesSessionCoreTest.kt
2026-11-06 21:48:38 +01:00
Blendman974 02dba88131 fix: show silent-mode feedback and promptly detect exit
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 21:48:08 +01:00
Blendman974 19944c7855 Merge temporary switcher window guard
# Conflicts:
#	app/ui/shell/shell.ts
2026-11-06 06:34:51 +01:00
Blendman974 8be15d890c fix: hide switcher move action for temporary windows
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 06:33:47 +01:00
Blendman974 428142aef7 Merge open app switcher ordering
# Conflicts:
#	app/ui/shell/shell.ts
#	docs/validation-launcher-layout.md
2026-11-06 06:32:58 +01:00
Blendman974 c22f2e5f13 feat: reorder open app windows from the system menu
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 06:31:29 +01:00
Blendman974 53391a310e Merge default close-app selection 2026-11-06 06:05:32 +01:00
Blendman974 cfda10d47f fix: select close current app by default in system menu
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
2026-11-06 06:05:31 +01:00
Blendman974 46cc5c5e5b test: validate launcher alongside all feature branches 2026-11-06 06:02:17 +01:00
Blendman974 c9151710dc Merge launcher folders and custom layout 2026-11-06 06:00:42 +01:00
Blendman974 dded80a850 feat: manage launcher folders and custom icon positions
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 06:00:25 +01:00
Blendman974 5ae06c1818 Merge configurable menu gestures
# Conflicts:
#	app/ui/shell/shell.ts
#	tests/ai-chat.test.cjs
#	tests/brightness-picker.test.cjs
2026-11-06 04:46:29 +01:00
Blendman974 bac1c8a239 feat: allow swapping app and system menu hold gestures
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 04:44:39 +01:00
Blendman974 883fe6bee9 docs: document independent feature PRs and combined builds 2026-11-06 04:35:20 +01:00
Blendman974 cb4635d959 Merge multilingual Whisper transcription
# Conflicts:
#	app/g2/dashboard-controller.ts
2026-11-06 04:31:25 +01:00
Blendman974 3f392712ec Merge app system menu controls 2026-11-06 04:30:15 +01:00
Blendman974 ec6fc8d40e Merge head-up wake controls 2026-11-06 04:30:15 +01:00
Blendman974 675c051461 Merge shared calendar selection 2026-11-06 04:30:15 +01:00
Blendman974 f060deaf6f Merge notification preferences and bulk actions 2026-11-06 04:30:15 +01:00
Blendman974 6f80ca75bd Merge Android 16 KiB page compatibility 2026-11-06 04:30:14 +01:00
Blendman974 acd114370b Merge Nix Android and Wear OS build environment 2026-11-06 04:30:14 +01:00
Blendman974 5929259d84 feat: configure and synchronize head-up wake and angle
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
2026-11-06 04:30:07 +01:00
Blendman974 55bd5f891c feat: add application close, full-screen and keep-awake controls
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 04:26:16 +01:00
Blendman974 0b9e4cb367 feat: select languages for cloud and local Whisper transcription
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 04:22:48 +01:00
Blendman974 1959e57d02 feat: configure notification filtering and dismiss all notifications
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
Secret scan / gitleaks (pull_request) Canceled after 0s
2026-11-06 04:21:43 +01:00
Blendman974 9e6bb8473a feat: share calendar selection across Android readers
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 04:21:43 +01:00
Blendman974 75bceb3b92 fix: support Android 16 KiB pages in native llama library
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 04:21:43 +01:00
Blendman974 1b2095aa5b build: add reproducible Nix Android and Wear OS environment
Secret scan / gitleaks (pull_request) Canceled after 0s
Tests / test (pull_request) Canceled after 0s
Tests / test-kotlin (pull_request) Canceled after 0s
Tests / stats-server (pull_request) Canceled after 0s
Tests / evenhub-extensions (pull_request) Canceled after 0s
2026-11-06 04:21:21 +01:00
290 changed files with 6894 additions and 16224 deletions

No files matched your search

+3
View File
@@ -0,0 +1,3 @@
watch_file devenv.nix devenv.yaml devenv.lock
eval "$(devenv direnvrc)"
use devenv
+5 -2
View File
@@ -49,6 +49,11 @@ faceclaw_settings.xml
faceclaw_settings*.json
# Developer configs
.devenv*
.direnv/
.android/
devenv.local.nix
local.properties
build_paths.sh
**/xcuserdata/
@@ -58,8 +63,6 @@ native/kotlin/.kotlin/
native/kotlin/build/
native/kotlin/shared/build/
native/kotlin/local.properties
# Go SSH transport build outputs (scripts/go-ssh-build.cjs)
native/ssh/build/
native/kotlin/plugin/platforms/android/*.aar
native/kotlin/plugin/platforms/ios/
!native/kotlin/plugin/platforms/
-6
View File
@@ -23,12 +23,6 @@ and Mentra Labs, Inc. (MIT License): https://github.com/tetramo-labs/glassly and
https://github.com/tetramo-labs/glassly-rayneo-cfw. The iO wire codecs in
native/kotlin/shared/.../rayneo/RayNeoIoProtocol.kt follow their protocol work.
Bluetooth keyboard support on the G2 uses the keyboard bridge Tetramo, Inc. wrote
for their Glassly custom firmware (https://github.com/tetramo-labs/g2flash).
Faceclaw's client for it, in app/g2/keyboard/ and app/apps/terminal/terminal-keys.ts,
follows the reference client in that repository and the keyboard handling in the
Glassly app (MIT License): https://github.com/tetramo-labs/glassly.
iPhone notification icons are Pebble's notification and timeline icons, from
https://github.com/pebble-dev/iconography: the original Pebble Technology icons
and Rebble community additions. They are licensed under the Apache License 2.0;
+8
View File
@@ -165,6 +165,14 @@ tasks.register("prepareFaceclawNativeLibs") {
def faceclawLlamaCppTag = "b10333"
def faceclawFindSdkCmakeDir = {
// NixOS and other managed environments can provide patched host tools.
def explicitCmake = System.getenv("ANDROID_CMAKE_HOME")
if (explicitCmake != null) {
def explicitDir = file(explicitCmake)
if (faceclawHostExecutable(file("$explicitDir/bin"), "cmake").exists()
&& faceclawHostExecutable(file("$explicitDir/bin"), "ninja").exists()) return explicitDir
throw new GradleException("ANDROID_CMAKE_HOME must contain bin/cmake and bin/ninja: $explicitDir")
}
def sdkRoot = System.getenv("ANDROID_HOME") ?: System.getenv("ANDROID_SDK_ROOT")
if (sdkRoot == null) return null
def cmakeRoot = file("$sdkRoot/cmake")
@@ -25,18 +25,21 @@ internal object AndroidSpeechEngines {
private const val FEATURE_DIM = 80
/** Sherpa-onnx configuration for the on-device model files in [modelDir]. */
fun recognizerConfig(modelDir: File, kind: VoiceModelKind): OfflineRecognizerConfig {
fun recognizerConfig(
modelDir: File, kind: VoiceModelKind,
whisper: WhisperModelFiles = WhisperModelFiles.forLanguage(null),
): OfflineRecognizerConfig {
val modelConfig = OfflineModelConfig.builder()
.setNumThreads(1)
if (kind == VoiceModelKind.WHISPER) {
modelConfig
.setWhisper(OfflineWhisperModelConfig.builder()
.setEncoder(File(modelDir, "base.en-encoder.int8.onnx").absolutePath)
.setDecoder(File(modelDir, "base.en-decoder.int8.onnx").absolutePath)
.setLanguage("en")
.setEncoder(File(modelDir, whisper.encoder).absolutePath)
.setDecoder(File(modelDir, whisper.decoder).absolutePath)
.setLanguage(whisper.language)
.setTask("transcribe")
.build())
.setTokens(File(modelDir, "base.en-tokens.txt").absolutePath)
.setTokens(File(modelDir, whisper.tokens).absolutePath)
} else {
modelConfig
.setMoonshine(OfflineMoonshineModelConfig.builder()
@@ -274,6 +274,8 @@ class FaceclawBleCommunicator(context: Context, rightAddress: String?, leftAddre
fun enableWearDetectionAndRequestState() = core.enableWearDetectionAndRequestState()
fun configureHeadUpWake(enabled: Int, angle: Int) = core.configureHeadUpWake(enabled, angle)
fun startBandwidthBenchmark(messageSize: Int, windowSize: Int, durationMs: Int): Boolean =
core.startBandwidthBenchmark(messageSize, windowSize, durationMs)
@@ -290,13 +292,6 @@ class FaceclawBleCommunicator(context: Context, rightAddress: String?, leftAddre
/** See [GlassesSessionCore.getRingStatus]. */
fun getRingStatus(): String = core.getRingStatus()
/** See [GlassesSessionCore.setHeadUpTrigger]. */
fun setHeadUpTrigger(degrees: Int, originalSwitch: Int, originalAngle: Int) =
core.setHeadUpTrigger(degrees, originalSwitch, originalAngle)
/** See [GlassesSessionCore.getHeadUpStatus]. */
fun getHeadUpStatus(): String = core.getHeadUpStatus()
/** See [GlassesSessionCore.setGlassesRingLink]. */
fun setGlassesRingLink(action: String, fallbackAddress: String?, fallbackName: String?): Boolean =
core.setGlassesRingLink(action, fallbackAddress, fallbackName)
@@ -329,13 +324,6 @@ class FaceclawBleCommunicator(context: Context, rightAddress: String?, leftAddre
fun sendFaceclawMicControl(record: ByteArray?, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean =
core.sendFaceclawMicControl(record, label, rightTemple, leftTemple)
fun addKeyboardEventListener(listener: FaceclawKeyboardEventListener?) = core.addKeyboardEventListener(listener)
fun removeKeyboardEventListener(listener: FaceclawKeyboardEventListener?) = core.removeKeyboardEventListener(listener)
fun sendFaceclawKeyboardControl(record: ByteArray?, label: String): Boolean =
core.sendFaceclawKeyboardControl(record, label)
fun startG2AudioForwarding(listener: FaceclawAudioPacketListener?): Boolean = core.startG2AudioForwarding(listener)
fun stopG2AudioForwarding() = core.stopG2AudioForwarding()
@@ -231,17 +231,16 @@ class FaceclawBleManager(context: Context) {
fun connect(address: String?, timeoutMs: Int): Boolean = connect(address, timeoutMs, false)
/**
* EXPERIMENTAL (2026-09-11): the ring specifically needs autoConnect=true.
* Even's own vendor app connects the ring with Android's autoConnect flag
* set (confirmed via its own BLE debug log, "connect() - device: ...,
* auto: true"), while every other faceclaw connection (both glasses arms)
* uses autoConnect=false. Direct connect (false) is faster to establish
* but Android supervises it less patiently; a live test showed the
* ring-only direct connection dying almost exactly 5 seconds in,
* repeatedly, with nothing else competing for it. autoConnect=true can
* take much longer to actually complete (Android manages it as a
* background reconnect, not an immediate attempt), so this needs its own
* longer timeout. Like [connect], waits on an attempt already pending.
* autoConnect=true is used for the ring and both glasses arms. The ring specifically
* needs it (Even's own vendor app connects the ring with Android's autoConnect flag
* set, confirmed via its own BLE debug log, "connect() - device: ...,
* auto: true"). A live test showed a ring-only direct connection dying almost
* exactly 5 seconds in, repeatedly, with nothing else competing for it, and the
* glasses arms showed the same "less patiently supervised" direct-connect drop.
* Direct connect (false) is faster to establish but Android supervises it less
* patiently. autoConnect=true can take much longer to actually complete (Android
* manages it as a background reconnect, not an immediate attempt), so it needs its
* own longer timeout. Like [connect], waits on an attempt already pending.
*/
fun connect(address: String?, timeoutMs: Int, autoConnect: Boolean): Boolean {
val key = requireAddress(address)
@@ -54,7 +54,8 @@ class FaceclawCalendarProvider private constructor() {
* boundaries the provider stores them as.
*/
@JvmStatic
fun getUpcomingEventsJson(context: Context?, maxEvents: Int, windowMs: Long): String {
@JvmOverloads
fun getUpcomingEventsJson(context: Context?, maxEvents: Int, windowMs: Long, selectedCalendarIdsJson: String = "null"): String {
if (context == null || maxEvents <= 0) {
return "[]"
}
@@ -73,11 +74,21 @@ class FaceclawCalendarProvider private constructor() {
val events = ArrayList<Event>()
var cursor: Cursor? = null
try {
// Filter instances before the cutoff/limit so excluded calendars
// cannot crowd out selected calendars, including recurrences.
val ids = if (selectedCalendarIdsJson == "null") null else JSONArray(selectedCalendarIdsJson)
if (ids != null && ids.length() == 0) return "[]"
val args = ids?.let { values -> Array(values.length()) { index ->
values.getString(index).also { require(it.matches(Regex("[0-9]+"))) }
} }
val selection = args?.let {
CalendarContract.Instances.CALENDAR_ID + " IN (" + it.joinToString(",") { "?" } + ")"
}
cursor = context.contentResolver.query(
uri,
PROJECTION,
null,
null,
selection,
args,
CalendarContract.Instances.BEGIN + " ASC")
if (cursor != null) {
val utc = Calendar.getInstance(TimeZone.getTimeZone("UTC"))
@@ -118,6 +129,33 @@ class FaceclawCalendarProvider private constructor() {
return out.toString()
}
@JvmStatic
fun getCalendarsJson(context: Context?): String {
if (context == null) return "[]"
val out = JSONArray()
try {
context.contentResolver.query(
CalendarContract.Calendars.CONTENT_URI,
arrayOf(CalendarContract.Calendars._ID, CalendarContract.Calendars.CALENDAR_DISPLAY_NAME,
CalendarContract.Calendars.ACCOUNT_NAME, CalendarContract.Calendars.ACCOUNT_TYPE),
null, null, CalendarContract.Calendars.CALENDAR_DISPLAY_NAME + " ASC",
)?.use { cursor ->
while (cursor.moveToNext()) {
out.put(JSONObject().apply {
put("id", cursor.getLong(0).toString())
put("name", cursor.getString(1) ?: "")
put("accountName", cursor.getString(2) ?: "")
put("accountType", cursor.getString(3) ?: "")
})
}
}
} catch (t: Throwable) {
Log.w(TAG, "failed to read calendars", t)
return "[]"
}
return out.toString()
}
private fun readEvent(cursor: Cursor, utc: Calendar, local: Calendar): Event {
val allDay = cursor.getInt(4) != 0
var startMs = cursor.getLong(2)
@@ -0,0 +1,47 @@
package com.faceclaw.app
import android.content.Context
import android.security.keystore.KeyGenParameterSpec
import android.security.keystore.KeyProperties
import android.util.AtomicFile
import java.io.File
import java.security.KeyStore
import javax.crypto.Cipher
import javax.crypto.KeyGenerator
import javax.crypto.SecretKey
import javax.crypto.spec.GCMParameterSpec
/** This credential cannot be restored onto a device lacking its Keystore key. */
object FaceclawHealthExportSecret {
private const val ALIAS = "faceclaw.health.export"
private fun file(context: Context) = AtomicFile(File(context.noBackupFilesDir, "health-export-key"))
private fun key(): SecretKey {
val store = KeyStore.getInstance("AndroidKeyStore").apply { load(null) }
(store.getKey(ALIAS, null) as? SecretKey)?.let { return it }
return KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, "AndroidKeyStore").apply {
init(KeyGenParameterSpec.Builder(ALIAS, KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE).build())
}.generateKey()
}
@JvmStatic @Synchronized fun read(context: Context): String {
val held = file(context)
if (!held.baseFile.exists()) return ""
return try {
val bytes = held.readFully()
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
cipher.init(Cipher.DECRYPT_MODE, key(), GCMParameterSpec(128, bytes.copyOfRange(0, 12)))
String(cipher.doFinal(bytes.copyOfRange(12, bytes.size)), Charsets.UTF_8)
} catch (_: Exception) { "" } // Require re-entry after key loss; never log the credential.
}
@JvmStatic @Synchronized fun write(context: Context, value: String) {
val held = file(context)
if (value.isEmpty()) { held.delete(); return }
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
cipher.init(Cipher.ENCRYPT_MODE, key())
val bytes = cipher.iv + cipher.doFinal(value.toByteArray(Charsets.UTF_8))
val stream = held.startWrite()
try { stream.write(bytes); held.finishWrite(stream) }
catch (error: Exception) { held.failWrite(stream); throw error }
}
}
@@ -125,6 +125,27 @@ class FaceclawMediaNotificationListenerService : NotificationListenerService() {
return "connecting"
}
/** Fresh, read-only phone DND state; unavailable filters retain normal wake behavior. */
@JvmStatic
fun isPhoneDoNotDisturbEnabled(context: Context?): Boolean {
val filter = try {
val manager = context?.getSystemService(Context.NOTIFICATION_SERVICE) as? NotificationManager
manager?.currentInterruptionFilter
} catch (t: Throwable) {
null
} ?: try {
activeService?.currentInterruptionFilter
} catch (t: Throwable) {
null
}
return when (filter) {
NotificationManager.INTERRUPTION_FILTER_PRIORITY,
NotificationManager.INTERRUPTION_FILTER_NONE,
NotificationManager.INTERRUPTION_FILTER_ALARMS -> true
else -> false
}
}
/** Run on the main thread whenever getListenerState may have changed. */
@JvmStatic
fun addListenerStateObserver(observer: Runnable?) {
@@ -504,8 +525,22 @@ class FaceclawMediaNotificationListenerService : NotificationListenerService() {
}
}
@JvmStatic
fun dismissAllNotifications(): Boolean {
val service = activeService ?: return false
return try {
service.cancelAllNotifications()
true
} catch (t: Throwable) {
Log.w(TAG, "failed to dismiss all notifications", t)
false
}
}
private fun emitNotificationPosted(key: String?) {
emitNotificationEvent(key) { listener, eventKey -> listener.onNotificationPosted(eventKey) }
// Capture before mainHandler posting: a later DND change must not alter this arrival.
val canWakeScreen = !isPhoneDoNotDisturbEnabled(activeService)
emitNotificationEvent(key) { listener, eventKey -> listener.onNotificationPosted(eventKey, canWakeScreen) }
}
private fun emitNotificationRemoved(key: String?) {
@@ -1,212 +0,0 @@
package com.faceclaw.app
import android.content.ClipData
import android.content.ClipboardManager
import android.content.Context
import android.os.Build
import android.os.Handler
import android.os.Looper
import android.security.keystore.KeyGenParameterSpec
import android.security.keystore.KeyProperties
import android.security.keystore.StrongBoxUnavailableException
import android.util.Base64
import android.util.Log
import com.faceclaw.ssh.faceclawssh.Channel
import com.faceclaw.ssh.faceclawssh.Faceclawssh
import com.faceclaw.ssh.faceclawssh.Listener
import com.faceclaw.ssh.faceclawssh.Signer
import java.security.KeyPairGenerator
import java.security.KeyStore
import java.security.PrivateKey
import java.security.Signature
import java.security.spec.ECGenParameterSpec
import javax.crypto.Cipher
import javax.crypto.KeyGenerator
import javax.crypto.SecretKey
import javax.crypto.spec.GCMParameterSpec
/**
* One SSH channel for the TypeScript side (app/native/ssh.ts), backed by the
* gomobile-bound Go transport in native/ssh. `spec` is the Go openSpec JSON.
* Events arrive from Go goroutines and are posted to the Looper of the
* thread that constructed this object (a JS isolate's own thread), like
* FaceclawWebSocket.
*/
class FaceclawSshChannel(spec: String, useKey: Boolean, listener: FaceclawSshListener) {
private val handler = Handler(Looper.myLooper() ?: Looper.getMainLooper())
private val channel: Channel
init {
val signer = if (useKey) FaceclawSshKeys.signer() else null
channel = Faceclawssh.open(spec, signer, object : Listener {
override fun onEvent(kind: String?, payload: String?) {
handler.post {
try {
listener.onEvent(kind, payload)
} catch (t: Throwable) {
Log.w(TAG, "listener onEvent($kind) failed", t)
}
}
}
})
}
fun send(text: String): Boolean = channel.send(text)
fun close() = channel.close()
private companion object {
const val TAG = "FaceclawSsh"
}
}
/**
* The device's SSH client key and the Terminal app's saved SSH passwords.
*
* The client key is an ECDSA P-256 key generated inside Android Keystore
* (StrongBox when the device has one) and never exportable: only its public
* half leaves the keystore, as an authorized_keys line. Passwords are
* encrypted with an AES-GCM key that also lives in Keystore, and stored in a
* private SharedPreferences file kept apart from the settings store (which is
* exported by the settings port).
*/
object FaceclawSshKeys {
private const val TAG = "FaceclawSsh"
private const val CLIENT_KEY_ALIAS = "faceclaw-ssh-client"
private const val SECRET_KEY_ALIAS = "faceclaw-ssh-secrets"
private const val SECRETS_PREFS = "faceclaw_ssh_secrets"
private const val GCM_TAG_BITS = 128
private fun keyStore(): KeyStore = KeyStore.getInstance("AndroidKeyStore").apply { load(null) }
@JvmStatic
@Synchronized
fun hasClientKey(): Boolean = keyStore().containsAlias(CLIENT_KEY_ALIAS)
/** Generate the client key if it doesn't exist yet. */
@JvmStatic
@Synchronized
fun ensureClientKey() {
if (keyStore().containsAlias(CLIENT_KEY_ALIAS)) return
fun spec(strongBox: Boolean): KeyGenParameterSpec {
val builder = KeyGenParameterSpec.Builder(CLIENT_KEY_ALIAS, KeyProperties.PURPOSE_SIGN)
.setAlgorithmParameterSpec(ECGenParameterSpec("secp256r1"))
.setDigests(KeyProperties.DIGEST_SHA256)
if (strongBox && Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) builder.setIsStrongBoxBacked(true)
return builder.build()
}
val generator = KeyPairGenerator.getInstance(KeyProperties.KEY_ALGORITHM_EC, "AndroidKeyStore")
try {
generator.initialize(spec(strongBox = true))
generator.generateKeyPair()
} catch (e: Exception) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P && e is StrongBoxUnavailableException) {
Log.i(TAG, "StrongBox unavailable; generating the SSH key in the TEE")
} else {
Log.w(TAG, "StrongBox key generation failed; retrying in the TEE", e)
}
generator.initialize(spec(strongBox = false))
generator.generateKeyPair()
}
Log.i(TAG, "generated SSH client key")
}
/** X.509 SubjectPublicKeyInfo DER of the client key, or null if none. */
@JvmStatic
fun publicKey(): ByteArray? = keyStore().getCertificate(CLIENT_KEY_ALIAS)?.publicKey?.encoded
/** The client key's authorized_keys line (generating the key if needed). */
@JvmStatic
fun authorizedKey(comment: String): String {
ensureClientKey()
return Faceclawssh.authorizedKey(publicKey(), comment)
}
/** The client key's SHA256 fingerprint (generating the key if needed). */
@JvmStatic
fun fingerprint(): String {
ensureClientKey()
return Faceclawssh.fingerprint(publicKey())
}
/** A Go Signer over the client key, generating the key if needed. */
@JvmStatic
fun signer(): Signer? = try {
ensureClientKey()
val publicKey = publicKey() ?: throw IllegalStateException("client key has no certificate")
object : Signer {
override fun publicKey(): ByteArray = publicKey
override fun sign(data: ByteArray?): ByteArray {
val privateKey = keyStore().getKey(CLIENT_KEY_ALIAS, null) as PrivateKey
return Signature.getInstance("SHA256withECDSA").run {
initSign(privateKey)
update(data ?: ByteArray(0))
sign()
}
}
}
} catch (t: Throwable) {
Log.w(TAG, "SSH client key unavailable", t)
null
}
private fun secretKey(): SecretKey {
val store = keyStore()
(store.getKey(SECRET_KEY_ALIAS, null) as? SecretKey)?.let { return it }
val generator = KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, "AndroidKeyStore")
generator.init(
KeyGenParameterSpec.Builder(SECRET_KEY_ALIAS, KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
.setKeySize(256)
.build()
)
return generator.generateKey()
}
private fun prefs(context: Context) =
context.applicationContext.getSharedPreferences(SECRETS_PREFS, Context.MODE_PRIVATE)
@JvmStatic
@Synchronized
fun setSecret(context: Context, id: String, value: String) {
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
cipher.init(Cipher.ENCRYPT_MODE, secretKey())
val sealed = cipher.iv + cipher.doFinal(value.toByteArray(Charsets.UTF_8))
prefs(context).edit().putString(id, Base64.encodeToString(sealed, Base64.NO_WRAP)).apply()
}
/** The saved secret, or "" if none (or it can no longer be decrypted). */
@JvmStatic
@Synchronized
fun getSecret(context: Context, id: String): String {
val encoded = prefs(context).getString(id, null) ?: return ""
return try {
val sealed = Base64.decode(encoded, Base64.NO_WRAP)
val cipher = Cipher.getInstance("AES/GCM/NoPadding")
val ivLength = 12
cipher.init(Cipher.DECRYPT_MODE, secretKey(), GCMParameterSpec(GCM_TAG_BITS, sealed, 0, ivLength))
String(cipher.doFinal(sealed, ivLength, sealed.size - ivLength), Charsets.UTF_8)
} catch (t: Throwable) {
Log.w(TAG, "saved SSH secret unreadable; dropping it", t)
prefs(context).edit().remove(id).apply()
""
}
}
@JvmStatic
@Synchronized
fun removeSecret(context: Context, id: String) {
prefs(context).edit().remove(id).apply()
}
/** Put text on the phone's clipboard (from any thread). */
@JvmStatic
fun copyToClipboard(context: Context, label: String, text: String) {
val appContext = context.applicationContext
Handler(Looper.getMainLooper()).post {
val clipboard = appContext.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager
clipboard.setPrimaryClip(ClipData.newPlainText(label, text))
}
}
}
@@ -32,17 +32,6 @@ class FaceclawVoiceController(context: Context) {
"decoder_model_merged.ort",
"tokens.txt"
)
// Second on-device model: sherpa-onnx's offline Whisper backend (base.en,
// int8-quantized -- see the model-choice note in asr-model.ts). Directory
// shared with the TS-side download flow, same convention as ASR_MODEL_DIR.
// Whisper re-encodes the whole buffer on every call, so the shared session
// skips live partials for it; see VoiceCaptureSession.processRecognizer.
private const val ASR_WHISPER_MODEL_DIR = "sherpa-onnx-whisper-base-en-int8"
private val ASR_WHISPER_MODEL_FILES = arrayOf(
"base.en-encoder.int8.onnx",
"base.en-decoder.int8.onnx",
"base.en-tokens.txt"
)
// 50 ms chunks match the G2 packet cadence the rest of the pipeline
// (endpointing, transcript pacing) is tuned for.
private const val PHONE_MIC_CHUNK_SAMPLES = SAMPLE_RATE / 20
@@ -84,6 +73,8 @@ class FaceclawVoiceController(context: Context) {
private val mainHandler = Handler(Looper.getMainLooper())
@Volatile
private var communicator: FaceclawBleCommunicator? = null
private var whisperModel = WhisperModelFiles.forLanguage(null)
private var frenchWhisper = false
private val host = object : VoiceCaptureHost {
override val dispatcher = CallbackDispatcher { action -> mainHandler.post { action() } }
@@ -114,21 +105,6 @@ class FaceclawVoiceController(context: Context) {
}
}
override fun micArrayLink(): MicArrayLink? {
val currentCommunicator = communicator ?: return null
return object : MicArrayLink {
override fun startForwarding(listener: FaceclawAudioPacketListener): Boolean =
currentCommunicator.startG2AudioForwarding(listener)
override fun stopForwarding() = currentCommunicator.stopG2AudioForwarding()
override fun isForwarding(): Boolean = currentCommunicator.isAudioCaptureActive()
override fun sendControl(record: ByteArray, label: String, rightTemple: Boolean, leftTemple: Boolean): Boolean =
currentCommunicator.sendFaceclawMicControl(record, label, rightTemple, leftTemple)
}
}
override fun createPacketDecoder(): AudioPacketDecoder =
if (communicator?.isRayNeoIo() == true) AndroidSpeechEngines.OpusDecoderAdapter()
else AndroidSpeechEngines.Lc3DecoderAdapter()
@@ -139,7 +115,7 @@ class FaceclawVoiceController(context: Context) {
override fun loadTranscriber(kind: VoiceModelKind): OfflineTranscriber {
val modelDir = findAsrModelDir(kind) ?: throw IllegalStateException("voice model missing")
return AndroidSpeechEngines.SherpaTranscriber(AndroidSpeechEngines.recognizerConfig(modelDir, kind))
return AndroidSpeechEngines.SherpaTranscriber(AndroidSpeechEngines.recognizerConfig(modelDir, kind, whisperModel))
}
override fun speakerEmbedder(modelPath: String): SpeakerEmbedder {
@@ -155,11 +131,6 @@ class FaceclawVoiceController(context: Context) {
private val session = VoiceCaptureSession(host)
// The recording of this controller's live captures (see startRecording). It outlives any one
// capture, and is only handed to the session while recording is enabled.
private var recording: MicRecordingWriter? = null
private var recordingEnabled = true
fun setListener(listener: FaceclawVoiceControllerListener?) {
session.setListener(listener)
}
@@ -183,6 +154,17 @@ class FaceclawVoiceController(context: Context) {
session.setOnboardModelKind(kind)
}
/** Select the dedicated French fine-tune before setting the capture language. */
fun setFrenchWhisper(enabled: Boolean) {
frenchWhisper = enabled
whisperModel = WhisperModelFiles.forSelection(whisperModel.language, enabled)
}
/** Language for the next on-device Whisper capture; empty enables auto-detection. */
fun setOnboardLanguage(language: String?) {
whisperModel = WhisperModelFiles.forSelection(language, frenchWhisper)
}
/**
* When true, watch the decoded PCM and fire {@code onSpeechEnd} once the
* speaker stops. Used by hands-free ("Hey Even") capture, which has no
@@ -221,88 +203,6 @@ class FaceclawVoiceController(context: Context) {
session.setBeamFilter(enabled, centerDeg, halfWidthDeg)
}
/**
* Capture glasses sessions from the temples' own microphones (the custom firmware's
* mic_control stream) instead of the stock mixed stream. [configureRecord] is the CONFIGURE
* record to arm both temples with; null goes back to the mixed stream. Must be set before
* {@link #start}.
*/
fun setMicArray(configureRecord: ByteArray?) {
session.setMicArray(configureRecord)
}
/**
* Have the next {@link #start} play the recording at [path] (a .mka file) from [startMs] in
* place of any microphone; null goes back to the microphones. Progress arrives through the
* listener's onReplay.
*/
fun setReplay(path: String?, startMs: Long) {
session.setReplay(path, startMs)
}
/** Continue the running replay from [positionMs] into its recording. */
fun seekReplay(positionMs: Long) {
session.seekReplay(positionMs)
}
fun setReplayPaused(paused: Boolean) {
session.setReplayPaused(paused)
}
/**
* Record live captures to a new Matroska audio file at [path], from now until
* {@link #stopRecording}, across however many captures run in between: every source's audio
* as it arrives, before any processing. [configJson] describes the capture and is stored in
* the file. Returns false when the file cannot be created.
*/
@Synchronized
fun startRecording(path: String, configJson: String?): Boolean {
stopRecording()
return try {
val parent = File(path).parentFile
if (parent != null && !parent.exists()) parent.mkdirs()
recording = MicRecordingWriter(openFileOutput(path), configJson, System.currentTimeMillis())
session.setRecorder(if (recordingEnabled) recording else null)
true
} catch (t: Throwable) {
Log.w(TAG, "could not create recording $path", t)
recording = null
false
}
}
/**
* Whether captures are written to the recording right now: off while the capture running
* isn't one that is to be recorded.
*/
@Synchronized
fun setRecordingEnabled(enabled: Boolean) {
recordingEnabled = enabled
session.setRecorder(if (enabled) recording else null)
}
/** Whether a recording is open and still being written (false once writing to it failed). */
@Synchronized
fun isRecording(): Boolean = recording?.failed == false
/**
* Note a change in what is being captured (a JSON object) at the recording's current
* position; the audio after it is placed afresh on the recording's timeline.
*/
@Synchronized
fun noteRecordingEvent(json: String) {
recording?.event(json)
}
/** Complete the recording's file. Returns its length in ms, or -1 when none was open. */
@Synchronized
fun stopRecording(): Long {
val current = recording ?: return -1
recording = null
session.setRecorder(null)
return current.finish()
}
fun start(requestedMode: String?) {
session.start(requestedMode, 0)
}
@@ -337,8 +237,8 @@ class FaceclawVoiceController(context: Context) {
*/
private fun findAsrModelDir(kind: VoiceModelKind): File? {
val whisper = kind == VoiceModelKind.WHISPER
val dirName = if (whisper) ASR_WHISPER_MODEL_DIR else ASR_MODEL_DIR
val fileNames = if (whisper) ASR_WHISPER_MODEL_FILES else ASR_MODEL_FILES
val dirName = if (whisper) whisperModel.dirName else ASR_MODEL_DIR
val fileNames = if (whisper) whisperModel.files else ASR_MODEL_FILES.toList()
val modelDir = File(appContext.filesDir, ASR_ROOT + File.separator + dirName)
for (fileName in fileNames) {
val file = File(modelDir, fileName)
@@ -29,3 +29,8 @@ add_subdirectory(${LLAMA_CPP_SOURCE_DIR} llama.cpp-build EXCLUDE_FROM_ALL)
add_library(faceclaw_llama SHARED faceclaw_llama.cpp)
target_link_libraries(faceclaw_llama PRIVATE llama android log)
# NDK r27 defaults to 4 KiB. Align both LOAD segments and RELRO boundaries
# for Android devices using 16 KiB pages, including debug APKs.
target_link_options(faceclaw_llama PRIVATE
"-Wl,-z,max-page-size=16384"
"-Wl,-z,common-page-size=16384")
-31
View File
@@ -1,31 +0,0 @@
#import <Foundation/Foundation.h>
/**
* One SSH channel for the TypeScript side (app/native/ssh.ios.ts), backed by
* the gomobile-bound Go transport in native/ssh. `spec` is the Go openSpec
* JSON. Events arrive on Go threads and queue here; the owning JS isolate
* drains them with takeEvents (a JSON array of {kind, payload}).
*/
@interface FaceclawSshChannel : NSObject
- (instancetype)initWithSpec:(NSString *)spec useKey:(BOOL)useKey;
- (NSString *)takeEvents;
- (BOOL)send:(NSString *)text;
- (void)close;
@end
/**
* The device's SSH client key (a non-exportable Secure Enclave P-256 key; a
* plain keychain key on the simulator, which has no Secure Enclave) and the
* Terminal app's saved SSH passwords (keychain generic passwords).
*/
@interface FaceclawSshKeys : NSObject
/** "ecdsa-sha2-nistp256 AAAA... comment"; generates the key if needed. nil on failure. */
+ (NSString *)authorizedKeyWithComment:(NSString *)comment;
/** "SHA256:..." fingerprint; generates the key if needed. nil on failure. */
+ (NSString *)fingerprint;
+ (void)setSecret:(NSString *)value forId:(NSString *)secretId;
/** The saved secret, or "" if none. */
+ (NSString *)secretForId:(NSString *)secretId;
+ (void)removeSecretForId:(NSString *)secretId;
+ (void)copyToClipboard:(NSString *)text;
@end
-234
View File
@@ -1,234 +0,0 @@
#import "FaceclawSsh.h"
#import <Faceclawssh/Faceclawssh.h>
#import <Security/Security.h>
#import <UIKit/UIKit.h>
static NSString *const kClientKeyTag = @"com.faceclaw.ssh.client";
static NSString *const kSecretService = @"com.faceclaw.ssh.secret";
#pragma mark - Client key
static NSData *ClientKeyTagData(void) {
return [kClientKeyTag dataUsingEncoding:NSUTF8StringEncoding];
}
static SecKeyRef CopyClientKey(void) {
NSDictionary *query = @{
(id)kSecClass: (id)kSecClassKey,
(id)kSecAttrApplicationTag: ClientKeyTagData(),
(id)kSecAttrKeyType: (id)kSecAttrKeyTypeECSECPrimeRandom,
(id)kSecReturnRef: @YES,
};
CFTypeRef key = NULL;
if (SecItemCopyMatching((__bridge CFDictionaryRef)query, &key) != errSecSuccess) return NULL;
return (SecKeyRef)key;
}
static SecKeyRef CreateClientKey(BOOL secureEnclave) {
SecAccessControlRef access = SecAccessControlCreateWithFlags(
kCFAllocatorDefault, kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly,
secureEnclave ? kSecAccessControlPrivateKeyUsage : 0, NULL);
if (!access) return NULL;
NSMutableDictionary *attributes = [@{
(id)kSecAttrKeyType: (id)kSecAttrKeyTypeECSECPrimeRandom,
(id)kSecAttrKeySizeInBits: @256,
(id)kSecPrivateKeyAttrs: @{
(id)kSecAttrIsPermanent: @YES,
(id)kSecAttrApplicationTag: ClientKeyTagData(),
(id)kSecAttrAccessControl: (__bridge id)access,
},
} mutableCopy];
if (secureEnclave) attributes[(id)kSecAttrTokenID] = (id)kSecAttrTokenIDSecureEnclave;
CFErrorRef error = NULL;
SecKeyRef key = SecKeyCreateRandomKey((__bridge CFDictionaryRef)attributes, &error);
CFRelease(access);
if (!key && error) {
NSLog(@"FaceclawSsh: key generation failed (secure enclave %d): %@", secureEnclave, (__bridge NSError *)error);
CFRelease(error);
}
return key;
}
/** The client key, generating it on first use. Caller releases. */
static SecKeyRef CopyOrCreateClientKey(void) {
@synchronized([FaceclawSshKeys class]) {
SecKeyRef key = CopyClientKey();
if (key) return key;
key = CreateClientKey(YES);
#if TARGET_OS_SIMULATOR
if (!key) key = CreateClientKey(NO);
#endif
return key;
}
}
/** Uncompressed X9.62 public point (0x04 || X || Y). */
static NSData *PublicKeyData(SecKeyRef key) {
SecKeyRef publicKey = SecKeyCopyPublicKey(key);
if (!publicKey) return nil;
NSData *data = (__bridge_transfer NSData *)SecKeyCopyExternalRepresentation(publicKey, NULL);
CFRelease(publicKey);
return data;
}
@interface FaceclawSshSigner : NSObject <FaceclawsshSigner>
@end
@implementation FaceclawSshSigner {
SecKeyRef _key;
NSData *_publicKey;
}
- (instancetype)initWithKey:(SecKeyRef)key {
if ((self = [super init])) {
_key = (SecKeyRef)CFRetain(key);
_publicKey = PublicKeyData(key);
}
return self;
}
- (void)dealloc {
if (_key) CFRelease(_key);
}
- (NSData *)publicKey {
return _publicKey;
}
- (NSData *)sign:(NSData *)data error:(NSError **)error {
CFErrorRef cfError = NULL;
NSData *signature = (__bridge_transfer NSData *)SecKeyCreateSignature(
_key, kSecKeyAlgorithmECDSASignatureMessageX962SHA256, (__bridge CFDataRef)(data ?: [NSData data]), &cfError);
if (!signature) {
if (error) *error = (__bridge_transfer NSError *)cfError;
else if (cfError) CFRelease(cfError);
}
return signature;
}
@end
#pragma mark - Channel
@interface FaceclawSshEventQueue : NSObject <FaceclawsshListener>
@property(nonatomic, strong) NSMutableArray *events;
@property(nonatomic) BOOL ended;
@end
@implementation FaceclawSshEventQueue
- (instancetype)init {
if ((self = [super init])) _events = [NSMutableArray array];
return self;
}
- (void)onEvent:(NSString *)kind payload:(NSString *)payload {
@synchronized(self) {
if (_ended) return;
[_events addObject:@{@"kind": kind ?: @"", @"payload": payload ?: @""}];
}
}
@end
@implementation FaceclawSshChannel {
FaceclawsshChannel *_channel;
FaceclawSshEventQueue *_queue;
}
- (instancetype)initWithSpec:(NSString *)spec useKey:(BOOL)useKey {
if ((self = [super init])) {
_queue = [FaceclawSshEventQueue new];
FaceclawSshSigner *signer = nil;
if (useKey) {
SecKeyRef key = CopyOrCreateClientKey();
if (key) {
signer = [[FaceclawSshSigner alloc] initWithKey:key];
CFRelease(key);
}
}
_channel = FaceclawsshOpen(spec, signer, _queue);
}
return self;
}
- (NSString *)takeEvents {
NSArray *events;
@synchronized(_queue) {
if (_queue.events.count == 0) return @"[]";
events = [_queue.events copy];
[_queue.events removeAllObjects];
}
NSData *data = [NSJSONSerialization dataWithJSONObject:events options:0 error:nil];
return [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding] ?: @"[]";
}
- (BOOL)send:(NSString *)text {
return [_channel send:text];
}
- (void)close {
@synchronized(_queue) {
_queue.ended = YES;
[_queue.events removeAllObjects];
}
[_channel close];
}
@end
#pragma mark - Keys and secrets
@implementation FaceclawSshKeys
+ (NSString *)authorizedKeyWithComment:(NSString *)comment {
SecKeyRef key = CopyOrCreateClientKey();
if (!key) return nil;
NSData *publicKey = PublicKeyData(key);
CFRelease(key);
NSError *error = nil;
NSString *line = FaceclawsshAuthorizedKey(publicKey, comment ?: @"", &error);
if (error) NSLog(@"FaceclawSsh: authorizedKey failed: %@", error);
return line;
}
+ (NSString *)fingerprint {
SecKeyRef key = CopyOrCreateClientKey();
if (!key) return nil;
NSData *publicKey = PublicKeyData(key);
CFRelease(key);
NSError *error = nil;
NSString *fingerprint = FaceclawsshFingerprint(publicKey, &error);
if (error) NSLog(@"FaceclawSsh: fingerprint failed: %@", error);
return fingerprint;
}
+ (NSDictionary *)secretQuery:(NSString *)secretId {
return @{
(id)kSecClass: (id)kSecClassGenericPassword,
(id)kSecAttrService: kSecretService,
(id)kSecAttrAccount: secretId ?: @"",
};
}
+ (void)setSecret:(NSString *)value forId:(NSString *)secretId {
NSDictionary *query = [self secretQuery:secretId];
SecItemDelete((__bridge CFDictionaryRef)query);
NSMutableDictionary *item = [query mutableCopy];
item[(id)kSecValueData] = [(value ?: @"") dataUsingEncoding:NSUTF8StringEncoding];
item[(id)kSecAttrAccessible] = (id)kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly;
OSStatus status = SecItemAdd((__bridge CFDictionaryRef)item, NULL);
if (status != errSecSuccess) NSLog(@"FaceclawSsh: saving secret failed: %d", (int)status);
}
+ (NSString *)secretForId:(NSString *)secretId {
NSMutableDictionary *query = [[self secretQuery:secretId] mutableCopy];
query[(id)kSecReturnData] = @YES;
query[(id)kSecMatchLimit] = (id)kSecMatchLimitOne;
CFTypeRef result = NULL;
if (SecItemCopyMatching((__bridge CFDictionaryRef)query, &result) != errSecSuccess) return @"";
NSData *data = (__bridge_transfer NSData *)result;
return [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding] ?: @"";
}
+ (void)removeSecretForId:(NSString *)secretId {
SecItemDelete((__bridge CFDictionaryRef)[self secretQuery:secretId]);
}
+ (void)copyToClipboard:(NSString *)text {
dispatch_async(dispatch_get_main_queue(), ^{
UIPasteboard.generalPasteboard.string = text ?: @"";
});
}
@end
-1
View File
@@ -2,7 +2,6 @@ module FaceclawGraphics {
header "FaceclawGraphics.h"
header "FaceclawFontRenderer.h"
header "FaceclawSocket.h"
header "FaceclawSsh.h"
header "FaceclawLc3Decoder.h"
header "FaceclawSpeech.h"
header "FaceclawLocation.h"
+2 -4
View File
@@ -80,10 +80,8 @@ import it into the new version (see [config transfer](scripts/config-transfer.md
* **Integration with Android notifications**: shows the same notification
icons your phone does, and popups when notifications arrive.
* **Terminal mirroring.** Mirror terminal apps such as Claude Code or Codex
CLI with [g2mirror](https://github.com/jimrandomh/g2mirror), or attach to
tmux sessions over SSH (no server-side setup beyond tmux 3.2+), view them
on the glasses, and send them inputs with the voice assistant. g2mirror
can also be tunnelled over SSH instead of a VPN.
CLI with [g2mirror](https://github.com/jimrandomh/g2mirror), view them on
the glasses, and send them inputs with the voice assistant.
* **Media player controls** including playlist and media library navigation,
compatible with most Android media players.
* **Turn-by-turn directions** (requires a Mapbox API token).
+1 -6
View File
@@ -55,10 +55,8 @@ export type GlanceboardProvider = {
isEnabled: () => boolean;
/** Whether a single tap shows the board (for tapTimeoutMs). */
showOnTap: () => boolean;
/** How long a tap keeps the board up (and a head-tilt, when its head-down is unlikely to come). */
/** How long a tap or head-tilt keeps the board up. */
tapTimeoutMs: () => number;
/** How long a head-tilt showing may last without a head-down (null = no limit). */
tiltTimeoutMs: () => number | null;
/** Whether a long-press holds the board up until released. */
showOnLongPress: () => boolean;
/** Whether the head-tilt wake goes to the board rather than the regular UI. */
@@ -189,8 +187,6 @@ export async function launchWorkerAppWindow(
matchExistingBy?: "appId" | "windowId";
/** Deliver watch swipes raw instead of the fallback (see WorkerWindowSpec). */
acceptsDirectional?: boolean;
/** Deliver hardware-keyboard keys as keys (see WorkerWindowSpec). */
acceptsKeys?: boolean;
},
): Promise<void> {
const existing = shell
@@ -211,7 +207,6 @@ export async function launchWorkerAppWindow(
icon: options.icon,
focus: true,
acceptsDirectional: options.acceptsDirectional,
acceptsKeys: options.acceptsKeys,
});
ctx.appendLog(`launched ${options.windowId}`);
}
-1
View File
@@ -312,7 +312,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
}
case "navigation-sensors":
case "text-input":
case "key":
case "tool-call":
break;
}
+5
View File
@@ -3,6 +3,7 @@ import { invalidateCalendarCache, onCalendarChanged } from "../../native/calenda
import { makeImageWindowIcon, windowIcon } from "../../ui/shell/chrome-layer";
import { CalendarLayer } from "./calendar";
import { renderCalendarDateIcon } from "./calendar-icon";
import { calendarSelectionMenuItems } from "./calendar-filter";
import {
createInProcessWindow,
YieldAtRootLayer,
@@ -44,6 +45,10 @@ export function createCalendarAppWindow(options: InProcessAppOptions): InProcess
icon: "calendar",
drawIcon: makeImageWindowIcon(renderCalendarDateIcon, windowIcon("calendar", "Ca")),
closeable: true,
menuItems: () => calendarSelectionMenuItems().map(item => ({ ...item, onSelect: (ctx, menu) => {
ctx.stack.pop();
return item.onSelect(ctx, menu);
} })),
actions: options.actions,
baseLayer: new YieldAtRootLayer(new CalendarLayer(requestPermission)),
submitFrame: options.submitFrame,
+94
View File
@@ -0,0 +1,94 @@
import { getDefaultSmallFont } from "../../graphics/ui-fonts";
import { type GrayImage } from "../../graphics/image";
import { truncateText } from "../../graphics/textwrap";
import { readCalendarSelection, setCalendarSelection } from "../../native/calendar-selection";
import { readAvailableCalendars } from "../../native/calendar-sources";
import { hasCalendarPermission } from "../../native/calendar-permissions";
import { type InputEvent } from "../../ui/gestures";
import { type Layer, type LayerContext, type PaintBelow } from "../../ui/layers";
import { drawToggleMenuItem, MenuLayer, type MenuItem } from "../../ui/menu";
/** Shared preference for every Android calendar reader. */
export class CalendarFilterLayer implements Layer {
private menu: MenuLayer | null = null;
private width = 0;
private keys: string[] = [];
private currentMenu(ctx: LayerContext): MenuLayer {
const calendars = readAvailableCalendars().sort((a, b) =>
a.name.localeCompare(b.name) || a.accountName.localeCompare(b.accountName) || a.id.localeCompare(b.id));
const selection = readCalendarSelection();
const known = new Set(calendars.map(calendar => calendar.id));
const keys = ["all", "none"];
const toggle = (id: string) => {
// Leaving all mode snapshots the current calendars. Newly added calendars
// then stay excluded until selected individually or all mode is restored.
const ids = new Set(readCalendarSelection() ?? calendars.map(calendar => calendar.id));
if (ids.has(id)) ids.delete(id); else ids.add(id);
setCalendarSelection(Array.from(ids));
};
const items: MenuItem[] = [{
label: "All calendars",
description: "Include all calendars, including calendars added later.",
onSelect: () => setCalendarSelection(null),
render: ({ image, x, y, width, selected }) => drawToggleMenuItem(image, getDefaultSmallFont(), x, y, width,
"All calendars", readCalendarSelection() === null, selected),
}, {
label: "No calendars",
onSelect: () => setCalendarSelection([]),
render: ({ image, x, y, width, selected }) => drawToggleMenuItem(image, getDefaultSmallFont(), x, y, width,
"No calendars", readCalendarSelection()?.length === 0, selected),
}];
const rows = [
...calendars.map(calendar => ({ id: calendar.id,
label: `${calendar.name || "(unnamed)"} / ${calendar.accountName || "Local"} (${calendar.accountType || "local"}) #${calendar.id}` })),
...(selection ?? []).filter(id => !known.has(id)).map(id => ({ id, label: `Unavailable calendar #${id}` })),
];
for (const row of rows) {
keys.push(`calendar:${row.id}`);
items.push({
label: row.label,
description: row.label,
onSelect: () => toggle(row.id),
render: ({ image, x, y, width, selected }) => {
const font = getDefaultSmallFont();
const ids = readCalendarSelection();
drawToggleMenuItem(image, font, x, y, width, truncateText(font, row.label, width - 48),
ids === null || ids.includes(row.id), selected);
},
});
}
if (!calendars.length) items.push({
label: hasCalendarPermission() ? "No calendars available" : "Grant calendar permission on your phone",
disabled: true, onSelect: () => {},
});
const key = this.menu?.selectedIndex != null ? this.keys[this.menu.selectedIndex] : "all";
const index = Math.max(0, keys.indexOf(key ?? "all"));
this.keys = keys;
const width = ctx.stack.getBaseSize().width;
if (!this.menu || this.width !== width) {
this.width = width;
this.menu = new MenuLayer("Calendars displayed", items, {
x: 8, y: 8, width: width - 16, opaque: true, showBorder: false,
footer: "Shared by Calendar and Glanceboard",
}).selectItem(index);
} else this.menu.setItems(items, index);
return this.menu;
}
paint(ctx: LayerContext, paintBelow: PaintBelow): GrayImage {
return this.currentMenu(ctx).paint(ctx, paintBelow);
}
async handleInput(event: InputEvent, ctx: LayerContext): Promise<void> {
await this.currentMenu(ctx).handleInput(event, ctx);
}
}
export function calendarSelectionMenuItems(): MenuItem[] {
return global.isAndroid ? [{
label: "Calendars displayed",
description: "Choose the calendars used by Calendar, Glanceboard and other calendar views.",
onSelect: ctx => ctx.stack.push(new CalendarFilterLayer()),
}] : [];
}
+2 -9
View File
@@ -9,7 +9,6 @@ import { hasCalendarPermission } from "../../native/calendar-permissions";
import { type Layer, type LayerContext } from "../../ui/layers";
import { lineStep } from "../../ui/metrics";
import type { Viewport } from "../../ui/viewport";
import { listingDayMs, spanLabel, spansDays } from "./event-days";
/** How far a row's text sits inside its selection box; the title lines up with it. */
const ROW_TEXT_INSET = 10;
@@ -144,20 +143,14 @@ export class CalendarLayer implements Layer {
function buildEventRows(font: UiFont, events: CalendarEvent[]): EventRow[] {
const rows: EventRow[] = [];
let previousDayKey = "";
const nowMs = Date.now();
for (const event of events) {
// An event still going since an earlier day is listed under today, not its first day.
const listedMs = listingDayMs(event, nowMs);
const dayKey = dayKeyOf(listedMs);
const dayHeader = dayKey === previousDayKey ? null : dayHeaderLabel(listedMs);
const dayKey = dayKeyOf(event.startMs);
const dayHeader = dayKey === previousDayKey ? null : dayHeaderLabel(event.startMs);
previousDayKey = dayKey;
const lines: string[] = [];
const timeLabel = event.allDay ? "All day" : formatEventTime(event.startMs);
lines.push(`${timeLabel} ${event.title || "(untitled)"}`);
if (spansDays(event)) {
lines.push(spanLabel(event));
}
if (event.location) {
lines.push(event.location);
}
-43
View File
@@ -1,43 +0,0 @@
/**
* Which day an event is listed under, and how a multi-day span reads. Pure (no
* native or UI imports) so it can be unit-tested.
*/
const WEEKDAYS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
export type DayEvent = { startMs: number; endMs: number; allDay: boolean };
function midnight(timestampMs: number): number {
const date = new Date(timestampMs);
return new Date(date.getFullYear(), date.getMonth(), date.getDate()).getTime();
}
/** The last moment the event covers: all-day ends are exclusive (the midnight after its last day). */
function lastMomentMs(event: DayEvent): number {
return event.allDay ? Math.max(event.startMs, event.endMs - 1) : Math.max(event.startMs, event.endMs);
}
/**
* The day to list an event under: its start, or today for one that began on an
* earlier day and is still going. Without this, the third day of a three-day
* event shows under its first day and never under today.
*/
export function listingDayMs(event: DayEvent, nowMs: number): number {
return Math.max(midnight(event.startMs), Math.min(midnight(nowMs), midnight(lastMomentMs(event))));
}
/** Whether the event covers more than one calendar day. */
export function spansDays(event: DayEvent): boolean {
return midnight(lastMomentMs(event)) > midnight(event.startMs);
}
/** "Mon 16 – Wed 18 Nov", or with both months when they differ ("Mon 30 Nov – Tue 1 Dec"). */
export function spanLabel(event: DayEvent): string {
const start = new Date(event.startMs);
const last = new Date(lastMomentMs(event));
const day = (date: Date) => `${WEEKDAYS[date.getDay()]} ${date.getDate()}`;
return start.getMonth() === last.getMonth() && start.getFullYear() === last.getFullYear()
? `${day(start)} – ${day(last)} ${MONTHS[last.getMonth()]}`
: `${day(start)} ${MONTHS[start.getMonth()]} – ${day(last)} ${MONTHS[last.getMonth()]}`;
}
+3 -5
View File
@@ -1,7 +1,7 @@
import { getDefaultMediumFont, getDefaultSmallFont } from "../../graphics/ui-fonts";
import { GrayImage } from "../../graphics/image";
import { clamp } from "../../util/numeric-util";
import { addImuListener, imuPitchDegrees, imuSourceLabel, setImuReportEnabled, type ImuReading } from "../../native/imu";
import { addImuListener, imuSourceLabel, setImuReportEnabled, type ImuReading } from "../../native/imu";
import { type InputEvent } from "../../ui/gestures";
import { Layer, type LayerContext } from "../../ui/layers";
import { shell } from "../../ui/shell/shell";
@@ -23,8 +23,7 @@ const BAR_RIGHT_INSET = 4;
/**
* Accelerometer demo: turns the IMU stream on while this page is visible (the
* Developer window is foregrounded and the screen is on), off otherwise, and
* shows the latest x/y/z reading (Faceclaw's frame, imu-types.ts) with a
* simple per-axis bar, and the head's pitch.
* shows the latest x/y/z reading with a simple per-axis bar.
*
* The IMU stream is experimental — the firmware may never populate it — so the
* page also reports how many samples have arrived.
@@ -93,8 +92,7 @@ export class AccelerometerDemoLayer implements Layer {
const bodyX = padding.left + BODY_INDENT;
image.drawText(small, padding.left, top, "Accelerometer", 220);
const pitch = this.reading ? `pitch ${Math.round(imuPitchDegrees(this.reading))}° ` : "";
const status = pitch + (this.enabled ? "IMU on" : "IMU off");
const status = this.enabled ? "IMU on" : "IMU off";
image.drawText(small, right - small.measureText(status), top, status, 150);
if (this.reading) {
-8
View File
@@ -6,7 +6,6 @@ import { BuzzerDemoLayer } from "./buzzer-demo";
import { AccelerometerDemoLayer } from "./accelerometer-demo";
import { BandwidthBenchmarkLayer } from "./bandwidth-benchmark";
import { LightSensorDemoLayer } from "./light-sensor-demo";
import { MouseTestLayer } from "./mouse-test";
import { MicrophonesMonitorLayer } from "./microphones-monitor";
import { ResourceUsageLayer } from "./resource-usage";
import { RingStatusLayer } from "./ring-status";
@@ -71,13 +70,6 @@ function debugTestsMenu(openInputEvents: OpenPage, openMicrophones: OpenPage): M
label: "Microphones",
onSelect: openMicrophones,
},
{
// The touchpad or mouse of a keyboard paired through the glasses.
label: "Mouse cursor",
onSelect: (ctx) => {
ctx.stack.push(new MouseTestLayer(DEVELOPER_WINDOW_ID, ctx.actions.requestRender));
},
},
{
label: "BLE bandwidth",
onSelect: (ctx) => {
-206
View File
@@ -1,206 +0,0 @@
import { getDefaultSmallFont } from "../../graphics/ui-fonts";
import { GrayImage } from "../../graphics/image";
import type { HidPointerSample } from "../../g2/keyboard/hid";
import { keyboardService } from "../../g2/keyboard/service";
import { type InputEvent } from "../../ui/gestures";
import { Layer, type LayerContext } from "../../ui/layers";
import { shell } from "../../ui/shell/shell";
import { clamp } from "../../util/numeric-util";
// How often to reconcile holding the pointer with window visibility.
const RECONCILE_INTERVAL_MS = 400;
/** A touchpad reports a hundred times a second; the page repaints at most this often. */
const RENDER_INTERVAL_MS = 33;
/** Cursor pixels per count the touchpad reports. */
const POINTER_GAIN = 1;
/** Left-click marks kept on the page. */
const MAX_MARKS = 12;
/** The arrow, tip at its top left: '#' is the outline, '.' the fill. */
const CURSOR = [
"#",
"##",
"#.#",
"#..#",
"#...#",
"#....#",
"#.....#",
"#......#",
"#.......#",
"#........#",
"#.....#####",
"#..#..#",
"#.# #..#",
"## #..#",
"# #..#",
" #..#",
" ##",
];
/**
* Mouse cursor test: moves an arrow with the touchpad or mouse of the
* keyboard paired through the glasses (app/g2/keyboard/), and shows its
* buttons and wheel. The main UI has no cursor; this is where one is tried
* out.
*
* While the page is visible it takes the pointer, so the buttons and wheel
* show here instead of acting as select, back and scroll. A left click
* leaves a mark where the arrow is and a right click clears them. Leave
* with Esc, or a double tap on the ring or a temple.
*/
export class MouseTestLayer implements Layer {
private x = -1;
private y = -1;
private size = { width: 0, height: 0 };
private buttons = 0;
private wheel = 0;
private marks: Array<{ x: number; y: number }> = [];
private samples = 0;
private rate = 0;
private rateWindowStart = Date.now();
private removed = false;
private release: (() => void) | null = null;
private reconcileTimer: ReturnType<typeof setInterval> | null = null;
private renderTimer: ReturnType<typeof setTimeout> | null = null;
constructor(
private readonly windowId: string,
private readonly requestRender: () => void,
) {
this.reconcileTimer = setInterval(() => this.reconcile(), RECONCILE_INTERVAL_MS);
this.reconcile();
}
/** Hold the pointer only while the page can be seen; elsewhere its buttons go back to being keys. */
private reconcile(): void {
if (this.removed) return;
const visible = shell.isWindowVisible(this.windowId);
if (visible && !this.release) {
this.release = keyboardService.takePointer((sample) => this.onSample(sample));
} else if (!visible && this.release) {
this.release();
this.release = null;
this.buttons = 0;
}
// The rate reads zero once the pointer has been still for a second.
const now = Date.now();
if (now - this.rateWindowStart >= 1000) {
const rate = this.samples;
this.samples = 0;
this.rateWindowStart = now;
if (rate !== this.rate) {
this.rate = rate;
this.scheduleRender();
}
}
}
private onSample(sample: HidPointerSample): void {
if (this.removed) return;
this.samples++;
const { width, height } = this.size;
if (width > 0 && height > 0) {
if (this.x < 0) {
this.x = width / 2;
this.y = height / 2;
}
this.x = clamp(this.x + sample.dx * POINTER_GAIN, 0, width - 1);
this.y = clamp(this.y + sample.dy * POINTER_GAIN, 0, height - 1);
}
const pressed = sample.buttons & ~this.buttons;
this.buttons = sample.buttons;
this.wheel += sample.wheel;
if (pressed & 1 && this.x >= 0) {
this.marks.push({ x: Math.round(this.x), y: Math.round(this.y) });
if (this.marks.length > MAX_MARKS) this.marks.shift();
}
if (pressed & 2) this.marks = [];
this.scheduleRender();
}
private scheduleRender(): void {
if (this.renderTimer !== null || this.removed) return;
this.renderTimer = setTimeout(() => {
this.renderTimer = null;
if (!this.removed) this.requestRender();
}, RENDER_INTERVAL_MS);
}
onRemoved(): void {
this.removed = true;
if (this.reconcileTimer !== null) clearInterval(this.reconcileTimer);
this.reconcileTimer = null;
if (this.renderTimer !== null) clearTimeout(this.renderTimer);
this.renderTimer = null;
this.release?.();
this.release = null;
}
paint(ctx: LayerContext): GrayImage {
const font = getDefaultSmallFont();
const { width, height, padding } = ctx.stack.getBaseSize();
this.size = { width, height };
const image = new GrayImage(width, height, 0);
const left = padding.left;
const right = width - padding.right;
const top = padding.top;
const line = font.lineHeight + 4;
image.drawText(font, left, top, "Mouse cursor", 220);
const snapshot = keyboardService.snapshot();
const problem = snapshot.state !== "connected"
? "No keyboard is connected."
: !snapshot.pointer ? "This keyboard has no touchpad or mouse buttons." : null;
if (problem) {
image.drawText(font, left, top + 2 * line, problem, 190);
image.drawText(font, left, top + 3 * line, "Pair one under Settings > Keyboard.", 120);
return image;
}
const position = this.x < 0 ? "move the pointer" : `${Math.round(this.x)}, ${Math.round(this.y)}`;
image.drawText(font, right - font.measureText(position), top, position, 150);
// Buttons: a box each, filled while held.
let x = left;
const boxY = top + line;
for (const [index, label] of ["L", "M", "R"].entries()) {
// HID numbers them left, right, middle; they sit left, middle, right.
const held = !!(this.buttons & [1, 4, 2][index]!);
const boxW = font.measureText(label) + 10;
if (held) image.fillRect(x, boxY, boxW, font.lineHeight + 2, 200);
else image.drawRect(x, boxY, boxW, font.lineHeight + 2, 90);
image.drawText(font, x + 5, boxY + 1, label, held ? 1 : 170);
x += boxW + 4;
}
image.drawText(font, x + 8, boxY + 1, `wheel ${this.wheel}`, 150);
const rate = `${this.rate}/s`;
image.drawText(font, right - font.measureText(rate), boxY + 1, rate, 110);
const hint = "Left click marks, right click clears. Esc or double tap: back";
image.drawText(font, left, height - padding.bottom - font.lineHeight, hint, 90);
for (const mark of this.marks) {
image.drawLine(mark.x - 4, mark.y, mark.x + 4, mark.y, 140);
image.drawLine(mark.x, mark.y - 4, mark.x, mark.y + 4, 140);
}
if (this.x >= 0) drawCursor(image, Math.round(this.x), Math.round(this.y));
return image;
}
async handleInput(event: InputEvent, ctx: LayerContext): Promise<void> {
if (event.type === "double-click") {
ctx.stack.pop();
}
}
}
function drawCursor(image: GrayImage, x: number, y: number): void {
for (const [row, pixels] of CURSOR.entries()) {
for (let col = 0; col < pixels.length; col++) {
const pixel = pixels[col];
if (pixel === " ") continue;
const px = x + col;
const py = y + row;
if (px < 0 || py < 0 || px >= image.width || py >= image.height) continue;
image.setPixel(px, py, pixel === "#" ? 255 : 70);
}
}
}
+24 -14
View File
@@ -4,7 +4,7 @@
* makes the content area exactly the 576x288 surface EvenHub apps expect.
*/
import { GrayImage } from "../../graphics/image";
import { EVENHUB_SCREEN_WIDTH } from "./compositor";
import { EVENHUB_SCREEN_WIDTH, EVENHUB_SCREEN_HEIGHT } from "./compositor";
import { directionalFallback, isDirectionalInput, type InputEvent } from "../../ui/gestures";
import { type Layer, type LayerContext } from "../../ui/layers";
import {
@@ -26,21 +26,26 @@ class EvenHubAppLayer implements Layer {
* extension's source field should see that they came from the watch.
*/
readonly acceptsDirectional = true;
private tallCanvas = false;
constructor(private readonly session: EvenHubSession) {}
setTallCanvas(tall: boolean): void {
this.tallCanvas = tall;
}
paint(ctx: LayerContext): GrayImage {
const size = ctx.stack.getBaseSize();
// EvenHub apps lay out for the stock 576-wide surface. In the Full
// Screen display mode the window is 640 wide; paint at stock width and
// centre it rather than leaving all the slack on the right.
if (size.width > EVENHUB_SCREEN_WIDTH) {
const content = this.session.paint({ width: EVENHUB_SCREEN_WIDTH, height: size.height }, ctx.stack.isFocused());
const image = new GrayImage(size.width, size.height, 0);
content.composeInto(image, Math.floor((size.width - EVENHUB_SCREEN_WIDTH) / 2), 0);
return image;
}
return this.session.paint(size, ctx.stack.isFocused());
// Host full screen hides chrome; it does not enlarge the stock app canvas.
// Only the app's explicit Faceclaw extended-layout request opts into height.
const canvas = { width: EVENHUB_SCREEN_WIDTH, height: this.tallCanvas ? size.height : EVENHUB_SCREEN_HEIGHT };
const content = this.session.paint(canvas, ctx.stack.isFocused());
if (size.width === canvas.width && size.height === canvas.height) return content;
const image = new GrayImage(size.width, size.height, 0);
// Clip text to its logical canvas before placing it in the larger host window.
image.bitBlt(content.withDrawsBaked(), Math.max(0, Math.floor((size.width - canvas.width) / 2)),
Math.max(0, Math.floor((size.height - canvas.height) / 2)));
return image;
}
handleInput(event: InputEvent): void {
@@ -62,6 +67,7 @@ export function createEvenHubWindow(
requestShellRender: () => void,
): InProcessWindow {
const fallbackIcon = windowIcon("package", session.manifest.name.charAt(0).toUpperCase() || "E");
const appLayer = new EvenHubAppLayer(session);
const created = createInProcessWindow({
appId,
windowId,
@@ -90,10 +96,11 @@ export function createEvenHubWindow(
})),
// Glasses-first: the app runs without the phone showing it; this reveals
// its phone UI (config pages, etc.) over the dashboard on demand.
{ label: "Show phone UI", onSelect: (ctx: LayerContext) => { ctx.stack.pop(); onShowPhone(); } },
...(session.manifest.packageId === "dev.guillaume.evenanchor" && session.contextMenuItems().length > 0
? [] : [{ label: "Show phone UI", onSelect: (ctx: LayerContext) => { ctx.stack.pop(); onShowPhone(); } }]),
],
actions: options.actions,
baseLayer: new EvenHubAppLayer(session),
baseLayer: appLayer,
submitFrame: options.submitFrame,
setSurfaceVisible: options.setSurfaceVisible,
removeSurface: options.removeSurface,
@@ -111,7 +118,10 @@ export function createEvenHubWindow(
pushOverlay: (layer) => created.stack.push(layer),
// The extended layout uses the full 576x452 app area ("max"); stock apps
// stay in the 576x288 band ("medium").
setTallCanvas: (tall) => created.setHeightMode(tall ? "max" : "medium"),
setTallCanvas: (tall) => {
appLayer.setTallCanvas(tall);
created.setHeightMode(tall ? "max" : "medium");
},
setAttention: (attention) => shell.setWindowAttention(windowId, attention),
// Only while the app listens for text: otherwise the shell's Type Into App
// destinations (voice dialog, phone keyboard, watch) would send into a void.
+1 -15
View File
@@ -2,7 +2,6 @@ import { readTextFile, type DirectoryEntry } from "../../native/file-access";
import { isFontFile } from "../../native/font-files";
import { installFontFile, isFontInstalled } from "../../graphics/installed-fonts";
import { isDecodableImageFile } from "../../native/image-files";
import { isMicRecordingFile } from "../../native/mic-recording";
import { isPlayableVideoFile } from "../../native/video-player";
import { readEvenHubPackageManifest } from "../evenhub/installed-apps";
import { EvenHubPermissionDialogLayer } from "../evenhub/permission-dialog";
@@ -39,8 +38,6 @@ export type FilesAppOptions = InProcessAppOptions & {
openFontWindow: (title: string, path: string) => void;
/** Open a video file's player (settings panel first) as its own shell window. */
openVideoWindow: (title: string, path: string) => void;
/** Replay a recording (.mka) through Transcribe, in a window of its own. */
openRecordingInTranscribe: (title: string, path: string) => void;
};
/**
@@ -53,7 +50,7 @@ export function createFilesAppWindow(options: FilesAppOptions): InProcessWindow
const browser = new FileBrowserLayer({
isSupportedFile: (name) =>
TEXT_FILE.test(name) || isFontFile(name) || EHPK_FILE.test(name) ||
(!global.isIOS && (isDecodableImageFile(name) || isPlayableVideoFile(name) || isMicRecordingFile(name))),
(!global.isIOS && (isDecodableImageFile(name) || isPlayableVideoFile(name))),
// The browser handles double-click itself (up a level), so it is not
// wrapped in YieldAtRootLayer; it yields explicitly from the top level.
onLeave: () => shell.yieldFocusToSidebar(),
@@ -238,17 +235,6 @@ function fileOpenActions(entry: DirectoryEntry, options: FilesAppOptions): FileI
},
];
}
if (isMicRecordingFile(entry.name)) {
return [
{
label: "Open in Transcribe",
onSelect: (ctx) => {
ctx.stack.pop();
options.openRecordingInTranscribe(entry.name, entry.path);
},
},
];
}
if (isPlayableVideoFile(entry.name)) {
return [
{
-4
View File
@@ -14,7 +14,6 @@ import {
FILES_WINDOW_ID,
} from "./files-app";
import { createVideoPlayerWindow } from "./video-window";
import { openRecordingInTranscribe } from "../transcribe";
// Document windows are closeable and non-singleton; the serial keeps their
// windowIds unique.
@@ -92,9 +91,6 @@ const filesApp: AppDefinition = {
},
openFontWindow: (title, path) => openFontDocumentWindow(ctx, title, path),
openVideoWindow: (title, path) => openVideoWindow(ctx, title, path),
openRecordingInTranscribe: (title, path) => {
openRecordingInTranscribe(ctx, path, title);
},
}),
),
// A document arriving via Android's Share intent opens as its own window.
-1
View File
@@ -305,7 +305,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
break;
case "navigation-sensors":
case "text-input":
case "key":
case "tool-call":
break;
}
-1
View File
@@ -236,7 +236,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
case "navigation-sensors":
case "screen":
case "text-input":
case "key":
case "tool-call":
break;
}
+18 -42
View File
@@ -5,9 +5,7 @@ import { enumSettingMenuItem, onAnySettingChanged, toggleSettingMenuItem } from
import { uiFontOverridePickerMenuItem } from "../../ui/font-picker";
import { type InputEvent } from "../../ui/gestures";
import { type Layer, type LayerContext } from "../../ui/layers";
import { drawSelectionHighlight, isMenuItemDisabled, menuHeadingItem, MenuLayer, openModalMenu, submenuItem, type MenuItem } from "../../ui/menu";
import { liveHeadPitchFooter } from "../../g2/head-angle";
import { headUpControlsAvailable } from "../../native/glasses-controls";
import { drawSelectionHighlight, isMenuItemDisabled, MenuLayer, openModalMenu, submenuItem, type MenuItem } from "../../ui/menu";
import { Menu } from "../../ui/menu-core";
import { LIST_ROW_TEXT_INSET, lineStep, listRowHeight } from "../../ui/metrics";
import { createInProcessWindow, type InProcessAppOptions, type InProcessWindow } from "../../ui/shell/in-process-window";
@@ -22,9 +20,7 @@ import {
glanceLayout,
glanceLayoutSetting,
glanceShowLinesSetting,
glanceAngleSetting,
glanceShowOnHeadTiltSetting,
glanceTiltTimeoutSetting,
glanceShowOnLongPressSetting,
glanceSlotChoiceLabel,
glanceSlotSettings,
@@ -323,45 +319,25 @@ function openSlotPicker(ctx: LayerContext, slotIndex: number): void {
}
/**
* Settings, in four sections: Layout (the slot grid), Trigger (the sleep
* gestures that show the board, and for how long), Display (slot lines,
* widget font, stereo depth) and Cards (a row per widget on the board, for
* its own settings; left out while the board is empty).
* Settings: layout, sleep gestures, duration, slot lines, widget font, and
* stereo depth, then a row per widget on the board for its own settings.
*/
function openGlanceSettings(ctx: LayerContext): void {
const items = (): MenuItem[] => {
const cards = widgetSettingsMenuItems();
return [
menuHeadingItem("Layout"),
// The layout decides which slots (and so which widgets) are on the board.
enumSettingMenuItem(glanceLayoutSetting, { onChange: () => layer.setItems(items()) }),
menuHeadingItem("Trigger"),
enumSettingMenuItem(glanceTapDurationSetting),
toggleSettingMenuItem(glanceShowOnLongPressSetting),
toggleSettingMenuItem(glanceShowOnHeadTiltSetting),
enumSettingMenuItem(glanceTiltTimeoutSetting),
// Only Android can change the glasses' head-up setting (glasses-controls.ts).
// The controller's settings listener sends a new angle to the glasses; the
// picker shows the head's current angle to choose by.
...(headUpControlsAvailable()
? [enumSettingMenuItem(glanceAngleSetting, {
pickerFooter: liveHeadPitchFooter(() => shell.isWindowVisible(GLANCEBOARD_WINDOW_ID)),
})]
: []),
menuHeadingItem("Display"),
toggleSettingMenuItem(glanceShowLinesSetting),
uiFontOverridePickerMenuItem(glanceFont, {
title: "Glanceboard font",
description:
"Typeface for the Glanceboard's widgets. Same as UI font follows the Font setting in Settings > Customization.",
}),
enumSettingMenuItem(glanceDepthSetting),
...(cards.length ? [menuHeadingItem("Cards"), ...cards] : []),
];
};
const items = (): MenuItem[] => [
// The layout decides which slots (and so which widgets) are on the board.
enumSettingMenuItem(glanceLayoutSetting, { onChange: () => layer.setItems(items()) }),
enumSettingMenuItem(glanceTapDurationSetting),
toggleSettingMenuItem(glanceShowOnLongPressSetting),
toggleSettingMenuItem(glanceShowOnHeadTiltSetting),
toggleSettingMenuItem(glanceShowLinesSetting),
uiFontOverridePickerMenuItem(glanceFont, {
title: "Glanceboard font",
description:
"Typeface for the Glanceboard's widgets. Same as UI font follows the Font setting in Settings > Customization.",
}),
enumSettingMenuItem(glanceDepthSetting),
...widgetSettingsMenuItems(),
];
const layer = new MenuLayer("Glanceboard settings", items(), PAGE_MENU_LAYOUT);
ctx.stack.push(layer);
}
+8 -66
View File
@@ -49,9 +49,8 @@ export function glanceLayout(): GlanceLayout {
/**
* Whether sleep-time gestures show the board at all. Off restores the plain
* behaviour: a tap or hold while asleep does nothing, a head-tilt wakes the
* regular UI, double-tap wakes; and the head-up angle goes back to the glasses'
* own.
* behaviour: a tap, hold or head-tilt while asleep does nothing (a head-tilt
* wakes the regular UI), double-tap wakes.
*/
export const glanceboardEnabledSetting = new ConfigSettingBoolean({
id: "glanceboard-enabled",
@@ -86,10 +85,9 @@ function formatTapDuration(value: GlanceTapDuration): string {
}
/**
* How long a tap keeps the board up; each further tap restarts it. "off"
* means a single tap does not show the board at all. A head-tilt showing
* lasts until the head comes back down, falling back to this duration (the
* default one when "off") where the head-down is unlikely to arrive.
* How long a tap (or head-tilt) keeps the board up; each further tap restarts
* it. "off" means a single tap does not show the board at all; a head-tilt
* then uses the default duration.
*/
export const glanceTapDurationSetting = new ConfigSettingEnum<GlanceTapDuration>({
id: "glanceboard-tap-duration",
@@ -99,7 +97,7 @@ export const glanceTapDurationSetting = new ConfigSettingEnum<GlanceTapDuration>
values: TAP_DURATION_VALUES,
formatValue: formatTapDuration,
description:
"How long a single tap keeps the Glanceboard on screen. Disabled: a single tap does not show the Glanceboard.",
"How long a single tap (or a head-tilt) keeps the Glanceboard on screen. Disabled: a single tap does not show the Glanceboard.",
});
/** Whether a single tap while asleep shows the board. */
@@ -121,71 +119,15 @@ export const glanceShowOnLongPressSetting = new ConfigSettingBoolean({
description: "A long-press or short-then-long-press while asleep shows the Glanceboard until the press is released.",
});
/** The head-tilt wake shows the board (until the head comes back down) instead of the regular UI. */
/** The head-tilt wake shows the board (for the tap duration) instead of the regular UI. */
export const glanceShowOnHeadTiltSetting = new ConfigSettingBoolean({
id: "glanceboard-show-on-head-tilt",
label: "Show on head tilt",
storageKey: "glanceboard.showOnHeadTilt",
defaultValue: true,
description: "Tilting your head up while asleep shows the Glanceboard instead of waking the regular UI. It stays up until you look back down.",
description: "Tilting your head up while asleep shows the Glanceboard instead of waking the regular UI.",
});
export type GlanceTiltTimeout = "60s" | "30s" | "15s" | "10s" | "never";
/**
* How long a head-tilt showing may last if the head-down never comes (the
* board normally hides when the head comes back down). "never" removes the
* limit; a double-tap still dismisses the board.
*/
export const glanceTiltTimeoutSetting = new ConfigSettingEnum<GlanceTiltTimeout>({
id: "glanceboard-head-tilt-timeout",
label: "Head tilt timeout",
storageKey: "glanceboard.headTiltTimeout",
defaultValue: "60s",
values: ["60s", "30s", "15s", "10s", "never"],
formatValue: (value) => (value === "never" ? "Never" : value.replace("s", " s")),
description:
"The longest a head-tilt keeps the Glanceboard up when looking back down is not detected. Never: it stays until you look down or double-tap. " +
"When looking down is unlikely to register (a low Head tilt angle, or the display was suspended), the tap duration applies if shorter.",
});
/** The Head tilt timeout in ms, or null for none. */
export function glanceTiltTimeoutMs(): number | null {
const value = glanceTiltTimeoutSetting.get();
return value === "never" ? null : Number.parseInt(value, 10) * 1000;
}
const GLANCE_ANGLE_VALUES = ["glasses", "5", "10", "15", "20", "25", "30", "35", "40", "45", "50", "55", "60"] as const;
export type GlanceAngle = (typeof GLANCE_ANGLE_VALUES)[number];
/**
* How far the head must tilt up for a head-tilt wake, as the glasses' own
* pitch reading (degrees). "glasses" keeps the glasses' own setting (from the
* Even app), writing back their original values if Faceclaw changed them. A
* number is written to the glasses' head-up setting on every connection while
* the Glanceboard is enabled (head-angle.ts). Android only.
*/
export const glanceAngleSetting = new ConfigSettingEnum<GlanceAngle>({
id: "glanceboard-head-tilt-angle",
label: "Head tilt angle",
storageKey: "glanceboard.headTiltAngle",
defaultValue: "glasses",
values: GLANCE_ANGLE_VALUES,
formatValue: (value) => (value === "glasses" ? "Glasses default" : `${value}°`),
description:
"How far you tilt your head up to wake the display; the list shows your head's angle now. Glasses default uses the glasses' own setting from the Even app. " +
"The board hides when your head drops 20° below this angle; under 25°, looking straight ahead may not count, so it hides after the tap duration instead.",
});
/**
* The trigger Faceclaw should maintain, in degrees: the Head tilt angle while
* the Glanceboard is enabled, else 0 (the glasses' own setting).
*/
export function glanceAngleTriggerDegrees(): number {
const value = glanceAngleSetting.get();
return value === "glasses" || !glanceboardEnabledSetting.get() ? 0 : Number(value);
}
const DEPTH_VALUES = ["-64", "-48", "-32", "-16", "0", "16", "32", "48", "64"] as const;
export type GlanceDepth = (typeof DEPTH_VALUES)[number];
-2
View File
@@ -9,7 +9,6 @@ import {
glanceShowOnLongPressSetting,
glanceShowOnTap,
glanceTapTimeoutMs,
glanceTiltTimeoutMs,
} from "./glanceboard-settings";
const glanceboardApp: AppDefinition = {
@@ -25,7 +24,6 @@ const glanceboardApp: AppDefinition = {
isEnabled: () => glanceboardEnabledSetting.get(),
showOnTap: glanceShowOnTap,
tapTimeoutMs: glanceTapTimeoutMs,
tiltTimeoutMs: glanceTiltTimeoutMs,
showOnLongPress: () => glanceShowOnLongPressSetting.get(),
showOnHeadTilt: () => glanceShowOnHeadTiltSetting.get(),
depth: glanceDepth,
+2 -1
View File
@@ -1,5 +1,6 @@
import { type GlanceWidgetDefinition, type GlanceWidgetId } from "../widget";
import { CalendarWidget } from "./calendar-widget";
import { calendarSelectionMenuItems } from "../../calendar/calendar-filter";
import { CompassWidget } from "./compass-widget";
import { MusicWidget, musicSettingsMenuItems } from "./music-widget";
import { NightscoutWidget } from "./nightscout-widget";
@@ -11,7 +12,7 @@ import { HealthWidget } from "./health-widget";
/** Every widget a board slot can hold. */
export const GLANCE_WIDGETS: readonly GlanceWidgetDefinition[] = [
{ id: "system-card", label: "System card", settingsMenuItems: systemCardSettingsMenuItems, create: () => new SystemCardWidget() },
{ id: "calendar", label: "Calendar", tall: true, create: () => new CalendarWidget() },
{ id: "calendar", label: "Calendar", tall: true, settingsMenuItems: calendarSelectionMenuItems, create: () => new CalendarWidget() },
{ id: "terminal", label: "Terminal", tall: true, create: () => new TerminalWidget() },
{ id: "t3code", label: "T3 Code", tall: true, create: () => new T3CodeWidget() },
{ id: "nightscout", label: "Nightscout", create: () => new NightscoutWidget() },
+5 -15
View File
@@ -50,26 +50,16 @@ export const musicShowAlbumArtSetting = new ConfigSettingBoolean({
"Show the album art beside the track details. When off, the title, artist, album and progress bar use the full width of the card.",
});
export const musicShowIdleNoteSetting = new ConfigSettingBoolean({
id: "glanceboard-music-show-idle-note",
label: "Show note when idle",
storageKey: "glanceboard.music.showIdleNote",
defaultValue: true,
description: "Show a dim music note on the card while nothing is playing. When off, the card is left blank.",
});
/** The music card's submenu in Glanceboard > Settings. */
export function musicSettingsMenuItems(): MenuItem[] {
return [toggleSettingMenuItem(musicShowAlbumArtSetting), toggleSettingMenuItem(musicShowIdleNoteSetting)];
return [toggleSettingMenuItem(musicShowAlbumArtSetting)];
}
/**
* Now playing: album art (unless turned off), title/artist/album and a
* progress bar for the active media session (Apple Music on iOS), the Music
* app's header at card size. With nothing playing the card holds a dim note,
* or nothing with that turned off. The media bridge runs for the whole
* session; the widget subscribes and ticks once a second only while
* something is playing.
* app's header at card size. The media bridge runs for the whole session; the
* widget subscribes and ticks once a second only while something is playing.
*/
export class MusicWidget implements GlanceWidget {
private media: MediaControllerState = mediaControllerBridge.snapshot();
@@ -118,8 +108,8 @@ export class MusicWidget implements GlanceWidget {
image.drawText(small, PAD, PAD + small.lineHeight + 6, mediaControllerBridge.accessLabel, 140);
return;
}
// Nothing playing: a dim note, centred, and nothing else; or a blank card.
const note = musicShowIdleNoteSetting.get() ? idleNoteImage() : null;
// Nothing playing: a dim note, centred, and nothing else.
const note = idleNoteImage();
if (note) image.drawImage(note, ((image.width - note.width) / 2) | 0, ((image.height - note.height) / 2) | 0);
return;
}
+136 -26
View File
@@ -8,7 +8,11 @@ import { IconGrid, ICON_GRID_ICON_SIZE } from "../../ui/icon-grid";
import { Layer, LayerActions, LayerContext } from "../../ui/layers";
import { MenuLayer, type MenuItem } from "../../ui/menu";
import { WINDOW_MENU_LAYOUT } from "../../ui/window-menu";
import { onAnySettingChanged } from "../../ui/dashboard-settings";
import {
onAnySettingChanged, launcherLayoutSetting, enumSettingMenuItem,
ConfigSettingString, EditTextSettingLayer,
} from "../../ui/dashboard-settings";
import { getLauncherLayoutFingerprint, orderLauncherEntries, moveLauncherEntry } from "./launcher-layout";
import { createInProcessWindow } from "../../ui/shell/in-process-window";
import {
getFolderAssignments,
@@ -16,6 +20,7 @@ import {
getFolderStateFingerprint,
setAppFolder,
unusedNewFolderName,
getExplicitFolderNames, createFolder, renameFolder, folderNameError, disbandFolder,
} from "./launcher-folders";
import { shell, type ShellWindow } from "../../ui/shell/shell";
@@ -63,6 +68,14 @@ type LauncherGridEntry =
* and is re-read every paint, so assistant folder tools take effect without
* the layer holding any copy of the state.
*/
const entryKey = (entry: LauncherGridEntry) => entry.kind === "app" ? `app:${entry.appId}` : `folder:${entry.name}`;
/** Reuse the phone text editor; dismiss it if the glasses leave this page. */
class FolderNameEditorLayer extends EditTextSettingLayer {
constructor(setting: ConfigSettingString, private readonly actions: LayerActions) { super(setting); }
onRemoved(): void { void this.actions.endTextSettingEdit(); }
}
class LauncherGridLayer implements Layer {
// Watch swipes are spatial: up/down move between rows, left/right between
// columns. From the leftmost column, left keeps going out: it leaves the
@@ -71,6 +84,12 @@ class LauncherGridLayer implements Layer {
readonly acceptsDirectional = true;
/** Folder whose contents the grid is showing, or null for the top grid. */
private currentFolder: string | null = null;
private editingFolder = false;
private moving: { key: string; label: string; folder: string | null } | null = null;
private sortEntries(entries: LauncherGridEntry[]): LauncherGridEntry[] {
return orderLauncherEntries(entries, entryKey, this.currentFolder, launcherLayoutSetting.get() === "custom");
}
private readonly grid = new IconGrid<LauncherGridEntry>({
items: () => this.entries(),
describe: (entry) => ({
@@ -101,25 +120,23 @@ class LauncherGridLayer implements Layer {
const members = (getFolders().get(this.currentFolder) ?? [])
.map((appId) => byAppId.get(appId))
.filter(Boolean) as LauncherAppEntry[];
if (members.length > 0) {
return members
.map((app): LauncherGridEntry => ({
if (members.length > 0 || getExplicitFolderNames().includes(this.currentFolder)) {
return this.sortEntries(members.map((app): LauncherGridEntry => ({
kind: "app",
label: app.label,
icon: app.icon,
appId: app.appId,
renderIcon: app.renderIcon,
}))
.sort((a, b) => a.label.localeCompare(b.label));
})));
}
this.currentFolder = null;
}
const assignments = getFolderAssignments();
const entries: LauncherGridEntry[] = [];
// A folder cell appears only when it holds at least one known app; stale
// assignments to removed apps are ignored.
// Legacy folders need a known member; explicitly created folders may be empty.
const explicit = new Set(getExplicitFolderNames());
for (const [name, members] of getFolders()) {
if (members.some((appId) => byAppId.has(appId))) {
if (explicit.has(name) || members.some((appId) => byAppId.has(appId))) {
entries.push({ kind: "folder", label: name, name });
}
}
@@ -134,7 +151,7 @@ class LauncherGridLayer implements Layer {
});
}
}
return entries.sort((a, b) => a.label.localeCompare(b.label));
return this.sortEntries(entries);
}
/** Leave the current folder, putting the selection back on its cell. */
@@ -148,6 +165,16 @@ class LauncherGridLayer implements Layer {
/** Enter a folder or launch an app; true when that left the grid (a launch). */
private async openEntry(entry: LauncherGridEntry): Promise<boolean> {
if (this.moving) {
const moving = this.moving;
this.moving = null;
if (launcherLayoutSetting.get() === "custom" && moving.folder === this.currentFolder) {
moveLauncherEntry(this.currentFolder, this.entries().map(entryKey), moving.key, entryKey(entry));
const index = this.entries().findIndex(item => entryKey(item) === moving.key);
if (index >= 0) this.grid.selectIndex(index);
}
return false;
}
if (entry.kind === "folder") {
this.currentFolder = entry.name;
this.grid.resetSelection();
@@ -163,6 +190,13 @@ class LauncherGridLayer implements Layer {
/** Leave the open folder, else yield to the sidebar; true when that left the grid. */
private back(): boolean {
if (this.moving) {
const key = this.moving.key;
this.moving = null;
const index = this.entries().findIndex(entry => entryKey(entry) === key);
if (index >= 0) this.grid.selectIndex(index);
return false;
}
if (this.currentFolder !== null) {
this.exitFolder();
return false;
@@ -172,14 +206,54 @@ class LauncherGridLayer implements Layer {
}
/**
* App-specific entries for the window's context menu. Only item mode
* pins the selection to a single cell, and only apps can move to folders,
* so anywhere else the menu falls back to just the defaults.
* Layout and folder creation are always available. Per-icon actions need
* item mode so row navigation never edits an ambiguous selection.
*/
menuItems(): MenuItem[] {
if (this.editingFolder) return [];
const entry = this.grid.selectedItem;
if (!entry || entry.kind !== "app") return [];
return [
const items: MenuItem[] = [
enumSettingMenuItem(launcherLayoutSetting, { onChange: () => { this.moving = null; this.grid.resetSelection(); } }),
{
label: "New folder",
onSelect: ctx => {
const name = unusedNewFolderName();
if (createFolder(name)) {
this.currentFolder = null;
this.moving = null;
this.grid.resetSelection();
this.grid.selectIndex(this.entries().findIndex(item => item.kind === "folder" && item.name === name));
}
ctx.stack.pop();
},
},
];
const folder = entry?.kind === "folder" ? entry.name : this.currentFolder;
if (folder !== null) {
items.push({ label: "Rename folder", onSelect: ctx => this.editFolderName(folder, ctx) }, {
label: "Delete folder",
onSelect: ctx => {
ctx.stack.pop();
ctx.stack.push(new MenuLayer("Delete folder? Apps stay installed.", [
{ label: "Delete folder", onSelect: confirm => {
disbandFolder(folder);
if (this.currentFolder === folder) this.currentFolder = null;
this.moving = null;
this.grid.resetSelection();
confirm.stack.pop();
} },
{ label: "Cancel", onSelect: confirm => confirm.stack.pop() },
], WINDOW_MENU_LAYOUT));
},
});
}
if (this.moving) items.push({ label: "Cancel move", onSelect: ctx => { this.back(); ctx.stack.pop(); } });
else if (entry && launcherLayoutSetting.get() === "custom") items.push({
label: "Move icon",
onSelect: ctx => { this.moving = { key: entryKey(entry), label: entry.label, folder: this.currentFolder }; ctx.stack.pop(); },
});
if (!entry || entry.kind !== "app") return items;
return [...items,
{
label: "Move to folder",
onSelect: (ctx) => {
@@ -218,6 +292,32 @@ class LauncherGridLayer implements Layer {
];
}
private editFolderName(folder: string, ctx: LayerContext): void {
ctx.stack.pop();
const setting = new ConfigSettingString({
id: "launcher-folder-name", label: "Folder name", storageKey: "launcher.folderNameDraft",
defaultValue: folder, normalize: value => value ?? "",
validate: value => folderNameError(value, folder), reportInAnalytics: false,
});
setting.set(folder);
this.editingFolder = true;
const editor = new FolderNameEditorLayer(setting, ctx.actions);
ctx.stack.push(editor);
void ctx.actions.startTextSettingsEdit([setting], "Rename folder", () => {
this.editingFolder = false;
if (renameFolder(folder, setting.get())) {
if (this.currentFolder === folder) this.currentFolder = setting.get().trim();
this.moving = null;
}
ctx.stack.popIfTop(layer => layer === editor);
ctx.actions.requestRender();
}, undefined, () => {
this.editingFolder = false;
ctx.stack.popIfTop(layer => layer === editor);
ctx.actions.requestRender();
});
}
/**
* The folder-picker submenu: every existing folder except the app's own,
* plus a generated-name "New folder" (ring input has no text entry) and,
@@ -239,7 +339,9 @@ class LauncherGridLayer implements Layer {
items.push({
label: "New folder",
onSelect: (ctx) => {
setAppFolder(appId, unusedNewFolderName());
const name = unusedNewFolderName();
createFolder(name);
setAppFolder(appId, name);
ctx.stack.pop();
},
});
@@ -259,18 +361,25 @@ class LauncherGridLayer implements Layer {
const font = getDefaultSmallFont();
const { width, height, padding } = ctx.stack.getBaseSize();
const image = new GrayImage(width, height, 0);
const entries = this.entries();
if (this.moving && (launcherLayoutSetting.get() !== "custom" || this.moving.folder !== this.currentFolder ||
!entries.some(entry => entryKey(entry) === this.moving!.key))) this.moving = null;
const gridX = padding.left;
const gridWidth = width - padding.left - padding.right;
// The grid scrolls, so it runs to the window's top and bottom edges, the
// padding suggested for them inside it: its end rows keep clear of a
// border there, and a row cut off for scrolling meets the border rather
// than a blank strip. A folder's name stays put above its grid instead,
// in a band that scales with the font.
// Scrolling rows reach the window edges, with the suggested padding
// inside the grid. Folder and move headers stay fixed above it.
let gridTop = 0;
let padTop = padding.top;
if (this.currentFolder !== null) {
const nameX = gridX + FOLDER_NAME_INSET_X;
image.drawText(font, nameX, padding.top, truncateText(font, this.currentFolder, gridWidth - 2 * FOLDER_NAME_INSET_X), 160);
const nameX = gridX + FOLDER_NAME_INSET_X;
const nameWidth = gridWidth - 2 * FOLDER_NAME_INSET_X;
if (this.moving) {
image.drawText(font, nameX, padding.top, truncateText(font, `Move ${this.moving.label}`, nameWidth), 220);
image.drawText(font, nameX, padding.top + font.lineHeight + 2,
truncateText(font, "Select position; double tap cancels", nameWidth), 160);
gridTop = padding.top + 2 * (font.lineHeight + 2) + FOLDER_NAME_GAP;
padTop = 0;
} else if (this.currentFolder !== null) {
image.drawText(font, nameX, padding.top, truncateText(font, this.currentFolder, nameWidth), 160);
gridTop = padding.top + font.lineHeight + FOLDER_NAME_GAP;
padTop = 0;
}
@@ -280,6 +389,7 @@ class LauncherGridLayer implements Layer {
}
async handleInput(event: InputEvent, ctx: LayerContext): Promise<void> {
if (this.moving && event.type === "double-click") { this.back(); return; }
await this.grid.handleInput(event, ctx);
}
@@ -317,9 +427,9 @@ export function createLauncherWindow(options: LauncherOptions): ShellWindow {
// keeps every unrelated setting change from repainting the launcher. The
// launcher is pinned for the app's lifetime, so the subscription never needs
// tearing down.
let lastState = `${getFolderStateFingerprint()}\n${launcherAppsFingerprint(options.apps())}`;
let lastState = `${getFolderStateFingerprint()}\n${getLauncherLayoutFingerprint()}\n${launcherAppsFingerprint(options.apps())}`;
onAnySettingChanged(() => {
const state = `${getFolderStateFingerprint()}\n${launcherAppsFingerprint(options.apps())}`;
const state = `${getFolderStateFingerprint()}\n${getLauncherLayoutFingerprint()}\n${launcherAppsFingerprint(options.apps())}`;
if (state === lastState) return;
lastState = state;
created.requestRender();
+51 -11
View File
@@ -1,14 +1,50 @@
/**
* Launcher folder state: which apps are grouped into which named folder.
* Stored as one JSON string (appId -> folder name) in the settings store, so
* it persists and its changes broadcast to every isolate. An unset value means
* the default grouping; any edit persists the whole map, so clearing out the
* defaults sticks. Folders exist only while at least one app is assigned to
* them — there is no separate create/delete step.
*/
/** Launcher assignments keep their legacy format; explicitly created folders may be empty. */
import { getStringSetting, setStringSetting } from "../../native/settings-store";
import { renameLauncherFolderOrder, removeLauncherFolderOrder } from "./launcher-layout";
const STORAGE_KEY = "launcher.folders";
const NAMES_KEY = "launcher.folderNames";
/** Only user-created/renamed folders persist after their last member leaves. */
export function getExplicitFolderNames(): string[] {
try {
const value = JSON.parse(getStringSetting(NAMES_KEY, "[]"));
return Array.isArray(value) ? [...new Set(value.filter((name): name is string =>
typeof name === "string" && !!name.trim()).map(name => name.trim()))] : [];
} catch { return []; }
}
export function folderNameError(input: string, previous?: string): string | null {
const name = input.trim();
if (!name) return "Enter a folder name.";
if (name.length > FOLDER_NAME_MAX_LENGTH) return `Use at most ${FOLDER_NAME_MAX_LENGTH} characters.`;
if (/[\u0000-\u001f\u007f]/.test(name)) return "Use a single-line folder name.";
if (previous !== undefined && !getFolders().has(previous)) return "This folder no longer exists.";
for (const existing of getFolders().keys()) {
if (existing !== previous && existing.toLowerCase() === name.toLowerCase()) return "A folder with this name already exists.";
}
return null;
}
export function createFolder(input: string): boolean {
if (folderNameError(input)) return false;
setStringSetting(NAMES_KEY, JSON.stringify([...getExplicitFolderNames(), input.trim()]));
return true;
}
export function renameFolder(previous: string, input: string): boolean {
if (folderNameError(input, previous)) return false;
const name = input.trim();
if (name === previous) return true;
const assignments = getFolderAssignments();
for (const appId of Object.keys(assignments)) if (assignments[appId] === previous) assignments[appId] = name;
// Preserve position and per-folder order when the visible name changes.
renameLauncherFolderOrder(previous, name);
setStringSetting(NAMES_KEY, JSON.stringify([...getExplicitFolderNames().filter(n => n !== previous), name]));
setStringSetting(STORAGE_KEY, JSON.stringify(assignments));
return true;
}
/** The out-of-the-box grouping, applied only while the setting is unset. */
const DEFAULT_ASSIGNMENTS: Record<string, string> = {
@@ -32,7 +68,7 @@ export function getFolderAssignments(): Record<string, string> {
try {
const parsed = JSON.parse(raw);
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return {};
const assignments: Record<string, string> = {};
const assignments: Record<string, string> = Object.create(null);
for (const appId of Object.keys(parsed)) {
const folder = parsed[appId];
if (typeof folder === "string" && folder.trim()) {
@@ -58,6 +94,7 @@ export function getFolders(): Map<string, string[]> {
unsorted.set(folder, [appId]);
}
}
for (const name of getExplicitFolderNames()) if (!unsorted.has(name)) unsorted.set(name, []);
const folders = new Map<string, string[]>();
for (const name of Array.from(unsorted.keys()).sort((a, b) => a.localeCompare(b))) {
folders.set(name, unsorted.get(name)!);
@@ -86,6 +123,8 @@ export function disbandFolder(folder: string): number {
moved++;
}
}
setStringSetting(NAMES_KEY, JSON.stringify(getExplicitFolderNames().filter(name => name !== folder)));
removeLauncherFolderOrder(folder);
if (moved > 0) {
setStringSetting(STORAGE_KEY, JSON.stringify(assignments));
}
@@ -95,7 +134,7 @@ export function disbandFolder(folder: string): number {
/**
* An unused name for a ring-created folder ("New folder", "New folder 2", …).
* Ring input has no text entry, so new folders start with a generated name;
* the assistant's folder tools can regroup the apps under a better one.
* Rename folder uses the existing phone text editor for a custom name.
*/
export function unusedNewFolderName(): string {
const existing = new Set(Array.from(getFolders().keys()).map((name) => name.toLowerCase()));
@@ -119,5 +158,6 @@ export function resolveFolderName(input: string): string {
/** Raw stored value, for cheap change detection by the launcher window. */
export function getFolderStateFingerprint(): string {
return getStringSetting(STORAGE_KEY, "");
return JSON.stringify([getStringSetting(STORAGE_KEY, ""), getStringSetting(NAMES_KEY, ""),
getStringSetting("launcher.folderNameDraft", "")]);
}
+60
View File
@@ -0,0 +1,60 @@
import { getStringSetting, setStringSetting } from "../../native/settings-store";
const ORDER_KEY = "launcher.order";
const scopeKey = (folder: string | null) => folder === null ? "root" : `folder:${folder}`;
function readOrders(): Record<string, string[]> {
try {
const value = JSON.parse(getStringSetting(ORDER_KEY, "{}"));
if (!value || typeof value !== "object" || Array.isArray(value)) return {};
return Object.fromEntries(Object.entries(value).filter(([, order]) => Array.isArray(order))
.map(([scope, order]) => [scope, [...new Set((order as unknown[]).filter((id): id is string => typeof id === "string"))]]));
} catch { return {}; }
}
export function setLauncherOrder(folder: string | null, ids: readonly string[]): void {
const orders = readOrders();
orders[scopeKey(folder)] = [...new Set(ids)];
setStringSetting(ORDER_KEY, JSON.stringify(orders));
}
/** Unseen apps append alphabetically; changing sort modes never destroys saved order. */
export function orderLauncherEntries<T extends { label: string }>(
entries: T[], key: (entry: T) => string, folder: string | null, custom: boolean,
): T[] {
const order = custom ? readOrders()[scopeKey(folder)] ?? [] : [];
const rank = new Map(order.map((id, index) => [id, index]));
return entries.sort((a, b) => (rank.get(key(a)) ?? Infinity) - (rank.get(key(b)) ?? Infinity) ||
a.label.localeCompare(b.label) || key(a).localeCompare(key(b)));
}
/** Move into the target's position, without moving apps across folders. */
export function moveLauncherEntry(folder: string | null, ids: string[], source: string, target: string): boolean {
const from = ids.indexOf(source), to = ids.indexOf(target);
if (from < 0 || to < 0) return false;
const next = ids.slice();
next.splice(from, 1);
next.splice(to, 0, source);
setLauncherOrder(folder, next);
return true;
}
export function renameLauncherFolderOrder(previous: string, next: string): void {
const orders = readOrders();
const oldScope = scopeKey(previous), newScope = scopeKey(next);
if (orders[oldScope]) orders[newScope] = orders[oldScope];
delete orders[oldScope];
if (orders.root) orders.root = orders.root.map(id => id === `folder:${previous}` ? `folder:${next}` : id);
setStringSetting(ORDER_KEY, JSON.stringify(orders));
}
export function removeLauncherFolderOrder(folder: string): void {
const orders = readOrders();
delete orders[scopeKey(folder)];
if (orders.root) orders.root = orders.root.filter(id => id !== `folder:${folder}`);
setStringSetting(ORDER_KEY, JSON.stringify(orders));
}
export function getLauncherLayoutFingerprint(): string {
return JSON.stringify([getStringSetting("launcher.layout", "alphabetical"), getStringSetting(ORDER_KEY, "")]);
}
-21
View File
@@ -56,27 +56,6 @@ export function saveMicConfig(config: MicConfig): void {
setStringSetting(CONFIG_SETTING_KEY, JSON.stringify(config));
}
/**
* The configuration a capture of the temples' own microphones arms them
* with: `mics` of 2 is one microphone per temple (the front one), 4 is both
* of each. The front end and direction detection follow Microphone setup
* (the Microphones app), except that both of a temple's microphones only
* come through the codec front end. Always raw PCM16 at 16 kHz, which is
* what the firmware streams and transcription takes.
*/
export function micArrayCaptureConfig(mics: 2 | 4): MicConfig {
const base = loadMicConfig();
return {
...base,
source: mics === 4 ? "codec" : base.source,
channelMask: mics === 4 ? 0b11 : 0b01,
codec: "raw",
format: "pcm16",
sampleRateHz: 16_000,
armHardware: true,
};
}
class MicArrayController {
private readonly statusListeners = new Set<(statuses: MicArrayStatuses) => void>();
private statuses: MicArrayStatuses = { left: null, right: null };
@@ -272,7 +272,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
}
case "navigation-sensors":
case "text-input":
case "key":
case "tool-call":
break;
}
-3
View File
@@ -370,9 +370,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
frameTimings.logFrame(message.frameId, `input received in ${message.windowId} worker`);
handleInput(window, message.event as InputEvent, message.frameId);
break;
case "key":
// Only windows opened with acceptsKeys are sent keys.
break;
case "text-input":
if (editing) {
// Voice input as an alternative to the phone keyboard.
+10 -1
View File
@@ -1,6 +1,6 @@
import { NotificationsListLayer } from "../../ui/notifications";
import { NOTIFICATIONS_SETTINGS_SECTION } from "../../ui/dashboard/settings-menus";
import { onAndroidNotificationsChanged } from "../../native/notification-icons";
import { ALL_NOTIFICATIONS, dismissAllNotifications, onAndroidNotificationsChanged, readActiveNotifications } from "../../native/notification-icons";
import {
createInProcessWindow,
YieldAtRootLayer,
@@ -36,6 +36,15 @@ export function createNotificationsAppWindow(
ctx.stack.pop();
openSettings(NOTIFICATIONS_SETTINGS_SECTION);
},
}, {
label: "Dismiss all notifications",
description: "Dismiss notifications on the phone and glasses. System-protected notifications may remain.",
disabled: () => readActiveNotifications(ALL_NOTIFICATIONS).length === 0,
onSelect: (ctx) => {
ctx.stack.pop();
dismissAllNotifications();
created.requestRender();
},
}],
actions: options.actions,
baseLayer: new YieldAtRootLayer(new NotificationsListLayer()),
-1
View File
@@ -433,7 +433,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
}
case "navigation-sensors":
case "text-input":
case "key":
case "tool-call":
break;
}
-3
View File
@@ -227,9 +227,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
frameTimings.logFrame(message.frameId, `input received in ${message.windowId} worker`);
handleInput(window, message.event as InputEvent, message.frameId);
break;
case "key":
// Only windows opened with acceptsKeys are sent keys.
break;
case "text-input":
if (editingSetting) {
// Voice input as an alternative to the phone keyboard.
-3
View File
@@ -424,9 +424,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
frameTimings.logFrame(message.frameId, `input received in ${message.windowId} worker`);
handleInput(window, message.event as InputEvent, message.frameId);
break;
case "key":
// Only windows opened with acceptsKeys are sent keys.
break;
case "text-input":
if (window && window.windowId === message.windowId) handleTextInput(message.text.trim());
break;
+7 -89
View File
@@ -1,21 +1,13 @@
import { getStringSetting, setStringSetting } from "../../native/settings-store";
/**
* Terminal-app connection registry: the hosts the app knows about, stored as
* JSON under one settings key so every isolate sees the same list and change
* notifications ride the ordinary settings-store channel.
* Terminal-app connection registry: the g2mirror hosts the app knows about,
* stored as JSON under one settings key so every isolate sees the same list
* and change notifications ride the ordinary settings-store channel.
*
* A connection is entered as a connection string:
* - g2mirror://<token>@<host>[:port]: a g2mirror server reached directly
* (the same form g2mirror-view accepts) over a plain websocket (default
* port 8737); g2mirrors://... for TLS (default 443).
* - ssh://<user>@<host>[:port]: tmux on an SSH host (default port 22).
* - ssh://<user>@<host>[:port]?g2mirror=<token>[@<host>[:port]]: a g2mirror
* server tunnelled over SSH; the address after the token is where the
* server listens as seen from the SSH host (default 127.0.0.1:8737).
* SSH connections pin the host's key (hostKeys, trusted on first connection
* after the user checks its fingerprint); their passwords live in the
* platform keystore (app/native/ssh), not here.
* A connection is entered as a g2mirror:// connection string (the same form
* g2mirror-view accepts): g2mirror://<token>@<host>[:port] for plain
* websockets (default port 8737), g2mirrors://... for TLS (default 443).
*/
export const TERMINAL_CONNECTIONS_KEY = "terminal.connections";
@@ -33,8 +25,6 @@ export type TerminalConnection = {
serverName: string | null;
/** Auto-connect while the Terminal app is open. Toggled by Connect/Disconnect. */
enabled: boolean;
/** SSH connections: trusted host keys (SSH wire format, base64). */
hostKeys: string[];
};
export type ParsedConnectionString = {
@@ -64,79 +54,8 @@ export function parseConnectionString(url: string): ParsedConnectionString | nul
return { secure, host, port, authToken };
}
export type ParsedSshConnection = {
user: string;
host: string;
port: number;
} & (
| { mode: "tmux" }
| {
mode: "g2mirror";
authToken: string;
/** Where g2mirror-server listens, from the SSH host's point of view. */
targetHost: string;
targetPort: number;
}
);
function decodeComponent(text: string): string {
try {
return decodeURIComponent(text);
} catch {
return text;
}
}
/** Parse an ssh://user@host[:port][?g2mirror=token[@host[:port]]] string; null if malformed. */
export function parseSshConnectionString(url: string): ParsedSshConnection | null {
const match = /^ssh:\/\/([^@\s/?]+)@(\[[^\]\s]+\]|[^:/@\s?]+)(?::(\d+))?\/?(?:\?(.*))?$/.exec(url.trim());
if (!match) return null;
const user = decodeComponent(match[1]!);
const host = match[2]!.replace(/^\[(.*)\]$/, "$1");
const port = match[3] ? parseInt(match[3], 10) : 22;
if (!user || !(port >= 1 && port <= 65535)) return null;
const params = new Map<string, string>();
for (const part of (match[4] ?? "").split("&").filter(Boolean)) {
const equals = part.indexOf("=");
if (equals < 0) return null;
params.set(part.slice(0, equals), decodeComponent(part.slice(equals + 1)));
}
for (const key of params.keys()) {
if (key !== "g2mirror") return null;
}
const g2mirror = params.get("g2mirror");
if (g2mirror === undefined) return { user, host, port, mode: "tmux" };
// token[@host[:port]], split at the last "@" so the token may contain one.
const at = g2mirror.lastIndexOf("@");
const authToken = at < 0 ? g2mirror : g2mirror.slice(0, at);
const target = at < 0 ? "" : g2mirror.slice(at + 1);
const targetMatch = /^(\[[^\]\s]+\]|[^:\s]*)(?::(\d+))?$/.exec(target);
if (!authToken || !targetMatch) return null;
const targetHost = targetMatch[1] || "127.0.0.1";
const targetPort = targetMatch[2] ? parseInt(targetMatch[2], 10) : 8737;
if (!(targetPort >= 1 && targetPort <= 65535)) return null;
return { user, host, port, mode: "g2mirror", authToken, targetHost, targetPort };
}
export type ParsedAnyConnection =
| ({ kind: "g2mirror" } & ParsedConnectionString)
| ({ kind: "ssh" } & ParsedSshConnection);
/** Parse any supported connection string; null if it's none of them. */
export function parseAnyConnectionString(url: string): ParsedAnyConnection | null {
const direct = parseConnectionString(url);
if (direct) return { kind: "g2mirror", ...direct };
const ssh = parseSshConnectionString(url);
return ssh ? { kind: "ssh", ...ssh } : null;
}
/** The [user@]host[:port] part of a connection string (never a token); "" if it doesn't parse. */
/** The host[:port] part of a connection string; "" if it doesn't parse. */
export function connectionStringHostLabel(url: string): string {
const ssh = parseSshConnectionString(url);
if (ssh) {
const host = ssh.host.includes(":") ? `[${ssh.host}]` : ssh.host;
return `${ssh.user}@${host}${ssh.port === 22 ? "" : `:${ssh.port}`}`;
}
const parsed = parseConnectionString(url);
if (!parsed) return "";
const defaultPort = parsed.secure ? 443 : 8737;
@@ -160,7 +79,6 @@ export function loadConnections(): TerminalConnection[] {
url: String(item?.url ?? ""),
serverName: typeof item?.serverName === "string" && item.serverName ? item.serverName : null,
enabled: item?.enabled !== false,
hostKeys: Array.isArray(item?.hostKeys) ? item.hostKeys.filter((key: unknown) => typeof key === "string") : [],
}))
.filter((connection) => connection.id.length > 0 && connection.url.length > 0);
} catch {
-2
View File
@@ -14,8 +14,6 @@ const terminalApp: AppDefinition = {
// Session windows stay independent: launching reopens the hub even
// while terminal sessions are open.
matchExistingBy: "windowId",
// The hub window can show a session too, so it takes keys like one.
acceptsKeys: true,
}),
};
+36 -500
View File
@@ -16,17 +16,11 @@
* Each view has its own websocket connection (the protocol allows one attached
* session per connection) and its own xterm emulator.
*
* Connections are connection strings stored as JSON in the settings store
* (see connections.ts): g2mirror://<token>@<host>[:port] for a g2mirror
* server (g2mirrors:// for TLS), ssh://<user>@<host> for tmux over SSH
* (TmuxClient), or ssh://<user>@<host>?g2mirror=<token> for a g2mirror server
* tunnelled over SSH. Each enabled connection gets a control client that
* backs the session list and carries bell/title/activity notifications;
* control connections live as long as the worker so bells keep flowing even
* if the hub window is closed. SSH connections that need the user (an
* unverified or changed host key, a password, an unauthorized key) record
* the failure on the control (sshIssue) and offer a page that resolves it
* instead of retrying.
* Connections are g2mirror://<token>@<host>[:port] strings (g2mirrors:// for
* TLS), stored as JSON in the settings store (see connections.ts). Each
* enabled connection gets a control websocket that backs the session list and
* carries unsolicited bell/title notifications; control connections live as
* long as the worker so bells keep flowing even if the hub window is closed.
* The host's server_name (from the init reply) is cached per connection and
* used as its display name, falling back to the connection string's
* host[:port] before the first successful handshake.
@@ -61,24 +55,7 @@ import {
import * as frameTimings from "../../native/frame-timings";
import { getActiveDisplay } from "../../native/active-display";
import { GESTURE_DOUBLE_CLICK, type InputEvent } from "../../ui/gestures";
import type { KeyEvent } from "../../g2/keyboard/keys";
import { encodeTerminalKey, terminalScrollbackKey } from "./terminal-keys";
import { G2MirrorClient, type G2MirrorSession, type G2MirrorState } from "../../native/g2mirror-client";
import type { TerminalClient } from "./terminal-client";
import { TmuxClient, type OpenLineChannel } from "./tmux-client";
import {
copyToPhoneClipboard,
describeSshFailure,
getSshSecret,
openSshChannel,
removeSshSecret,
setSshSecret,
sshClientKey,
type SshEndpoint,
type SshFailure,
type SshHostKey,
} from "../../native/ssh";
import { wrapText } from "../../graphics/textwrap";
import { G2MirrorClient, type G2MirrorClientOptions, type G2MirrorSession, type G2MirrorState } from "../../native/g2mirror-client";
import { onSettingsStoreChanged } from "../../native/settings-store";
import { clamp } from "../../util/numeric-util";
import {
@@ -87,22 +64,12 @@ import {
terminalDisplayLayoutSetting,
terminalLaunchPresetsSetting,
terminalNewConnectionSetting,
terminalSshPasswordSetting,
terminalWakeOnBellSetting,
textSettingMenuItem,
toggleSettingMenuItem,
} from "../../ui/dashboard-settings";
import { terminalFontPickerMenuItem } from "../../ui/font-picker";
import {
connectionDisplayName,
connectionStringHostLabel,
loadConnections,
parseAnyConnectionString,
saveConnections,
TERMINAL_CONNECTIONS_KEY,
updateConnection,
type TerminalConnection,
} from "./connections";
import { connectionDisplayName, loadConnections, parseConnectionString, saveConnections, TERMINAL_CONNECTIONS_KEY, updateConnection, type TerminalConnection } from "./connections";
import { TerminalEmulator } from "./terminal-emulator";
import { drawSubmenuIndicator, submenuItem, type MenuItem } from "../../ui/menu";
import { Menu, type MenuDrawArgs } from "../../ui/menu-core";
@@ -145,10 +112,6 @@ const RECONNECT_MAX_DELAY_MS = 60_000;
const NEW_CONNECTION_DRAFT_KEY = "terminal.newConnectionDraft";
// Storage key of terminalLaunchPresetsSetting, edited from the window menu.
const LAUNCH_PRESETS_KEY = "terminal.launchPresets";
// Storage key of terminalSshPasswordSetting (the password page's draft).
const SSH_PASSWORD_DRAFT_KEY = "terminal.sshPasswordDraft";
// Comment on this device's SSH public key in authorized_keys.
const SSH_KEY_COMMENT = "faceclaw";
type BaseWindow = {
windowId: string;
@@ -185,24 +148,10 @@ type BaseWindow = {
/**
* What the hub window is showing: the session list, the Manage Connections
* section, the Add-connection screen (type the connection string on the
* phone, click to save), or a page of text with a click action (verify a
* host key, enter a password, show this device's SSH key).
* section, or the Add-connection screen (type the g2mirror:// string on the
* phone, click to save).
*/
type HubMode = "sessions" | "connections" | "add" | "page";
type HubPage = {
title: string;
/** Body text, wrapped to the window; re-read on every paint. */
body: () => string;
/** Bottom hint line. */
hint: string;
onClick?: () => void;
/** Double-click. */
onBack: () => void;
/** Runs whenever the page goes away (back, click, window closed). */
onLeave?: () => void;
};
type HubMode = "sessions" | "connections" | "add";
type HubWindow = BaseWindow & {
kind: "hub";
@@ -219,8 +168,6 @@ type HubWindow = BaseWindow & {
sessionOrder: string[];
/** Parse-failure message shown on the Add-connection screen. */
addError: string;
/** The page shown in mode "page". */
page: HubPage | null;
};
type ViewWindow = BaseWindow & {
@@ -239,7 +186,7 @@ type ViewWindow = BaseWindow & {
label: string;
/** Sidebar-icon character (">3"); "" if every glyph was taken at open time. */
glyph: string;
client: TerminalClient;
client: G2MirrorClient;
emulator: TerminalEmulator;
receivedData: boolean;
attachRequested: boolean;
@@ -273,8 +220,8 @@ type PendingView = {
socket: string;
label: string;
glyph: string;
/** How to make the view's client, captured when it was requested from a live control. */
options: ClientRecipe;
/** Client options captured when the view was requested from a live control. */
options: G2MirrorClientOptions;
};
const windows = new Map<string, TerminalWindow>();
@@ -355,13 +302,7 @@ let screenOn = true;
*/
type ControlConnection = {
config: TerminalConnection;
client: TerminalClient | null;
/**
* An SSH failure only the user can fix (host key to verify, password or
* key needed). Shown on the host's heading with a row that opens the fix;
* reconnects wait for it. Cleared on a successful connection.
*/
sshIssue: SshFailure | null;
client: G2MirrorClient | null;
state: G2MirrorState | null;
unsubscribers: Array<() => void>;
reconnectTimer: ReturnType<typeof setTimeout> | null;
@@ -494,13 +435,6 @@ global.onmessage = (event: { data: WorkerAppMessage }) => {
}
break;
}
case "key": {
const window = windows.get(message.windowId);
if (!window) break;
window.focused = message.focused;
handleKey(window, message.event as KeyEvent);
break;
}
case "render": {
const window = windows.get(message.windowId);
if (!window) break;
@@ -570,8 +504,7 @@ onSettingsStoreChanged((key) => {
reportIdle();
return;
case NEW_CONNECTION_DRAFT_KEY:
case SSH_PASSWORD_DRAFT_KEY:
// Live keystrokes from the phone editor; repaint the add/password screen.
// Live keystrokes from the phone editor; repaint the add screen.
renderHubWindows();
return;
case "terminal.autoReconnect":
@@ -645,7 +578,6 @@ function openWindow(windowId: string, surfaceId: string, title: string, viewport
list: null,
sessionOrder: [],
addError: "",
page: null,
});
// The hub carries the app's assistant tools; declare them once it exists.
post({ type: "set-tools", windowId, tools: TERMINAL_TOOLS });
@@ -678,10 +610,6 @@ function closeWindow(windowId: string): void {
// Window closed mid-add: shut the phone editor down.
endAddConnection(window);
}
if (window.kind === "hub" && window.mode === "page") {
// Likewise for a page's phone editor (the password draft).
closeHubPage(window, "sessions");
}
// Likewise for a setting being edited from the window menu.
window.menu?.close();
windows.delete(windowId);
@@ -699,120 +627,17 @@ function closeWindow(windowId: string): void {
}
}
/**
* How to make clients for a connection: "g2mirror" dials a g2mirror server
* directly, "ssh-g2mirror" reaches one through an SSH tunnel, "tmux" runs
* tmux control mode over SSH.
*/
type ClientRecipe = {
kind: "g2mirror" | "ssh-g2mirror" | "tmux";
/**
* `role` "control" watches bells (tmux keeps the bell bookkeeping for that
* client only). `onSshFailure` hears why an SSH channel failed.
*/
create(cols: number, rows: number, role: "control" | "view", onSshFailure?: (failure: SshFailure) => void): TerminalClient;
};
/** Client recipe for a connection string; null if the string doesn't parse. */
function clientOptionsFor(connection: TerminalConnection): ClientRecipe | null {
const parsed = parseAnyConnectionString(connection.url);
/** Client options for a connection string; null if the string doesn't parse. */
function clientOptionsFor(connection: TerminalConnection): G2MirrorClientOptions | null {
const parsed = parseConnectionString(connection.url);
if (!parsed) return null;
if (parsed.kind === "g2mirror") {
const { secure, host, port, authToken } = parsed;
return {
kind: "g2mirror",
create: (cols, rows) => new G2MirrorClient({ secure, host, port, authToken, deviceName: DEVICE_NAME, cols, rows }),
};
}
const connectionId = connection.id;
const label = connectionStringHostLabel(connection.url);
// Read at every channel open, so a newly trusted key or saved password
// applies to the next (re)connect, views included.
const endpoint = (): SshEndpoint => ({
host: parsed.host,
port: parsed.port,
user: parsed.user,
hostKeys: loadConnections().find((candidate) => candidate.id === connectionId)?.hostKeys ?? [],
password: getSshSecret(sshPasswordSecretId(connectionId)),
useKey: true,
});
if (parsed.mode === "g2mirror") {
const { authToken, targetHost, targetPort } = parsed;
const urlHost = targetHost.includes(":") && !targetHost.startsWith("[") ? `[${targetHost}]` : targetHost;
const url = `ws://${urlHost}:${targetPort}/`;
return {
kind: "ssh-g2mirror",
create: (cols, rows, _role, onSshFailure) =>
new G2MirrorClient({
secure: false,
host: targetHost,
port: targetPort,
authToken,
deviceName: DEVICE_NAME,
cols,
rows,
transport: {
label: `${label} over SSH`,
open: (listener) => {
const channel = openSshChannel(endpoint(), { type: "websocket", url }, {
onOpen: () => listener.onOpen(),
onText: (text) => listener.onTextMessage(text),
onClosed: (info) => listener.onClosed(info.code ?? 1000, info.reason ?? ""),
onFailure: (failure) => {
onSshFailure?.(failure);
listener.onFailure(describeSshFailure(failure));
},
});
return { sendText: (text) => channel.send(text), sendBinary: () => false, close: () => channel.close() };
},
},
}),
};
}
return {
kind: "tmux",
create: (cols, rows, role, onSshFailure) => {
const openChannel: OpenLineChannel = (command, listener) =>
openSshChannel(endpoint(), { type: "exec", command }, {
onOpen: () => listener.onOpen(),
onText: (line) => listener.onLine(line),
onStderr: (line) => listener.onStderr(line),
onClosed: (info) => listener.onClosed(info.exitStatus ?? null),
onFailure: (failure) => {
onSshFailure?.(failure);
listener.onFailure(describeSshFailure(failure));
},
});
return new TmuxClient({ openChannel, label, cols, rows, stateKey: role === "control" ? connectionId : "" });
},
...parsed,
deviceName: DEVICE_NAME,
...viewGrid(),
};
}
function sshPasswordSecretId(connectionId: string): string {
return `terminal.password.${connectionId}`;
}
/** Failures that retrying can't fix: the user has to verify a key or supply credentials. */
function sshFailureNeedsUser(failure: SshFailure): boolean {
return failure.kind === "hostkey-unknown" || failure.kind === "hostkey-mismatch" || failure.kind === "auth";
}
/**
* A control's SSH channel failed. If the user has to act, remember why, and
* if they're looking at the hub's lists, open the page that fixes it.
*/
function noteSshFailure(control: ControlConnection, failure: SshFailure): void {
if (!sshFailureNeedsUser(failure) || controls.get(control.config.id) !== control || !control.client) return;
control.sshIssue = failure;
renderHubWindows();
for (const window of windows.values()) {
if (window.kind === "hub" && window.foreground && (window.mode === "sessions" || window.mode === "connections")) {
openSshIssuePage(window, control);
break;
}
}
}
/**
* Reconcile the live control clients with the stored connection list: start
* enabled connections, stop disabled or removed ones, restart ones whose
@@ -838,7 +663,6 @@ function syncControlsFromSettings(): void {
control = {
config: connection,
client: null,
sshIssue: null,
state: null,
unsubscribers: [],
reconnectTimer: null,
@@ -860,13 +684,12 @@ function syncControlsFromSettings(): void {
function startControl(control: ControlConnection): void {
stopControl(control);
const recipe = clientOptionsFor(control.config);
if (!recipe) {
const options = clientOptionsFor(control.config);
if (!options) {
control.state = null;
return;
}
const grid = viewGrid();
const client = recipe.create(grid.cols, grid.rows, "control", (failure) => noteSshFailure(control, failure));
const client = new G2MirrorClient(options);
const connectionId = control.config.id;
control.client = client;
control.state = client.state();
@@ -880,7 +703,6 @@ function startControl(control: ControlConnection): void {
control.state = state;
if (state.phase === "connected") {
control.reconnectDelayMs = RECONNECT_MIN_DELAY_MS;
control.sshIssue = null;
maybeCacheServerName(control, state.serverName);
}
if (state.phase === "failed") scheduleControlReconnect(control);
@@ -955,8 +777,6 @@ function scheduleControlReconnect(control: ControlConnection): void {
if (!terminalAutoReconnectSetting.get()) return;
if (windows.size === 0 && !hasTerminalBackgroundWork()) return;
if (!control.config.enabled) return;
// Waiting on the user (host key, password) or on the host (no usable tmux).
if (control.sshIssue || control.state?.retryable === false) return;
const delayMs = control.reconnectDelayMs;
control.reconnectDelayMs = Math.min(control.reconnectDelayMs * 2, RECONNECT_MAX_DELAY_MS);
control.reconnectTimer = setTimeout(() => {
@@ -1123,7 +943,7 @@ function createViewWindow(
const cell = getTerminalFontConfig();
const gridCols = Math.floor(viewport.width / cell.cellWidth);
const gridRows = Math.floor(viewport.height / cell.cellHeight);
const client = view.options.create(gridCols, gridRows, "view");
const client = new G2MirrorClient({ ...view.options, cols: gridCols, rows: gridRows });
const window: ViewWindow = {
kind: "view",
hub: null,
@@ -1418,11 +1238,6 @@ function handleInput(window: TerminalWindow, event: InputEvent, frameId: number)
renderAndSubmit(window, frameId);
return;
}
if (window.kind === "hub" && window.mode === "page") {
window.page?.onBack();
renderAndSubmit(window, frameId);
return;
}
if (window.kind === "hub" && window.mode === "connections") {
setHubMode(window, "sessions");
renderAndSubmit(window, frameId);
@@ -1445,35 +1260,6 @@ function handleInput(window: TerminalWindow, event: InputEvent, frameId: number)
frameTimings.finishFrame(frameId, "discarded: terminal view ignored input");
}
/**
* A key from a hardware keyboard (the glasses' keyboard bridge) that no
* keybinding claimed. A session with nothing over it gets the key as its
* terminal would: Escape and the arrows belong to the program, and the way
* out is Escape with Shift, Ctrl or Alt, which never arrives here (it is a
* built-in keybinding for "back"). Anywhere else (the hub's lists, an open
* menu) the key is handed back, to act as navigation.
*/
function handleKey(window: TerminalWindow, event: KeyEvent): void {
if (window.kind === "view" && !window.menu?.isOpen()) {
activeViewId = window.windowId;
const page = terminalScrollbackKey(event);
if (page !== null) {
handleViewScroll(window, page, 0);
return;
}
const bytes = encodeTerminalKey(event, window.emulator.applicationCursorKeys());
if (bytes === null) return;
// Typing follows the live screen, as in any terminal emulator.
if (window.scrollTop !== null) {
window.scrollTop = null;
scheduleRender(window);
}
window.client.sendInput(bytes);
return;
}
post({ type: "key-unhandled", windowId: window.windowId, event });
}
/** Top visible absolute line index when following the live bottom. */
function followTop(window: ViewWindow): number {
return window.historyNext + window.emulator.bufferLength() - window.gridRows;
@@ -1600,154 +1386,6 @@ function setHubMode(window: HubWindow, mode: HubMode): void {
window.addError = "";
}
function openHubPage(window: HubWindow, page: HubPage): void {
if (window.mode === "add") endAddConnection(window);
if (window.mode === "page") window.page?.onLeave?.();
window.page = page;
window.mode = "page";
scheduleRender(window);
}
/** Leave the current page (if any) for `mode`. */
function closeHubPage(window: HubWindow, mode: HubMode): void {
const page = window.page;
window.page = null;
page?.onLeave?.();
setHubMode(window, mode);
scheduleRender(window);
}
/** The list a page returns to: the one it was opened from. */
function hubListMode(window: HubWindow): HubMode {
return window.mode === "connections" ? "connections" : "sessions";
}
function openSshIssuePage(window: HubWindow, control: ControlConnection): void {
const issue = control.sshIssue;
if (!issue) return;
if (issue.kind === "hostkey-unknown" && issue.hostKey) {
openHostKeyPage(window, control, issue.hostKey);
} else if (issue.kind === "hostkey-mismatch" && issue.hostKey) {
openHostKeyChangedPage(window, control, issue.hostKey);
} else if (issue.kind === "auth" && issue.passwordAllowed) {
openPasswordPage(window, control, Boolean(issue.passwordTried));
} else if (issue.kind === "auth") {
openClientKeyPage(window, `${connectionDisplayName(control.config)} didn't accept this device's key (and doesn't take passwords).\n`);
}
}
function hostKeyTypeName(type: string): string {
if (type === "ssh-ed25519") return "ED25519";
if (type.startsWith("ecdsa-")) return "ECDSA";
if (type === "ssh-rsa") return "RSA";
return type;
}
function hostKeyFile(type: string): string {
if (type === "ssh-ed25519") return "/etc/ssh/ssh_host_ed25519_key.pub";
if (type.startsWith("ecdsa-")) return "/etc/ssh/ssh_host_ecdsa_key.pub";
return "/etc/ssh/ssh_host_rsa_key.pub";
}
/** First connection to an SSH host: show the key's fingerprint; click trusts it. */
function openHostKeyPage(window: HubWindow, control: ControlConnection, hostKey: SshHostKey): void {
const returnMode = hubListMode(window);
openHubPage(window, {
title: "Verify host key",
body: () =>
`First connection to ${connectionStringHostLabel(control.config.url)}. Its ${hostKeyTypeName(hostKey.type)} host key fingerprint is:\n` +
`${hostKey.fingerprint}\n\n` +
`To check it, run this on the host and compare:\nssh-keygen -lf ${hostKeyFile(hostKey.type)}`,
hint: `click trust and connect ${GESTURE_DOUBLE_CLICK} cancel`,
onClick: () => {
const hostKeys = [...control.config.hostKeys.filter((key) => key !== hostKey.key), hostKey.key];
control.config.hostKeys = hostKeys;
updateConnection(control.config.id, { hostKeys });
control.sshIssue = null;
closeHubPage(window, returnMode);
connectControl(control);
},
onBack: () => closeHubPage(window, returnMode),
});
}
function openHostKeyChangedPage(window: HubWindow, control: ControlConnection, hostKey: SshHostKey): void {
const returnMode = hubListMode(window);
openHubPage(window, {
title: "HOST KEY CHANGED",
body: () =>
`${connectionStringHostLabel(control.config.url)} presented a different host key than the one trusted before: ` +
`${hostKeyTypeName(hostKey.type)} ${hostKey.fingerprint}\n\n` +
"Someone may be intercepting the connection, so Faceclaw won't connect. If the host was reinstalled or its keys " +
'were regenerated, use "Forget host key" on this connection in Manage Connections, then verify the new fingerprint.',
hint: `${GESTURE_DOUBLE_CLICK} back`,
onBack: () => closeHubPage(window, returnMode),
});
}
/** Ask for the host's password on the phone (masked editor); click saves it and connects. */
function openPasswordPage(window: HubWindow, control: ControlConnection, rejected: boolean): void {
const returnMode = hubListMode(window);
terminalSshPasswordSetting.set("");
post({ type: "start-text-setting-edit", settingId: terminalSshPasswordSetting.id });
openHubPage(window, {
title: "SSH password",
body: () => {
const typed = terminalSshPasswordSetting.get();
return (
(rejected ? "The host refused the saved password.\n" : "") +
`Type the password for ${connectionStringHostLabel(control.config.url)} in the phone app, then click to connect. ` +
"It is saved encrypted on this phone.\n\n" +
(typed ? "*".repeat(Math.min(typed.length, 32)) : "(nothing typed yet)")
);
},
hint: `click connect ${GESTURE_DOUBLE_CLICK} cancel`,
onClick: () => {
const password = terminalSshPasswordSetting.get();
if (!password) return;
setSshSecret(sshPasswordSecretId(control.config.id), password);
control.sshIssue = null;
closeHubPage(window, returnMode);
connectControl(control);
},
onBack: () => closeHubPage(window, returnMode),
onLeave: () => {
terminalSshPasswordSetting.set("");
post({ type: "end-text-setting-edit" });
},
});
}
/** This device's SSH public key (the private half never leaves the keystore); click copies it. */
function openClientKeyPage(window: HubWindow, intro = ""): void {
const returnMode = hubListMode(window);
let key: { authorizedKey: string; fingerprint: string } | null = null;
let error = "";
try {
key = sshClientKey(SSH_KEY_COMMENT);
} catch (caught) {
error = String((caught as Error)?.message ?? caught);
}
let copied = false;
openHubPage(window, {
title: "This device's SSH key",
body: () =>
key
? `${intro}Add this line to ~/.ssh/authorized_keys on each host to log in with this phone's key:\n` +
`${key.authorizedKey}\n${key.fingerprint}` +
(copied ? "\n\nCopied to the phone's clipboard." : "")
: `Couldn't create this device's key: ${error}`,
hint: key ? `click copy to phone clipboard ${GESTURE_DOUBLE_CLICK} back` : `${GESTURE_DOUBLE_CLICK} back`,
onClick: key
? () => {
copyToPhoneClipboard("SSH public key", key!.authorizedKey);
copied = true;
}
: undefined,
onBack: () => closeHubPage(window, returnMode),
});
}
/**
* One control connection's sessions in this hub window's display order:
* most-recently-updated first as of when the window became visible, with
@@ -1803,7 +1441,6 @@ function hubSessionItems(window: HubWindow): HubItem[] {
];
const multiHost = controls.size > 1;
const canLaunch = launchPresetNames().length > 0;
const isTmux = (control: ControlConnection) => clientOptionsFor(control.config)?.kind === "tmux";
// Every configured host is listed under its own heading, in stored order,
// whatever its connection state: a connected host lists its sessions, any
// other shows its state on the heading plus a Connect row (unless it is
@@ -1813,17 +1450,10 @@ function hubSessionItems(window: HubWindow): HubItem[] {
const name = connectionDisplayName(control.config);
if (!isControlConnected(control)) {
items.push({ label: `${name} (${controlStatusWord(control)})`, heading: true });
if (control.sshIssue && control.config.enabled) {
items.push({ label: sshIssueAction(control.sshIssue), onSelect: () => openSshIssuePage(window, control) });
continue;
}
const connecting = control.config.enabled && control.state?.phase === "connecting";
if (!connecting && clientOptionsFor(control.config)) {
// A failure retrying can't fix (tmux missing or too old) says why;
// selecting it retries once the host is fixed.
const unfixable = control.config.enabled && control.state?.retryable === false;
items.push({
label: unfixable ? control.state!.status : control.reconnectTimer ? "Connect now" : "Connect",
label: control.reconnectTimer ? "Connect now" : "Connect",
onSelect: () => connectControl(control),
});
}
@@ -1832,8 +1462,7 @@ function hubSessionItems(window: HubWindow): HubItem[] {
items.push({
label: name,
heading: true,
// tmux hosts can always start a shell, presets or not.
onSelect: canLaunch || isTmux(control) ? () => launchOnHost(window, control) : undefined,
onSelect: canLaunch ? () => launchOnHost(window, control) : undefined,
});
const sessions = orderedSessions(window, control);
for (const session of sessions) {
@@ -1849,11 +1478,7 @@ function hubSessionItems(window: HubWindow): HubItem[] {
}
if (!sessions.length) {
items.push({
label: isTmux(control)
? "(no tmux sessions; select the host to start one)"
: multiHost
? "(no live sessions)"
: "(no live sessions; run g2mirror <command>)",
label: multiHost ? "(no live sessions)" : "(no live sessions; run g2mirror <command>)",
onSelect: () => control.client?.listSessions(),
});
}
@@ -1874,10 +1499,6 @@ function hubConnectionItems(window: HubWindow): HubItem[] {
label: "Add connection",
onSelect: () => beginAddConnection(window),
},
{
label: "This device's SSH key",
onSelect: () => openClientKeyPage(window),
},
{
label: "Done",
onSelect: () => setHubMode(window, "sessions"),
@@ -1888,13 +1509,6 @@ function hubConnectionItems(window: HubWindow): HubItem[] {
function controlStatusWord(control: ControlConnection): string {
if (!control.config.enabled) return "off";
const issue = control.sshIssue;
if (issue) {
if (issue.kind === "hostkey-unknown") return "verify host key";
if (issue.kind === "hostkey-mismatch") return "HOST KEY CHANGED";
if (issue.passwordTried) return "password rejected";
return issue.passwordAllowed ? "password needed" : "key not authorized";
}
switch (control.state?.phase) {
case "connected":
case "attached":
@@ -1927,32 +1541,6 @@ function openConnectionActions(window: HubWindow, control: ControlConnection): v
connectControl(control);
},
},
...(control.config.hostKeys.length
? [
{
label: "Forget host key",
onSelect: (ctx: { stack: { pop(): unknown } }) => {
ctx.stack.pop();
control.config.hostKeys = [];
updateConnection(control.config.id, { hostKeys: [] });
control.sshIssue = null;
if (control.config.enabled) startControl(control);
renderHubWindows();
},
},
]
: []),
...(isSshConnection(control) && getSshSecret(sshPasswordSecretId(control.config.id))
? [
{
label: "Forget saved password",
onSelect: (ctx: { stack: { pop(): unknown } }) => {
ctx.stack.pop();
removeSshSecret(sshPasswordSecretId(control.config.id));
},
},
]
: []),
{
label: `Remove ${name}`,
onSelect: (ctx) => {
@@ -1970,11 +1558,6 @@ function openConnectionActions(window: HubWindow, control: ControlConnection): v
windowMenu(window).open(items);
}
function isSshConnection(control: ControlConnection): boolean {
const kind = clientOptionsFor(control.config)?.kind;
return kind === "tmux" || kind === "ssh-g2mirror";
}
/** Enable and (re)connect now; also the manual retry path, so reset backoff. */
function connectControl(control: ControlConnection): void {
control.reconnectDelayMs = RECONNECT_MIN_DELAY_MS;
@@ -1995,7 +1578,6 @@ function disconnectControl(control: ControlConnection): void {
function removeControl(control: ControlConnection): void {
stopControl(control);
removeSshSecret(sshPasswordSecretId(control.config.id));
controls.delete(control.config.id);
saveConnections(loadConnections().filter((connection) => connection.id !== control.config.id));
renderHubWindows();
@@ -2004,7 +1586,7 @@ function removeControl(control: ControlConnection): void {
/**
* Enter the Add-connection screen: clear the draft and ask the shell to open
* the phone text editor on it. The user types (or voice-inputs) the
* connection string, then clicks to save or double-clicks to cancel.
* g2mirror:// string, then clicks to save or double-clicks to cancel.
*/
function beginAddConnection(window: HubWindow): void {
terminalNewConnectionSetting.set("");
@@ -2014,15 +1596,15 @@ function beginAddConnection(window: HubWindow): void {
function saveAddConnection(window: HubWindow): void {
const url = terminalNewConnectionSetting.get();
if (!parseAnyConnectionString(url)) {
if (!parseConnectionString(url)) {
window.addError = url
? "Not an ssh://user@host or g2mirror://token@host string."
: "Enter a connection string first.";
? "Not a g2mirror://token@host connection string."
: "Enter a g2mirror:// connection string first.";
scheduleRender(window);
return;
}
const connections = loadConnections();
connections.push({ id: `c${Date.now()}`, url, serverName: null, enabled: true, hostKeys: [] });
connections.push({ id: `c${Date.now()}`, url, serverName: null, enabled: true });
saveConnections(connections);
endAddConnection(window);
setHubMode(window, "connections");
@@ -2052,15 +1634,6 @@ function handleHubInput(window: HubWindow, event: InputEvent, frameId: number):
frameTimings.finishFrame(frameId, "discarded: terminal add-connection ignored input");
return;
}
if (window.mode === "page") {
if (event.type === "click" && window.page?.onClick) {
window.page.onClick();
renderAndSubmit(window, frameId);
return;
}
frameTimings.finishFrame(frameId, "discarded: terminal page ignored input");
return;
}
const list = hubMenu(window);
list.setItems(hubItems(window));
switch (event.type) {
@@ -2227,7 +1800,6 @@ function openViewWindow(control: ControlConnection, socket: string, label: strin
focus: true,
// Default to tall sessions; the app display setting can override this.
heightMode: "max",
acceptsKeys: true,
});
}
@@ -2248,11 +1820,6 @@ function launchPresetNames(): string[] {
*/
function launchOnHost(window: HubWindow, control: ControlConnection): void {
const presets = launchPresetNames();
if (presets.length === 0) {
// A tmux host with no presets configured: a new session running the login shell.
launchFromUi(control, "shell");
return;
}
if (presets.length === 1) {
launchFromUi(control, presets[0]!);
return;
@@ -2297,9 +1864,6 @@ function paintHub(window: HubWindow): GrayImage {
if (window.mode === "add") {
return paintAddConnection(window);
}
if (window.mode === "page" && window.page) {
return paintHubPage(window, window.page);
}
const { viewportWidth: width, viewportHeight: height, padding } = window;
const image = new GrayImage(width, height, 0);
const font = chromeFont();
@@ -2314,7 +1878,7 @@ function paintHub(window: HubWindow): GrayImage {
let listTop = header.listTop;
if (window.mode === "sessions" && controls.size === 0) {
image.drawText(font, header.titleX, listTop, "Add an ssh:// (tmux) or g2mirror:// connection to get started, see:", 150);
image.drawText(font, header.titleX, listTop, "Add a g2mirror:// connection to get started, see:", 150);
image.drawText(font, header.titleX, listTop + step, "https://github.com/jimrandomh/g2mirror", 190);
listTop += 2 * step + 6;
}
@@ -2356,8 +1920,8 @@ function paintAddConnection(window: HubWindow): GrayImage {
image.drawText(font, padding.left, padding.top, "Add connection", 220);
const bodyX = padding.left + PAGE_BODY_INDENT;
const instructionsY = padding.top + 10 + step;
image.drawText(font, bodyX, instructionsY, "Type ssh://user@host (tmux) or g2mirror://token@host in", 170);
image.drawText(font, bodyX, instructionsY + step, "the phone app (or use voice input from the menu).", 170);
image.drawText(font, bodyX, instructionsY, "Type the g2mirror://token@host string in the", 170);
image.drawText(font, bodyX, instructionsY + step, "phone app (or use voice input from the menu).", 170);
const draftY = instructionsY + 3 * step;
const draft = terminalNewConnectionSetting.get();
image.drawText(font, bodyX, draftY, truncateLabel(draft || "(empty)", 52), 220);
@@ -2368,34 +1932,6 @@ function paintAddConnection(window: HubWindow): GrayImage {
return image;
}
/** A page of text: title at the top padding, wrapped body, hint on the bottom padding. */
function paintHubPage(window: HubWindow, page: HubPage): GrayImage {
const { viewportWidth: width, viewportHeight: height, padding } = window;
const image = new GrayImage(width, height, 0);
const font = chromeFont();
const step = lineStep(font);
image.drawText(font, padding.left, padding.top, page.title, 220);
const bodyX = padding.left + PAGE_BODY_INDENT;
const hintY = height - padding.bottom - font.lineHeight;
const lines = wrapText(font, page.body(), Math.max(1, width - padding.right - bodyX), { breakLongWords: true });
let y = padding.top + 10 + step;
for (const line of lines) {
if (y + step > hintY) break;
image.drawText(font, bodyX, y, line, 190);
y += step;
}
image.drawText(font, bodyX, hintY, page.hint, 110);
return image;
}
/** The action row under a host whose SSH connection needs the user. */
function sshIssueAction(issue: SshFailure): string {
if (issue.kind === "hostkey-unknown") return "Verify host key...";
if (issue.kind === "hostkey-mismatch") return "Host key changed!";
if (issue.passwordAllowed) return issue.passwordTried ? "Re-enter password..." : "Enter password...";
return "Show this device's SSH key...";
}
function truncateLabel(text: string, maxLength: number): string {
return text.length <= maxLength ? text : `${text.slice(0, Math.max(0, maxLength - 3))}...`;
}
-44
View File
@@ -1,44 +0,0 @@
import type {
G2MirrorHistoryReply,
G2MirrorPhase,
G2MirrorSession,
G2MirrorState,
} from "../../native/terminal-session-types";
/**
* What the Terminal worker needs from a session backend. G2MirrorClient (a
* g2mirror server, directly or tunnelled over SSH) and TmuxClient (tmux
* control mode over SSH) both provide it. The model is g2mirror's: a client
* starts "connecting", reaches "connected" (it can list and launch sessions,
* and reports bells/activity/titles for them), and can then attach to one
* session, view it (snapshot + live output), page its scrollback, and type
* into it.
*/
export interface TerminalClient {
state(): G2MirrorState;
onStateChange(listener: (state: G2MirrorState) => void): () => void;
/** Raw VT100/xterm bytes to feed the emulator. Snapshot implies reset-first. */
onTerminalData(listener: (data: Uint8Array, kind: "snapshot" | "output") => void): () => void;
onSessionAttached(listener: (command: string) => void): () => void;
onSessionDetached(listener: (reason: string) => void): () => void;
onBell(listener: (socket: string, lastBellAtMs: number) => void): () => void;
onActivity(listener: (socket: string, lastOutputAtMs: number) => void): () => void;
onTitle(listener: (socket: string, title: string) => void): () => void;
/** Fired after a snapshot's terminal data, with its scrollback splice point. */
onSnapshot(listener: (historyNext: number, historyOldest: number) => void): () => void;
onHistoryLines(listener: (reply: G2MirrorHistoryReply) => void): () => void;
setViewport(cols: number, rows: number): void;
start(): void;
stop(): void;
listSessions(): void;
launchSession(preset: string): Promise<string>;
connectSession(socket: string): void;
disconnectSession(): void;
view(): void;
unview(): void;
requestHistory(before: number, limit?: number): void;
sendInput(text: string): void;
submitInput(text: string): void;
}
export type { G2MirrorHistoryReply, G2MirrorPhase, G2MirrorSession, G2MirrorState };
-5
View File
@@ -82,9 +82,4 @@ export class TerminalEmulator {
cursorCol(): number {
return this.term.buffer.active.cursorX;
}
/** DECCKM: the session asked for application cursor keys (ESC O A rather than ESC [ A). */
applicationCursorKeys(): boolean {
return this.term.modes.applicationCursorKeysMode;
}
}
-118
View File
@@ -1,118 +0,0 @@
/**
* What a terminal emulator sends for a key from a hardware keyboard, by the
* classic xterm conventions: Ctrl chords become control characters, Alt
* prefixes ESC, and modified cursor and function keys carry the xterm
* modifier parameter (1 + shift + 2*alt + 4*ctrl). Follows the Glassly
* Terminal miniapp's encoder (terminal/keys.ts, MIT).
* No platform imports: this runs under node in the tests.
*/
import type { KeyEvent } from "../../g2/keyboard/keys";
const ESC = "\x1b";
/** Keys xterm encodes as `ESC [ <n> ~`, with a modifier as `;m`. */
const TILDE_KEYS: Record<string, number> = {
Insert: 2,
Delete: 3,
PageUp: 5,
PageDown: 6,
F5: 15,
F6: 17,
F7: 18,
F8: 19,
F9: 20,
F10: 21,
F11: 23,
F12: 24,
};
/** Keys with a single-letter final byte: SS3 in application cursor mode, CSI otherwise. */
const LETTER_KEYS: Record<string, string> = {
ArrowUp: "A",
ArrowDown: "B",
ArrowRight: "C",
ArrowLeft: "D",
Home: "H",
End: "F",
};
/** F1-F4 are always SS3 (`ESC O P`), or CSI with a modifier. */
const SS3_KEYS: Record<string, string> = { F1: "P", F2: "Q", F3: "R", F4: "S" };
/** Ctrl with a punctuation or digit key, per xterm (^[ ^\ ^] ^^ ^_ ^? ^@). */
const CONTROL_PUNCTUATION: Record<string, string> = {
"[": "\x1b",
"\\": "\x1c",
"]": "\x1d",
"^": "\x1e",
"6": "\x1e",
"_": "\x1f",
"-": "\x1f",
"?": "\x7f",
"/": "\x7f",
"8": "\x7f",
"@": "\x00",
"2": "\x00",
};
/**
* The text to write to the terminal for one key event, or null when the key
* sends nothing: releases, bare modifiers, Meta chords, and keys a terminal
* has no bytes for (volume, media). `appCursor` is DECCKM, which the emulator
* tracks from the session's output.
*/
export function encodeTerminalKey(event: KeyEvent, appCursor: boolean): string | null {
if (event.phase !== "down") return null;
const { key, modifiers } = event;
if (modifiers.meta) return null;
const modifierParam = 1 + (modifiers.shift ? 1 : 0) + (modifiers.alt ? 2 : 0) + (modifiers.ctrl ? 4 : 0);
const modified = (prefix: string, final: string): string | null =>
modifierParam > 1 ? `${ESC}[${prefix};${modifierParam}${final}` : null;
switch (key) {
case "Enter":
return modifiers.alt ? `${ESC}\r` : "\r";
case "Escape":
return ESC;
case "Tab":
return modifiers.shift ? `${ESC}[Z` : "\t";
case "Backspace":
if (modifiers.alt) return `${ESC}\x7f`;
return modifiers.ctrl ? "\x08" : "\x7f";
case " ":
return modifiers.ctrl ? "\x00" : modifiers.alt ? `${ESC} ` : " ";
}
const letter = LETTER_KEYS[key];
if (letter) return modified("1", letter) ?? (appCursor ? `${ESC}O${letter}` : `${ESC}[${letter}`);
const tilde = TILDE_KEYS[key];
if (tilde) return modified(String(tilde), "~") ?? `${ESC}[${tilde}~`;
const ss3 = SS3_KEYS[key];
if (ss3) return modified("1", ss3) ?? `${ESC}O${ss3}`;
if (modifiers.ctrl) {
// From the unshifted key: ^A..^Z, then the punctuation set.
const base = key.length === 1 ? key.toLowerCase() : "";
let char: string | undefined;
if (base >= "a" && base <= "z") char = String.fromCharCode(base.charCodeAt(0) - 96);
else char = CONTROL_PUNCTUATION[key];
if (char === undefined) return null;
return modifiers.alt ? `${ESC}${char}` : char;
}
// With Alt held the translator leaves `text` out (it is a chord, not
// typing); a terminal still wants ESC and the character.
const text = event.text ?? (modifiers.alt && key.length === 1 ? key : undefined);
if (text === undefined) return null;
return modifiers.alt ? `${ESC}${text}` : text;
}
/** Shift+PageUp / Shift+PageDown page the emulator's own scrollback, as in xterm. */
export function terminalScrollbackKey(event: KeyEvent): -1 | 1 | null {
if (event.phase !== "down") return null;
const { shift, ctrl, alt, meta } = event.modifiers;
if (!shift || ctrl || alt || meta) return null;
if (event.key === "PageUp") return -1;
if (event.key === "PageDown") return 1;
return null;
}
File diff suppressed because it is too large. Load diff
+2 -29
View File
@@ -1,32 +1,5 @@
import { type AppContext, type AppDefinition } from "../app-definition";
import {
createTranscribeAppWindow,
createTranscribeReplayWindow,
TRANSCRIBE_SURFACE_ID,
TRANSCRIBE_WINDOW_ID,
} from "./transcribe-app";
// Replay windows are closeable and non-singleton; the serial keeps their windowIds unique.
let nextReplaySerial = 1;
/**
* Open a recording (.mka) in Transcribe's replay mode, as a window of its
* own. Returns false where recordings can't be replayed.
*/
export function openRecordingInTranscribe(ctx: AppContext, path: string, name: string): boolean {
if (!ctx.actions.openVoiceReplay) return false;
const windowId = `transcribe:replay:${nextReplaySerial++}`;
void ctx
.launchInProcessApp(windowId, `window:${windowId}`, (options) => {
const window = createTranscribeReplayWindow(windowId, path, name, options);
if (!window) throw new Error("recordings can't be replayed here");
return window;
})
.catch((error) => {
ctx.appendLog(`transcribe replay window failed: ${error}`);
});
return true;
}
import { type AppDefinition } from "../app-definition";
import { createTranscribeAppWindow, TRANSCRIBE_SURFACE_ID, TRANSCRIBE_WINDOW_ID } from "./transcribe-app";
const transcribeApp: AppDefinition = {
appId: "transcribe",
-23
View File
@@ -1,23 +0,0 @@
/** A position in a recording as m:ss, or h:mm:ss from an hour on (or when `withHours`). */
export function formatReplayTime(ms: number, withHours = false): string {
const total = Math.max(0, Math.floor(ms / 1000));
const hours = Math.floor(total / 3600);
const minutes = Math.floor((total % 3600) / 60);
const seconds = String(total % 60).padStart(2, "0");
if (hours === 0 && !withHours) return `${minutes}:${seconds}`;
return `${hours}:${String(minutes).padStart(2, "0")}:${seconds}`;
}
/** The time picker's fields, in display and click order. */
export const TIME_FIELDS = ["hours", "minutes", "seconds"] as const;
export type TimeField = (typeof TIME_FIELDS)[number];
const FIELD_SECONDS: Record<TimeField, number> = { hours: 3600, minutes: 60, seconds: 1 };
/**
* Step a position (whole seconds) by `delta` of a field's unit, carrying
* into the neighboring fields, within [0, maxSeconds].
*/
export function stepReplaySeconds(seconds: number, field: TimeField, delta: number, maxSeconds: number): number {
return Math.max(0, Math.min(maxSeconds, seconds + delta * FIELD_SECONDS[field]));
}
-147
View File
@@ -1,147 +0,0 @@
import { GrayImage } from "../../graphics/image";
import { getDefaultMediumFont, getDefaultSmallFont } from "../../graphics/ui-fonts";
import { GESTURE_CLICK, GESTURE_DOUBLE_CLICK, GESTURE_SCROLL, gestureHints, type InputEvent } from "../../ui/gestures";
import type { Layer, LayerContext, PaintBelow } from "../../ui/layers";
import { CONTEXT_MENU_DIM } from "../../ui/menu";
import { centeredTextY } from "../../ui/metrics";
import { formatReplayTime, stepReplaySeconds, TIME_FIELDS, type TimeField } from "./replay-time";
const PADDING = 12;
const FIELD_GAP = 6;
const FIELD_PAD_X = 6;
/** Fill and outline of the active field (the Timers and date pickers' field style). */
const ACTIVE_FILL = 35;
const ACTIVE_STROKE = 130;
export type TimePickerOptions = {
title: string;
/** Where the picker starts, and the furthest it goes, in ms. */
initialMs: number;
maxMs: number;
onPick: (positionMs: number) => void;
};
/**
* A modal position picker for a recording: hours (when the recording has
* any), minutes and seconds, over the dimmed content beneath. Input follows
* the date picker's: scroll adjusts the active field (up is later, carrying
* into the fields beside it), click moves to the next field and on the last
* one picks, double-click steps back a field and on the first one cancels.
* A watch's swipes move between fields and adjust.
*/
export class TimePickerLayer implements Layer {
readonly acceptsDirectional = true;
readonly dimUnderneath = CONTEXT_MENU_DIM;
private readonly fields: readonly TimeField[];
private readonly maxSeconds: number;
private seconds: number;
private field = 0;
constructor(private readonly options: TimePickerOptions) {
this.maxSeconds = Math.max(0, Math.floor(options.maxMs / 1000));
this.seconds = Math.max(0, Math.min(this.maxSeconds, Math.floor(options.initialMs / 1000)));
this.fields = this.maxSeconds >= 3600 ? TIME_FIELDS : TIME_FIELDS.slice(1);
}
handleInput(event: InputEvent, ctx: LayerContext): void {
switch (event.type) {
case "scroll-up":
case "swipe-up":
this.adjust(1);
return;
case "scroll-down":
case "swipe-down":
this.adjust(-1);
return;
case "swipe-right":
this.field = Math.min(this.fields.length - 1, this.field + 1);
return;
case "swipe-left":
this.field = Math.max(0, this.field - 1);
return;
case "click":
if (this.field < this.fields.length - 1) {
this.field++;
} else {
ctx.stack.pop();
this.options.onPick(this.seconds * 1000);
}
return;
case "double-click":
if (this.field > 0) this.field--;
else ctx.stack.pop();
return;
default:
return;
}
}
private adjust(delta: number): void {
this.seconds = stepReplaySeconds(this.seconds, this.fields[this.field]!, delta, this.maxSeconds);
}
private hints(field: number): string {
return gestureHints([
[GESTURE_SCROLL, "adjust"],
[GESTURE_CLICK, field < this.fields.length - 1 ? "next" : "jump"],
[GESTURE_DOUBLE_CLICK, field > 0 ? "back" : "cancel"],
]);
}
paint(ctx: LayerContext, paintBelow: PaintBelow): GrayImage {
const image = paintBelow();
const small = getDefaultSmallFont();
const medium = getDefaultMediumFont();
const focused = ctx.stack.isFocused();
const base = ctx.stack.getBaseSize();
const values = [
String(Math.floor(this.seconds / 3600)),
String(Math.floor((this.seconds % 3600) / 60)).padStart(this.fields.length === 3 ? 2 : 1, "0"),
String(this.seconds % 60).padStart(2, "0"),
].slice(TIME_FIELDS.length - this.fields.length);
// Sized for the widest value a field can show, so the dialog holds still while the time changes.
const fieldWidths = this.fields.map(
(field) => Math.max(medium.measureText("000"), small.measureText(field)) + 2 * FIELD_PAD_X,
);
const fieldsWidth = fieldWidths.reduce((sum, width) => sum + width, 0) + FIELD_GAP * (fieldWidths.length - 1);
const boxHeight = medium.lineHeight + 6;
const length = `of ${formatReplayTime(this.maxSeconds * 1000)}`;
const hintsWidth = Math.max(...this.fields.map((_, index) => small.measureText(this.hints(index))));
const width = Math.min(
base.width,
Math.max(fieldsWidth, hintsWidth, small.measureText(this.options.title), small.measureText(length)) + 2 * PADDING,
);
const titleHeight = small.lineHeight + 12;
const bodyHeight = boxHeight + 2 + small.lineHeight + 10 + small.lineHeight;
const height = Math.min(base.height, 8 + titleHeight + bodyHeight + 12 + small.lineHeight + 10);
const x = Math.max(0, Math.floor((base.width - width) / 2));
const y = Math.max(0, Math.floor((base.height - height) / 2));
// Fill 1, not 0: 0 is the transparent color key on the shell surface.
image.fillRoundedRect(x, y, width, height, 1);
image.drawRoundedRect(x, y, width, height, 72);
image.drawText(small, x + PADDING, y + 8, this.options.title, 220);
const top = y + 8 + titleHeight;
let fieldX = x + PADDING;
for (let index = 0; index < this.fields.length; index++) {
const fieldWidth = fieldWidths[index]!;
const active = index === this.field;
if (active) {
if (focused) image.fillRoundedRect(fieldX, top, fieldWidth, boxHeight, ACTIVE_FILL, 6);
image.drawRoundedRect(fieldX, top, fieldWidth, boxHeight, ACTIVE_STROKE, 6);
}
const value = values[index]!;
image.drawText(medium, fieldX + Math.floor((fieldWidth - medium.measureText(value)) / 2),
centeredTextY(medium, top, boxHeight), value, active ? 250 : 170);
const caption = this.fields[index]!;
image.drawText(small, fieldX + Math.floor((fieldWidth - small.measureText(caption)) / 2),
top + boxHeight + 2, caption, active ? 190 : 110);
fieldX += fieldWidth + FIELD_GAP;
}
image.drawText(small, x + PADDING, top + bodyHeight - small.lineHeight, length, 190);
image.drawText(small, x + PADDING, y + height - 10 - small.lineHeight, this.hints(this.field), 110);
return image;
}
}
+4 -101
View File
@@ -1,11 +1,8 @@
import { imageFromAsciiArt, type GrayImage } from "../../graphics/image";
import { voiceControlBridge } from "../../native/voice-control";
import { enumSettingMenuItem, saveRecordingSetting, toggleSettingMenuItem } from "../../ui/dashboard-settings";
import { type ContinuousCaptureOptions } from "../../ui/layers";
import { saveRecordingSetting, toggleSettingMenuItem } from "../../ui/dashboard-settings";
import { withRecordingIcon } from "../../ui/listening-header";
import { TimePickerLayer } from "./time-picker";
import { TranscribeLayer, transcriptTimestamp } from "./transcribe";
import { transcribeMicCount, transcribeMicsSetting } from "./transcribe-settings";
import {
createInProcessWindow,
YieldAtRootLayer,
@@ -48,8 +45,8 @@ export function micTrayIcon(recording: boolean): GrayImage {
}
export type TranscribeAppOptions = InProcessAppOptions & {
/** Begin (or retarget) continuous mic capture, kept on while the window is open. */
startContinuousVoiceCapture: (options?: ContinuousCaptureOptions) => void;
/** Begin continuous mic capture (kept on while the window is open). */
startContinuousVoiceCapture: (options?: { recordAs?: string }) => void;
stopContinuousVoiceCapture: () => void;
};
@@ -62,10 +59,6 @@ export function createTranscribeAppWindow(options: TranscribeAppOptions): InProc
// One timestamp per session names both the audio recording and a saved transcript.
const stamp = transcriptTimestamp();
const layer = new TranscribeLayer(stamp);
// At launch, and again when the Microphones setting changes: a running
// capture moves over to the microphones now asked for.
const startCapture = () =>
options.startContinuousVoiceCapture({ recordAs: `transcribe-${stamp}`, mics: transcribeMicCount() });
const app = createInProcessWindow({
appId: "transcribe",
windowId: TRANSCRIBE_WINDOW_ID,
@@ -85,7 +78,6 @@ export function createTranscribeAppWindow(options: TranscribeAppOptions): InProc
},
// The controller starts and stops the recording as the setting changes.
toggleSettingMenuItem(saveRecordingSetting),
enumSettingMenuItem(transcribeMicsSetting, { onChange: startCapture }),
],
submitFrame: options.submitFrame,
setSurfaceVisible: options.setSurfaceVisible,
@@ -105,96 +97,7 @@ export function createTranscribeAppWindow(options: TranscribeAppOptions): InProc
shell.setTrayIcon(TRAY_ICON_ID, micTrayIcon(voiceControlBridge.recording));
app.requestRender();
});
startCapture();
options.startContinuousVoiceCapture({ recordAs: `transcribe-${stamp}` });
shell.setTrayIcon(TRAY_ICON_ID, micTrayIcon(voiceControlBridge.recording));
return app;
}
/**
* Transcribe in replay mode: a window of its own that transcribes a
* recording (a .mka file from Save Recording) in place of the microphones,
* in real time and through the same pipeline. It neither uses the mic nor
* shows the mic tray icon, so it runs beside a live Transcribe window. The
* header shows the position in the recording; a tap pauses and resumes; the
* menu jumps to a time.
*/
export function createTranscribeReplayWindow(
windowId: string,
path: string,
name: string,
options: InProcessAppOptions,
): InProcessWindow | null {
const replay = options.actions.openVoiceReplay?.(path) ?? null;
if (!replay) return null;
// Names a saved transcript after the recording it came from.
const layer = new TranscribeLayer(name.replace(/\.[^.]+$/, ""), replay, name);
const app = createInProcessWindow({
appId: "transcribe",
windowId,
title: name,
iconLetter: "Tr",
icon: "mic",
closeable: true,
// Relaunching Transcribe after a restart would open the microphone, not this recording.
restoresApp: false,
// Like a playing video: something to watch, not to touch.
keepsScreenOn: () => replay.state === "playing",
actions: options.actions,
baseLayer: new YieldAtRootLayer(layer),
menuItems: () => [
{
label: "Jump to time",
// The length isn't known until the recording has been opened.
disabled: () => replay.durationMs <= 0,
onSelect: (ctx) => {
ctx.stack.pop();
ctx.stack.push(
new TimePickerLayer({
title: "Jump to time",
initialMs: replay.positionMs,
maxMs: replay.durationMs,
onPick: (positionMs) => {
replay.jumpTo(positionMs);
app.requestRender();
},
}),
);
},
},
{
label: replay.state === "paused" ? "Resume" : "Pause",
disabled: () => replay.state !== "playing" && replay.state !== "paused",
onSelect: (ctx) => {
ctx.stack.pop();
replay.setPaused(replay.state === "playing");
},
},
{
label: "Restart from 0:00",
onSelect: (ctx) => {
ctx.stack.pop();
replay.jumpTo(0);
},
},
{
label: "Save",
onSelect: (ctx) => {
ctx.stack.pop();
layer.saveTranscript();
},
},
],
submitFrame: options.submitFrame,
setSurfaceVisible: options.setSurfaceVisible,
removeSurface: options.removeSurface,
onClosed: () => {
layer.onRemoved();
replay.close();
options.onClosed();
},
});
// Started first, so the layer picks its opening status up as it subscribes.
replay.start();
layer.start(app.requestRender, () => shell.isWindowVisible(windowId));
return app;
}
@@ -1,40 +0,0 @@
import { ConfigSettingEnum, glassesTypeSetting } from "../../ui/dashboard-settings";
import { type MicCount } from "../../ui/layers";
import { micControlSupported } from "../microphones/mic-control";
export type TranscribeMics = "1" | "2" | "4";
const MICS_LABELS: Record<TranscribeMics, string> = {
"1": "1 (mixed)",
"2": "2 (one per temple)",
"4": "4 (two per temple)",
};
/** Whether the paired glasses can stream their microphones separately (G2, with the microphone channel in use). */
export function separateMicsSupported(): boolean {
return glassesTypeSetting.get() === "g2" && micControlSupported();
}
/**
* How many of the glasses' microphones Transcribe captures. With more than
* one, the temples stream their own microphones (each temple on its own
* clock) instead of the glasses' mixed stream; a recording keeps every one,
* and what is transcribed is a mix made on the phone.
*/
export const transcribeMicsSetting = new ConfigSettingEnum<TranscribeMics>({
id: "transcribe-mics",
label: "Microphones",
storageKey: "transcribe.mics",
defaultValue: "1",
values: ["1", "2", "4"],
formatValue: (value) => MICS_LABELS[value],
isDisabled: (value) => value !== "1" && !separateMicsSupported(),
description:
"Which of the glasses' microphones to capture. 1 is the glasses' own mixed stream. 2 and 4 stream each temple's microphones separately (custom firmware), so a saved recording has every microphone on its own channel; the transcript then comes from a mix made on the phone.",
});
/** The setting as a capture option; 1 where the glasses can't stream their microphones separately. */
export function transcribeMicCount(): MicCount {
if (!separateMicsSupported()) return 1;
return Number(transcribeMicsSetting.get()) as MicCount;
}
+13 -72
View File
@@ -3,20 +3,15 @@ import { getDefaultSmallFont } from "../../graphics/ui-fonts";
import { lineStep } from "../../ui/metrics";
import { MicLevelMeter, paintListeningHeader, RECORDING_FAILED_NOTICE } from "../../ui/listening-header";
import { writeTextToDownloads } from "../../native/file-access";
import { voiceControlBridge, type VoiceEventSource, type VoiceReplay } from "../../native/voice-control";
import { voiceControlBridge } from "../../native/voice-control";
import { Layer, type LayerContext } from "../../ui/layers";
import { type InputEvent } from "../../ui/gestures";
import { formatReplayTime } from "./replay-time";
import { TranscriptModel, TranscriptScroll, transcriptLayout, wrapTranscribeText } from "./transcript-model";
/**
* Live transcription view. Continuous mic capture is owned by the app wrapper
* (createTranscribeAppWindow); this layer accumulates the transcript, shows
* it, and saves it to Downloads from the app menu.
*
* Given a replay, it transcribes that recording instead of the microphone:
* the header shows the position in the recording, a tap pauses and resumes,
* and each jump leaves its time as a line in the transcript.
*/
export class TranscribeLayer implements Layer {
private status = "Starting microphone...";
@@ -29,67 +24,25 @@ export class TranscribeLayer implements Layer {
private unsubscribeTranscript: (() => void) | null = null;
private unsubscribePause: (() => void) | null = null;
private unsubscribeStatus: (() => void) | null = null;
private unsubscribeReplay: (() => void) | null = null;
private readonly source: VoiceEventSource;
/**
* `stamp` names the saved transcript, matching the session's audio
* recording. `replay` is the recording to transcribe in place of the
* microphone; `replayName` is what to call it.
*/
constructor(
private readonly stamp: string = transcriptTimestamp(),
private readonly replay: VoiceReplay | null = null,
private readonly replayName = "",
) {
this.source = replay ? replay.events : voiceControlBridge;
if (replay) this.status = "Opening the recording...";
}
/** `stamp` names the saved transcript, matching the session's audio recording. */
constructor(private readonly stamp: string = transcriptTimestamp()) {}
/** `isVisible` says whether the window is on screen (the level meter and replay clock repaint only then). */
/** `isVisible` says whether the window is on screen (the level meter repaints only then). */
start(requestRender: () => void, isVisible: () => boolean): void {
this.meter.start(requestRender, isVisible, this.source);
this.unsubscribeTranscript = this.source.onTranscript((event) => {
this.meter.start(requestRender, isVisible);
this.unsubscribeTranscript = voiceControlBridge.onTranscript((event) => {
this.transcript.accept(event);
requestRender();
});
this.unsubscribePause = this.source.onSpeechPause(() => this.transcript.pause());
// The current status arrives while subscribing, before the window has a
// surface to render to; the window paints it when it is first shown.
let subscribing = true;
this.unsubscribeStatus = this.source.onStatus((state) => {
this.unsubscribePause = voiceControlBridge.onSpeechPause(() => this.transcript.pause());
this.unsubscribeStatus = voiceControlBridge.onStatus((state) => {
this.status = state.status;
this.listening = state.listening;
this.detail = state.detail;
this.saveNotice = "";
if (!subscribing) requestRender();
requestRender();
});
subscribing = false;
const replay = this.replay;
if (replay) {
let shownSecond = -1;
let shownState = replay.state;
this.unsubscribeReplay = replay.onChange((jumpedToMs) => {
if (jumpedToMs !== undefined) this.transcript.mark(`[${formatReplayTime(jumpedToMs)}]`);
// The clock ticks in whole seconds; repaint when what it shows changed.
const second = Math.floor(replay.positionMs / 1000);
if (jumpedToMs === undefined && second === shownSecond && replay.state === shownState) return;
shownSecond = second;
shownState = replay.state;
if (isVisible()) requestRender();
});
}
}
/** The header's status for a replay: where it is in the recording, and anything worth saying beside that. */
private replayStatus(replay: VoiceReplay): string {
if (replay.state === "failed" || (replay.state === "opening" && replay.durationMs === 0)) return this.status;
const withHours = replay.durationMs >= 3_600_000;
const clock = `${formatReplayTime(replay.positionMs, withHours)} / ${formatReplayTime(replay.durationMs, withHours)}`;
if (replay.state === "paused") return `${clock} · Paused`;
if (replay.state === "ended") return `${clock} · Finished`;
// Anything but plain listening is worth showing: the model loading, a provider reconnecting.
return this.listening ? clock : `${clock} · ${this.status}`;
}
paint(ctx: LayerContext): GrayImage {
@@ -97,19 +50,14 @@ export class TranscribeLayer implements Layer {
const viewport = ctx.stack.getBaseSize();
const image = new GrayImage(viewport.width, viewport.height, 0);
const replay = this.replay;
const status = replay
? this.replayStatus(replay)
: this.source.recordingFailed ? RECORDING_FAILED_NOTICE : this.status;
const playing = !replay || replay.state === "playing";
const status = voiceControlBridge.recordingFailed ? RECORDING_FAILED_NOTICE : this.status;
const bodyTop = paintListeningHeader(
image, font, "Transcribe", this.saveNotice || status, this.listening && playing ? this.meter : null, viewport.padding,
this.source.recording,
image, font, "Transcribe", this.saveNotice || status, this.listening ? this.meter : null, viewport.padding,
voiceControlBridge.recording,
);
const step = lineStep(font) + 2;
const layout = transcriptLayout(viewport, bodyTop, font.lineHeight, step);
const placeholder = replay ? `Replaying ${this.replayName}` : "Listening...";
const text = this.transcript.text || (this.listening ? placeholder : this.detail);
const text = this.transcript.text || (this.listening ? "Listening..." : this.detail);
const wrapped = wrapTranscribeText((line) => font.measureText(line), text, layout.textWidth);
const firstLine = this.scroll.layout(wrapped.length, layout.pageLines);
@@ -134,11 +82,6 @@ export class TranscribeLayer implements Layer {
ctx.actions.requestRender();
return;
}
if (event.type === "click" && this.replay) {
this.replay.setPaused(this.replay.state === "playing");
ctx.actions.requestRender();
return;
}
if (event.type === "double-click") {
ctx.stack.pop();
}
@@ -146,8 +89,6 @@ export class TranscribeLayer implements Layer {
onRemoved(): void {
this.meter.stop();
this.unsubscribeReplay?.();
this.unsubscribeReplay = null;
this.unsubscribePause?.();
this.unsubscribePause = null;
this.unsubscribeTranscript?.();
-12
View File
@@ -36,18 +36,6 @@ export class TranscriptModel {
if (this.live) this.breakAfterLive = true;
else if (this.finalized) this.separator = "\n";
}
/**
* A line of its own (where a replay jumped to), ending whatever was in
* progress: what follows it starts a new paragraph.
*/
mark(label: string): void {
const before = this.text;
this.finalized = before ? `${before}\n${label}` : label;
this.live = "";
this.separator = "\n";
this.breakAfterLive = false;
}
}
/** An absolute history position, with an explicit follow-tail mode. */
+8 -4
View File
@@ -4,6 +4,8 @@ import {
resolveTranslateLanguage,
resolveTranslateProvider,
translateLanguageSetting,
translateModelSetting,
translateTargetLanguageSetting,
translateProviderSetting,
translateShowOriginalSetting,
} from "./translate-settings";
@@ -12,7 +14,7 @@ import { transcriptTimestamp } from "../transcribe/transcribe";
import { voiceControlBridge } from "../../native/voice-control";
import { type ContinuousCaptureOptions } from "../../ui/layers";
import { openSettingsSubMenu } from "../../ui/dashboard/settings-panel";
import { enumSettingMenuItem, saveRecordingSetting, toggleSettingMenuItem } from "../../ui/dashboard-settings";
import { enumSettingMenuItem, saveRecordingSetting, textSettingMenuItem, toggleSettingMenuItem } from "../../ui/dashboard-settings";
import {
createInProcessWindow,
YieldAtRootLayer,
@@ -29,8 +31,8 @@ const TRAY_ICON_ID = "translate";
const NO_PROVIDER_PROBLEM =
"Translate needs a transcription provider that understands any language: add an ElevenLabs, OpenAI, or Soniox API key (Settings > API Keys), or a Whisper server URL (Settings > Voice).";
const NO_MODEL_PROBLEM = global.isIOS
? "Translate needs a language model: configure a cloud or OpenAI-compatible model in Settings > Assistant."
: "Translate needs a language model: configure one in Settings > Assistant, or download the on-phone model.";
? "Translate needs a language model: set an OpenAI model in Translate settings and add its API key, or configure a cloud or OpenAI-compatible model in Settings > Assistant."
: "Translate needs a language model: set an OpenAI model in Translate settings and add its API key, or configure one in Settings > Assistant, or download the on-phone model.";
export type TranslateAppOptions = InProcessAppOptions & {
/** Begin (or retarget) continuous mic capture, kept on while the window is open. */
@@ -39,7 +41,7 @@ export type TranslateAppOptions = InProcessAppOptions & {
};
/**
* The Translate app: live speech in any language, translated into English by
* The Translate app: live speech in any language, translated into the selected output language by
* the assistant's model, in its own window. Like Transcribe it holds
* continuous mic capture while open, but always with a multilingual cloud
* provider (Translate settings), which a shared capture switches to.
@@ -95,6 +97,8 @@ export function createTranslateAppWindow(options: TranslateAppOptions): InProces
openSettingsSubMenu(ctx, "Translate settings", [
enumSettingMenuItem(translateLanguageSetting, { onChange: startCapture }),
enumSettingMenuItem(translateProviderSetting, { onChange: startCapture }),
enumSettingMenuItem(translateTargetLanguageSetting),
textSettingMenuItem(translateModelSetting),
toggleSettingMenuItem(translateShowOriginalSetting),
]);
},
+14 -7
View File
@@ -1,6 +1,6 @@
/**
* Streamed translation against the assistant's configured model (Settings >
* Assistant). Each call translates one stretch of speech into English; see
* Streamed translation against Translate's OpenAI override, or the assistant's
* configured model when no override is set. Each call translates one stretch of speech into the selected output language; see
* TranslationModel for how stretches are batched and given context.
*/
import { streamAnthropicMessage } from "../../native/anthropic";
@@ -13,7 +13,8 @@ import {
anthropicApiKeySetting, assistantModelSetting, assistantOpenAiCompatibleKeySetting,
assistantOpenAiCompatibleModelSetting, assistantOpenAiCompatibleUrlSetting, openAiApiKeySetting,
} from "../../ui/dashboard-settings";
import { TRANSLATE_SYSTEM_PROMPT, buildTranslateUserMessage } from "../../prompts";
import { buildTranslateSystemPrompt, buildTranslateUserMessage } from "../../prompts";
import { translateModelSetting, translateTargetLanguageSetting, spokenLanguageName } from "./translate-settings";
import { type TranslationTurn } from "./translation-model";
/**
@@ -23,6 +24,11 @@ import { type TranslationTurn } from "./translation-model";
* for whichever API key is set, else the on-phone model.
*/
export function resolveTranslationModel(): ResolvedAssistantModel | null {
const translationModel = translateModelSetting.get().trim();
if (translationModel) {
const apiKey = openAiApiKeySetting.get().trim();
return apiKey ? { selection: "auto", provider: "openai", model: translationModel, apiKey } : null;
}
const selection = assistantModelSetting.get();
return resolveAssistantModel(selection === "hermes" ? "auto" : selection, {
anthropic: anthropicApiKeySetting.get(),
@@ -44,13 +50,14 @@ export type TranslationRequest = {
};
export function streamTranslation(model: ResolvedAssistantModel, request: TranslationRequest): LlmStreamHandle {
const targetLanguage = spokenLanguageName(translateTargetLanguageSetting.get());
const options: LlmStreamOptions = {
apiKey: model.apiKey,
model: model.model,
system: TRANSLATE_SYSTEM_PROMPT,
system: buildTranslateSystemPrompt(targetLanguage),
effort: model.effort,
maxTokens: 1024,
messages: [{ role: "user", content: buildTranslateUserMessage(request.speech, request.context, request.language) }],
messages: [{ role: "user", content: buildTranslateUserMessage(request.speech, request.context, request.language, targetLanguage) }],
onTextDelta: (_delta, textSoFar) => request.onProgress(cleanTranslation(textSoFar)),
onDone: (result) => {
if (result.stopReason === "refusal") request.onError("The model declined to translate this.");
@@ -70,7 +77,7 @@ export function streamTranslation(model: ResolvedAssistantModel, request: Transl
}
}
/** The context turns are tagged <english>; a model may echo the tag around its reply. */
/** The context turns are tagged <translation>; a model may echo the tag around its reply. */
function cleanTranslation(text: string): string {
return text.replace(/^\s*<english>/i, "").replace(/<\/english>\s*$/i, "").trim();
return text.replace(/^\s*<(?:translation|english)>/i, "").replace(/<\/(?:translation|english)>\s*$/i, "").trim();
}
+23 -1
View File
@@ -1,4 +1,4 @@
import { ConfigSettingBoolean, ConfigSettingEnum, voiceProviderSetting, type VoiceProvider } from "../../ui/dashboard-settings";
import { ConfigSettingBoolean, ConfigSettingEnum, ConfigSettingString, voiceProviderSetting, type VoiceProvider } from "../../ui/dashboard-settings";
/**
* The transcription providers that recognize any spoken language, in picker
@@ -52,6 +52,7 @@ export const translateProviderSetting = new ConfigSettingEnum<TranslateProvider>
* form every multilingual provider accepts), in picker order.
*/
const SPOKEN_LANGUAGES = {
en: "English",
ar: "Arabic",
zh: "Chinese",
nl: "Dutch",
@@ -106,3 +107,24 @@ export const translateShowOriginalSetting = new ConfigSettingBoolean({
defaultValue: false,
description: "Show what was said, dimmed, above each translation.",
});
/** Empty preserves the existing Assistant-based routing. */
export const translateModelSetting = new ConfigSettingString({
id: "translate-model",
label: "Translation model (OpenAI)",
storageKey: "translate.model",
defaultValue: "",
inputKind: "text",
normalize: (value) => value.trim(),
description: "OpenAI model ID used only by Translate, for example gpt-4o-mini. Uses the OpenAI key from Settings > API Keys. Leave empty to use the Assistant model. Changes apply to subsequent translation requests; Assistant settings are unchanged.",
});
export const translateTargetLanguageSetting = new ConfigSettingEnum<SpokenLanguage>({
id: "translate-target-language",
label: "Output language",
storageKey: "translate.targetLanguage",
defaultValue: "en",
values: Object.keys(SPOKEN_LANGUAGES) as SpokenLanguage[],
formatValue: (value) => SPOKEN_LANGUAGES[value],
description: "Language of subsequent translations. Independent of the spoken language and Assistant settings. Existing translations stay unchanged.",
});
+1 -1
View File
@@ -18,7 +18,7 @@ const NO_MODEL_NOTICE = "No language model is set up.";
* Live translation view. Continuous mic capture is owned by the app wrapper
* (createTranslateAppWindow); this layer feeds speech to the assistant's
* model one block at a time (provisionally while a block is still being
* spoken), shows the English as it streams in, and saves the session to
* spoken), shows the translation as it streams in, and saves the session to
* Downloads from the app menu.
*/
export class TranslateLayer implements Layer {
+1 -1
View File
@@ -3,7 +3,7 @@ import { wrapTranscribeText } from "../transcribe/transcript-model";
type TranscriptEvent = { text: string; isFinal: boolean };
/**
* One block of speech (up to a pause) and its English translation. The last
* One block of speech (up to a pause) and its translation. The last
* block may still be live, its transcript revising; it is translated
* provisionally as it grows, then again once final if its text changed.
*/
-62
View File
@@ -1,62 +0,0 @@
/**
* Date and calendar-event text for the assistant's tool results. Pure (no
* native imports) so it can be unit-tested.
*/
const WEEKDAYS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
export type FormattableEvent = {
title: string;
startMs: number;
endMs: number;
allDay: boolean;
location?: string;
};
export function formatEvent(event: FormattableEvent): string {
const location = event.location ? ` @ ${event.location}` : "";
return `- ${event.title || "(untitled)"} — ${formatEventSpan(event)}${location}`;
}
/**
* When an event happens, end included: a multi-day all-day event names its last
* day (all-day ends are exclusive, at the midnight after it), a timed event its
* end time. Without the end, an event that started yesterday and runs through
* tomorrow reads as yesterday's.
*/
export function formatEventSpan(event: Pick<FormattableEvent, "startMs" | "endMs" | "allDay">): string {
const start = new Date(event.startMs);
if (event.allDay) {
const lastDay = new Date(Math.max(event.startMs, event.endMs - 1));
return sameDay(start, lastDay)
? `${formatDate(start)} (all day)`
: `${formatDate(start)} – ${formatDate(lastDay)} (all day)`;
}
if (!(event.endMs > event.startMs)) return formatDateTime(start);
const end = new Date(event.endMs);
return sameDay(start, end)
? `${formatDateTime(start)} – ${formatTime(end)}`
: `${formatDateTime(start)} – ${formatDateTime(end)}`;
}
export function formatDateTime(date: Date): string {
return `${formatDate(date)}, ${formatTime(date)}`;
}
export function formatDate(date: Date): string {
return `${WEEKDAYS[date.getDay()]} ${MONTHS[date.getMonth()]} ${date.getDate()}`;
}
export function formatTime(date: Date): string {
let hours = date.getHours();
const minutes = date.getMinutes().toString().padStart(2, "0");
const meridiem = hours >= 12 ? "PM" : "AM";
hours = hours % 12;
if (hours === 0) hours = 12;
return `${hours}:${minutes} ${meridiem}`;
}
function sameDay(a: Date, b: Date): boolean {
return a.getFullYear() === b.getFullYear() && a.getMonth() === b.getMonth() && a.getDate() === b.getDate();
}
+27 -2
View File
@@ -1,11 +1,10 @@
import { readUpcomingEventsAsync } from "../native/calendar";
import { readUpcomingEventsAsync, type CalendarEvent } from "../native/calendar";
import { hasCalendarPermission } from "../native/calendar-permissions";
import { mediaControllerBridge } from "../native/media-controller";
import { spotifyService } from "../native/spotify";
import { dismissNotification, readActiveNotifications } from "../native/notification-icons";
import { shell } from "../ui/shell/shell";
import { toolRegistry, type ToolRegistry, type ToolResult } from "./tool-registry";
import { formatDateTime, formatEvent } from "./date-format";
/**
* Registers the always-available system tools into the registry. Called once at
@@ -13,6 +12,9 @@ import { formatDateTime, formatEvent } from "./date-format";
* media, notifications); nothing here needs an app window to be open.
*/
const WEEKDAYS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
let registered = false;
export function registerSystemTools(registry: ToolRegistry = toolRegistry): void {
@@ -242,12 +244,35 @@ function describeNowPlaying(): string {
return "Nothing is currently playing.";
}
function formatEvent(event: CalendarEvent): string {
const when = event.allDay ? `${formatDate(new Date(event.startMs))} (all day)` : formatDateTime(new Date(event.startMs));
const location = event.location ? ` @ ${event.location}` : "";
return `- ${event.title || "(untitled)"} — ${when}${location}`;
}
function formatNotification(notification: { key: string; appName: string; title: string; text: string; customText?: string[] }): string {
const title = notification.title || notification.customText?.join(" ") || "(no title)";
const body = notification.text ? `: ${notification.text}` : "";
return `- [${notification.appName}] ${title}${body} (key: ${notification.key})`;
}
function formatDateTime(date: Date): string {
return `${formatDate(date)}, ${formatTime(date)}`;
}
function formatDate(date: Date): string {
return `${WEEKDAYS[date.getDay()]} ${MONTHS[date.getMonth()]} ${date.getDate()}`;
}
function formatTime(date: Date): string {
let hours = date.getHours();
const minutes = date.getMinutes().toString().padStart(2, "0");
const meridiem = hours >= 12 ? "PM" : "AM";
hours = hours % 12;
if (hours === 0) hours = 12;
return `${hours}:${minutes} ${meridiem}`;
}
function clampNumber(value: unknown, min: number, max: number, fallback: number): number {
const num = typeof value === "number" ? value : Number(value);
if (!Number.isFinite(num)) return fallback;
+4 -3
View File
@@ -193,7 +193,7 @@ function registerTools(options: WindowToolsOptions, registry: ToolRegistry): voi
const lines: string[] = [];
for (const [name, members] of getFolders()) {
const known = members.filter((appId) => appIds.indexOf(appId) >= 0).sort();
if (known.length) lines.push(`${name}: ${known.map(describeApp).join(", ")}`);
if (known.length || members.length === 0) lines.push(`${name}: ${known.length ? known.map(describeApp).join(", ") : "(empty)"}`);
}
const assignments = getFolderAssignments();
const ungrouped = appIds.filter((appId) => !assignments[appId]);
@@ -239,7 +239,7 @@ function registerTools(options: WindowToolsOptions, registry: ToolRegistry): voi
{
name: "apps.remove_from_folder",
description:
"Move an app out of its launcher folder back to the top-level grid. A folder disappears when its last app is removed.",
"Move an app out of its launcher folder back to the top-level grid. Default folders disappear when their last app is removed; user-created folders can stay empty.",
inputSchema: {
type: "object",
properties: {
@@ -278,8 +278,9 @@ function registerTools(options: WindowToolsOptions, registry: ToolRegistry): voi
(args) => {
const folder = resolveFolderName(String(args?.folder ?? ""));
if (!folder) return err("folder must be a non-empty name");
const existed = getFolders().has(folder);
const moved = disbandFolder(folder);
if (moved === 0) {
if (moved === 0 && !existed) {
const names = Array.from(getFolders().keys());
return err(
names.length
+2
View File
@@ -1,3 +1,4 @@
import { startHealthExport } from "./health/health-export"
import { Application } from '@nativescript/core'
import { registerShareIntentHandler } from './native/share-intents'
import { installNativeUserAgent } from './util/http'
@@ -18,6 +19,7 @@ Application.on(Application.launchEvent, () => {
// they are closed would be lost if the process restarted first, so this keeps
// them reaching disk regardless. No-op when there is nothing new.
startLiveHealthSync()
startHealthExport()
// Collect from the ring on a wall-clock cadence (:01 and :31). Without this
// nothing drives a pull on a timer at all - only onRingReady() on reconnect -
+88 -145
View File
@@ -16,17 +16,13 @@ import { mediaControllerBridge } from "../native/media-controller";
import { nightscoutBridge } from "../native/nightscout-bridge";
import { weatherBridge } from "../native/weather";
import { ALL_NOTIFICATIONS, onAndroidNotificationPosted, onAndroidNotificationRemoved, readActiveNotifications } from "../native/notification-icons";
import { isPhoneDoNotDisturbEnabled } from "../native/phone-do-not-disturb";
import { watchNotificationListenerBinding } from "../native/notification-access";
import { openEvenAppSettings, readEvenAppNotificationState } from "../native/even-app-conflict";
import { defaultRingName, setGlassesRingLink } from "../native/ring-status";
import { applyGlanceAngle, glassesHeadUpTrigger, syncGlanceAngleOriginal } from "./head-angle";
import { glanceAngleTriggerDegrees, glanceTapTimeoutMs } from "../apps/glanceboard/glanceboard-settings";
import { GLANCE_TILT_MAX_MS, glanceTiltMaxMs } from "./glance-state";
import { getBooleanSetting, setBooleanSetting } from "../native/settings-store";
import { grayImageToPreviewSource } from "../native/gray-image-preview";
import { firmwareIncompatibilityMessage, hasCompatibleFirmware } from "./firmware-compat";
import { keyboardService } from "./keyboard/service";
import type { KeyEvent } from "./keyboard/keys";
import { hasExtractedEvenHubFonts } from "./firmware-builder";
import { isHaltedSessionPhase, resumeAutoReconnect, suppressAutoReconnect } from "./reconnect-policy";
import { WearRemote, type WearRemoteInputKind } from "./wear-remote";
@@ -58,7 +54,7 @@ import { type GlanceEvent } from "./glance-state";
import { isPreviewOnlyMode } from "../phone-ui/onboarding-state";
import { claimGlassesOnboardingAutoLaunch, ONBOARDING_APP_ID } from "../apps/onboarding/onboarding-progress";
import { beginRenderPass, endRenderPass } from "../util/render-freshness";
import { VoiceReplay, voiceControlBridge } from "../native/voice-control";
import { voiceControlBridge } from "../native/voice-control";
import { voiceActivity } from "../ui/shell/voice-activity";
import { GrayImage } from "../graphics/image";
import { flattenPlanesWithDraws, planesFingerprint, type Plane } from "../graphics/plane";
@@ -81,9 +77,9 @@ import { type InProcessAppOptions, type InProcessWindow } from "../ui/shell/in-p
import { loadPersistedOpenApps, savePersistedOpenApps } from "../ui/shell/open-apps-persistence";
import { appViewportRect, isOnSwitcherEdge, screenHeight, screenWidth, sidebarStripVisible, windowEdges, type ScreenRect, type WindowHeightMode } from "../ui/shell/geometry";
import { sameEdges } from "../ui/viewport";
import { type ContinuousCaptureOptions, type LayerActions, type MicCount, type TextSettingsEditToggle, type TranscriptionChoice } from "../ui/layers";
import { type ContinuousCaptureOptions, type LayerActions, type TextSettingsEditToggle, type TranscriptionChoice } from "../ui/layers";
import { type KeyboardInputSession } from "../ui/shell/keyboard-input";
import { appSwitcherPositionSetting, glassesModelName, glassesTypeSetting, statusBarPositionSetting, statusBarVisibilitySetting, windowBorderSetting, assistantAllowProactiveSetting, assistantBackendSetting, assistantBridgeHostSetting, assistantBridgePortSetting, assistantBridgeTokenSetting, getBrightnessPreferences, displayLayoutSetting, navigateDisplayLayoutSetting, terminalDisplayLayoutSetting, elevenLabsApiKeySetting, getStringSettingById, openAiApiKeySetting, nightscoutApiTokenSetting, firmwareDebugFlagsSetting, whisperServerApiKeySetting, whisperServerLanguageSetting, whisperServerModelSetting, whisperServerUrlSetting, lockScreenEnabledSetting, nightscoutSiteUrlSetting, onAnySettingChanged, previewColorSetting, effectiveRingConnectionMode, type RingConnectionMode, saveRecordingSetting, sonioxApiKeySetting, screenTimeoutSetting, screenTimeoutSettingToMs, shouldPopUpNewNotification, suspendEvenHubWhenScreenOffSetting, voiceProviderSetting, wakeWordActionSetting, type ConfigSettingString } from "../ui/dashboard-settings";
import { appSwitcherPositionSetting, glassesModelName, glassesTypeSetting, statusBarPositionSetting, statusBarVisibilitySetting, windowBorderSetting, assistantAllowProactiveSetting, assistantBackendSetting, assistantBridgeHostSetting, assistantBridgePortSetting, assistantBridgeTokenSetting, getBrightnessPreferences, getHeadUpPreferences, displayLayoutSetting, navigateDisplayLayoutSetting, terminalDisplayLayoutSetting, elevenLabsApiKeySetting, getStringSettingById, openAiApiKeySetting, openAiTranscriptionModelSetting, nightscoutApiTokenSetting, firmwareDebugFlagsSetting, whisperServerApiKeySetting, getVoiceTranscriptionLanguage, whisperServerModelSetting, whisperServerUrlSetting, lockScreenEnabledSetting, nightscoutSiteUrlSetting, onAnySettingChanged, previewColorSetting, effectiveRingConnectionMode, type RingConnectionMode, saveRecordingSetting, sonioxApiKeySetting, screenTimeoutSetting, screenTimeoutSettingToMs, shouldPopUpNewNotification, suspendEvenHubWhenScreenOffSetting, voiceProviderSetting, wakeWordActionSetting, type ConfigSettingString } from "../ui/dashboard-settings";
import { isIgnoringBatteryOptimizations, requestIgnoreBatteryOptimizations } from "../native/battery-optimization";
import {
getInstalledEvenHubAppById,
@@ -103,8 +99,6 @@ import { createNotificationsAppWindow, NOTIFICATIONS_SURFACE_ID, NOTIFICATIONS_W
import { isInstalledPackagePresent } from "../apps/evenhub/updates";
import { wearerVerificationOptions } from "../apps/microphones/speakers";
import { micSession } from "../apps/microphones/mic-session";
import { micArrayCaptureConfig, micControlSupported } from "../apps/microphones/mic-control";
import { encodeMicControl } from "../apps/microphones/mic-protocol";
import {
chargingPreviewMessage,
glassesDisplayAsleep,
@@ -126,6 +120,7 @@ export type DashboardSnapshot = {
* glasses, and this says which harmless thing is actually going on.
*/
displayPreviewMessage: string;
silentModeFeedback: string;
activeTextSettingId: string | null;
activeTextEditorTitle: string;
activeTextSettingTitle: string;
@@ -301,8 +296,12 @@ class DashboardController {
/** The screen area displayPreview shows (Preview scaling), or null for the whole panel. */
private displayPreviewCrop: ScreenRect | null = null;
private silentMode = false;
private silentModeFeedback = "";
private silentModeFeedbackGeneration = 0;
private silentModeFeedbackTimer: ReturnType<typeof setTimeout> | null = null;
// Brightness value last pushed to the glasses; see pushBrightness.
private lastPushedBrightness: string | null = null;
private lastPushedHeadUp: string | null = null;
// Last battery level the glasses reported, kept across disconnects so a
// drop-off right after a low reading can be explained as a flat battery.
private lastHeadsetBattery: number | null = null;
@@ -340,7 +339,6 @@ class DashboardController {
private evenHubResumePromise: Promise<boolean> | null = null;
/** Faceclaw's firmware at the required revision reported in; its extensions can be used. */
private customFirmwareConfirmed = false;
private lastGlanceAngleDegrees = glanceAngleTriggerDegrees();
private faceclawWakeLeaseState: boolean | null = null;
/** Ring connection mode the current session was started with (it applies per connection). */
private ringModeAtConnect: RingConnectionMode = "glasses";
@@ -368,7 +366,7 @@ class DashboardController {
private offAndroidNotification: (() => void) | null = null;
private offAndroidNotificationRemoved: (() => void) | null = null;
/** New notifications wait here briefly before popping up (see NotificationPopupDelay). */
private readonly notificationPopups = new NotificationPopupDelay((notificationKey) => this.popUpAndroidNotification(notificationKey));
private readonly notificationPopups = new NotificationPopupDelay((notificationKey, canWakeScreen) => this.popUpAndroidNotification(notificationKey, canWakeScreen));
private lastInput = "waiting...";
private lastSys = "none yet";
private shellRenderInProgress = false;
@@ -418,7 +416,6 @@ class DashboardController {
startContinuousVoiceCapture: (client: string, options?: ContinuousCaptureOptions) =>
this.startContinuousVoiceCapture(client, options),
stopContinuousVoiceCapture: (client: string) => this.stopContinuousVoiceCapture(client),
openVoiceReplay: (path: string) => this.openVoiceReplay(path),
playBuzzerSequence: (payload: Uint8Array) => this.playBuzzerSequence(payload),
};
this.sharedActions = sharedActions;
@@ -461,12 +458,13 @@ class DashboardController {
this.openInProcessAppInBackground(NOTIFICATIONS_WINDOW_ID, NOTIFICATIONS_SURFACE_ID,
(options) => createNotificationsAppWindow(options, (section) => this.openSettings(section))),
onSwitcherBarHeightChanged: () => this.applySwitcherBarHeight(),
onWindowLayoutChanged: () => this.applySwitcherBarHeight(),
isLocked: () => this.glassesLocked,
onScreenStateChanged: (on) => {
// Any wake of the regular UI replaces a showing Glanceboard.
if (on) this.glance.dismiss();
this.handleScreenStateChanged(on);
if (on) this.requestShellRender();
if (on) keyboardService.retryNow();
this.emit();
},
});
@@ -476,7 +474,7 @@ class DashboardController {
app.boot?.(this.buildAppContext(app));
}
timerEngine.onChange(() => this.refreshAlarmReliabilityWarning());
this.offAndroidNotification = onAndroidNotificationPosted((notificationKey) => this.handleAndroidNotificationPosted(notificationKey));
this.offAndroidNotification = onAndroidNotificationPosted((notificationKey, canWakeScreen) => this.handleAndroidNotificationPosted(notificationKey, canWakeScreen));
this.offAndroidNotificationRemoved = onAndroidNotificationRemoved((notificationKey) => this.handleAndroidNotificationRemoved(notificationKey));
watchNotificationListenerBinding();
readActiveNotifications(ALL_NOTIFICATIONS);
@@ -487,6 +485,7 @@ class DashboardController {
// Apply a firmware-debug-flags toggle live while connected (Java dedups).
this.pushFirmwareDebugFlags();
this.pushBrightness();
this.pushHeadUpPreferences();
this.syncEvenHubScreenOffSetting();
this.syncVoiceWake();
// Display mode size first: a reflow repositions every surface too, so
@@ -497,8 +496,8 @@ class DashboardController {
// Either reflow hands windows their new edges along with their size.
if (!resized && !appsResized) this.applyWindowEdgesIfChanged();
this.syncAssistantBridgeIfChanged();
this.retargetContinuousCapture();
this.syncLockScreenSettingIfChanged();
this.applyGlanceAngleIfChanged();
this.syncContinuousRecording();
// A Preview color or scaling change should show on the mirror at once,
// not at the next composited frame (cheap: floor-limited, dedup'd on
@@ -520,29 +519,6 @@ class DashboardController {
assistantAvailable: () => shell.isAssistantAvailable(),
assistant: text => shell.sendToAssistant(text),
});
// A keyboard paired through the glasses (CFW keyboard bridge). The
// service decides what each key means (app/g2/keyboard/key-router.ts);
// this is how it reaches the display and the input pipeline.
keyboardService.setHost({
ready: () => this.phase === "connected",
locked: () => this.glassesLocked,
wake: () => {
void this.handleInputEvent({
kind: "display-wake",
containerName: "",
eventType: OsEventTypeList.DOUBLE_CLICK_EVENT,
eventSource: EventSourceType.TOUCH_EVENT_FORM_DUMMY_NULL,
systemExitReasonCode: 0,
frameId: 0,
}).catch((error) => this.appendLog(`keyboard wake failed: ${this.formatError(error)}`));
},
gesture: (gesture, origin) => this.injectSyntheticRingInput(gesture, origin),
log: (line) => this.appendLog(line),
// A keyboard that is switched off would otherwise keep the glasses'
// radio looking for it around the clock. One still connected when the
// display sleeps stays connected, and its next key wakes the display.
shouldReconnect: () => this.phase === "connected" && shell.isScreenOn(),
});
this.wearRemote = new WearRemote({
apps: launchableApps,
injectInput: (kind) => this.injectSyntheticRingInput(kind, "watch"),
@@ -1174,6 +1150,19 @@ class DashboardController {
});
}
private pushHeadUpPreferences(force = false): void {
if (!global.isAndroid || glassesTypeSetting.get() !== "g2" || !this.communicator) return;
const { enabled, angle } = getHeadUpPreferences();
if (enabled < 0 && angle < 0) return;
const value = JSON.stringify({ enabled, angle });
if (!force && value === this.lastPushedHeadUp) return;
this.lastPushedHeadUp = value;
void this.communicator.configureHeadUpWake(enabled, angle).catch(error => {
this.lastPushedHeadUp = null;
this.appendLog(`head-up configuration failed: ${this.formatError(error)}`);
});
}
/** Save the occupied part of the composited screen as a 4-bit grayscale PNG. */
saveScreenshot(): string {
const path = this.display?.saveScreenshot(shell.screenshotCropRect()) ?? "";
@@ -1232,6 +1221,7 @@ class DashboardController {
status: this.status,
displayPreview: this.displayPreview,
displayPreviewMessage: this.displayPreviewMessage(),
silentModeFeedback: this.silentModeFeedback,
activeTextSettingId: primaryTextSetting?.id ?? null,
activeTextEditorTitle: this.activeTextEditorTitle,
activeTextSettingTitle: primaryTextSetting?.editorTitle ?? "",
@@ -1270,6 +1260,43 @@ class DashboardController {
* charging and silent mode both make the display unavailable while BLE stays
* up, and a battery that just ran out simply stops answering.
*/
private handleSilentMode(silent: boolean): void {
if (this.silentMode === silent) return;
this.silentMode = silent;
const generation = ++this.silentModeFeedbackGeneration;
if (this.silentModeFeedbackTimer !== null) clearTimeout(this.silentModeFeedbackTimer);
this.silentModeFeedbackTimer = null;
this.silentModeFeedback = silent
? "Silent mode enabled — hold both touchpads to exit"
: "Silent mode disabled";
if (!silent) {
void this.showSilentModeExit(generation).catch(error => {
this.appendLog(`silent-mode exit feedback failed: ${this.formatError(error)}`);
});
this.silentModeFeedbackTimer = setTimeout(() => {
this.silentModeFeedbackTimer = null;
this.silentModeFeedback = "";
this.emit();
}, 5000);
}
this.emit();
}
private async showSilentModeExit(generation: number): Promise<void> {
if (this.phase !== "connected" || this.silentMode || this.glassesLocked) return;
// The firmware powered the lenses down, and Faceclaw may have gone to
// sleep/suspended EvenHub meanwhile. Reusing the ordinary wake barrier is
// essential: rendering an alert into the retained surface alone is invisible.
this.cancelEvenHubSuspendTimer();
shell.wake("sidebar");
const ready = await this.ensureEvenHubSessionActive();
if (!ready || generation !== this.silentModeFeedbackGeneration || this.silentMode ||
this.phase !== "connected" || this.glassesLocked) return;
shell.showAlert("Silent mode disabled");
shell.foregroundWindow()?.requestRender();
this.requestShellRender();
}
private displayPreviewMessage(): string {
if (this.phase === "charging") {
return chargingPreviewMessage(this.glassesDisplayState());
@@ -1629,6 +1656,7 @@ class DashboardController {
// The paired glasses' screen: the G2's 640x480 or the RayNeo iO's 540x220.
await communicator.configureCompositorScreen(screenWidth(), screenHeight());
await communicator.configureBrightness(getBrightnessPreferences());
this.pushHeadUpPreferences(true);
this.voiceWakeSent = null;
this.syncVoiceWake(communicator);
this.offState = communicator.onStateChange((state) => {
@@ -1668,6 +1696,7 @@ class DashboardController {
// mode-7 control message once the dashboard container is warmed up.
this.pushFirmwareDebugFlags();
this.pushBrightness(true);
this.pushHeadUpPreferences(true);
this.handRingBackToGlassesIfNeeded();
}
if (mappedPhase !== "connected" && mappedPhase !== "charging") {
@@ -1689,8 +1718,6 @@ class DashboardController {
if (mappedPhase === "connected") {
this.syncEvenHubScreenOffSetting();
this.ensureWearStateTracking();
// The session reapplies it on reconnects; restate it for a new session.
applyGlanceAngle();
this.maybeWakeAfterChargingReconnect();
} else if (mappedPhase !== "charging") {
this.cancelEvenHubSuspendTimer();
@@ -1704,9 +1731,7 @@ class DashboardController {
});
});
this.offSilentMode = communicator.onSilentMode((silent) => {
if (this.silentMode === silent) return;
this.silentMode = silent;
this.emit();
this.handleSilentMode(silent);
});
this.offWearState = communicator.onWearState((wearing) => {
this.handleWearState(wearing);
@@ -1799,7 +1824,6 @@ class DashboardController {
this.appendLog(`wake takeover lease sync failed: ${this.formatError(error)}`);
});
this.ensureWearStateTracking();
this.syncKeyboardLink();
}
// The shared session halts on incompatible firmware by itself
// ("incompatible-firmware" phase); this only explains why.
@@ -1853,7 +1877,6 @@ class DashboardController {
this.requestShellRender();
this.updateCompositePreview();
this.updateConnectedForegroundNotification();
syncGlanceAngleOriginal();
}, SHELL_REFRESH_INTERVAL_MS);
this.previewTimer = setInterval(() => this.updateCompositePreview(), PREVIEW_INTERVAL_MS);
this.screenTimeoutTimer = setInterval(() => {
@@ -2291,30 +2314,6 @@ class DashboardController {
return undefined;
}
/**
* How many of the glasses' microphones the continuous-capture windows call
* for (the most any asks for), where the glasses can stream them
* separately: G2 with the microphone channel in use. Otherwise 1, the
* mixed stream.
*/
private continuousMics(): MicCount {
if (glassesTypeSetting.get() !== "g2" || !micControlSupported()) return 1;
let mics: MicCount = 1;
for (const options of this.continuousClients.values()) {
if ((options.mics ?? 1) > mics) mics = options.mics!;
}
return mics;
}
/**
* A recording as a capture of its own (see VoiceReplay), transcribed with
* the provider and processing settings a microphone capture would get.
* "My voice only" is left out: a recording is of whoever spoke.
*/
private openVoiceReplay(path: string): VoiceReplay {
return new VoiceReplay(path, () => ({ ...this.voiceCaptureOptions(null), speakerVerification: undefined }));
}
/**
* Continuous capture's claim on the mic (the Microphones app and EvenHub
* mic apps stand down while it is held) follows whether the capture really
@@ -2337,33 +2336,16 @@ class DashboardController {
private voiceCaptureOptions(communicator: FaceclawCommunicatorBridge | null) {
// A window that needs its own provider and language sets them for every
// capture while it is open, so push-to-talk still shares its stream.
// Otherwise only Whisper server is told a language (its own setting).
// Otherwise use the shared transcription-language preference.
const choice = this.transcriptionChoice();
const provider = choice?.provider ?? voiceProviderSetting.get();
// Separate microphones only come from connected glasses.
const mics = communicator ? this.continuousMics() : 1;
const micConfig = mics === 1 ? null : micArrayCaptureConfig(mics);
return {
communicator: communicator?.getNativeCommunicator() ?? null,
usePhoneMic: communicator === null,
micArray: micConfig ? { configure: encodeMicControl("configure", micConfig) } : null,
// Stored in a recording of the capture (see Save Recording).
captureInfo: {
glasses: communicator ? glassesModelName() : null,
mics,
micConfig: micConfig && {
source: micConfig.source,
channelMask: micConfig.channelMask,
codec: micConfig.codec,
format: micConfig.format,
sampleRateHz: micConfig.sampleRateHz,
directionDetection: micConfig.beamform,
},
},
provider,
language: choice ? choice.language : provider === "whisper-server" ? whisperServerLanguageSetting.get() : "",
language: choice ? choice.language : getVoiceTranscriptionLanguage(provider),
elevenLabsApiKey: elevenLabsApiKeySetting.get(),
openAiApiKey: openAiApiKeySetting.get(),
openAiApiKey: openAiApiKeySetting.get(), openAiTranscriptionModel: openAiTranscriptionModelSetting.get(),
sonioxApiKey: sonioxApiKeySetting.get(),
whisperServerUrl: whisperServerUrlSetting.get(),
whisperServerModel: whisperServerModelSetting.get(),
@@ -2542,16 +2524,6 @@ class DashboardController {
frameTimings.logFrame(frameId, `TS input handler start: ${event.kind} ${eventLabel(event.kind, event.eventType)}`);
let frameOwned = false;
try {
if (event.kind === "sys-event" && event.eventType === OsEventTypeList.HEAD_DOWN_EVENT) {
// Only the Glanceboard cares (it ends a head-tilt showing); never
// shell input, so it can't count as activity or wake anything.
const glanceEvent = this.glance.eventForGesture("head-down");
if (glanceEvent) {
await this.glance.handleEvent(glanceEvent, frameId);
frameOwned = true;
}
return;
}
const inputEvent = rawInputEventToInputEvent(event);
if (!acceptInput(inputEvent)) return;
// The gesture, plus which app is on screen and whether input goes to it,
@@ -2670,49 +2642,15 @@ class DashboardController {
}
}
/** Re-send the head-tilt angle when it changed (the setting, from either settings UI, or the Glanceboard toggle). */
private applyGlanceAngleIfChanged(): void {
const degrees = glanceAngleTriggerDegrees();
if (degrees === this.lastGlanceAngleDegrees) return;
this.lastGlanceAngleDegrees = degrees;
applyGlanceAngle();
}
/**
* Tell the keyboard bridge whether the session can host a keyboard: a G2
* on firmware with the bridge, connected and in use. While the glasses
* charge in their case the link stays up but nobody is typing, so the
* firmware is told to let the keyboard go.
*/
private syncKeyboardLink(): void {
const up =
this.phase === "connected" &&
this.customFirmwareConfirmed &&
glassesTypeSetting.get() === "g2";
// Charging, or a disconnect the user asked for: the link is still up, so
// the DISABLE gets out ahead of everything the teardown sends.
if (up) keyboardService.linkReady();
else keyboardService.linkLost(this.phase === "charging" || this.phase === "disconnecting");
}
/**
* The Glanceboard event a sleep-time input means, or null when the shell
* should see the input as usual. A display-wake carries the head-tilt
* gesture (HEAD_UP_EVENT) as a tilt, which lasts until the head-down; the
* double-tap display-wake stays the regular UI's wake.
* gesture (HEAD_UP_EVENT) as a press; the double-tap display-wake stays the
* regular UI's wake.
*/
private glanceEventFor(inputEvent: InputEvent, event: RawInputEvent): GlanceEvent | null {
if (inputEvent.type === "display-wake") {
if (event.eventType !== OsEventTypeList.HEAD_UP_EVENT) return null;
const glanceEvent = this.glance.eventForGesture("head-tilt");
if (glanceEvent?.type !== "tilt") return glanceEvent;
const maxMs = glanceTiltMaxMs({
capMs: glanceEvent.maxMs === undefined ? GLANCE_TILT_MAX_MS : glanceEvent.maxMs,
resumedSession: this.evenHubSessionSuspended || this.evenHubResumePromise !== null,
triggerDegrees: glassesHeadUpTrigger() ?? (glanceAngleTriggerDegrees() || null),
tapTimeoutMs: glanceTapTimeoutMs(),
});
return { type: "tilt", maxMs };
return event.eventType === OsEventTypeList.HEAD_UP_EVENT ? this.glance.eventForGesture("head-tilt") : null;
}
return this.glance.eventForGesture(inputEvent.type);
}
@@ -2811,7 +2749,6 @@ class DashboardController {
}
},
endTextSettingEdit: () => this.endTextSettingEdit(),
unhandledKey: (event) => keyboardService.handleUnhandledKey(event as KeyEvent),
});
this.appHosts.set(appId, host);
return host;
@@ -2845,11 +2782,10 @@ class DashboardController {
if (!this.openAppsRestored || this.suppressOpenAppsPersist) return;
const open: string[] = [];
for (const window of shell.getWindows()) {
if (window.appId === "launcher" || window.restoresApp === false) continue;
if (window.appId === "launcher") continue;
if (!open.includes(window.appId)) open.push(window.appId);
}
const foreground = shell.foregroundWindow();
savePersistedOpenApps({ open, foreground: foreground && foreground.restoresApp !== false ? foreground.appId : null });
savePersistedOpenApps({ open, foreground: shell.foregroundWindow()?.appId ?? null });
}
/**
@@ -3106,12 +3042,13 @@ class DashboardController {
this.schedulePreviewUpdate();
}
private handleAndroidNotificationPosted(notificationKey: string): void {
private handleAndroidNotificationPosted(notificationKey: string, canWakeScreen = true): void {
// The tray shows the notification at once; its popup waits out a short
// delay, so one withdrawn right away (see NotificationPopupDelay) never shows.
const notification = readActiveNotifications(ALL_NOTIFICATIONS).find((item) => item.key === notificationKey);
if (notification && shouldPopUpNewNotification(notification.packageName)) {
this.notificationPopups.post(notificationKey, notificationPopupDelayMs(notification));
this.notificationPopups.post(notificationKey, notificationPopupDelayMs(notification),
canWakeScreen && !isPhoneDoNotDisturbEnabled());
}
this.requestShellRender();
}
@@ -3129,10 +3066,13 @@ class DashboardController {
}
/** A posted notification waited out its popup delay: show it, if it is still there. */
private popUpAndroidNotification(notificationKey: string): void {
private popUpAndroidNotification(notificationKey: string, canWakeScreen = true): void {
try {
const notification = readActiveNotifications(ALL_NOTIFICATIONS).find((item) => item.key === notificationKey);
if (!notification || !shouldPopUpNewNotification(notification.packageName)) return;
// Keep the arrival-time restriction and re-check DND immediately before
// waking: a delayed popup must neither outlive DND nor bypass a new DND.
if (!shell.isScreenOn() && (!canWakeScreen || isPhoneDoNotDisturbEnabled())) return;
// New notifications open a shell modal over the app viewport; if the
// screen was off, wake for it and go back to sleep when it is closed.
// Waking while already on would steal focus, so only wake from sleep.
@@ -3177,7 +3117,6 @@ class DashboardController {
private setPhase(phase: ConnectionPhase): void {
if (this.phase === phase) return;
this.phase = phase;
this.syncKeyboardLink();
if (phase !== "connected" && phase !== "charging") {
shell.setBatteryLevels({ ring: null, ringCharging: null });
}
@@ -3190,6 +3129,10 @@ class DashboardController {
// Kept across "connecting": silent mode blocks app launches, so it can
// itself cause the reconnect churn, and Java re-reports it either way.
this.silentMode = false;
this.silentModeFeedbackGeneration++;
if (this.silentModeFeedbackTimer !== null) clearTimeout(this.silentModeFeedbackTimer);
this.silentModeFeedbackTimer = null;
this.silentModeFeedback = "";
}
this.emit();
}
-17
View File
@@ -68,23 +68,6 @@ export function extractG2Mac(mfg: Uint8Array): string | null {
return humanReadableAddressFromWireBytes(mfg.subarray(16, 22));
}
/**
* G2 manufacturer data (hex) carrying `serial` and `address`, for an arm known
* only from a saved pairing: what `extractG2Serial` and `extractG2Mac` read
* back, with the serial zero-padded and no flag byte. Null when the serial is
* not up to 14 printable ASCII characters or the address is not a MAC.
*/
export function g2ManufacturerData(serial: string, address: string): string | null {
const wire = wireBytesFromHumanReadableAddress(address);
if (!wire || !/^[\x20-\x7e]{1,14}$/.test(serial)) return null;
const bytes = new Uint8Array(22);
bytes[0] = 0x45;
bytes[1] = 0x52;
for (let i = 0; i < serial.length; i++) bytes[2 + i] = serial.charCodeAt(i);
bytes.set(wire, 16);
return bytesToHex(bytes);
}
/** Preserve the final G2 manufacturer-data byte without guessing its meaning. */
export function extractG2AdvertisementFlag(mfg: Uint8Array): number | null {
if (mfg.length < 23 || !hasEvenManufacturerSignature(mfg)) return null;
-7
View File
@@ -79,12 +79,6 @@ export const OsEventTypeList = {
HEAD_UP_EVENT: 12,
/** Faceclaw/18: R1 touch-down, before click/hold/slide recognition. */
RING_PRESS_EVENT: 14,
/**
* Faceclaw/38: the IMU head-down (the head back below the head-up
* threshold minus 20 degrees), forwarded while a page is up. Ends a
* head-tilt Glanceboard; nothing else uses it.
*/
HEAD_DOWN_EVENT: 15,
} as const;
export const OsEventTypeName: Record<number, string> = {
@@ -102,5 +96,4 @@ export const OsEventTypeName: Record<number, string> = {
11: "SHORT_THEN_LONG_PRESS_EVENT",
12: "HEAD_UP_EVENT",
14: "RING_PRESS_EVENT",
15: "HEAD_DOWN_EVENT",
};
+1 -1
View File
@@ -26,7 +26,7 @@ export type FirmwareInfo = {
};
/** The Faceclaw firmware revision this build of the app needs. */
export const REQUIRED_FACECLAW_FIRMWARE_VERSION = 40;
export const REQUIRED_FACECLAW_FIRMWARE_VERSION = 38;
const FACECLAW_PREFIX = "Faceclaw/";
const LEGACY_PREFIX = "EVENCFW";
+35 -101
View File
@@ -19,42 +19,12 @@ export type FirmwarePatchSet = {
export const CFW_PATCH_SET: FirmwarePatchSet = {
"base": "g2_2.3.0.24.bin",
"baseSha256": "187ccf2bcc5c17a212106e8a376745511e8289c4232b634a7ea94b9bf25a0979",
"outputSha256": "2a31417f04cec16ecd4e4c3e575d89ce77fb3c5386b47cdb49b9753aaefb1c32",
"outputSha256": "4aac023f49ec6965186d9106e328a516f20a2915ab400d561f662f984023a432",
"patches": [
{
"offset": 1379947,
"old": "0ef009fa",
"new": "08f3a6fc",
"desc": "DM callback allocation: retain keyboard lifecycle on queue exhaustion"
},
{
"offset": 1380027,
"old": "0ef0e1f9",
"new": "08f304fc",
"desc": "ATT callback allocation: expose lost keyboard events as relay gaps"
},
{
"offset": 2649417,
"old": "0a681279",
"new": "d2f155fc",
"desc": "SMP Pairing I/O capability: KeyboardDisplay only for owned keyboard"
},
{
"offset": 2651811,
"old": "0a681279",
"new": "d1f1a8ff",
"desc": "SMP Pairing I/O capability: KeyboardDisplay only for owned keyboard"
},
{
"offset": 1385975,
"old": "38b584b0",
"new": "06f3c2bf",
"desc": "BLE app dispatcher: isolate explicitly enabled keyboard connection"
},
{
"offset": 1045797,
"old": "38b584b0",
"new": "5bf309ba",
"new": "5cf371bb",
"desc": "ring receiver entry: timestamped unfiltered reports under framebuffer lease"
},
{
@@ -66,49 +36,49 @@ export const CFW_PATCH_SET: FirmwarePatchSet = {
{
"offset": 1437923,
"old": "fef788ff",
"new": "fbf23aff",
"new": "fdf2a2f8",
"desc": "cfw_ancs_event"
},
{
"offset": 1437951,
"old": "fef7e8fb",
"new": "fbf268fc",
"new": "fcf2d0fd",
"desc": "cfw_ancs_open"
},
{
"offset": 1437969,
"old": "fef7e7fb",
"new": "fbf283fc",
"new": "fcf2ebfd",
"desc": "cfw_ancs_close"
},
{
"offset": 1388387,
"old": "39694d00",
"new": "55467d00",
"new": "25597d00",
"desc": "connection-bound ANCS EUS write callback"
},
{
"offset": 971497,
"old": "fff717f8",
"new": "6df339fb",
"new": "6ef3a1fc",
"desc": "bl faceclaw_input_bridge_received (ring relay and ordered private transport before worker pool)"
},
{
"offset": 971845,
"old": "fef769ff",
"new": "6df38bfa",
"new": "6ef3f3fb",
"desc": "bl faceclaw_input_bridge_received (ring relay and ordered private transport before worker pool)"
},
{
"offset": 979579,
"old": "fdf74ef8",
"new": "6bf370fb",
"new": "6cf3d8fc",
"desc": "bl faceclaw_input_bridge_received (ring relay and ordered private transport before worker pool)"
},
{
"offset": 1428895,
"old": "f8f76cfd",
"new": "f9f2d0fb",
"new": "fbf250fe",
"desc": "bl cfw_receive_packet (private SID-f0 probe before TPL reassembly)"
},
{
@@ -132,7 +102,7 @@ export const CFW_PATCH_SET: FirmwarePatchSet = {
{
"offset": 1058287,
"old": "fff754fb",
"new": "58f3daf9",
"new": "59f342fb",
"desc": "bl ble_classify_link (only a <= 11.25 ms / latency-0 link is 'already fast')"
},
{
@@ -141,184 +111,148 @@ export const CFW_PATCH_SET: FirmwarePatchSet = {
"new": "5ff43332",
"desc": "reserve final 1 KiB of primary TLSF arena for CFW context anchor"
},
{
"offset": 1130241,
"old": "5ff44d22",
"new": "5ff40622",
"desc": "LVGL arena size 0xcd000 -> 0x86000 (fixed CFW buffers follow it)"
},
{
"offset": 2697223,
"old": "9c212020",
"new": "9c192320",
"desc": "EvenHub arena base 0x2020219c -> 0x2023199c"
},
{
"offset": 2697133,
"old": "5ff4e121",
"new": "5ff48221",
"desc": "EvenHub arena clear length 0x70800 -> 0x41000"
},
{
"offset": 2697149,
"old": "5ff4e121",
"new": "5ff48221",
"desc": "EvenHub TLSF pool size 0x70800 -> 0x41000"
},
{
"offset": 2245731,
"old": "5ff4e125",
"new": "5ff48225",
"desc": "EvenHub heap total in the stock usage log 0x70800 -> 0x41000"
},
{
"offset": 4394495,
"old": "9b212020",
"new": "9bb11b20",
"desc": "MPU region 6 (non-cacheable) ends with the shrunk LVGL arena"
},
{
"offset": 1248487,
"old": "d4f712fb",
"new": "27f3a0f9",
"new": "29f344fd",
"desc": "bl settings_send_wrapper (append caps field 100)"
},
{
"offset": 1246183,
"old": "f5f7e2f8",
"new": "27f3befb",
"new": "29f3e8fe",
"desc": "bl settings_decode_wrapper (Faceclaw lease field 101)"
},
{
"offset": 975219,
"old": "02f083fc",
"new": "69f320fc",
"new": "6bf34aff",
"desc": "bl faceclaw_display_start @ 0x467d68 (fail-open wake takeover)"
},
{
"offset": 975411,
"old": "02f023fc",
"new": "69f38efc",
"new": "6bf3b8ff",
"desc": "bl faceclaw_display_start_headup @ 0x467e28 (fail-open wake takeover)"
},
{
"offset": 975103,
"old": "07f085fb",
"new": "6cf33efb",
"new": "6df3a6fc",
"desc": "bl headup_gate (idle input gate: tap/long/release -> field-102 events, stock result kept)"
},
{
"offset": 830549,
"old": "19f03efb",
"new": "8ff313fd",
"new": "90f37bfe",
"desc": "bl gesture_ring_press_mode (ring touch-down 14, head up 12 / down 15; stock dispatch preserved)"
},
{
"offset": 830795,
"old": "1ff071ff",
"new": "8ff3ccfc",
"new": "90f334fe",
"desc": "bl gesture_press (owned event 9 before stock menu gate)"
},
{
"offset": 830887,
"old": "1ff043ff",
"new": "8ff3c6fc",
"new": "90f32efe",
"desc": "bl gesture_short_long (Faceclaw lease gates event 11 vs stock Menu path)"
},
{
"offset": 831763,
"old": "1ff08dfd",
"new": "8ff3fcfa",
"new": "90f364fc",
"desc": "bl gesture_release (owned event 10 before stock menu gate)"
},
{
"offset": 1570209,
"old": "7fb50600",
"new": "d8f231bd",
"new": "dbf2d5b8",
"desc": "even_ai_display_ctrl entry -> conditional Faceclaw lease trampoline"
},
{
"offset": 1051545,
"old": "f6f787f9",
"new": "54f359fe",
"new": "55f3c9f8",
"desc": "bl display_copy_hook @ 0x47a78e (640x480 direct framebuffer)"
},
{
"offset": 1051861,
"old": "f6f7e9f8",
"new": "54f3bbfd",
"new": "55f32bf8",
"desc": "bl display_copy_hook @ 0x47a8ca (640x480 direct framebuffer)"
},
{
"offset": 1262237,
"old": "d9f7d0fd",
"new": "23f3edfa",
"new": "25f3bffd",
"desc": "bl faceclaw_send_wear_event @ 0x4ade92 (outside onboarding)"
},
{
"offset": 1262337,
"old": "d9f79efd",
"new": "23f3bbfa",
"new": "25f38dfd",
"desc": "bl faceclaw_send_wear_event @ 0x4adef6 (outside onboarding)"
},
{
"offset": 1309405,
"old": "67f02dff",
"new": "17f397fb",
"new": "19f35bfe",
"desc": "bl compass_decode_capture (sample-matched GAF diagnostics)"
},
{
"offset": 1307865,
"old": "fff7c9fb",
"new": "17f319ff",
"new": "1af3ddf9",
"desc": "bl compass_report_event (stock UI + heading with diagnostics over BLE)"
},
{
"offset": 1300937,
"old": "00210170",
"new": "19f3f7fb",
"new": "1bf3bbfe",
"desc": "bl compass_preserve_accuracy (retain magnetic calibration under Faceclaw framebuffer lease)"
},
{
"offset": 4537963,
"old": "",
"new": "b0b581ea00039b0701d0034637e0c2b110f0030315d000f1010ca2f1010e0023cc5c9e45c45403f1010404d0634413f003032346f4d121440319121b102a14d302e00346102a10d3d1f800c0d1f804e0cc688d68103adc609d60c3f804e043f810cb0f2a01f11001eed8042a07d300bf51f8045b043a032a43f8045bf8d8002a08bfb0bd0139013b11f8015f013a03f8015ff9d1b0bd00bf884218bf002a00d1704710b50b1a38bf431a934207d288420ad281ea00039b0726d0034669e0fff79fffbde81040704781ea000300eb020c9b0701eb020e42d15fea8c732bd0a0f1010c00bf8b1813f801ec8418531e04f801ec04d0624412f003021a46f2d101eb030e00eb030c17e0830736d0a2f1010c00f1010e002300bfcc5c9c45c45403f1010404d0734413f003032346f4d1121b2144031922e01346042b0ed3aef10402acf1040152f80449043b032b41f80449f8d802f1040e01f1040c1a4653b1aef10101acf1010300bf11f80149013a03f80149f9d1bde8104070470346042a06d351f8044b043a032a43f8044bf8d8002af0d00139013b00bf11f8014f013a03f8014ff9d1e6e700bf80b5002a18bf10f0030301d186460ce000f1010c864600bf013a0ef8011b04d01cf003030cf1010cf6d1102a11d35ffa81fc4ff001330cfb03fc00bf103acef80cc0cef808c0cef804c04ef810cb0f2af4d8042a0bd35ffa81fc4ff001330cfb03fc00bf043a032a4ef804cbfad8002a08bf80bdaef10103013a03f8011ffbd180bd00bf2de9f04fc7b04ff29c5bc2f2290b4ff6ff784ff47e740027012647f67f754ff07f0a1191cde90f3007e000bf48f2a351c0f24501bff32f8f8847dbf80090bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0ad101684ef66d62ccf2ff02914203d14068b0f57e7f4ed04ff47e7002f0f9f9002800f027816ff0070142180831b1f57c7f177257729772d772177357739773d773f2d14ef66d61ccf2ff01016044608673a0f8c85180f8c7a1a0f82080a0f82280a0f82480a0f82680a0f82880a0f82a80a0f82c80a0f82e80a0f83080a0f83280a0f83480a0f83680a0f83880a0f83a80a0f83c80a0f83e805be8001f494599d1bff35f8f4be8000121b15be8001f4945f8d08fe74ff47e7a00254ff0010947f67f777f2407e000bf48f2a351c0f24501bff32f8f8847dbf80060bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0ad101684ef66d62ccf2ff02914203d14168b1f57e7f50d04ff47e7002f07ff9002800f0ad806ff0070142180831b1f57c7f157255729572d572157355739573d573f2d14ef66d61ccf2ff010160c0f804a080f80e90a0f8c87180f8c741a0f82080a0f82280a0f82480a0f82680a0f82880a0f82a80a0f82c80a0f82e80a0f83080a0f83280a0f83480a0f83680a0f83880a0f83a80a0f83c80a0f83e805be8001fb14297d1bff35f8f4be8000121b15be8001fb142f8d08de7d0f8984242f6f76ac0f2440abff35f8fdcb30af19a0220464ff0ff314ff0ff379047002840f02e81119800284ed010980778a7f11700022848d811980c2821d1109d6878a97808431cd0e8780138022818d8aa79eb796e7a42ea03222b7a287942ea034242ea0663aa7aee7a6979192f42ea062200f07f81172f40f09581002a00f04e834ff0ff37f8e004460020d047c8b104f5267151e8004f54b9bff35f8f41e80002002a00f0998051e8004f002cf6d00021bff32f8f0029bff35f8f00f089800446a5e74ff0ff37dbe047f6df40c0f24f000e94804700274ff47e79012647f67f744ff07f0a13978df8507006e048f2a351c0f24501bff32f8f8847dbf80050bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0ad101684ef66d62ccf2ff02914203d14168b1f57e7f57d04ff47e7002f095f8002850d06ff0070100bf42180831b1f57c7f177257729772d772177357739773d773f2d14ef66d61ccf2ff010160c0f804908673a0f8c84180f8c7a1a0f82080a0f82280a0f82480a0f82680a0f82880a0f82a80a0f82c80a0f82e80a0f83080a0f83280a0f83480a0f83680a0f83880a0f83a80a0f83c80a0f83e805be8001fa94298d1bff35f8f00bf4be8000159b15be8001fa942f8d08de70af5a97188471ae701216be70020109f42f6f76ac0f2440a9fb1119989b1397801f07f02b2f10b0418bf0124b2f11c0518bf01258a291edc0b2920d01c291ed03ae00f9012a801f06dfe4ff0ff37012612a801f037ff054646b99df8500028b94af21f30c0f247004c3080470f98002818bf056117e09c2901d08b291cd180b14af21f31c0f247010646884796f86811bff35f8f29b996f824133046bff35f8f51b14ff0ff370e9c0af5807120468847384647b0bde8f08f0f9012a804ea050601f030fe387800f07f041e2c05d11199384606f01ffe0746bae71d2c00f07e0018bf1a2819d1dbf800100322cff680720a40b2f1005fbff35f8f12d10a684ef66d63ccf2ff039a420bd14a68b2f57e7f04bf002281f8a12303e0a4f115010129e1d90f2c2cdd142c40f3ad81152c00f0c882162c00f0d2821f2c40f095831198012840f0648438781f2840f0608445f64d54c0f24604a047012841f05e8041f2de01c0f2010104f5e2777944e02029220023b8470746002818bf4ff0ff3767e7092c00f3b880052c00f0ee81072c40f06b830d9600264ff0010947f67f777f243ae000294ff0ff3708bf002b7ff47caf1328bff479af102a4ff0000138bf01211228b2fa82f218bf510901297ff46caf00f087bc182f4ff0ff3708bf00297ff463af002b7ff460af352800f37083002800f05f84012800f06b84312840f05d84082abff451af00f06cbc48f2a351c0f24501bff32f8f8847dbf80050bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0bd101684ef66d62ccf2ff02914204d14168b1f57e7f00f077854ff47e7001f01dff002800f06f856ff0070142180831b1f57c7f167256729672d672167356739673d673f2d14ef66d61ccf2ff0101604ff47e71416080f80e90a0f8c87180f8c741a0f82080a0f82280a0f82480a0f82680a0f82880a0f82a80a0f82c80a0f82e80a0f83080a0f83280a0f83480a0f83680a0f83880a0f83a80a0f83c80a0f83e805be8001fa94295d1bff35f8f4be80001002900f028855be8001fa942f6d089e70a2c00f0bb810b2c40f0b382dbf800500320cff680702840b0f1005fbff35f8f40f0808728684ef66d61ccf2ff01884240f0788768680027b0f57e7f7ff48daed5f8900342f68d49c0f244090028a5f8a07318bfc8470021a5f8d013c5f8cc13c5f8701185f86c11bff35f8f85f86811686c85f86b11c5f87411c5f82013c5f88813a5f88c13a5f8481040b1c847686c09f166018847002804bf002068644bf63940c0f251008047dbf800400320cff680702040b0f1005fbff35f8f46d120684ef66Line truncated
"new": "b0b581ea00039b0701d0034637e0c2b110f0030315d000f1010ca2f1010e0023cc5c9e45c45403f1010404d0634413f003032346f4d121440319121b102a14d302e00346102a10d3d1f800c0d1f804e0cc688d68103adc609d60c3f804e043f810cb0f2a01f11001eed8042a07d300bf51f8045b043a032a43f8045bf8d8002a08bfb0bd0139013b11f8015f013a03f8015ff9d1b0bd00bf884218bf002a00d1704710b50b1a38bf431a934207d288420ad281ea00039b0726d0034669e0fff79fffbde81040704781ea000300eb020c9b0701eb020e42d15fea8c732bd0a0f1010c00bf8b1813f801ec8418531e04f801ec04d0624412f003021a46f2d101eb030e00eb030c17e0830736d0a2f1010c00f1010e002300bfcc5c9c45c45403f1010404d0734413f003032346f4d1121b2144031922e01346042b0ed3aef10402acf1040152f80449043b032b41f80449f8d802f1040e01f1040c1a4653b1aef10101acf1010300bf11f80149013a03f80149f9d1bde8104070470346042a06d351f8044b043a032a43f8044bf8d8002af0d00139013b00bf11f8014f013a03f8014ff9d1e6e700bf80b5002a18bf10f0030301d186460ce000f1010c864600bf013a0ef8011b04d01cf003030cf1010cf6d1102a11d35ffa81fc4ff001330cfb03fc00bf103acef80cc0cef808c0cef804c04ef810cb0f2af4d8042a0bd35ffa81fc4ff001330cfb03fc00bf043a032a4ef804cbfad8002a08bf80bdaef10103013a03f8011ffbd180bd00bf2de9f04fc7b04ff29c5bc2f2290b4ff6ff754ff47d7400274ff0010847f67f7a7f261191cde90f3007e000bf48f2a351c0f24501bff32f8f8847dbf80090bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0bd101684ef66c62ccf2ff02914204d14068b0f57d7f00f096804ff47d7006f004fd002800f0cb814ff60c41cff6ff7100bf42182e3182f8f47382f8f57382f8f67382f8f77382f8f87382f8f97382f8fa7382f8fb7382f8fc7382f8fd7382f8fe7382f8ff7382f8007482f8017482f8027482f8037482f8047482f8057482f8067482f8077482f8087482f8097482f80a7482f80b7482f80c7482f80d7482f80e7482f80f7482f8107482f8117482f8127482f8137482f8147482f8157482f8167482f8177482f8187482f8197482f81a7482f81b7482f81c7482f81d7482f81e7482f81f7482f8207482f82174a0d14ef66c61ccf2ff010160446080f80e80a0f8c4a180f8c361058445848584c584058545858585c585058645868586c586058745878587c5875be8001f49457ff452afbff35f8f4be8000121b15be8001f4945f8d047e74ff47d7800274ff0010a47f67f797f2607e000bf48f2a351c0f24501bff32f8f8847dbf80040bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0bd101684ef66c62ccf2ff02914204d14168b1f57d7f00f098804ff47d7006f042fc002800f009814ff60c41cff6ff7100bf42182e3182f8f47382f8f57382f8f67382f8f77382f8f87382f8f97382f8fa7382f8fb7382f8fc7382f8fd7382f8fe7382f8ff7382f8007482f8017482f8027482f8037482f8047482f8057482f8067482f8077482f8087482f8097482f80a7482f80b7482f80c7482f80d7482f80e7482f80f7482f8107482f8117482f8127482f8137482f8147482f8157482f8167482f8177482f8187482f8197482f81a7482f81b7482f81c7482f81d7482f81e7482f81f7482f8207482f82174a0d14ef66c61ccf2ff010160c0f8048080f80ea0a0f8c49180f8c361058445848584c584058545858585c585058645868586c586058745878587c5875be8001fa1427ff451afbff35f8f00bf4be8000121b15be8001fa142f8d045e7d0f8948242f6f76ac0f2440ab8f1000fbff35f8f4ad00af19a0240464ff0ff314ff0ff379047002840f090811198002861d010980778a7f1170002285bd811980c2840f07d81109c6078a178084300f07781e0780138022800f27281a279e379667a42ea0322237a207942ea034242ea0663a27ae67a6179192f42ea062200f0de81172f40f0f481002a40f05981fe2800f0b086812800f0a386002840f0b68651ea030040f04c8100f000bf04460020d047e8b104f5257151e8008fb8f1000f0cd1bff35f8f00bf41e80002002a00f0e58051e8008fb8f1000ff5d00021bff32f8f0029bff35f8f00f0d480804692e74ff0ff372ae147f6df40c0f24f00cdf83880804700274ff47d784ff0010947f67f744ff07f0a1397adf8507007e000bf48f2a351c0f24501bff32f8f8847dbf80060bff35f8fdbf800000321cff680710140b1f1005fbff35f8f0bd101684ef66c62ccf2ff02914204d14168b1f57d7f00f09f804ff47d7006f0f8fa002800f097804ff60c41cff6ff7100bf42182e3182f8f47382f8f57382f8f67382f8f77382f8f87382f8f97382f8fa7382f8fb7382f8fc7382f8fd7382f8fe7382f8ff7382f8007482f8017482f8027482f8037482f8047482f8057482f8067482f8077482f8087482f8097482f80a7482f80b7482f80c7482f80d7482f80e7482f80f7482f8107482f8117482f8127482f8137482f8147482f8157482f8167482f8177482f8187482f8197482f81a7482f81b7482f81c7482f81d7482f81e7482f81f7482f8207482f82174a0d14ef66c61ccf2ff010160c0f8048080f80e90a0f8c44180f8c3a1058445848584c584058545858585c585058645868586c586058745878587c5875be8001fb1427ff451afbff35f8f00bf4be8000159b15be8001fb142f8d045e70af5a9718847bce6012120e70020109f42f6f76addf83880c0f2440a9fb1119989b1397801f07f02b2f10b0418bf0124b2f11c0618bf01268a2924dc0b2926d01c2924d03ce00f9012a803f045ff4ff0ff3701249df85100002812a818bf4ff0ff3704f05ff8054644b99df8500028b94af21f30c0f247004c3080470f98002818bf056114e09c2901d08b2918d168b14af21f31c0f247010746884797f8681121b997f824133846109f49b14ff0ff370af5807140468847384647b0bde8f08f0f9012a8344003f007ff387800f07f061e2e05d11199384607f094fe0746b9e71d2e00f07e0018bf1a2819d1dbf800100322cff680720a40b2f1005fbff35f8f12d10a684ef66c63ccf2ff039a420bd14a68b2f57d7f04bf002281f8a12303e0a6f115010129e1d90f2e2cdd142e40f31082152e00f05d83162e00f069831f2e40f02c841198012840f0fc8438781f2840f0f88445f64d55c0f24605a847012841f0828140f24c71c0f2010105f5e2777944e02029220023b8470746002818bf4ff0ff3766e7092e00f3fe80052e00f03e82072e40f0028400274ff0010947f67f764ff07f080d943ae000294ff0ff3708bf002b7ff47caf1328bff479af102a4ff0000138bf01211228b2fa82f218bf510901297ff46caf0Line truncated
"desc": "append injected blobs to main-app payload"
},
{
"offset": 779123,
"old": "805a3900",
"new": "10783a00",
"new": "68703a00",
"desc": "main-app subheader payload size (ps)"
},
{
"offset": 152,
"old": "005b3900",
"new": "90783a00",
"new": "e8703a00",
"desc": "main-app TOC entry size (ps + 128)"
},
{
"offset": 779243,
"old": "805a3904",
"new": "10783a04",
"new": "68703a04",
"desc": "main-app preamble length (low 24 bits)"
},
{
"offset": 779247,
"old": "effdbcea",
"new": "adc5378e",
"new": "0b0c48b0",
"desc": "[5] ota/s200_firmware_ota.bin preamble crc32"
},
{
"offset": 156,
"old": "c0c6322a",
"new": "0c2e1c3e",
"new": "52937d21",
"desc": "[5] ota/s200_firmware_ota.bin component crc32c (TOC)"
},
{
"offset": 779127,
"old": "c0c6322a",
"new": "0c2e1c3e",
"new": "52937d21",
"desc": "[5] ota/s200_firmware_ota.bin component crc32c (subheader)"
}
]
+2 -3
View File
@@ -82,7 +82,7 @@ export class GlanceHost {
* The glance event a sleep-time gesture means under the board's settings,
* or null when the shell should see the input as usual: nothing while the
* board is disabled, no press from a tap when "Show on tap" is disabled,
* no hold when "Show on long press" is off, no tilt from a head-tilt when
* no hold when "Show on long press" is off, no press from a head-tilt when
* "Show on head tilt" is off (it then wakes the regular UI). A release
* always passes so a hold in progress can end.
*/
@@ -92,8 +92,7 @@ export class GlanceHost {
if (gesture === "click" && !provider.showOnTap()) return null;
if ((gesture === "long-press" || gesture === "short-then-long-press") && !provider.showOnLongPress()) return null;
if (gesture === "head-tilt" && !provider.showOnHeadTilt()) return null;
const event = glanceEventForGesture(gesture, this.isVisible());
return event?.type === "tilt" ? { type: "tilt", maxMs: provider.tiltTimeoutMs() } : event;
return glanceEventForGesture(gesture, this.isVisible());
}
/**
+12 -82
View File
@@ -1,10 +1,9 @@
/**
* The Glanceboard's visibility rules, as a pure reducer so they can be tested
* without the display plumbing. The board is an alternate sleep-time display:
* while the shell is asleep a press (single tap) shows it for a short while, a
* head-tilt wake shows it until the head comes back down, a hold (long-press or
* short-then-long-press) shows it until release, and a double-tap dismisses it
* in favour of the regular UI. GlanceHost feeds
* while the shell is asleep a press (single tap, or a head-tilt wake) shows it
* for a short while, a hold (long-press or short-then-long-press) shows it until release,
* and a double-tap dismisses it in favour of the regular UI. GlanceHost feeds
* these events in and applies the resulting visibility to the compositor.
*
* No NativeScript imports: this module is compiled and run under plain node
@@ -14,67 +13,17 @@
/** Fallback press timeout when the host supplies none; each further press restarts it. */
export const GLANCE_TIMEOUT_MS = 3000;
/**
* Default cap for a head-tilt showing: the board normally stays up until the
* head comes back down (as the stock dashboard does), but a lost head-down
* event should not leave it on indefinitely. The user's Head tilt timeout
* setting replaces it (and may remove it).
*/
export const GLANCE_TILT_MAX_MS = 60_000;
/**
* The firmware reports head-down once the pitch falls 20° below the head-up
* trigger (a fixed hysteresis in DRV_IMUCheckHeadUpEvent). Under this trigger,
* looking level again does not get below that, so the head-down may never come.
*/
export const GLANCE_LOW_TILT_TRIGGER_DEGREES = 25;
/**
* How long a head-tilt showing may last without a head-down: capMs (the Head
* tilt timeout; null = no limit), except when the head-down is unlikely to
* arrive, where the tap duration is used if shorter: a tilt that woke a
* suspended EvenHub session (the head-down is only forwarded once the page is
* back, so a quick glance can end before that), or a trigger too low for a
* level gaze to count as looking down. triggerDegrees is the glasses'
* trigger, or null when not known.
*/
export function glanceTiltMaxMs(options: {
capMs: number | null;
resumedSession: boolean;
triggerDegrees: number | null;
tapTimeoutMs: number;
}): number | null {
const lowTrigger =
options.triggerDegrees !== null && options.triggerDegrees > 0 && options.triggerDegrees < GLANCE_LOW_TILT_TRIGGER_DEGREES;
if (!options.resumedSession && !lowTrigger) return options.capMs;
return options.capMs === null ? options.tapTimeoutMs : Math.min(options.capMs, options.tapTimeoutMs);
}
export type GlanceState = {
visible: boolean;
/** A long-press is holding the board up; the timer does not run meanwhile. */
holding: boolean;
/**
* A head-tilt is holding the board up: it stays until the firmware reports
* the head back down (or tiltMaxMs passes).
*/
tilted: boolean;
/** The current head-tilt's cap (0 when not tilted, null for none); a hold ending mid-tilt re-arms it. */
tiltMaxMs: number | null;
/** When the board hides on its own (epoch ms), or null while held / hidden. */
hideAtMs: number | null;
};
export type GlanceEvent =
/** Single tap: show, and (re)start the auto-hide timer. */
/** Single tap or head-tilt wake: show, and (re)start the auto-hide timer. */
| { type: "press" }
/**
* Head-tilt (IMU head-up): show until the head comes back down, or until
* maxMs passes (default GLANCE_TILT_MAX_MS; null = no limit).
*/
| { type: "tilt"; maxMs?: number | null }
/** The IMU reported the head back down: ends a head-tilt showing. */
| { type: "head-down" }
/** Long-press: show and keep showing until the release. */
| { type: "hold" }
/** The long-press ended. */
@@ -84,7 +33,7 @@ export type GlanceEvent =
/** Double-tap (or any other wake of the regular UI): hide now. */
| { type: "dismiss" };
export const GLANCE_HIDDEN: GlanceState = Object.freeze({ visible: false, holding: false, tilted: false, tiltMaxMs: 0, hideAtMs: null });
export const GLANCE_HIDDEN: GlanceState = Object.freeze({ visible: false, holding: false, hideAtMs: null });
export function reduceGlance(
state: GlanceState,
@@ -94,26 +43,14 @@ export function reduceGlance(
): GlanceState {
switch (event.type) {
case "press":
// A tap during a hold or a head-tilt changes nothing: those decide when it ends.
if (state.holding || state.tilted) return state;
return { visible: true, holding: false, tilted: false, tiltMaxMs: 0, hideAtMs: nowMs + timeoutMs };
case "tilt": {
const maxMs = event.maxMs === undefined ? GLANCE_TILT_MAX_MS : event.maxMs;
return { visible: true, holding: state.holding, tilted: true, tiltMaxMs: maxMs, hideAtMs: state.holding ? null : tiltHideAt(nowMs, maxMs) };
}
case "head-down":
// Only a head-tilt showing ends with the head; a tap's timer or a hold keeps running.
if (!state.tilted) return state;
return state.holding ? { ...state, tilted: false, tiltMaxMs: 0 } : GLANCE_HIDDEN;
// A tap during a hold changes nothing: the hold decides when it ends.
if (state.holding) return state;
return { visible: true, holding: false, hideAtMs: nowMs + timeoutMs };
case "hold":
return { visible: true, holding: true, tilted: state.tilted, tiltMaxMs: state.tiltMaxMs, hideAtMs: null };
return { visible: true, holding: true, hideAtMs: null };
case "release":
// Releases delivered late or after a disabled hold gesture mean nothing here.
if (!state.holding) return state;
// Head still up: the board stays, as if the tilt had just shown it.
return state.tilted
? { visible: true, holding: false, tilted: true, tiltMaxMs: state.tiltMaxMs, hideAtMs: tiltHideAt(nowMs, state.tiltMaxMs) }
: GLANCE_HIDDEN;
return state.holding ? GLANCE_HIDDEN : state;
case "timeout":
if (!state.visible || state.holding || state.hideAtMs === null || nowMs < state.hideAtMs) return state;
return GLANCE_HIDDEN;
@@ -122,10 +59,6 @@ export function reduceGlance(
}
}
function tiltHideAt(nowMs: number, maxMs: number | null): number | null {
return maxMs === null ? null : nowMs + maxMs;
}
/**
* Which glance event, if any, a shell input gesture means while the screen is
* off. A double-tap only concerns the board while it is showing (it then
@@ -133,16 +66,13 @@ function tiltHideAt(nowMs: number, maxMs: number | null): number | null {
* already handles or ignores while asleep.
*/
export function glanceEventForGesture(
gesture: "click" | "double-click" | "long-press" | "short-then-long-press" | "long-press-release" | "head-tilt" | "head-down" | (string & {}),
gesture: "click" | "double-click" | "long-press" | "short-then-long-press" | "long-press-release" | "head-tilt" | (string & {}),
boardVisible: boolean,
): GlanceEvent | null {
switch (gesture) {
case "click":
return { type: "press" };
case "head-tilt":
return { type: "tilt" };
case "head-down":
return boardVisible ? { type: "head-down" } : null;
return { type: "press" };
case "long-press":
case "short-then-long-press":
return { type: "hold" };
-151
View File
@@ -1,151 +0,0 @@
import { addImuListener, imuPitchDegrees, setImuReportEnabled, type ImuReading } from "../native/imu";
import { getHeadUpStatus, setHeadUpTrigger, type HeadUpStatus } from "../native/glasses-controls";
import { getStringSetting, setStringSetting } from "../native/settings-store";
import { glanceAngleTriggerDegrees } from "../apps/glanceboard/glanceboard-settings";
// Same ImuReportPace code the accelerometer demo uses (not literal Hz).
const IMU_REPORT_PACE = 200;
/** How often the live reading checks visibility and repaints a changed value. */
const LIVE_TICK_MS = 250;
/** Weight of each new sample in the smoothed pitch. */
const LIVE_SMOOTHING = 0.3;
/** A reading older than this (or none this long after the stream started) is shown as missing. */
const LIVE_STALE_MS = 3_000;
/**
* The head's pitch, live, for the Head tilt angle picker. The IMU report
* stream runs while `visible()` holds (checked every tick, so it stops while
* the screen is off or the window is in the background) and until stop().
* text() is the footer line; a changed value repaints at most once a tick.
*/
class LiveHeadPitch {
private pitch: number | null = null;
private lastSampleAt = 0;
private streaming = false;
private streamingSince = 0;
private shown = "";
private readonly unsubscribe: () => void;
private readonly timer: ReturnType<typeof setInterval>;
constructor(
private readonly visible: () => boolean,
private readonly repaint: () => void,
) {
this.unsubscribe = addImuListener((reading) => this.onReading(reading));
this.timer = setInterval(() => this.tick(), LIVE_TICK_MS);
this.tick();
}
text(): string {
return this.shown;
}
stop(): void {
clearInterval(this.timer);
this.unsubscribe();
if (this.streaming) setImuReportEnabled(false, IMU_REPORT_PACE);
this.streaming = false;
}
private onReading(reading: ImuReading): void {
// Glasses temples only (1 right, 3 left, 0 unspecified); the ring is 2.
if (!this.streaming || reading.source === 2) return;
if (!(Math.hypot(reading.x, reading.y, reading.z) > 1e-6)) return;
const pitch = imuPitchDegrees(reading);
this.pitch = this.pitch === null ? pitch : this.pitch + LIVE_SMOOTHING * (pitch - this.pitch);
this.lastSampleAt = Date.now();
}
private tick(): void {
const now = Date.now();
const visible = this.visible();
if (visible !== this.streaming) {
setImuReportEnabled(visible, IMU_REPORT_PACE);
this.streaming = visible;
this.streamingSince = now;
this.pitch = null;
}
const text = this.format(now);
if (text !== this.shown) {
this.shown = text;
this.repaint();
}
}
private format(now: number): string {
if (this.pitch !== null && now - this.lastSampleAt < LIVE_STALE_MS) return `Head now: ${Math.round(this.pitch)}°`;
return now - Math.max(this.streamingSince, this.lastSampleAt) < LIVE_STALE_MS ? "Head now: …" : "Head now: no data";
}
}
/**
* The Head tilt angle picker's footer: the head's current pitch, read while
* the picker is open and `visible()` holds (enumSettingMenuItem pickerFooter).
*/
export function liveHeadPitchFooter(visible: () => boolean): { text: () => string; open: (repaint: () => void) => () => void } {
let reader: LiveHeadPitch | null = null;
return {
text: () => reader?.text() ?? "",
open: (repaint) => {
const opened = new LiveHeadPitch(visible, repaint);
reader = opened;
return () => {
opened.stop();
if (reader === opened) reader = null;
};
},
};
}
// The glasses' own head-up switch and angle from before Faceclaw first changed
// them, as "switch,angle", or "" when none is held. Kept here (not only in the
// session) so "Glasses default" can still restore them after an app restart.
const ORIGINAL_KEY = "glanceboard.headTiltOriginal";
function savedOriginal(): { switch: number; angle: number } | null {
const [sw, angle] = getStringSetting(ORIGINAL_KEY, "").split(",").map(Number);
return (sw === 0 || sw === 1) && Number.isInteger(angle) ? { switch: sw, angle } : null;
}
/**
* Push the wanted trigger to the session (on change and on each connect):
* the Head tilt angle while the Glanceboard is enabled, else none, which
* writes back the glasses' own values if Faceclaw changed them.
*/
export function applyGlanceAngle(): boolean {
const sent = setHeadUpTrigger(glanceAngleTriggerDegrees(), savedOriginal());
syncGlanceAngleOriginal();
return sent;
}
/**
* Keep the persisted original in step with the session: save the values the
* session captured before its first write, and drop them once a read shows
* them restored. Called on apply, from the controller's periodic refresh, and
* when the settings row draws.
*/
export function syncGlanceAngleOriginal(status: HeadUpStatus | null = getHeadUpStatus()): void {
if (!status) return;
const saved = savedOriginal();
if (!saved) {
if (status.originalSwitch === 0 || status.originalSwitch === 1) {
setStringSetting(ORIGINAL_KEY, `${status.originalSwitch},${status.originalAngle}`);
}
return;
}
if (
glanceAngleTriggerDegrees() === 0 &&
status.desired < 0 &&
!status.restorePending &&
status.switch === saved.switch &&
status.angle === saved.angle
) {
setStringSetting(ORIGINAL_KEY, "");
}
}
/** The glasses' current head-up trigger (angle + calibration), 0 when off, or null when not read yet. */
export function glassesHeadUpTrigger(status: HeadUpStatus | null = getHeadUpStatus()): number | null {
if (!status || status.switch < 0) return null;
return status.switch === 1 ? status.angle + status.calibration : 0;
}
+13 -53
View File
@@ -29,8 +29,6 @@ import { OsEventTypeList } from './events'
import { loadDeviceAddresses } from './device-addresses'
import { updateGlassesPresence } from './glasses-presence'
import { firmwareIncompatibilityMessage, hasCompatibleFirmware } from './firmware-compat'
import { keyboardService } from './keyboard/service'
import type { KeyEvent } from './keyboard/keys'
import { isHaltedSessionPhase, resumeAutoReconnect, suppressAutoReconnect } from './reconnect-policy'
import { WearRemote, type WearRemoteInputKind } from './wear-remote'
import { deviceAddressError } from './ios-peripheral-identity'
@@ -54,7 +52,7 @@ import { mirrorToPanelFraction, phonePreviewCrop } from './phone-preview-crop'
import { onAnySettingChanged,
previewColorSetting, lockScreenEnabledSetting, getBrightnessPreferences, getStringSettingById,
nightscoutSiteUrlSetting, nightscoutApiTokenSetting, voiceProviderSetting, elevenLabsApiKeySetting,
openAiApiKeySetting, sonioxApiKeySetting, whisperServerApiKeySetting, whisperServerLanguageSetting,
openAiApiKeySetting, openAiTranscriptionModelSetting, sonioxApiKeySetting, whisperServerApiKeySetting, getVoiceTranscriptionLanguage,
whisperServerModelSetting, whisperServerUrlSetting, ringConnectionModeSetting, saveRecordingSetting, glassesTypeSetting, shouldPopUpNewNotification } from '../ui/dashboard-settings'
import { transcriptionKey, type VoiceProviderSelection } from '../native/cloud-stt-provider'
import type { PhoneGesture } from '../phone-ui/phone-gestures'
@@ -224,6 +222,12 @@ export class IosPreviewController {
shell.foregroundWindow()?.requestRender()
this.requestShellRender()
},
onWindowLayoutChanged: () => {
this.relayout()
shell.foregroundWindow()?.requestRender()
this.requestShellRender()
},
isLocked: () => this.glassesLocked,
onScreenStateChanged: on => {
if (on) this.glance.dismiss()
// A wake unblanks only once the next shell render has staged an awake
@@ -234,7 +238,6 @@ export class IosPreviewController {
if (!on) void this.display?.setScreenBlanked(true).catch(error => this.fail(error))
this.requestShellRender()
this.wearRemote?.schedulePublish()
if (on) keyboardService.retryNow()
},
})
const launcher = createLauncherWindow({
@@ -273,28 +276,6 @@ export class IosPreviewController {
assistantAvailable: () => shell.isAssistantAvailable(),
assistant: text => shell.sendToAssistant(text),
})
// A keyboard paired through the glasses (the CFW keyboard bridge). The service decides what
// each key means (app/g2/keyboard/key-router.ts); its gestures share the controller queue
// with physical input, like the watch's.
const queueKeyboardInput = (run: () => Promise<void>): Promise<void> => {
const task = this.inputQueue.then(() => { this.onActivity(); return run() })
this.inputQueue = task.catch(error => this.fail(error))
return task
}
keyboardService.setHost({
ready: () => this.state.phase === 'connected',
locked: () => this.glassesLocked,
wake: () => { void queueKeyboardInput(() => this.receiveInput(makeInputEvent({ type: 'display-wake' }))).catch(() => {}) },
gesture: (gesture, origin) => queueKeyboardInput(async () => {
await this.receiveInput(makeInputEvent({ type: gesture, source: origin } as InputEventPayload))
// A discrete hold includes its release.
if (gesture === 'long-press') await this.receiveInput(makeInputEvent({ type: 'long-press-release', source: origin }))
}),
log: line => this.logBluetooth(line),
// Only while the display is on: a keyboard that is switched off would otherwise keep the
// glasses' radio looking for it around the clock.
shouldReconnect: () => this.state.phase === 'connected' && shell.isScreenOn(),
})
// The header's EvenHub phone-UI button hides while that UI is showing.
onPhoneShownChanged(() => this.syncEvenHubPhoneUi())
registerSystemTools()
@@ -442,6 +423,7 @@ export class IosPreviewController {
const phone = readPhoneBatteryState()
this.logBluetooth(`Phone battery ${phone.battery ?? "unknown"}% charging=${phone.charging}`)
this.offSettings = onAnySettingChanged(() => {
this.syncContinuousCapture()
this.syncLockSetting()
this.syncContinuousRecording()
this.pushBrightness()
@@ -834,7 +816,6 @@ export class IosPreviewController {
},
setSurfaceVisible: (id, visible) => this.setWindowSurfaceVisible(id, visible),
removeSurface: id => this.removeWindowSurface(id),
unhandledKey: event => keyboardService.handleUnhandledKey(event as KeyEvent),
submitPixels: (id, pixels, width, height, draws) => {
void this.submitWorkerPixels(id, pixels, width, height, draws).catch(error => this.fail(error))
},
@@ -856,18 +837,6 @@ export class IosPreviewController {
this.state = { ...this.state, phase, status }
this.logBluetooth(status)
this.emitState()
this.syncKeyboardLink()
}
/**
* Tell the keyboard bridge whether the session can host a keyboard: a G2 on firmware with
* the bridge, connected. On a disconnect the user asked for, the link is still up when the
* phase turns, so the firmware is told to let the keyboard go first.
*/
private syncKeyboardLink(): void {
const up = !!this.communicator && this.state.phase === 'connected' && this.firmwareCompatible
&& glassesTypeSetting.get() === 'g2'
if (up) keyboardService.linkReady()
else keyboardService.linkLost(this.state.phase === 'disconnecting')
}
/** Mirror the session into glasses-presence (read by EvenHub's device status and Timers). */
private syncPresence(): void {
@@ -987,7 +956,6 @@ export class IosPreviewController {
const firmwareWarning = firmwareIncompatibilityMessage(info) ?? ''
if (firmwareWarning && firmwareWarning !== this.state.firmwareWarning) this.logBluetooth(`Firmware compatibility warning: ${firmwareWarning}`)
this.firmwareCompatible = !firmwareWarning && hasCompatibleFirmware(info)
this.syncKeyboardLink()
this.state = { ...this.state, leftVersion: info.leftVersion || this.state.leftVersion,
rightVersion: info.rightVersion || this.state.rightVersion, capabilities: info.extension || this.state.capabilities,
firmwareWarning }
@@ -1052,13 +1020,6 @@ export class IosPreviewController {
private handleRingEvent(input: RawInputEvent): void {
this.inputQueue = this.inputQueue.then(async () => {
if (this.state.phase !== 'connected') return
if (input.kind === 'sys-event' && input.eventType === OsEventTypeList.HEAD_DOWN_EVENT) {
// As on Android: only the Glanceboard uses it (it ends a head-tilt
// showing); it never reaches the shell or counts as activity.
const glanceEvent = this.glance.eventForGesture('head-down')
if (glanceEvent) await this.glance.handleEvent(glanceEvent, 0)
return
}
this.onActivity()
if (input.ringInput) this.state = { ...this.state, ring: true }
await this.receiveInput(rawInputEventToInputEvent(input),
@@ -1157,8 +1118,8 @@ export class IosPreviewController {
/**
* Continuous capture on behalf of `client` (one id per window: Transcribe,
* Translate, Teleprompter, Calculator); it runs while any client wants it, as
* on Android. `recordAs` names a recording (.mka) of the audio in Downloads
* when Save Recording is on.
* on Android. `recordAs` names a WAV of the audio in Downloads when Save
* Recording is on.
*/
private startContinuousVoiceCapture(client: string, options: ContinuousCaptureOptions = {}): void {
this.continuousClients.set(client, options)
@@ -1258,14 +1219,13 @@ export class IosPreviewController {
/**
* The transcription provider setting and the keys the cloud ones need. A
* window that needs its own provider and language (Translate) sets them for
* every capture while it is open; otherwise only Whisper server is told a
* language (its own setting).
* every capture while it is open; otherwise use the shared language preference.
*/
private voiceProvider(): VoiceProviderSelection {
const choice = [...this.continuousClients.values()].find(options => options.transcription)?.transcription
const provider = choice?.provider ?? voiceProviderSetting.get()
return { provider, language: choice ? choice.language : provider === 'whisper-server' ? whisperServerLanguageSetting.get() : '',
elevenLabsApiKey: elevenLabsApiKeySetting.get(), openAiApiKey: openAiApiKeySetting.get(), sonioxApiKey: sonioxApiKeySetting.get(),
return { provider, language: choice ? choice.language : getVoiceTranscriptionLanguage(provider),
elevenLabsApiKey: elevenLabsApiKeySetting.get(), openAiApiKey: openAiApiKeySetting.get(), openAiTranscriptionModel: openAiTranscriptionModelSetting.get(), sonioxApiKey: sonioxApiKeySetting.get(),
whisperServerUrl: whisperServerUrlSetting.get(), whisperServerModel: whisperServerModelSetting.get(),
whisperServerApiKey: whisperServerApiKeySetting.get() }
}
-291
View File
@@ -1,291 +0,0 @@
/**
* What keys from a hardware keyboard do, apart from typing: the built-in
* defaults, the user's custom keybindings, and how the two combine.
*
* A key becomes one of the gestures the ring and the watch already have, so
* nothing downstream knows about keyboards. Three layers, first match wins:
*
* 1. Custom keybindings (Settings > Keyboard > Keybindings). These apply
* everywhere, including where the focused window takes keys for itself
* (a terminal session), since that is the point of binding a remote's
* few buttons.
* 2. Built-in bindings that also apply there: Escape with Shift, Ctrl or
* Alt goes back, which is the way out of a terminal session; and a
* mouse's buttons and wheel, which are never typed.
* 3. Built-in navigation, only for keys no window took: the arrows are the
* watch's four swipes, Enter selects, Escape goes back.
*
* A keyboard's touchpad or mouse arrives here as keys too: "MouseButton1"
* (left), "MouseButton2" (right), "MouseButton3" (middle) and so on, and
* "WheelUp" / "WheelDown" / "WheelLeft" / "WheelRight" (see pointer.ts).
*
* No platform imports: this runs under node in the tests.
*/
import type { KeyEvent } from "./keys";
/** A gesture a key can stand for, named as the input pipeline names it. */
export type KeyGesture =
| "swipe-up"
| "swipe-down"
| "swipe-left"
| "swipe-right"
| "scroll-up"
| "scroll-down"
| "click"
| "double-click"
| "long-press"
| "short-then-long-press";
/**
* What a custom keybinding does. "none" turns off the key's built-in
* navigation meaning; it does not stop the key being typed into a window
* that takes keys.
*/
export type KeyAction = KeyGesture | "none";
/** The actions, in the order the picker lists them. */
export const KEY_ACTIONS: readonly { action: KeyAction; label: string }[] = [
{ action: "click", label: "Select" },
{ action: "double-click", label: "Back" },
{ action: "swipe-up", label: "Swipe up" },
{ action: "swipe-down", label: "Swipe down" },
{ action: "swipe-left", label: "Swipe left" },
{ action: "swipe-right", label: "Swipe right" },
{ action: "scroll-up", label: "Scroll up" },
{ action: "scroll-down", label: "Scroll down" },
{ action: "long-press", label: "System menu" },
{ action: "short-then-long-press", label: "App menu" },
{ action: "none", label: "Nothing" },
];
export function keyActionLabel(action: KeyAction): string {
return KEY_ACTIONS.find((entry) => entry.action === action)?.label ?? action;
}
function isKeyAction(value: unknown): value is KeyAction {
return KEY_ACTIONS.some((entry) => entry.action === value);
}
/** A physical key with the modifiers held: what a keybinding is keyed on. */
export type KeyChord = { code: string; shift: boolean; ctrl: boolean; alt: boolean; meta: boolean };
export type KeyBinding = { chord: KeyChord; action: KeyAction };
const MODIFIER_CODES = new Set([
"ControlLeft", "ControlRight", "ShiftLeft", "ShiftRight", "AltLeft", "AltRight", "MetaLeft", "MetaRight",
]);
/** Shift, Ctrl, Alt and Meta themselves: they qualify a chord rather than being one. */
export function isModifierCode(code: string): boolean {
return MODIFIER_CODES.has(code);
}
export function chordOf(event: KeyEvent): KeyChord {
const { shift, ctrl, alt, meta } = event.modifiers;
return { code: event.code, shift, ctrl, alt, meta };
}
/** Canonical text of a chord ("Ctrl+Shift+Escape"): the stored form, and the lookup key. */
export function chordId(chord: KeyChord): string {
return [chord.ctrl && "Ctrl", chord.alt && "Alt", chord.shift && "Shift", chord.meta && "Meta", chord.code]
.filter(Boolean).join("+");
}
/** The inverse of chordId, or null for anything it could not have produced. */
export function parseChordId(text: string): KeyChord | null {
const parts = text.split("+");
const code = parts.pop() ?? "";
if (!/^[A-Za-z][A-Za-z0-9]{0,39}$/.test(code) || isModifierCode(code)) return null;
const chord: KeyChord = { code, shift: false, ctrl: false, alt: false, meta: false };
for (const part of parts) {
if (part === "Ctrl" && !chord.ctrl) chord.ctrl = true;
else if (part === "Alt" && !chord.alt) chord.alt = true;
else if (part === "Shift" && !chord.shift) chord.shift = true;
else if (part === "Meta" && !chord.meta) chord.meta = true;
else return null;
}
return chord;
}
const CODE_LABELS: Record<string, string> = {
ArrowUp: "Up", ArrowDown: "Down", ArrowLeft: "Left", ArrowRight: "Right",
Escape: "Esc", PageUp: "Page Up", PageDown: "Page Down", CapsLock: "Caps Lock",
PrintScreen: "Print Screen", ScrollLock: "Scroll Lock", NumLock: "Num Lock", ContextMenu: "Menu",
Minus: "-", Equal: "=", BracketLeft: "[", BracketRight: "]", Backslash: "\\", IntlBackslash: "\\",
IntlHash: "#", Semicolon: ";", Quote: "'", Backquote: "`", Comma: ",", Period: ".", Slash: "/",
AudioVolumeUp: "Volume Up", AudioVolumeDown: "Volume Down", AudioVolumeMute: "Mute",
MediaPlayPause: "Play/Pause", MediaPlay: "Play", MediaPause: "Pause", MediaStop: "Stop",
MediaTrackNext: "Next Track", MediaTrackPrevious: "Previous Track",
BrightnessUp: "Brightness Up", BrightnessDown: "Brightness Down",
BrowserSearch: "Search", BrowserHome: "Browser Home", BrowserBack: "Browser Back", BrowserForward: "Browser Forward",
LaunchMail: "Mail", LaunchApplication1: "App 1", LaunchApplication2: "App 2",
MouseButton1: "Left button", MouseButton2: "Right button", MouseButton3: "Middle button",
WheelUp: "Wheel up", WheelDown: "Wheel down", WheelLeft: "Wheel left", WheelRight: "Wheel right",
};
/** A key as a person would name it. Unknown keys keep their HID usage number, which is all there is. */
function codeLabel(code: string): string {
const known = CODE_LABELS[code];
if (known) return known;
let match = /^Key([A-Z])$/.exec(code);
if (match) return match[1]!;
match = /^Digit([0-9])$/.exec(code);
if (match) return match[1]!;
match = /^Numpad(.+)$/.exec(code);
if (match) return `Numpad ${match[1]}`;
match = /^MouseButton([0-9]+)$/.exec(code);
if (match) return `Mouse button ${match[1]}`;
match = /^Usage([0-9a-f]+)$/.exec(code);
if (match) return `Key 0x${match[1]}`;
match = /^Consumer([0-9a-f]+)$/.exec(code);
if (match) return `Media key 0x${match[1]}`;
return code;
}
/** A chord for display: "Shift+Esc", "Left", "Volume Up". */
export function describeChord(chord: KeyChord): string {
return [chord.ctrl && "Ctrl", chord.alt && "Alt", chord.shift && "Shift", chord.meta && "Meta", codeLabel(chord.code)]
.filter(Boolean).join("+");
}
/** The most custom keybindings kept; far more than a keyboard has uses for. */
export const MAX_KEY_BINDINGS = 64;
/**
* Custom keybindings from their stored JSON. Lenient: whatever doesn't make
* sense is dropped, and a later entry for a chord replaces an earlier one.
*/
export function parseKeyBindings(raw: string): KeyBinding[] {
let parsed: unknown;
try {
parsed = raw ? JSON.parse(raw) : [];
} catch {
return [];
}
if (!Array.isArray(parsed)) return [];
const bindings = new Map<string, KeyBinding>();
for (const entry of parsed) {
if (!entry || typeof entry !== "object") continue;
const { key, action } = entry as { key?: unknown; action?: unknown };
const chord = typeof key === "string" ? parseChordId(key) : null;
if (!chord || !isKeyAction(action)) continue;
bindings.set(chordId(chord), { chord, action });
if (bindings.size >= MAX_KEY_BINDINGS) break;
}
return [...bindings.values()];
}
export function serializeKeyBindings(bindings: readonly KeyBinding[]): string {
return JSON.stringify(bindings.map((binding) => ({ key: chordId(binding.chord), action: binding.action })));
}
/** Custom keybindings as the lookup resolveKey takes. */
export function keyBindingMap(bindings: readonly KeyBinding[]): Map<string, KeyAction> {
return new Map(bindings.map((binding) => [chordId(binding.chord), binding.action]));
}
/** What a mouse does unless bound otherwise: the ring's tap, double tap, hold and scroll. */
const MOUSE_DEFAULTS: Record<string, KeyGesture | undefined> = {
MouseButton1: "click",
MouseButton2: "double-click",
MouseButton3: "long-press",
WheelUp: "scroll-up",
WheelDown: "scroll-down",
};
/**
* The built-in meaning of a key press, or null when it has none.
* `overridesTyping`: it holds even where the focused window takes keys.
* Built-ins go by the logical key, so both Enter keys select.
*/
export function defaultKeyAction(event: KeyEvent): { action: KeyGesture; overridesTyping: boolean } | null {
// A mouse's buttons and wheel are the ring's gestures: tap, double tap,
// hold, and scroll. Nothing types them, so they hold everywhere, whatever
// is held on the keyboard.
const mouse = MOUSE_DEFAULTS[event.key];
if (mouse) return { action: mouse, overridesTyping: true };
const { shift, ctrl, alt, meta } = event.modifiers;
if (meta) return null;
if (event.key === "Escape") {
// Plain Escape belongs to whatever takes keys (it is a keystroke in a
// terminal); with a modifier it always means "back out of here".
return { action: "double-click", overridesTyping: shift || ctrl || alt };
}
if (ctrl || alt) return null;
switch (event.key) {
case "ArrowUp":
return { action: "swipe-up", overridesTyping: false };
case "ArrowDown":
return { action: "swipe-down", overridesTyping: false };
case "ArrowLeft":
return { action: "swipe-left", overridesTyping: false };
case "ArrowRight":
return { action: "swipe-right", overridesTyping: false };
case "Enter":
return { action: "click", overridesTyping: false };
default:
return null;
}
}
/** The built-ins, for showing to the user. Keep in step with defaultKeyAction. */
export const DEFAULT_KEYS_HELP =
"Arrow keys: swipe up, down, left and right, as on the watch. Enter: select. Esc: back.\n\n" +
"In a terminal session those keys are typed instead, and Shift+Esc (or Ctrl+Esc, Alt+Esc) goes back.\n\n" +
"Mouse or touchpad: the left button selects, the right button goes back, the middle button opens the system menu, " +
"and the wheel scrolls. These work in a terminal session too.\n\n" +
"A custom keybinding replaces these for its key and works everywhere, terminals included. " +
"\"Nothing\" turns a key's meaning here off without stopping it being typed.";
/**
* Whether holding the key repeats the gesture. Moving up and down a list
* does. Sideways swipes don't: depending on the layout they enter and leave
* things, and a held key would flap between the two. Selecting, backing out
* and menus act once per press.
*/
export function gestureRepeats(gesture: KeyGesture): boolean {
return gesture === "scroll-up" || gesture === "scroll-down" || gesture === "swipe-up" || gesture === "swipe-down";
}
export type KeyResolution = {
/** The key is spoken for: don't offer it to the focused window. */
consume: boolean;
/**
* The gesture to perform: at once when `consume`, otherwise only if no
* window takes the key. Null for none (including a held key's repeats of
* a gesture that doesn't repeat).
*/
gesture: KeyGesture | null;
};
/** What one key press (or auto-repeat) means, given the custom keybindings. */
export function resolveKey(event: KeyEvent, custom: ReadonlyMap<string, KeyAction>): KeyResolution {
const repeatable = (gesture: KeyGesture): KeyGesture | null =>
event.repeat && !gestureRepeats(gesture) ? null : gesture;
const bound = custom.get(chordId(chordOf(event)));
if (bound !== undefined && bound !== "none") return { consume: true, gesture: repeatable(bound) };
// "none" takes the built-in meaning away and leaves the key to be typed.
if (bound === "none") return { consume: false, gesture: null };
const builtIn = defaultKeyAction(event);
if (!builtIn) return { consume: false, gesture: null };
return { consume: builtIn.overridesTyping, gesture: repeatable(builtIn.action) };
}
export type GestureOrigin = "ring" | "watch";
/**
* Which device a keyboard's gesture is presented as. Swipes are the watch's
* and scrolls the ring's; the two schemes differ in places (an icon grid
* moves by cell for the watch, by row and then cell for the ring). A tap, a
* double tap or a hold exists in both, so it goes with whichever the
* keyboard last moved with: arrow keys make the keyboard a watch, a remote
* bound to scroll makes it a ring. Before any movement, the watch.
*/
export class GestureScheme {
private scheme: GestureOrigin = "watch";
originFor(gesture: KeyGesture): GestureOrigin {
if (gesture.startsWith("swipe-")) this.scheme = "watch";
else if (gesture.startsWith("scroll-")) this.scheme = "ring";
return this.scheme;
}
}
-566
View File
@@ -1,566 +0,0 @@
/**
* The phone side of the CFW keyboard bridge for one glasses session: the
* client, the remembered keyboard, reconnecting it, and what the UI shows.
* The session hands it field-131 records and a way to send controls; the
* rest stays in here. Modelled on the Glassly app's G2KeyboardBridge (MIT).
*
* The firmware keeps nothing across a reboot (it comes back with the bridge
* off), and holds the keyboard's connection only while told to. So the
* remembered keyboard is brought back whenever the glasses link comes up and
* whenever the keyboard drops, backing off while it stays away. The
* discovered handle mapping is good for one connection and is never stored.
* No platform imports: this runs under node in the tests.
*/
import { KeyboardClient, KeyboardClientError, type KeyboardStatus } from "./client";
import type { HidKeyEvent, HidPointerSample } from "./hid";
import {
KeyboardDm,
KeyboardError,
mergeKeyboardAdverts,
parseKeyboardAddress,
type KeyboardAdvert,
} from "./protocol";
export type SavedKeyboard = { address: string; addressType: number; name: string };
export type KeyboardBridgeState =
/** No glasses link that can host a keyboard. */
| "unavailable"
| "idle"
| "scanning"
| "connecting"
| "pairing"
| "configuring"
| "connected"
| "error";
/** A pairing step waiting on the user. */
export type KeyboardPairingPrompt =
/** Type this on the keyboard, then Enter. */
| { kind: "display"; pin: number }
/** Confirm both sides show this value (answerCompare). */
| { kind: "compare"; value: number }
/** The keyboard shows a code to enter here (answerPasskey). */
| { kind: "entry" };
export type KeyboardBridgeSnapshot = {
state: KeyboardBridgeState;
saved: SavedKeyboard | null;
/** Name of the keyboard being connected or in use. */
activeName: string;
error: string;
pairing: KeyboardPairingPrompt | null;
/** Advertisers from the last scan, HID devices first, strongest first. */
scanResults: KeyboardAdvert[];
/** A user-started scan or connect is running. */
busy: boolean;
/** The remembered keyboard is away and the bridge keeps trying to bring it back. */
waiting: boolean;
/** The connected keyboard has a touchpad or mouse buttons. */
pointer: boolean;
};
export type KeyboardBridgeHost = {
/** Queue a control record on the live session; false when there is none. */
send(record: Uint8Array): boolean;
loadSaved(): SavedKeyboard | null;
storeSaved(saved: SavedKeyboard | null): void;
/** Key transitions from the keyboard (including forced releases). */
onKeys(events: HidKeyEvent[]): void;
/** A report from the keyboard's touchpad or mouse (including a forced release of its buttons). */
onPointer(sample: HidPointerSample): void;
/** The keyboard stopped being usable; drop held-key state without releases. */
onKeyboardGone(): void;
/** Something in snapshot() changed. */
onChanged(): void;
/**
* A line for the log. `routine` marks the ones that repeat for as long as
* a remembered keyboard stays away, which only a verbose log wants.
*/
log(line: string, routine?: boolean): void;
/** A fresh six-digit pairing passkey from a good random source. */
randomPasskey?(): number;
/**
* Whether looking for an absent keyboard is worthwhile now. Each attempt
* has the glasses' radio listening for the keyboard for up to half a
* minute, so the owner limits it to when the glasses are in use, and calls
* retryNow() when that starts. Default: always.
*/
shouldReconnect?(): boolean;
};
/** Delay before each automatic attempt after a failure; the last one repeats. */
const RECONNECT_BACKOFF_MS = [2_000, 5_000, 10_000, 20_000, 30_000];
/** After the glasses link comes up, before the first attempt. */
const RESUME_DELAY_MS = 1_000;
/** After the keyboard drops (it may just be going to sleep). */
const DROP_RETRY_DELAY_MS = 2_000;
const RELEASE_POLLS = 20;
const RELEASE_POLL_MS = 150;
export class KeyboardBridge {
private client: KeyboardClient | null = null;
private available = false;
private state: KeyboardBridgeState = "unavailable";
private saved: SavedKeyboard | null;
private activeName = "";
private error = "";
private pairing: KeyboardPairingPrompt | null = null;
private scanResults: KeyboardAdvert[] = [];
/** A user-started scan/connect holds the bridge. */
private busy = false;
/** The keyboard is connected, encrypted and subscribed. */
private ready = false;
private userDisconnected = false;
/** The keyboard was let go while the glasses were out of reach; tell the firmware when they are back. */
private releaseOnLink = false;
private reconnectTimer: ReturnType<typeof setTimeout> | null = null;
private reconnectAttempt = 0;
/** Bumped whenever the client is replaced; work started for an older one stops quietly. */
private generation = 0;
constructor(private readonly host: KeyboardBridgeHost) {
this.saved = host.loadSaved();
}
snapshot(): KeyboardBridgeSnapshot {
return {
state: this.state,
saved: this.saved,
activeName: this.activeName,
error: this.error,
pairing: this.pairing,
scanResults: sortAdverts(this.scanResults),
busy: this.busy,
waiting: this.available && !!this.saved && !this.ready && !this.userDisconnected && !this.busy,
pointer: this.ready && this.client?.configuration?.pointer === true,
};
}
/** True while key events can arrive. */
isConnected(): boolean {
return this.ready;
}
// ---------- Link lifecycle (driven by the session owner) ----------
/** The glasses link is up on firmware with the bridge: bring the remembered keyboard back. */
linkReady(): void {
if (this.available) return;
this.available = true;
this.userDisconnected = false;
this.reconnectAttempt = 0;
this.error = "";
const client = this.replaceClient();
this.setState("idle");
if (this.saved && !this.releaseOnLink) {
this.scheduleReconnect(RESUME_DELAY_MS);
} else if (this.releaseOnLink && client) {
this.userDisconnected = !!this.saved;
client.query().then((status) => {
if (status.enabled || status.conn) return client.disable();
}).catch(() => {});
}
this.releaseOnLink = false;
}
/** The glasses link is gone. Only the remembered keyboard survives. */
linkLost(): void {
if (!this.available && !this.client) return;
this.available = false;
this.cancelReconnect();
this.generation++;
this.client?.close();
this.client = null;
this.busy = false;
this.pairing = null;
this.error = "";
this.dropKeyboard();
this.setState("unavailable");
}
/**
* The app is about to close the glasses link on purpose: tell the firmware
* to let the keyboard go, so it isn't left attached to glasses nobody is
* listening to. Best effort; nothing waits for the answer.
*/
linkClosing(): void {
const client = this.client;
if (!client || !this.available || !client.status.enabled) return;
client.disable().catch(() => {});
}
/** One field-131 payload from the session. */
accept(payload: Uint8Array): void {
this.client?.accept(payload);
}
/**
* The keyboard may be reachable now, or is wanted now (the display woke):
* if it is away and the bridge is waiting out a backoff, try at once.
*/
retryNow(): void {
if (!this.available || !this.saved || this.ready || this.userDisconnected || this.busy) return;
// No timer means an attempt is already in flight.
if (this.reconnectTimer === null) return;
this.reconnectAttempt = 0;
this.scheduleReconnect(0);
}
// ---------- Commands (the settings UI) ----------
/** Scan for about five seconds; results accumulate in the snapshot as they arrive. */
async scan(): Promise<KeyboardAdvert[]> {
const client = this.beginUserOperation();
const generation = this.generation;
this.scanResults = [];
this.error = "";
this.setState("scanning");
try {
await this.releaseLink(client);
await client.enable();
await client.scan();
if (generation === this.generation) this.setState("idle");
} catch (e) {
if (generation === this.generation) this.fail(e);
throw friendly(e);
} finally {
if (generation === this.generation) this.endUserOperation();
}
return sortAdverts(this.scanResults);
}
/** Connect, pair and subscribe a keyboard, then remember it. */
async connect(keyboard: SavedKeyboard): Promise<void> {
if (!parseKeyboardAddress(keyboard.address) || keyboard.addressType < 0 || keyboard.addressType > 3) {
throw new KeyboardClientError("Invalid keyboard address");
}
const client = this.beginUserOperation();
const generation = this.generation;
this.userDisconnected = false;
try {
await this.connectWith(client, generation, keyboard, false);
} finally {
if (generation === this.generation) this.endUserOperation();
}
}
/** Try the remembered keyboard again now. */
async reconnect(): Promise<void> {
if (!this.saved) throw new KeyboardClientError("No remembered keyboard");
await this.connect(this.saved);
}
/** Drop the keyboard but keep it remembered. No automatic reconnect until the next explicit connect or glasses session. */
async disconnect(): Promise<void> {
this.userDisconnected = true;
this.cancelReconnect();
const client = this.replaceClient();
// A scan or connect this interrupts no longer owns the bridge.
this.busy = false;
this.dropKeyboard();
this.pairing = null;
this.error = "";
if (!client) {
this.releaseOnLink = true;
this.setState("unavailable");
return;
}
this.setState("idle");
try {
await client.query();
await client.disable();
} catch (e) {
this.host.log(`keyboard disconnect: ${messageOf(e)}`);
}
}
/** Forget the remembered keyboard. The glasses keep their bond with it. */
async forget(): Promise<void> {
this.saved = null;
this.host.storeSaved(null);
await this.disconnect();
this.scanResults = [];
this.activeName = "";
this.host.onChanged();
}
/** Answer a "compare" pairing prompt. */
async answerCompare(accept: boolean): Promise<void> {
if (!this.client || this.pairing?.kind !== "compare") throw new KeyboardClientError("No pairing comparison is pending");
this.pairing = null;
this.host.onChanged();
await this.client.confirm(accept);
}
/** Answer an "entry" pairing prompt with the code the keyboard shows. */
async answerPasskey(pin: number): Promise<void> {
if (!this.client || this.pairing?.kind !== "entry") throw new KeyboardClientError("No pairing code is pending");
this.pairing = null;
this.host.onChanged();
await this.client.authenticate(pin);
}
// ---------- Internals ----------
private beginUserOperation(): KeyboardClient {
if (!this.available) throw new KeyboardClientError("Connect to G2 glasses running Faceclaw firmware first");
if (this.busy) throw new KeyboardClientError("Another keyboard operation is in progress");
// An automatic attempt may be mid-flight (a sleeping keyboard holds one
// for up to 30 s): a fresh client fails it at once and takes over.
this.cancelReconnect();
const client = this.replaceClient()!;
this.dropKeyboard();
this.pairing = null;
this.busy = true;
return client;
}
private endUserOperation(): void {
this.busy = false;
this.host.onChanged();
}
/** Close the current client and, while the link is up, start a fresh one. */
private replaceClient(): KeyboardClient | null {
this.generation++;
this.client?.close();
this.client = null;
if (!this.available) return null;
const generation = this.generation;
const live = () => generation === this.generation;
const client = new KeyboardClient(
(record) => live() && this.host.send(record),
{
keys: (events) => { if (live() && this.ready) this.host.onKeys(events); },
pointer: (sample) => { if (live() && this.ready) this.host.onPointer(sample); },
advert: (advert) => { if (live()) this.addAdvert(advert); },
status: (status) => { if (live()) this.onStatus(status); },
dm: (event, status) => { if (live()) this.onDm(event, status); },
passkeyDisplay: (pin) => { if (live()) this.prompt({ kind: "display", pin }); },
passkeyRequest: () => { if (live()) this.prompt({ kind: "entry" }); },
compareRequest: (value) => { if (live()) this.prompt({ kind: "compare", value }); },
servicesChanged: () => { if (live()) this.onServicesChanged(); },
error: (message) => { if (live()) this.host.log(`keyboard: ${message}`); },
log: (line) => { if (live()) this.host.log(`keyboard setup: ${line}`); },
},
{ randomPasskey: this.host.randomPasskey ? () => this.host.randomPasskey!() : undefined },
);
this.client = client;
return client;
}
private async connectWith(
client: KeyboardClient,
generation: number,
keyboard: SavedKeyboard,
resume: boolean,
): Promise<void> {
const live = () => generation === this.generation;
this.error = "";
this.activeName = keyboard.name;
this.setState("connecting");
try {
const status = await client.query();
if (resume && status.enabled && status.connected && status.encrypted) {
// The glasses kept the keyboard through a phone-side drop (or an app
// restart): only the handle mapping is missing.
this.host.log("keyboard still connected to the glasses; rediscovering");
} else {
await this.releaseLink(client);
await client.enable();
await client.connect(keyboard.addressType, parseKeyboardAddress(keyboard.address)!);
}
if (!live()) return;
this.pairing = null;
this.setState("configuring");
const config = await client.configure();
if (!live()) return;
this.ready = true;
this.reconnectAttempt = 0;
// The list was for choosing; it is stale from here on.
this.scanResults = [];
if (!sameKeyboard(this.saved, keyboard)) {
this.saved = { ...keyboard };
this.host.storeSaved(this.saved);
}
this.host.log(
`keyboard ready: ${keyboard.name || keyboard.address}, ${config.watched.length} of ` +
`${config.reports.filter((r) => r.type === 1).length} input reports subscribed` +
`${config.pointer ? ", with a pointer" : ""}, epoch ${config.epoch}`);
this.setState("connected");
} catch (e) {
if (!live()) return;
this.ready = false;
this.pairing = null;
if (resume) {
// An automatic attempt: a keyboard that is asleep or out of range is
// the ordinary case, not an error to put in front of the user.
this.host.log(`keyboard reconnect: ${messageOf(friendly(e))}`, true);
this.setState("idle");
} else {
this.fail(e);
}
throw friendly(e);
}
}
/**
* The firmware refuses SCAN and CONNECT while it still owns a keyboard
* slot, and an interrupted connect keeps initiating until the close lands.
* DISABLE is epoch-checked and a fresh client hasn't seen the epoch yet,
* so QUERY first (it isn't, and its answer carries the epoch), then wait
* for the slot to clear.
*/
private async releaseLink(client: KeyboardClient): Promise<void> {
const status = await client.query();
if (!status.enabled && status.conn === 0) return;
await client.disable();
for (let i = 0; i < RELEASE_POLLS; i++) {
if ((await client.query()).conn === 0) return;
await delay(RELEASE_POLL_MS);
}
throw new KeyboardClientError("The glasses are still closing the previous keyboard link; try again in a moment");
}
private cancelReconnect(): void {
if (this.reconnectTimer !== null) clearTimeout(this.reconnectTimer);
this.reconnectTimer = null;
}
private scheduleReconnect(delayMs: number): void {
this.cancelReconnect();
this.reconnectTimer = setTimeout(() => {
this.reconnectTimer = null;
void this.attemptReconnect();
}, delayMs);
}
private async attemptReconnect(): Promise<void> {
const saved = this.saved;
const client = this.client;
if (!this.available || !client || !saved || this.userDisconnected || this.ready || this.busy) return;
if (this.host.shouldReconnect && !this.host.shouldReconnect()) {
this.parkFirmware(client);
this.scheduleReconnect(RECONNECT_BACKOFF_MS[RECONNECT_BACKOFF_MS.length - 1]!);
return;
}
const generation = this.generation;
try {
await this.connectWith(client, generation, saved, true);
} catch {
if (generation !== this.generation || !this.available || this.userDisconnected) return;
// Keyboards sleep, and the ring's own scan can hold the controller
// (BUSY): keep trying, less often, while the glasses stay up.
const backoff = RECONNECT_BACKOFF_MS[Math.min(this.reconnectAttempt, RECONNECT_BACKOFF_MS.length - 1)]!;
this.reconnectAttempt++;
this.scheduleReconnect(backoff);
}
}
/**
* While it is enabled the firmware bridge runs a 20 ms timer and reports
* its status every second, keyboard or no keyboard. When nothing will be
* tried for a while, switch it off; the next attempt enables it again.
*/
private parkFirmware(client: KeyboardClient): void {
if (!client.status.enabled) return;
client.disable().catch(() => {});
}
private onStatus(status: KeyboardStatus): void {
if (this.ready && (!status.connected || !status.encrypted)) this.keyboardDropped("status");
else if (this.state === "connecting" && status.connected && !status.encrypted) this.setState("pairing");
}
private onDm(event: number, status: number): void {
if (event === KeyboardDm.connClose || event === KeyboardDm.pairFailed || event === KeyboardDm.encryptFailed) {
this.host.log(`keyboard DM event 0x${event.toString(16)} status ${status}`);
if (this.ready) this.keyboardDropped("dm");
}
}
private onServicesChanged(): void {
this.host.log("keyboard services changed; rediscovering");
this.dropKeyboard();
this.setState("idle");
if (this.saved && !this.userDisconnected && !this.busy) this.scheduleReconnect(0);
}
private keyboardDropped(reason: string): void {
this.host.log(`keyboard disconnected (${reason})`);
this.dropKeyboard();
this.setState("idle");
if (this.saved && !this.userDisconnected && !this.busy) this.scheduleReconnect(DROP_RETRY_DELAY_MS);
}
/** The keyboard is no longer usable: tell the host once. */
private dropKeyboard(): void {
if (!this.ready) return;
this.ready = false;
this.host.onKeyboardGone();
}
private addAdvert(advert: KeyboardAdvert): void {
const index = this.scanResults.findIndex(
(a) => a.address === advert.address && a.addressType === advert.addressType);
if (index < 0) {
this.scanResults.push(advert);
} else {
const merged = mergeKeyboardAdverts(this.scanResults[index]!, advert);
const old = this.scanResults[index]!;
if (merged.name === old.name && merged.hid === old.hid && merged.rssi === old.rssi) return;
this.scanResults[index] = merged;
}
this.host.onChanged();
}
private prompt(pairing: KeyboardPairingPrompt): void {
this.pairing = pairing;
this.setState("pairing");
}
private setState(next: KeyboardBridgeState): void {
this.state = next;
if (next !== "pairing" && next !== "connecting") this.pairing = null;
this.host.onChanged();
}
private fail(error: unknown): void {
this.error = messageOf(friendly(error));
this.host.log(`keyboard error: ${this.error}`);
this.setState("error");
}
}
function sameKeyboard(a: SavedKeyboard | null, b: SavedKeyboard): boolean {
return !!a && a.address === b.address && a.addressType === b.addressType && a.name === b.name;
}
function sortAdverts(adverts: readonly KeyboardAdvert[]): KeyboardAdvert[] {
return [...adverts].sort((a, b) => Number(b.hid) - Number(a.hid) || b.rssi - a.rssi);
}
/**
* Firmware BUSY nearly always means the ring's own scan or reconnect holds
* the radio; say that rather than the raw control error.
*/
function friendly(error: unknown): Error {
if (error instanceof KeyboardClientError && error.code === KeyboardError.busy) {
return new KeyboardClientError(
"The glasses' radio is busy (usually the ring reconnecting). Wait a few seconds and try again.", error.code);
}
if (error instanceof KeyboardClientError && error.code === KeyboardError.noSpace) {
return new KeyboardClientError(
"The glasses have no room for another connection or pairing.", error.code);
}
return error instanceof Error ? error : new Error(String(error));
}
function messageOf(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function delay(ms: number): Promise<void> {
return new Promise<void>((resolve) => setTimeout(resolve, ms));
}
-754
View File
@@ -1,754 +0,0 @@
/**
* Transport-independent client for the CFW keyboard bridge: enable, scan,
* connect and pair a keyboard through the glasses, discover its HID service
* over the relayed ATT operations, subscribe its input reports, and decode
* them into key transitions. From the reference client in g2flash demos/
* (keyboard-client.ts), with the fixes its use in the Glassly app turned up.
*
* The firmware keeps almost no policy: it owns the connection and the bond,
* and forwards one ATT request at a time. Everything HID lives here. Handle
* numbers found by configure() are only good for one connection epoch.
* No platform imports: this runs under node in the tests.
*/
import {
HidKeys,
parseReportMap,
reportHasKeys,
reportHasPointer,
reportPointer,
type HidKeyEvent,
type HidPointerSample,
type HidReport,
} from "./hid";
import {
KeyboardDm,
KeyboardError,
KeyboardKind,
KeyboardOp,
KeyboardRecords,
keyboardControl,
keyboardErrorName,
le16,
parseKeyboardAdvert,
u16,
u32,
type KeyboardAdvert,
type KeyboardRecord,
} from "./protocol";
/** Queue one field-130 control record; false when the glasses link can't take it now. */
export type KeyboardTransport = (record: Uint8Array) => boolean;
export type KeyboardStatus = {
enabled: boolean;
/** The firmware's connection id for the keyboard, 0 when it owns none. */
conn: number;
connected: boolean;
encrypted: boolean;
scanning: boolean;
};
export type KeyboardClientEvents = {
/** Key transitions from one input report (or the releases a gap forces). */
keys?(events: HidKeyEvent[]): void;
/**
* One report from the keyboard's touchpad or mouse: the buttons now held
* (across all its pointer reports) and the movement and wheel since the
* last one. Also sent with everything zero when a gap forces a release.
*/
pointer?(sample: HidPointerSample): void;
advert?(advert: KeyboardAdvert): void;
status?(status: KeyboardStatus): void;
/** The stream may have lost records; held keys were released. */
gap?(): void;
/** Device-manager event from the glasses' BLE host (KeyboardDm). */
dm?(event: number, status: number): void;
/** The keyboard shows a six-digit passkey; answer with authenticate(). */
passkeyRequest?(): void;
/** We chose a passkey and answered with it; the user types it on the keyboard, then Enter. */
passkeyDisplay?(pin: number): void;
/** Numeric comparison: answer with confirm() once the user has compared. */
compareRequest?(value: number): void;
/** The keyboard's GATT table changed; the mapping is void until configure() runs again. */
servicesChanged?(): void;
error?(message: string): void;
/** Setup diagnostics: the keyboard's attribute table and what was made of it. Never key contents. */
log?(line: string): void;
};
export type KeyboardReport = {
/** Handle of the HID service declaration the report belongs to. */
service: number;
handle: number;
id: number;
/** 1 input, 2 output, 3 feature. */
type: number;
ccc?: number;
/** From the characteristic's declaration; undefined when the keyboard wouldn't say (see declaredProperties). */
properties?: number;
map: Uint8Array;
};
export type KeyboardConfiguration = {
epoch: number;
/** Every report found, input or not. */
reports: KeyboardReport[];
/** Handles of the input reports subscribed: those that carry keys, or a pointer's buttons and movement. */
watched: number[];
/** Whether any of them is a pointer's (a touchpad, mouse buttons, a wheel). */
pointer: boolean;
serviceChanged?: number;
};
export type KeyboardClientOptions = {
/** A fresh six-digit passkey for display pairing. Default: Math.random (pass a stronger source). */
randomPasskey?: () => number;
};
export class KeyboardClientError extends Error {
/** The firmware's control error code (KeyboardError), when that is what failed. */
readonly code: number | undefined;
/** The keyboard's ATT error code, when it refused a read or write. */
readonly attStatus: number | undefined;
constructor(message: string, code?: number, attStatus?: number) {
super(message);
this.name = "KeyboardClientError";
this.code = code;
this.attStatus = attStatus;
}
}
type Attribute = { handle: number; uuid: number | string };
type Pending = {
op: number;
resolve: (r: KeyboardRecord) => void;
reject: (e: Error) => void;
timer: ReturnType<typeof setTimeout>;
};
type Waiter = { test: () => boolean; resolve: () => void; reject: (e: Error) => void; timer: ReturnType<typeof setTimeout> };
type Input = { report: HidReport; id: number; service: number; keys: boolean; pointer: boolean };
const COMMAND_TIMEOUT_MS = 32_000;
const MAX_PENDING = 8;
const MAX_ATTRIBUTES = 512;
const MAX_VALUE = 512;
const MAX_WATCHES = 16;
const UUID_PRIMARY_SERVICE = 0x2800;
const UUID_SECONDARY_SERVICE = 0x2801;
const UUID_CHARACTERISTIC = 0x2803;
const UUID_CCC = 0x2902;
const UUID_EXTERNAL_REPORT_REFERENCE = 0x2907;
const UUID_REPORT_REFERENCE = 0x2908;
const UUID_HID_SERVICE = 0x1812;
const UUID_SERVICE_CHANGED = 0x2a05;
const UUID_REPORT_MAP = 0x2a4b;
const UUID_REPORT = 0x2a4d;
const UUID_PROTOCOL_MODE = 0x2a4e;
/** ATT error codes that end a paged read rather than failing it. */
const ATT_INVALID_OFFSET = 7;
const ATT_NOT_LONG = 11;
const ATT_NOT_FOUND = 10;
/**
* ATT UUIDs are little endian. A 128-bit UUID on the Bluetooth base collapses
* to its 16-bit form; vendor UUIDs stay as hex strings.
*/
export function attributeUuid(bytes: Uint8Array): number | string {
if (bytes.length === 2) return u16(bytes);
if (bytes.length !== 16) throw new KeyboardClientError("Invalid ATT UUID length");
const base = [0xfb, 0x34, 0x9b, 0x5f, 0x80, 0, 0, 0x80, 0, 0x10, 0, 0];
if (base.every((v, i) => bytes[i] === v) && bytes[14] === 0 && bytes[15] === 0) return u16(bytes, 12);
return Array.from(bytes, (x) => x.toString(16).padStart(2, "0")).join("");
}
export class KeyboardClient {
/** The firmware's connection generation; every control but QUERY/ENABLE must carry the live one. */
epoch = 0;
status: KeyboardStatus = { enabled: false, conn: 0, connected: false, encrypted: false, scanning: false };
configuration: KeyboardConfiguration | undefined;
private serial = 0;
private closed = false;
/** The link under connect() went away (close, failed open, failed pairing). */
private linkLost = false;
private readonly pending = new Map<number, Pending>();
private readonly waiters = new Set<Waiter>();
private readonly keys = new HidKeys();
/** Pointer buttons held, by the report that says so (as HidKeys does for keys). */
private readonly pointerButtons = new Map<number, number>();
private inputs = new Map<number, Input>();
private readonly records = new KeyboardRecords(() => this.gap());
private readonly randomPasskey: () => number;
constructor(
private readonly transport: KeyboardTransport,
private readonly events: KeyboardClientEvents = {},
options: KeyboardClientOptions = {},
) {
this.randomPasskey = options.randomPasskey ?? (() => Math.floor(Math.random() * 1_000_000));
}
// ---------- Inbound ----------
/** Feed one field-131 payload, in arrival order. */
accept(data: Uint8Array): void {
if (this.closed) return;
const r = this.records.push(data);
if (!r) return;
this.epoch = r.epoch;
if (r.kind === KeyboardKind.status && r.data.length === 10) {
this.status = {
enabled: !!r.data[0], conn: r.data[1]!, connected: !!r.data[2], encrypted: !!r.data[3], scanning: !!r.data[4],
};
if (!this.status.enabled || !this.status.connected || !this.status.encrypted) this.release();
this.events.status?.(this.status);
}
const pending = this.pending.get(r.request);
if (pending && (r.kind === KeyboardKind.status || r.kind === KeyboardKind.att)) {
this.pending.delete(r.request);
clearTimeout(pending.timer);
if (r.kind === KeyboardKind.status && r.code) {
pending.reject(new KeyboardClientError(
`Keyboard command ${opName(pending.op)}: firmware ${keyboardErrorName(r.code)}`, r.code));
} else {
pending.resolve(r);
}
}
if (r.kind === KeyboardKind.dm) this.acceptDm(r);
else if (r.kind === KeyboardKind.input) this.acceptInput(r);
else if (r.kind === KeyboardKind.advert) {
const advert = parseKeyboardAdvert(r.data);
if (advert) this.events.advert?.(advert);
}
this.checkWaiters();
}
private acceptDm(r: KeyboardRecord): void {
const event = r.code & 255;
const status = r.code >>> 8;
if (event === KeyboardDm.connClose || event === KeyboardDm.pairFailed || event === KeyboardDm.encryptFailed ||
(event === KeyboardDm.connOpen && status !== 0)) {
this.linkLost = true;
this.gap();
this.inputs.clear();
this.configuration = undefined;
}
if (event === KeyboardDm.authRequest && r.data.length === 2) {
if (r.data[0]) {
this.command(KeyboardOp.disconnect).catch(() => {});
this.events.error?.("This keyboard only pairs out of band, which isn't supported.");
} else if (r.data[1]) {
// KeyboardDisplay on our side: choose the passkey, show it, and
// answer at once; the user types it on the keyboard.
const pin = Math.min(999_999, Math.max(0, Math.floor(this.randomPasskey())));
this.events.passkeyDisplay?.(pin);
this.authenticate(pin).catch((e) => this.events.error?.(messageOf(e)));
} else {
this.events.passkeyRequest?.();
}
}
if (event === KeyboardDm.compare && r.data.length === 4) this.events.compareRequest?.(u32(r.data));
this.events.dm?.(event, status);
}
private acceptInput(r: KeyboardRecord): void {
const changed = this.configuration?.serviceChanged;
if (changed !== undefined && r.handle === changed) {
this.gap();
this.inputs.clear();
this.configuration = undefined;
this.events.servicesChanged?.();
return;
}
const input = this.inputs.get(r.handle);
if (!input || this.configuration?.epoch !== r.epoch) {
this.gap();
return;
}
if (input.keys) {
try {
const events = this.keys.update(r.handle, input.report, r.data);
if (events.length) this.events.keys?.(events);
} catch (e) {
this.release();
this.events.error?.(messageOf(e));
}
}
if (input.pointer) {
const sample = reportPointer(input.report, r.data);
this.pointerButtons.set(r.handle, sample.buttons);
this.events.pointer?.({ ...sample, buttons: this.heldButtons() });
}
}
private heldButtons(): number {
let buttons = 0;
for (const held of this.pointerButtons.values()) buttons |= held;
return buttons;
}
/** Nothing is held any more, as far as anyone can tell: say so. */
private release(): void {
const events = this.keys.reset();
if (events.length) this.events.keys?.(events);
const buttons = this.heldButtons();
this.pointerButtons.clear();
if (buttons) this.events.pointer?.({ buttons: 0, dx: 0, dy: 0, wheel: 0, pan: 0 });
}
private gap(): void {
this.release();
this.events.gap?.();
for (const [id, p] of this.pending) {
if (p.op < KeyboardOp.find) continue;
clearTimeout(p.timer);
this.pending.delete(id);
p.reject(new KeyboardClientError("Keyboard stream gap during ATT transaction"));
}
}
/** The phone's link to the glasses dropped: release held keys and fail everything in flight. */
lostPhoneLink(): void {
this.records.reset();
this.cancel("Glasses link lost");
}
/** Stop for good; later accept() calls are ignored. */
close(): void {
if (this.closed) return;
this.lostPhoneLink();
this.closed = true;
}
private cancel(reason: string): void {
for (const p of this.pending.values()) {
clearTimeout(p.timer);
p.reject(new KeyboardClientError(reason));
}
this.pending.clear();
for (const w of this.waiters) {
clearTimeout(w.timer);
w.reject(new KeyboardClientError(reason));
}
this.waiters.clear();
}
private checkWaiters(): void {
for (const w of [...this.waiters]) {
let done = false;
try {
done = w.test();
} catch (e) {
this.waiters.delete(w);
clearTimeout(w.timer);
w.reject(e instanceof Error ? e : new KeyboardClientError(String(e)));
continue;
}
if (!done) continue;
this.waiters.delete(w);
clearTimeout(w.timer);
w.resolve();
}
}
/** Resolve once `test` holds; it is re-evaluated after every accepted record. */
private waitFor(test: () => boolean, timeoutMs: number, what: string): Promise<void> {
if (this.closed) return Promise.reject(new KeyboardClientError("Keyboard client closed"));
if (test()) return Promise.resolve();
return new Promise<void>((resolve, reject) => {
const waiter: Waiter = {
test, resolve, reject,
timer: setTimeout(() => {
this.waiters.delete(waiter);
reject(new KeyboardClientError(`Timed out waiting for ${what}`));
}, timeoutMs),
};
this.waiters.add(waiter);
});
}
// ---------- Commands ----------
/** Send one control and await the STATUS or ATT record that answers it. */
command(op: number, body: readonly number[] = []): Promise<KeyboardRecord> {
if (this.closed) return Promise.reject(new KeyboardClientError("Keyboard client closed"));
if (this.pending.size >= MAX_PENDING) return Promise.reject(new KeyboardClientError("Too many keyboard commands"));
const request = this.serial = this.serial % 65535 + 1;
return new Promise<KeyboardRecord>((resolve, reject) => {
const timer = setTimeout(() => {
this.pending.delete(request);
reject(new KeyboardClientError(`Keyboard command ${opName(op)} timed out`));
}, COMMAND_TIMEOUT_MS);
this.pending.set(request, { op, resolve, reject, timer });
let queued = false;
try {
queued = this.transport(keyboardControl(op, request, this.epoch, body));
} catch {
queued = false;
}
if (!queued) {
clearTimeout(timer);
this.pending.delete(request);
reject(new KeyboardClientError("The glasses aren't ready for keyboard commands"));
}
});
}
async enable(): Promise<void> {
await this.command(KeyboardOp.enable);
}
/** Disconnect the keyboard and stop the bridge; the bond stays. */
async disable(): Promise<void> {
await this.command(KeyboardOp.disable);
this.release();
}
/** Ask for the firmware's state. Also the way to learn the live epoch: QUERY isn't epoch-checked. */
async query(): Promise<KeyboardStatus> {
await this.command(KeyboardOp.query);
return this.status;
}
/**
* Active scan for about five seconds; adverts arrive through events.advert.
* The firmware only clears `scanning` once the stock scanner is idle again,
* and a ring looking for its lens can hold that well past the window, so a
* lingering flag is not an error: the adverts already delivered stand.
*/
async scan(): Promise<void> {
await this.command(KeyboardOp.scan);
try {
await this.waitFor(() => !this.status.scanning, 8000, "the scan to finish");
} catch (e) {
if (this.closed) throw e;
}
}
/** Connect, then pair unless the link comes up encrypted. `address` is LSB first. */
async connect(addressType: number, address: Uint8Array): Promise<void> {
if (address.length !== 6 || !Number.isInteger(addressType) || addressType < 0 || addressType > 3) {
throw new KeyboardClientError("Invalid BLE address");
}
this.release();
this.inputs.clear();
this.configuration = undefined;
this.linkLost = false;
await this.command(KeyboardOp.connect, [addressType, ...address]);
await this.waitFor(() => this.status.connected || this.linkLost, 32_000, "the keyboard to connect");
if (!this.status.connected) throw new KeyboardClientError("The keyboard didn't answer");
if (!this.status.encrypted) {
// The keyboard's own security request may have started pairing already.
// BUSY is not success: the encrypted status below still decides.
try {
await this.command(KeyboardOp.pair);
} catch (e) {
if (!(e instanceof KeyboardClientError) || e.code !== KeyboardError.busy) throw e;
}
await this.waitFor(() => this.status.encrypted || this.linkLost, 65_000, "pairing to finish");
if (!this.status.encrypted) throw new KeyboardClientError("Pairing with the keyboard failed");
}
}
async disconnect(): Promise<void> {
await this.command(KeyboardOp.disconnect);
this.release();
}
/** Answer a pending passkey prompt. */
async authenticate(pin: number): Promise<void> {
if (!Number.isInteger(pin) || pin < 0 || pin > 999_999) throw new KeyboardClientError("Invalid pairing passkey");
await this.command(KeyboardOp.auth, [pin & 255, (pin >>> 8) & 255, pin >>> 16]);
}
/** Answer a pending numeric comparison. */
async confirm(accept: boolean): Promise<void> {
await this.command(KeyboardOp.compare, [accept ? 1 : 0]);
}
private async att(op: number, body: readonly number[]): Promise<KeyboardRecord> {
const r = await this.command(op, body);
if (r.kind !== KeyboardKind.att) throw new KeyboardClientError("Expected keyboard ATT response");
return r;
}
/** Read a whole attribute: Read, then Read Blob until a short or empty reply. Never assumes the MTU. */
async read(handle: number): Promise<Uint8Array> {
const parts: number[] = [];
for (;;) {
const r = await this.att(KeyboardOp.read, [...le16(handle), ...le16(parts.length)]);
const status = r.code >>> 8;
if (parts.length && (status === ATT_INVALID_OFFSET || status === ATT_NOT_LONG)) break;
if (status) throw new KeyboardClientError(`ATT read ${handle}: ${status}`, undefined, status);
if (!r.data.length) break;
for (const b of r.data) parts.push(b);
if (parts.length > MAX_VALUE) throw new KeyboardClientError("Attribute exceeds 512 bytes");
if (parts.length === MAX_VALUE) break;
}
return Uint8Array.from(parts);
}
/** Read a short attribute once (a Report Reference, a declaration, a service UUID). */
private async shortRead(handle: number): Promise<Uint8Array> {
const r = await this.att(KeyboardOp.read, [...le16(handle), 0, 0]);
if (r.code >>> 8) throw new KeyboardClientError(`ATT read ${handle}: ${r.code >>> 8}`, undefined, r.code >>> 8);
return r.data;
}
/** Write 1..18 bytes (an output report such as the LEDs, a CCC, the protocol mode). */
async write(handle: number, value: Uint8Array, withoutResponse = false): Promise<void> {
if (!value.length || value.length > 18) throw new KeyboardClientError("Keyboard write supports 1..18 bytes");
const r = await this.att(KeyboardOp.write, [...le16(handle), withoutResponse ? 1 : 0, value.length, ...value]);
if (r.code >>> 8) throw new KeyboardClientError(`ATT write ${handle}: ${r.code >>> 8}`, undefined, r.code >>> 8);
}
private async attributes(): Promise<Attribute[]> {
const attrs: Attribute[] = [];
let start = 1;
while (start <= 65535) {
const r = await this.att(KeyboardOp.find, [...le16(start), 255, 255]);
const status = r.code >>> 8;
if (status === ATT_NOT_FOUND) break; // the normal end of the table
if (status) throw new KeyboardClientError(`ATT Find Information: ${status}`);
const width = r.data[0] === 1 ? 4 : r.data[0] === 2 ? 18 : 0;
if (!width || r.data.length <= 1 || (r.data.length - 1) % width) {
throw new KeyboardClientError("Malformed Find Information response");
}
for (let i = 1; i < r.data.length; i += width) {
const handle = u16(r.data, i);
if (handle < start || attrs.length >= MAX_ATTRIBUTES) throw new KeyboardClientError("Invalid or excessive GATT handles");
attrs.push({ handle, uuid: attributeUuid(r.data.subarray(i + 2, i + width)) });
start = handle + 1;
}
}
return attrs;
}
/**
* The properties byte of a characteristic, from its declaration, or
* undefined when the keyboard won't say. A GATT client normally learns
* declarations through Read By Type, which the bridge doesn't relay, so
* this reads the declaration attribute itself. Most stacks answer that
* with the full declaration. Some keep only the properties byte there and
* build the rest for Read By Type alone (older Telink SDKs, common in
* small keyboards), and some refuse the read. None of that is a reason to
* turn a keyboard away: the report's handle and UUID are already known.
*/
private async declaredProperties(attrs: readonly Attribute[], a: Attribute): Promise<number | undefined> {
const decl = attrs.filter((d) => d.handle < a.handle && d.uuid === UUID_CHARACTERISTIC).pop();
if (!decl) return undefined;
let declaration: Uint8Array;
try {
declaration = await this.shortRead(decl.handle);
} catch (e) {
const status = attRefusal(e);
this.events.log?.(`declaration ${decl.handle} of ${a.handle}: read refused (ATT ${status})`);
return undefined;
}
if (declaration.length === 5 || declaration.length === 19) {
if (u16(declaration, 1) === a.handle && attributeUuid(declaration.subarray(3)) === a.uuid) return declaration[0]!;
this.events.log?.(`declaration ${decl.handle} of ${a.handle}: describes something else (${hex(declaration)})`);
return undefined;
}
if (declaration.length === 1) return declaration[0]!;
this.events.log?.(`declaration ${decl.handle} of ${a.handle}: unexpected value (${hex(declaration)})`);
return undefined;
}
/**
* Discover every HID service, its Report Map and each report's Report
* Reference; select Report Protocol and subscribe the input reports that
* carry keys or a pointer's buttons and movement. Needs an encrypted link.
* No handle is assumed.
*
* Only what input needs is required of the device: a Report Map that
* parses, and at least one such input report with a Report Reference and
* a CCC. A report that can't be placed is skipped and logged, as is
* anything wrong with an external report (the battery level, usually).
*/
async configure(): Promise<KeyboardConfiguration> {
const epoch = this.epoch;
const log = (line: string): void => this.events.log?.(line);
// A new discovery replaces any earlier mapping, whether or not it succeeds.
this.release();
this.inputs.clear();
this.configuration = undefined;
const attrs = await this.attributes();
log(`gatt ${attrs.map((a) => `${a.handle}=${typeof a.uuid === "number" ? a.uuid.toString(16) : a.uuid}`).join(" ")}`);
const reports: KeyboardReport[] = [];
const watched: KeyboardReport[] = [];
let pointerFound = false;
const skipped: string[] = [];
const descriptorsFor = (handle: number): Attribute[] => {
const end = attrs.find((a) => a.handle > handle &&
(a.uuid === UUID_PRIMARY_SERVICE || a.uuid === UUID_SECONDARY_SERVICE || a.uuid === UUID_CHARACTERISTIC))?.handle ?? 65536;
return attrs.filter((a) => a.handle > handle && a.handle < end);
};
/** Why a report was left out; the link going away is not a reason to carry on. */
const skip = (a: Attribute, reason: string): void => {
skipped.push(`${a.handle}: ${reason}`);
log(`report ${a.handle} skipped: ${reason}`);
};
const addReport = async (service: number, a: Attribute, map: Uint8Array, parsed: Map<number, HidReport>) => {
if (reports.some((r) => r.handle === a.handle)) return skip(a, "already listed by another HID service");
const descriptors = descriptorsFor(a.handle);
const ref = descriptors.find((d) => d.uuid === UUID_REPORT_REFERENCE);
if (!ref) return skip(a, "no Report Reference descriptor");
let reference: Uint8Array;
try {
reference = await this.shortRead(ref.handle);
} catch (e) {
return skip(a, `Report Reference ${ref.handle} unreadable (ATT ${attRefusal(e)})`);
}
if (reference.length !== 2) return skip(a, `Report Reference ${ref.handle} is ${hex(reference) || "empty"}`);
const id = reference[0]!;
const type = reference[1]!;
if (type < 1 || type > 3) return skip(a, `Report Reference type ${type}`);
const ccc = descriptors.find((d) => d.uuid === UUID_CCC)?.handle;
const properties = await this.declaredProperties(attrs, a);
const report: KeyboardReport = { service, handle: a.handle, id, type, ccc, properties, map };
if (type === 1) {
const layout = parsed.get(id);
if (!layout) return skip(a, `input report id ${id} is not in the Report Map`);
if (ccc === undefined || (properties !== undefined && !(properties & 0x10))) return skip(a, "input report can't notify");
const keys = reportHasKeys(layout);
const pointer = reportHasPointer(layout);
if (keys || pointer) watched.push(report);
else log(`report ${a.handle} (id ${id}) carries no keys or pointer; not subscribed`);
if (pointer) {
pointerFound = true;
log(`report ${a.handle} (id ${id}) is a pointer's${keys ? ", with keys" : ""}`);
}
}
reports.push(report);
log(`report ${a.handle}: id ${id} type ${type} ccc ${ccc ?? "-"} properties ${properties ?? "?"}`);
};
for (let i = 0; i < attrs.length; i++) {
const service = attrs[i]!;
if (service.uuid !== UUID_PRIMARY_SERVICE) continue;
if (attributeUuid(await this.shortRead(service.handle)) !== UUID_HID_SERVICE) continue;
const end = attrs.slice(i + 1).find((a) => a.uuid === UUID_PRIMARY_SERVICE || a.uuid === UUID_SECONDARY_SERVICE)?.handle ?? 65536;
const members = attrs.filter((a) => a.handle > service.handle && a.handle < end);
const mapAttr = members.find((a) => a.uuid === UUID_REPORT_MAP);
if (!mapAttr) throw new KeyboardClientError(`HID service ${service.handle} has no Report Map`);
const map = await this.read(mapAttr.handle);
log(`report map ${mapAttr.handle}: ${hex(map)}`);
const parsed = parseReportMap(map);
for (const mode of members.filter((a) => a.uuid === UUID_PROTOCOL_MODE)) {
try {
await this.write(mode.handle, Uint8Array.of(1), true);
} catch (e) {
// Report Protocol is the default after a connection; a keyboard
// that won't take the write is almost certainly in it already.
log(`protocol mode ${mode.handle}: write refused (ATT ${attRefusal(e)})`);
}
}
for (const a of members.filter((m) => m.uuid === UUID_REPORT)) await addReport(service.handle, a, map, parsed);
// HOGP lets a Report Map refer to reports in other services (commonly Battery).
for (const ext of descriptorsFor(mapAttr.handle).filter((a) => a.uuid === UUID_EXTERNAL_REPORT_REFERENCE)) {
let value: Uint8Array;
try {
value = await this.shortRead(ext.handle);
} catch (e) {
log(`external report reference ${ext.handle} skipped: unreadable (ATT ${attRefusal(e)})`);
continue;
}
if (value.length !== 2 && value.length !== 16) {
log(`external report reference ${ext.handle} skipped: value ${hex(value) || "empty"}`);
continue;
}
const uuid = attributeUuid(value);
for (const a of attrs.filter((m) => m.uuid === uuid && (m.handle < service.handle || m.handle >= end))) {
await addReport(service.handle, a, map, parsed);
}
}
}
if (this.closed || this.epoch !== epoch) throw new KeyboardClientError("The keyboard link changed during setup");
if (!watched.length) {
throw new KeyboardClientError(reports.length || skipped.length
? `No usable keyboard input report (${skipped.join("; ") || "none carries keys or a pointer"})`
: "This device has no HID service");
}
if (watched.length > MAX_WATCHES) throw new KeyboardClientError("Keyboard has more than 16 input reports");
const changed = attrs.find((a) => a.uuid === UUID_SERVICE_CHANGED);
const changedCcc = changed && descriptorsFor(changed.handle).find((a) => a.uuid === UUID_CCC);
const watches = watched.map((r) => r.handle);
if (changed && changedCcc && watches.length < MAX_WATCHES) watches.push(changed.handle);
const serviceChanged = changed && changedCcc && watches.includes(changed.handle) ? changed.handle : undefined;
const config: KeyboardConfiguration = {
epoch: this.epoch, reports, watched: watched.map((r) => r.handle), pointer: pointerFound, serviceChanged,
};
this.restore(config);
try {
await this.command(KeyboardOp.watch, [watches.length, ...watches.flatMap(le16)]);
if (serviceChanged !== undefined) {
try {
await this.write(changedCcc!.handle, Uint8Array.of(2, 0));
} catch (e) {
log(`service changed ${changedCcc!.handle}: subscription refused (ATT ${attRefusal(e)})`);
}
}
for (const r of watched) await this.write(r.ccc!, Uint8Array.of(1, 0));
} catch (e) {
this.inputs.clear();
this.configuration = undefined;
this.release();
throw e;
}
return config;
}
/**
* Install a mapping found by configure(). It is only good while the glasses
* keep the same connection (the same epoch): after a keyboard reconnect or
* a firmware boot, discover again.
*/
restore(config: KeyboardConfiguration): void {
if (!config.reports.length || config.reports.length > 64) throw new KeyboardClientError("Invalid keyboard configuration");
const inputs = new Map<number, Input>();
for (const r of config.reports) {
if (r.type !== 1 || !config.watched.includes(r.handle)) continue;
if (!Number.isInteger(r.handle) || r.handle < 1 || r.handle > 65535 || inputs.has(r.handle)) {
throw new KeyboardClientError("Invalid restored Report handle");
}
if (!r.map.length || r.map.length > MAX_VALUE) throw new KeyboardClientError("Invalid restored Report Map");
const report = parseReportMap(r.map).get(r.id);
if (!report) throw new KeyboardClientError("Restored Report ID missing");
inputs.set(r.handle, { report, id: r.id, service: r.service, keys: reportHasKeys(report), pointer: reportHasPointer(report) });
}
if (!inputs.size || inputs.size > MAX_WATCHES) throw new KeyboardClientError("Invalid restored input count");
this.release();
this.inputs = inputs;
this.configuration = config;
}
/** Output reports of the configured keyboard (the LED report is the usual one). */
outputReports(): KeyboardReport[] {
return this.configuration?.reports.filter((r) => r.type === 2) ?? [];
}
}
/**
* The ATT error code when `error` is the keyboard refusing a read or write.
* Anything else (the link dropping, a timeout, a stream gap) is rethrown:
* those end the setup, where a refusal only costs the one attribute.
*/
function attRefusal(error: unknown): number {
if (error instanceof KeyboardClientError && error.attStatus !== undefined) return error.attStatus;
throw error;
}
function hex(bytes: Uint8Array): string {
return Array.from(bytes, (b) => b.toString(16).padStart(2, "0")).join("");
}
function opName(op: number): string {
for (const [name, value] of Object.entries(KeyboardOp)) if (value === op) return name;
return String(op);
}
function messageOf(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
-268
View File
@@ -1,268 +0,0 @@
/**
* Bounded HID report-descriptor decoder for the keyboard bridge: turns a
* keyboard's Report Map into the layout of each input report, and input
* reports into the set of keys held. From the reference client in g2flash
* demos/ (keyboard-hid.ts).
*
* HID over GATT carries the Report ID out of band (in each characteristic's
* Report Reference descriptor), so byte zero of a report value is data: it is
* never stripped as USB HID would. The keyboard/keypad page (7) and the
* consumer page (12) produce keys. A mouse's or touchpad's report (buttons,
* relative X/Y, wheel) is read by reportPointer.
* No platform imports: this runs under node in the tests.
*/
export type HidUsage = { page: number; usage: number };
export type HidKeyEvent = HidUsage & { down: boolean };
type Global = { page: number; min: number; max: number; size: number; count: number; id: number };
type Field = {
bit: number;
size: number;
count: number;
variable: boolean;
/** Reports a change since the last report (a mouse's axes and wheel), not a state. */
relative: boolean;
min: number;
max: number;
usages: HidUsage[];
range?: [HidUsage, HidUsage];
};
export type HidReport = { id: number; bits: number; fields: Field[] };
const HID_PAGE_KEYBOARD = 7;
const HID_PAGE_CONSUMER = 12;
const REPORT_MAP_MAX = 512;
function fail(reason: string): never {
throw new Error(`Unsupported or malformed HID descriptor: ${reason}`);
}
const usageKey = (u: HidUsage): string => `${u.page}:${u.usage}`;
/** Input reports by Report ID (0 when the descriptor uses none). Throws on anything out of bounds. */
export function parseReportMap(bytes: Uint8Array): Map<number, HidReport> {
if (!bytes.length || bytes.length > REPORT_MAP_MAX) fail("report map must contain 1..512 bytes");
const reports = new Map<number, HidReport>();
let g: Global = { page: 0, min: 0, max: 0, size: 0, count: 0, id: 0 };
const stack: Global[] = [];
let usages: HidUsage[] = [];
let lo: HidUsage | undefined;
let hi: HidUsage | undefined;
let depth = 0;
const usage = (v: number, n: number): HidUsage =>
n === 4 ? { page: v >>> 16, usage: v & 65535 } : { page: g.page, usage: v };
for (let p = 0; p < bytes.length;) {
const prefix = bytes[p++]!;
if (prefix === 0xfe) fail("long items");
const n = [0, 1, 2, 4][prefix & 3]!;
if (p + n > bytes.length) fail("truncated item");
let v = 0;
for (let j = 0; j < n; j++) v += bytes[p++]! * 2 ** (8 * j);
const signed = n && v >= 2 ** (n * 8 - 1) ? v - 2 ** (n * 8) : v;
const type = (prefix >> 2) & 3;
const tag = prefix >> 4;
if (type === 1) {
switch (tag) {
case 0: g.page = v; break;
case 1: g.min = signed; break;
case 2: g.max = g.min < 0 ? signed : v; break;
case 7: g.size = v; break;
case 8: if (!v || v > 255) fail("Report ID"); g.id = v; break;
case 9: g.count = v; break;
case 10: if (stack.length >= 8) fail("global stack overflow"); stack.push({ ...g }); break;
case 11: if (!stack.length) fail("global stack underflow"); g = stack.pop()!; break;
case 3: case 4: case 5: case 6: break; // physical range, unit and exponent
default: fail("unknown global item");
}
} else if (type === 2) {
if (tag === 0) { if (usages.length >= 512) fail("too many usages"); usages.push(usage(v, n)); }
else if (tag === 1) lo = usage(v, n);
else if (tag === 2) hi = usage(v, n);
else if (tag === 10) fail("usage delimiters");
else if (tag > 9) fail("unknown local item");
} else if (type === 0) {
if (tag === 10) { if (++depth > 32) fail("collection depth"); }
else if (tag === 12) { if (--depth < 0) fail("collection underflow"); }
else if (tag === 8) {
if (!g.size || g.size > 32 || !g.count || g.count > 512) fail("field dimensions");
let r = reports.get(g.id);
if (!r) {
if (reports.size >= 16) fail("too many input reports");
r = { id: g.id, bits: 0, fields: [] };
reports.set(g.id, r);
}
const field: Field = {
bit: r.bits, size: g.size, count: g.count, variable: !!(v & 2), relative: !!(v & 4),
min: g.min, max: g.max, usages: [...usages],
};
if (lo || hi) {
if (!lo || !hi || lo.page !== hi.page || hi.usage < lo.usage) fail("usage range");
field.range = [lo!, hi!];
}
r.bits += g.size * g.count;
if (r.bits > 4096) fail("report exceeds 512 bytes");
// Constant fields are padding.
if (!(v & 1)) r.fields.push(field);
} else if (tag !== 9 && tag !== 11) fail("unknown main item");
usages = [];
lo = hi = undefined;
} else fail("reserved item type");
}
if (depth || stack.length) fail("unbalanced collection/global stack");
if (reports.has(0) && reports.size > 1) fail("mixed numbered and unnumbered input reports");
if (!reports.size) fail("no input reports");
return reports;
}
function usageAt(f: Field, index: number): HidUsage | undefined {
if (index < 0) return undefined;
if (index < f.usages.length) return f.usages[index];
const rangeIndex = index - f.usages.length;
if (f.range && rangeIndex <= f.range[1].usage - f.range[0].usage) {
return { page: f.range[0].page, usage: f.range[0].usage + rangeIndex };
}
// Variable items reuse their last usage when Report Count exceeds the usages.
if (f.variable) return f.range?.[1] ?? f.usages[f.usages.length - 1];
return undefined;
}
/**
* Whether an input report can ever say a key is held. False for a pointer's
* report (a trackpad or mouse on a combined device), whose movement would
* otherwise be relayed, and thrown away, a hundred times a second.
*/
export function reportHasKeys(report: HidReport): boolean {
const isKeyPage = (u: HidUsage): boolean => u.page === HID_PAGE_KEYBOARD || u.page === HID_PAGE_CONSUMER;
return report.fields.some((f) => !f.relative && (f.usages.some(isKeyPage) || (f.range !== undefined && isKeyPage(f.range[0]))));
}
/** One field's `index`th value, sign-extended when the field is signed. Bits past the data read as zero. */
function fieldValue(f: Field, index: number, data: Uint8Array): number {
let value = 0;
for (let j = 0; j < f.size; j++) {
const bit = f.bit + index * f.size + j;
if ((data[bit >>> 3] ?? 0) & (1 << (bit & 7))) value += 2 ** j;
}
if (f.min < 0 && value >= 2 ** (f.size - 1)) value -= 2 ** f.size;
return value;
}
/** What one report from a mouse or touchpad says. */
export type HidPointerSample = {
/** Buttons held, bit 0 for button 1 (left), bit 1 for button 2 (right), bit 2 for button 3 (middle). */
buttons: number;
/** Movement since the last report, in the device's counts; +x right, +y down. */
dx: number;
dy: number;
/** Wheel detents since the last report; positive is away from the user ("up"). */
wheel: number;
/** Horizontal wheel (AC Pan); positive is right. */
pan: number;
};
const HID_PAGE_GENERIC_DESKTOP = 1;
const HID_PAGE_BUTTON = 9;
const USAGE_X = 0x30;
const USAGE_Y = 0x31;
const USAGE_WHEEL = 0x38;
const USAGE_AC_PAN = 0x238;
/** Buttons past this are not tracked; a mouse with more is rare and has nothing to bind them to anyway. */
const MAX_POINTER_BUTTONS = 8;
/** Which part of a pointer sample a field's usage feeds, if any. */
function pointerRole(f: Field, u: HidUsage | undefined): "button" | "dx" | "dy" | "wheel" | "pan" | null {
if (!u || !f.variable) return null;
if (u.page === HID_PAGE_BUTTON) return !f.relative && u.usage >= 1 && u.usage <= MAX_POINTER_BUTTONS ? "button" : null;
if (!f.relative) return null; // an absolute X/Y is a touchscreen or a digitizer, not a mouse
if (u.page === HID_PAGE_GENERIC_DESKTOP) {
if (u.usage === USAGE_X) return "dx";
if (u.usage === USAGE_Y) return "dy";
if (u.usage === USAGE_WHEEL) return "wheel";
}
if (u.page === HID_PAGE_CONSUMER && u.usage === USAGE_AC_PAN) return "pan";
return null;
}
/** Whether an input report carries a mouse's buttons, movement or wheel. */
export function reportHasPointer(report: HidReport): boolean {
return report.fields.some((f) => {
for (let i = 0; i < f.count; i++) if (pointerRole(f, usageAt(f, i))) return true;
return false;
});
}
/**
* Read a mouse or touchpad report. Lenient about length, unlike reportKeys:
* some devices leave the wheel bytes off when there is nothing to say, and a
* missing byte is zero either way.
*/
export function reportPointer(report: HidReport, data: Uint8Array): HidPointerSample {
const sample: HidPointerSample = { buttons: 0, dx: 0, dy: 0, wheel: 0, pan: 0 };
for (const f of report.fields) {
for (let i = 0; i < f.count; i++) {
const u = usageAt(f, i);
const role = pointerRole(f, u);
if (!role) continue;
const value = fieldValue(f, i, data);
if (value < f.min || value > f.max) continue; // out of range: no reading
if (role === "button") {
if (value) sample.buttons |= 1 << (u!.usage - 1);
} else {
sample[role] += value;
}
}
}
return sample;
}
/** The keys one input report says are held. Throws on a length mismatch or a rollover report. */
export function reportKeys(report: HidReport, data: Uint8Array): HidUsage[] {
if (data.length !== Math.ceil(report.bits / 8)) throw new Error("HID input length does not match Report Map");
const keys = new Map<string, HidUsage>();
for (const f of report.fields) {
// A relative field is a movement, not a key that is down.
if (f.relative) continue;
for (let i = 0; i < f.count; i++) {
const value = fieldValue(f, i, data);
if (value < f.min || value > f.max) continue; // null state
const u = usageAt(f, f.variable ? i : value - f.min);
if (!u || (f.variable && value === 0) || !u.usage) continue;
if (u.page === HID_PAGE_KEYBOARD && u.usage <= 3) throw new Error("HID keyboard rollover/error report");
if (u.page === HID_PAGE_KEYBOARD || u.page === HID_PAGE_CONSUMER) keys.set(usageKey(u), u);
}
}
return [...keys.values()];
}
/**
* Held keys across a keyboard's input reports. Ownership is per report
* characteristic, then unioned: releasing a key in one report cannot release
* a modifier another report still holds.
*/
export class HidKeys {
private reports = new Map<number, HidUsage[]>();
private held = new Map<string, HidUsage>();
/** Apply one report from characteristic `handle`; returns the transitions it caused. */
update(handle: number, report: HidReport, data: Uint8Array): HidKeyEvent[] {
this.reports.set(handle, reportKeys(report, data));
return this.diff();
}
/** Release everything (a stream gap, a dropped link). */
reset(): HidKeyEvent[] {
this.reports.clear();
return this.diff();
}
private diff(): HidKeyEvent[] {
const next = new Map<string, HidUsage>();
for (const values of this.reports.values()) for (const u of values) next.set(usageKey(u), u);
const events: HidKeyEvent[] = [];
for (const [k, u] of this.held) if (!next.has(k)) events.push({ ...u, down: false });
for (const [k, u] of next) if (!this.held.has(k)) events.push({ ...u, down: true });
this.held = next;
return events;
}
}
-146
View File
@@ -1,146 +0,0 @@
/**
* Where a key from the hardware keyboard goes: to a keybinding being set up,
* to a gesture it is bound to, to the window with input focus, or to the
* built-in navigation. The meanings are in bindings.ts; this is the order of
* asking, and the bookkeeping around it. Shared by the Android and iOS
* controllers through KeyRouterHost.
* No platform imports: this runs under node in the tests.
*/
import {
GestureScheme,
chordOf,
isModifierCode,
resolveKey,
type GestureOrigin,
type KeyAction,
type KeyChord,
type KeyGesture,
} from "./bindings";
import type { KeyEvent } from "./keys";
export type KeyRouterHost = {
/** Connected to glasses that are in use. */
ready(): boolean;
locked(): boolean;
screenOn(): boolean;
/** Turn a dark display on, and nothing else. */
wake(): void;
/** Offer a key to the window with input focus; true when that window takes keys. */
offerToWindow(event: KeyEvent): boolean;
/** Perform a gesture as if it came from the ring or the watch. */
gesture(gesture: KeyGesture, origin: GestureOrigin): Promise<void>;
/** The custom keybindings, by chord id. */
bindings(): ReadonlyMap<string, KeyAction>;
/** A key was used for something other than a gesture or typing (count it against the screen timeout). */
noteActivity(): void;
log(line: string): void;
now?(): number;
};
/**
* A held key repeats twenty times a second, and a wheel or a two-finger
* scroll reports faster still: more than a list can repaint over the
* glasses' link. Those are dropped while a gesture is still being handled,
* and closer together than this.
*/
const REPEAT_MIN_INTERVAL_MS = 100;
export class KeyRouter {
private capture: ((chord: KeyChord) => void) | null = null;
/** Keys whose press was spoken for, so their release is too. */
private readonly consumed = new Set<string>();
private readonly scheme = new GestureScheme();
private gesturesInFlight = 0;
private lastGestureAtMs = 0;
constructor(private readonly host: KeyRouterHost) {}
/** One key transition from the keyboard. */
handleKey(event: KeyEvent): void {
const host = this.host;
if (!host.ready()) return;
if (host.locked() || !host.screenOn()) {
// A dark display takes no typing: the first press only wakes it, as
// the ring's double tap would, and a locked one stays locked.
if (event.phase === "down" && !event.repeat && !host.screenOn()) host.wake();
return;
}
if (this.capture) {
this.captureKey(event);
return;
}
if (event.phase === "up") {
if (!this.consumed.delete(event.code)) host.offerToWindow(event);
return;
}
const resolution = resolveKey(event, host.bindings());
const droppable = event.repeat || event.burst === true;
if (resolution.consume) {
this.consumed.add(event.code);
if (resolution.gesture) this.perform(resolution.gesture, droppable);
return;
}
if (host.offerToWindow(event)) return;
if (resolution.gesture) this.perform(resolution.gesture, droppable);
}
/**
* A key the focused window was offered and had no use for (a terminal
* showing its session list rather than a session): it gets the built-in
* navigation meaning after all.
*/
handleUnhandledKey(event: KeyEvent): void {
const host = this.host;
if (event.phase !== "down" || this.capture || !host.ready() || host.locked() || !host.screenOn()) return;
const resolution = resolveKey(event, host.bindings());
if (!resolution.consume && resolution.gesture) this.perform(resolution.gesture, event.repeat || event.burst === true);
}
/**
* Take the next key press instead of acting on it (setting up a
* keybinding). Modifiers held with it are part of the chord; pressed alone
* they are waited through. Returns a function that calls the capture off.
*/
captureNextKey(onChord: (chord: KeyChord) => void): () => void {
this.capture = onChord;
return () => {
if (this.capture === onChord) this.capture = null;
};
}
/** The keyboard went away: nothing is held any more. */
reset(): void {
this.consumed.clear();
}
private captureKey(event: KeyEvent): void {
this.host.noteActivity();
if (event.phase === "up") {
this.consumed.delete(event.code);
return;
}
if (event.repeat || isModifierCode(event.code)) return;
const capture = this.capture!;
this.capture = null;
this.consumed.add(event.code);
capture(chordOf(event));
}
private perform(gesture: KeyGesture, droppable: boolean): void {
const now = this.host.now ? this.host.now() : Date.now();
if (droppable && (this.gesturesInFlight > 0 || now - this.lastGestureAtMs < REPEAT_MIN_INTERVAL_MS)) return;
this.lastGestureAtMs = now;
void this.run(gesture, this.scheme.originFor(gesture));
}
private async run(gesture: KeyGesture, origin: GestureOrigin): Promise<void> {
this.gesturesInFlight++;
try {
await this.host.gesture(gesture, origin);
} catch (error) {
this.host.log(`keyboard gesture ${gesture} failed: ${error instanceof Error ? error.message : String(error)}`);
} finally {
this.gesturesInFlight--;
}
}
}
-306
View File
@@ -1,306 +0,0 @@
/**
* Key events as the rest of Faceclaw sees them, and the translation from the
* keyboard bridge's raw HID transitions: layout (US only for now), modifier
* and Caps Lock state, and typematic repeat, which a BLE keyboard never sends
* (its report just stays asserted). The translation tables follow the Glassly
* app's (GlassesKeyboardInput.ts, MIT).
* No platform imports: this runs under node in the tests, and in app workers.
*/
import type { HidKeyEvent } from "./hid";
export type KeyModifiers = { shift: boolean; ctrl: boolean; alt: boolean; meta: boolean };
/**
* One key transition, shaped like a DOM KeyboardEvent: `key` is the logical
* key with the layout applied ("a", "A", "Enter", "ArrowUp"), `code` the
* physical key ("KeyA"), `text` what a press types (absent for releases, for
* keys that type nothing, and for Ctrl/Alt/Meta chords, which are commands).
*/
export type KeyEvent = {
phase: "down" | "up";
key: string;
code: string;
text?: string;
modifiers: KeyModifiers;
/** A synthesized auto-repeat of a held key. */
repeat: boolean;
/**
* One of a stream that can arrive faster than it can be acted on (a wheel
* turning): whoever acts on it may drop some rather than fall behind.
*/
burst?: boolean;
/** Date.now() when the phone saw the transition. */
timestampMs: number;
};
const HID_PAGE_KEYBOARD = 7;
const HID_PAGE_CONSUMER = 12;
/** A mouse's buttons, which arrive here as keys: usage 1 is the left button, 2 the right, 3 the middle. */
export const HID_PAGE_BUTTON = 9;
const USAGE_CAPS_LOCK = 57;
/** [unshifted, shifted, code] for US-layout punctuation on the keyboard page. */
const PUNCTUATION: Record<number, [string, string, string]> = {
45: ["-", "_", "Minus"],
46: ["=", "+", "Equal"],
47: ["[", "{", "BracketLeft"],
48: ["]", "}", "BracketRight"],
49: ["\\", "|", "Backslash"],
50: ["#", "~", "IntlHash"],
51: [";", ":", "Semicolon"],
52: ["'", '"', "Quote"],
53: ["`", "~", "Backquote"],
54: [",", "<", "Comma"],
55: [".", ">", "Period"],
56: ["/", "?", "Slash"],
100: ["\\", "|", "IntlBackslash"],
};
const DIGITS = "1234567890";
const SHIFTED_DIGITS = "!@#$%^&*()";
/** [key, code] for keys that type nothing. */
const NAMED: Record<number, [string, string]> = {
40: ["Enter", "Enter"],
41: ["Escape", "Escape"],
42: ["Backspace", "Backspace"],
43: ["Tab", "Tab"],
57: ["CapsLock", "CapsLock"],
70: ["PrintScreen", "PrintScreen"],
71: ["ScrollLock", "ScrollLock"],
72: ["Pause", "Pause"],
73: ["Insert", "Insert"],
74: ["Home", "Home"],
75: ["PageUp", "PageUp"],
76: ["Delete", "Delete"],
77: ["End", "End"],
78: ["PageDown", "PageDown"],
79: ["ArrowRight", "ArrowRight"],
80: ["ArrowLeft", "ArrowLeft"],
81: ["ArrowDown", "ArrowDown"],
82: ["ArrowUp", "ArrowUp"],
83: ["NumLock", "NumLock"],
88: ["Enter", "NumpadEnter"],
101: ["ContextMenu", "ContextMenu"],
102: ["Power", "Power"],
224: ["Control", "ControlLeft"],
225: ["Shift", "ShiftLeft"],
226: ["Alt", "AltLeft"],
227: ["Meta", "MetaLeft"],
228: ["Control", "ControlRight"],
229: ["Shift", "ShiftRight"],
230: ["Alt", "AltRight"],
231: ["Meta", "MetaRight"],
};
/** [key and text, code] for the numeric keypad's operators. */
const NUMPAD: Record<number, [string, string]> = {
84: ["/", "NumpadDivide"],
85: ["*", "NumpadMultiply"],
86: ["-", "NumpadSubtract"],
87: ["+", "NumpadAdd"],
99: [".", "NumpadDecimal"],
103: ["=", "NumpadEqual"],
};
const CONSUMER: Record<number, string> = {
0xe9: "AudioVolumeUp",
0xea: "AudioVolumeDown",
0xe2: "AudioVolumeMute",
0xcd: "MediaPlayPause",
0xb0: "MediaPlay",
0xb1: "MediaPause",
0xb5: "MediaTrackNext",
0xb6: "MediaTrackPrevious",
0xb7: "MediaStop",
0x6f: "BrightnessUp",
0x70: "BrightnessDown",
0x221: "BrowserSearch",
0x223: "BrowserHome",
0x224: "BrowserBack",
0x225: "BrowserForward",
0x18a: "LaunchMail",
0x192: "LaunchApplication2",
0x194: "LaunchApplication1",
};
const MODIFIER_USAGES: Record<number, keyof KeyModifiers> = {
224: "ctrl", 228: "ctrl", 225: "shift", 229: "shift", 226: "alt", 230: "alt", 227: "meta", 231: "meta",
};
type Translated = { key: string; code: string; text?: string };
/** One HID usage under the US layout. Caps Lock only affects letters. */
export function translateUsage(page: number, usage: number, shift: boolean, capsLock: boolean): Translated {
if (page === HID_PAGE_KEYBOARD) {
if (usage >= 4 && usage <= 29) {
const lower = String.fromCharCode(97 + usage - 4);
const key = shift !== capsLock ? lower.toUpperCase() : lower;
return { key, code: `Key${lower.toUpperCase()}`, text: key };
}
if (usage >= 30 && usage <= 39) {
const index = usage - 30;
const key = shift ? SHIFTED_DIGITS[index]! : DIGITS[index]!;
return { key, code: `Digit${DIGITS[index]}`, text: key };
}
if (usage === 44) return { key: " ", code: "Space", text: " " };
const punctuation = PUNCTUATION[usage];
if (punctuation) {
const key = shift ? punctuation[1] : punctuation[0];
return { key, code: punctuation[2], text: key };
}
if (usage >= 58 && usage <= 69) return { key: `F${usage - 57}`, code: `F${usage - 57}` };
if (usage >= 104 && usage <= 115) return { key: `F${usage - 91}`, code: `F${usage - 91}` };
if (usage >= 89 && usage <= 97) {
const digit = String(usage - 88);
return { key: digit, code: `Numpad${digit}`, text: digit };
}
if (usage === 98) return { key: "0", code: "Numpad0", text: "0" };
const numpad = NUMPAD[usage];
if (numpad) return { key: numpad[0], code: numpad[1], text: numpad[0] };
const named = NAMED[usage];
if (named) return { key: named[0], code: named[1] };
return { key: "Unidentified", code: `Usage${usage.toString(16).padStart(2, "0")}` };
}
if (page === HID_PAGE_CONSUMER) {
const name = CONSUMER[usage];
return name ? { key: name, code: name } : { key: "Unidentified", code: `Consumer${usage.toString(16)}` };
}
if (page === HID_PAGE_BUTTON) return { key: `MouseButton${usage}`, code: `MouseButton${usage}` };
return { key: "Unidentified", code: `Page${page.toString(16)}Usage${usage.toString(16)}` };
}
export type KeyTranslatorOptions = {
emit: (event: KeyEvent) => void;
now?: () => number;
setTimer?: (callback: () => void, delayMs: number) => unknown;
clearTimer?: (handle: unknown) => void;
/** Hold time before the first repeat, and the gap between repeats. */
repeatDelayMs?: number;
repeatIntervalMs?: number;
};
const REPEAT_DELAY_MS = 400;
const REPEAT_INTERVAL_MS = 50;
/**
* Raw HID transitions in, KeyEvents out. Modifier state comes from the same
* stream, and the bridge reports a release for every key when the stream may
* have lost something, so a missed key-up cannot leave a modifier stuck or a
* key repeating forever.
*/
export class KeyTranslator {
private readonly emitEvent: (event: KeyEvent) => void;
private readonly now: () => number;
private readonly setTimer: (callback: () => void, delayMs: number) => unknown;
private readonly clearTimer: (handle: unknown) => void;
private readonly repeatDelayMs: number;
private readonly repeatIntervalMs: number;
private readonly held = new Set<string>();
private modifiers: KeyModifiers = { shift: false, ctrl: false, alt: false, meta: false };
private capsLock = false;
private repeatTimer: unknown = null;
private repeating: { page: number; usage: number } | null = null;
constructor(options: KeyTranslatorOptions) {
this.emitEvent = options.emit;
this.now = options.now ?? (() => Date.now());
this.setTimer = options.setTimer ?? ((callback, delayMs) => setTimeout(callback, delayMs));
this.clearTimer = options.clearTimer ?? ((handle) => clearTimeout(handle as ReturnType<typeof setTimeout>));
this.repeatDelayMs = options.repeatDelayMs ?? REPEAT_DELAY_MS;
this.repeatIntervalMs = options.repeatIntervalMs ?? REPEAT_INTERVAL_MS;
}
/** Apply the transitions one input report produced, in order. */
handle(events: readonly HidKeyEvent[]): void {
for (const event of events) this.handleOne(event);
}
private handleOne(event: HidKeyEvent): void {
const id = `${event.page}:${event.usage}`;
const modifier = event.page === HID_PAGE_KEYBOARD ? MODIFIER_USAGES[event.usage] : undefined;
if (event.down) {
if (this.held.has(id)) return;
this.held.add(id);
if (modifier) this.modifiers[modifier] = true;
if (event.page === HID_PAGE_KEYBOARD && event.usage === USAGE_CAPS_LOCK) this.capsLock = !this.capsLock;
this.emit("down", event.page, event.usage, false);
// The newest non-modifier key is the one that repeats, as on a PC. A
// mouse button neither repeats nor takes the repeat from a held key.
if (!modifier && event.page !== HID_PAGE_BUTTON) this.startRepeat(event.page, event.usage);
} else {
if (!this.held.delete(id)) return;
if (modifier) this.modifiers[modifier] = this.otherHeld(event.usage, modifier);
if (this.repeating?.page === event.page && this.repeating.usage === event.usage) this.stopRepeat();
this.emit("up", event.page, event.usage, false);
}
}
/**
* A press and release in one, for input that has no "held" (a wheel's
* detent): `name` is both the key and the code, e.g. "WheelUp". Carries
* the modifiers held on the keyboard, like any key.
*/
pulse(name: string): void {
const timestampMs = this.now();
for (const phase of ["down", "up"] as const) {
this.emitEvent({ phase, key: name, code: name, modifiers: { ...this.modifiers }, repeat: false, burst: true, timestampMs });
}
}
/** The keyboard went away: forget everything without emitting releases. */
reset(): void {
this.stopRepeat();
this.held.clear();
this.modifiers = { shift: false, ctrl: false, alt: false, meta: false };
this.capsLock = false;
}
/** Whether the same modifier's key on the other side is still down. */
private otherHeld(released: number, modifier: keyof KeyModifiers): boolean {
for (const [usage, name] of Object.entries(MODIFIER_USAGES)) {
if (name === modifier && Number(usage) !== released && this.held.has(`${HID_PAGE_KEYBOARD}:${usage}`)) return true;
}
return false;
}
private emit(phase: "down" | "up", page: number, usage: number, repeat: boolean): void {
const translated = translateUsage(page, usage, this.modifiers.shift, this.capsLock);
const event: KeyEvent = {
phase,
key: translated.key,
code: translated.code,
modifiers: { ...this.modifiers },
repeat,
timestampMs: this.now(),
};
if (phase === "down" && translated.text !== undefined &&
!this.modifiers.ctrl && !this.modifiers.alt && !this.modifiers.meta) {
event.text = translated.text;
}
this.emitEvent(event);
}
private startRepeat(page: number, usage: number): void {
this.stopRepeat();
this.repeating = { page, usage };
this.repeatTimer = this.setTimer(() => this.repeatTick(), this.repeatDelayMs);
}
private repeatTick(): void {
const key = this.repeating;
this.repeatTimer = null;
if (!key || !this.held.has(`${key.page}:${key.usage}`)) {
this.repeating = null;
return;
}
this.emit("down", key.page, key.usage, true);
this.repeatTimer = this.setTimer(() => this.repeatTick(), this.repeatIntervalMs);
}
private stopRepeat(): void {
if (this.repeatTimer !== null) this.clearTimer(this.repeatTimer);
this.repeatTimer = null;
this.repeating = null;
}
}
-83
View File
@@ -1,83 +0,0 @@
/**
* A keyboard's touchpad or mouse, as the rest of the keyboard support sees
* it. Its buttons and wheel become keys ("MouseButton1", "WheelUp"), so
* they get default meanings and custom keybindings like any key
* (bindings.ts). Its movement goes nowhere unless something has taken the
* pointer (the cursor test page): there is no cursor in the main UI.
* No platform imports: this runs under node in the tests.
*/
import type { HidKeyEvent, HidPointerSample } from "./hid";
import { HID_PAGE_BUTTON } from "./keys";
/** The names a wheel's detents arrive under, as keys. */
export type WheelKey = "WheelUp" | "WheelDown" | "WheelLeft" | "WheelRight";
export type PointerInputHost = {
/** Button transitions, as HID usages on the Button page. */
buttons(events: HidKeyEvent[]): void;
/** One wheel report in a direction, however many detents it claimed. */
wheel(key: WheelKey): void;
};
export class PointerInput {
/** The buttons the host has been told are down. */
private reported = 0;
/** The buttons actually down, as of the last sample. */
private held = 0;
/** Buttons that went down while the pointer was taken: not keys until they have come up. */
private ignored = 0;
private grab: ((sample: HidPointerSample) => void) | null = null;
constructor(private readonly host: PointerInputHost) {}
/** One report from the pointer (see KeyboardClientEvents.pointer). */
handle(sample: HidPointerSample): void {
this.held = sample.buttons;
if (this.grab) {
this.grab(sample);
return;
}
this.ignored &= sample.buttons;
this.report(sample.buttons & ~this.ignored);
// One step per report: a touchpad's two-finger scroll sends many small
// ones, and the gesture they turn into is rate-limited downstream.
if (sample.wheel) this.host.wheel(sample.wheel > 0 ? "WheelUp" : "WheelDown");
if (sample.pan) this.host.wheel(sample.pan > 0 ? "WheelRight" : "WheelLeft");
}
/**
* Take the pointer: every sample goes to `listener`, movement included,
* and its buttons and wheel stop acting as keys. Buttons held at that
* moment are released first, so nothing is left down. Returns a function
* that gives the pointer back.
*/
take(listener: (sample: HidPointerSample) => void): () => void {
this.report(0);
this.grab = listener;
return () => {
if (this.grab !== listener) return;
this.grab = null;
// A button still down was pressed for the listener, not as a key: its
// release is nobody's business either.
this.ignored = this.held;
};
}
/** The keyboard went away: nothing is held, and nobody is told. */
reset(): void {
this.reported = 0;
this.held = 0;
this.ignored = 0;
}
private report(buttons: number): void {
const changed = buttons ^ this.reported;
if (!changed) return;
const events: HidKeyEvent[] = [];
for (let bit = 0; bit < 8; bit++) {
if (changed & (1 << bit)) events.push({ page: HID_PAGE_BUTTON, usage: bit + 1, down: !!(buttons & (1 << bit)) });
}
this.reported = buttons;
this.host.buttons(events);
}
}
-297
View File
@@ -1,297 +0,0 @@
/**
* Wire format of the CFW keyboard bridge (Faceclaw/39). The master temple acts
* as a BLE central for one HID-over-GATT keyboard; the phone drives that link
* over private settings fields and gets the keyboard's raw input reports back.
* The firmware side is g2flash patches/keyboard_core.h, documented in its
* docs/keyboard.md. This follows the reference client in g2flash demos/
* (keyboard-protocol.ts), which came with the firmware from tetramo-labs.
*
* Phone -> glasses, settings field 130 (wrapped by the native session):
* ['K','B',1,op, request LE16, epoch LE16, body...] at most 80 bytes
* Glasses -> phone, settings field 131:
* ['K','B',1,kind, sequence LE32, epoch LE16, request LE16, code LE16,
* handle LE16, total LE16, offset LE16, timestamp LE32, fragment...]
*
* All integers are little endian. No platform imports: this runs under node
* in the tests.
*/
export const KEYBOARD_PROTOCOL_VERSION = 1;
/** Fixed part of a field-131 record, before its fragment. */
export const KEYBOARD_RECORD_HEADER = 24;
/** Most body bytes one record carries; longer bodies are fragmented. */
export const KEYBOARD_RECORD_CHUNK = 96;
/** Largest reassembled body (an ATT value or a HID report). */
export const KEYBOARD_BODY_MAX = 512;
/** Largest control body after the 8-byte prefix. */
export const KEYBOARD_CONTROL_BODY_MAX = 72;
export const KeyboardOp = {
query: 0,
enable: 1,
disable: 2,
scan: 3,
connect: 4,
disconnect: 5,
pair: 6,
auth: 7,
compare: 8,
find: 9,
read: 10,
write: 11,
watch: 12,
} as const;
export type KeyboardOpCode = (typeof KeyboardOp)[keyof typeof KeyboardOp];
export const KeyboardKind = {
status: 1,
att: 2,
input: 3,
dm: 4,
advert: 5,
} as const;
/** Control error codes, carried in a STATUS record's `code`. */
export const KeyboardError = {
ok: 0,
badRequest: 1,
disabled: 2,
busy: 3,
staleEpoch: 4,
noSpace: 5,
notConnected: 6,
notEncrypted: 7,
timeout: 8,
noMemory: 9,
} as const;
export function keyboardErrorName(code: number): string {
switch (code) {
case KeyboardError.ok: return "ok";
case KeyboardError.badRequest: return "bad request";
case KeyboardError.disabled: return "keyboard bridge disabled";
case KeyboardError.busy: return "busy";
case KeyboardError.staleEpoch: return "stale epoch";
case KeyboardError.noSpace: return "no connection or bond space";
case KeyboardError.notConnected: return "not connected";
case KeyboardError.notEncrypted: return "not encrypted";
case KeyboardError.timeout: return "timeout";
case KeyboardError.noMemory: return "allocation failure";
default: return `error ${code}`;
}
}
/** Cordio device-manager events the bridge relays (a DM record's `code & 0xff`). */
export const KeyboardDm = {
connOpen: 0x27,
connClose: 0x28,
pairComplete: 0x2a,
pairFailed: 0x2b,
encrypted: 0x2c,
encryptFailed: 0x2d,
authRequest: 0x2e,
compare: 0x35,
} as const;
/** ATT client events (an ATT record's `code & 0xff`; the status is `code >> 8`). */
export const KeyboardAtt = {
notification: 13,
indication: 14,
} as const;
export type KeyboardRecord = {
kind: number;
sequence: number;
epoch: number;
request: number;
code: number;
handle: number;
/** Glasses uptime in ms when the record was queued. */
timestamp: number;
data: Uint8Array;
};
export const u16 = (p: Uint8Array, off = 0): number => p[off]! | (p[off + 1]! << 8);
export const u32 = (p: Uint8Array, off = 0): number =>
(p[off]! | (p[off + 1]! << 8) | (p[off + 2]! << 16) | (p[off + 3]! << 24)) >>> 0;
export const le16 = (v: number): number[] => [v & 255, (v >>> 8) & 255];
/** A field-130 control record. The session wraps it in the settings package. */
export function keyboardControl(op: number, request: number, epoch: number, body: readonly number[] = []): Uint8Array {
if (!Number.isInteger(op) || op < 0 || op > KeyboardOp.watch || body.length > KEYBOARD_CONTROL_BODY_MAX) {
throw new Error("Keyboard control bounds");
}
return Uint8Array.from([75, 66, KEYBOARD_PROTOCOL_VERSION, op, ...le16(request), ...le16(epoch), ...body]);
}
/**
* Bounded, ordered reassembly of field-131 records. `onGap` fires whenever
* the stream may have lost something (a skipped sequence, an epoch change, a
* malformed or orphaned fragment): the caller then releases every held key
* and fails any setup transaction in flight. The firmware's once-a-second
* status record makes a lost final release visible even when no further keys
* arrive.
*/
export class KeyboardRecords {
private epoch: number | undefined;
private last: number | undefined;
private fragment: { record: KeyboardRecord; total: number; used: number } | undefined;
constructor(private readonly onGap: () => void) {}
/** Forget the stream position (the phone link dropped); counts as a gap. */
reset(): void {
this.epoch = undefined;
this.last = undefined;
this.fragment = undefined;
this.onGap();
}
/** Feed one field-131 payload; returns a complete record, or undefined. */
push(p: Uint8Array): KeyboardRecord | undefined {
const invalid = (): undefined => {
this.fragment = undefined;
this.onGap();
return undefined;
};
if (
p.length < KEYBOARD_RECORD_HEADER || p.length > KEYBOARD_RECORD_HEADER + KEYBOARD_RECORD_CHUNK ||
p[0] !== 75 || p[1] !== 66 || p[2] !== KEYBOARD_PROTOCOL_VERSION
) {
return invalid();
}
const r: KeyboardRecord = {
kind: p[3]!,
sequence: u32(p, 4),
epoch: u16(p, 8),
request: u16(p, 10),
code: u16(p, 12),
handle: u16(p, 14),
timestamp: u32(p, 20),
data: new Uint8Array(0),
};
const total = u16(p, 16);
const offset = u16(p, 18);
const n = p.length - KEYBOARD_RECORD_HEADER;
if (r.kind < KeyboardKind.status || r.kind > KeyboardKind.advert || total > KEYBOARD_BODY_MAX ||
offset + n > total || (total > 0 && n === 0)) {
return invalid();
}
if (this.epoch !== r.epoch) {
this.onGap();
this.fragment = undefined;
this.last = undefined;
this.epoch = r.epoch;
}
if (offset === 0) {
if (this.last !== undefined) {
const distance = (r.sequence - this.last) >>> 0;
if (distance === 0 || distance >= 0x80000000) return undefined; // duplicate or old delivery
}
if (this.fragment || (this.last !== undefined && r.sequence !== ((this.last + 1) >>> 0))) this.onGap();
this.last = r.sequence;
this.fragment = { record: { ...r, data: new Uint8Array(total) }, total, used: 0 };
}
const f = this.fragment;
if (!f || f.used !== offset || f.total !== total || f.record.sequence !== r.sequence ||
f.record.kind !== r.kind || f.record.request !== r.request || f.record.code !== r.code ||
f.record.handle !== r.handle || f.record.timestamp !== r.timestamp) {
return invalid();
}
f.record.data.set(p.subarray(KEYBOARD_RECORD_HEADER), offset);
f.used += n;
if (f.used !== total) return undefined;
this.fragment = undefined;
return f.record;
}
}
/** BLE address types as the controller reports them. */
export const KEYBOARD_ADDRESS_TYPES = ["public", "random", "public-identity", "random-identity"] as const;
/** One advertiser seen during a scan (an ADVERT record's body, parsed). */
export type KeyboardAdvert = {
addressType: number;
/** "aa:bb:cc:dd:ee:ff", most significant byte first. */
address: string;
rssi: number;
name: string;
/** Advertises the HID service, or a HID appearance. */
hid: boolean;
};
/** ADVERT body: addrType, addr[6] (LSB first), RSSI, event type, AD length, AD structures. */
export function parseKeyboardAdvert(body: Uint8Array): KeyboardAdvert | null {
if (body.length < 10 || body[0]! > 3) return null;
let name = "";
let hid = false;
for (let i = 10; i < body.length;) {
const n = body[i++]!;
if (n === 0 || i + n > body.length) break;
const type = body[i]!;
if (type === 8 || type === 9) name = decodeUtf8(body.subarray(i + 1, i + n));
// 16-bit service lists (incomplete / complete): 0x1812 is HID.
if (type === 2 || type === 3) {
for (let j = i + 1; j + 1 < i + n; j += 2) if (body[j] === 0x12 && body[j + 1] === 0x18) hid = true;
}
// Appearance 0x03C0..0x03FF is the HID category (0x03C1 keyboard); some
// keyboards advertise that without the service list.
if (type === 0x19 && n >= 3 && body[i + 2] === 0x03 && body[i + 1]! >= 0xc0) hid = true;
i += n;
}
return {
addressType: body[0]!,
address: formatKeyboardAddress(body.subarray(1, 7)),
rssi: (body[7]! << 24) >> 24,
name,
hid,
};
}
/**
* Fold a later record for the same advertiser into an earlier one. An active
* scan reports the advertisement and its scan response separately and the
* name usually only rides in the latter: keep a name once seen, HID once
* seen, and the best signal.
*/
export function mergeKeyboardAdverts(a: KeyboardAdvert, b: KeyboardAdvert): KeyboardAdvert {
return { ...a, rssi: Math.max(a.rssi, b.rssi), name: a.name || b.name, hid: a.hid || b.hid };
}
/** Controller byte order (LSB first) to the usual "aa:bb:cc:dd:ee:ff". */
export function formatKeyboardAddress(lsbFirst: Uint8Array): string {
return Array.from(lsbFirst).reverse().map((b) => b.toString(16).padStart(2, "0")).join(":");
}
/** "aa:bb:cc:dd:ee:ff" to the controller's LSB-first bytes, or null if malformed. */
export function parseKeyboardAddress(text: string): Uint8Array | null {
if (!/^(?:[0-9a-f]{2}:){5}[0-9a-f]{2}$/i.test(text)) return null;
return Uint8Array.from(text.split(":").reverse().map((part) => parseInt(part, 16)));
}
/** Lenient UTF-8 decode (advertised names); no TextDecoder in every runtime this loads in. */
function decodeUtf8(bytes: Uint8Array): string {
let out = "";
for (let i = 0; i < bytes.length;) {
const b = bytes[i++]!;
let code = b;
let extra = 0;
if (b >= 0xf0 && b < 0xf8) { code = b & 0x07; extra = 3; }
else if (b >= 0xe0) { code = b & 0x0f; extra = 2; }
else if (b >= 0xc0) { code = b & 0x1f; extra = 1; }
else if (b >= 0x80) { out += "�"; continue; }
if (i + extra > bytes.length) { out += "�"; break; }
let valid = true;
for (let k = 0; k < extra; k++) {
const c = bytes[i + k]!;
if ((c & 0xc0) !== 0x80) { valid = false; break; }
code = (code << 6) | (c & 0x3f);
}
if (!valid || code > 0x10ffff) { out += "�"; continue; }
i += extra;
if (code === 0) continue;
out += String.fromCodePoint(code);
}
return out;
}
-235
View File
@@ -1,235 +0,0 @@
/**
* The keyboard bridge as the app uses it: one KeyboardBridge for the life of
* the app, fed by whichever glasses session is live, its remembered keyboard
* and the custom keybindings kept in the settings store, and its key
* transitions turned into KeyEvents and routed. The platform's controller
* reports the link coming and going and performs the gestures keys stand
* for; the Settings app drives pairing through `bridge` and edits the
* keybindings.
*/
import { addKeyboardEventListener, keyboardLinkSession, sendKeyboardControlRecord } from "../../native/keyboard-link";
import { getStringSetting, setStringSetting } from "../../native/settings-store";
import { shell } from "../../ui/shell/shell";
import {
MAX_KEY_BINDINGS,
chordId,
keyBindingMap,
parseKeyBindings,
serializeKeyBindings,
type GestureOrigin,
type KeyAction,
type KeyBinding,
type KeyChord,
type KeyGesture,
} from "./bindings";
import { KeyboardBridge, type KeyboardBridgeSnapshot, type SavedKeyboard } from "./bridge";
import type { HidPointerSample } from "./hid";
import { KeyRouter } from "./key-router";
import { KeyTranslator, type KeyEvent } from "./keys";
import { PointerInput } from "./pointer";
declare const java: any;
declare const arc4random_uniform: ((bound: number) => number) | undefined;
/** JSON of the remembered keyboard ({ address, addressType, name }), or "" for none. */
const SAVED_KEYBOARD_KEY = "keyboard.saved";
/** JSON list of the custom keybindings ({ key, action }); see bindings.ts. */
const KEY_BINDINGS_KEY = "keyboard.bindings";
/** What the platform's controller does for the keyboard. */
export type KeyboardServiceHost = {
/** Connected to glasses that are in use (not charging in their case). */
ready(): boolean;
locked(): boolean;
/** Turn a dark display on, and nothing else (a locked display stays locked). */
wake(): void;
/** Perform a gesture as if it came from the ring or the watch. */
gesture(gesture: KeyGesture, origin: GestureOrigin): Promise<void>;
/** Routine lines only (failed background reconnects); the rest go to the device log directly. */
log(line: string): void;
/** Whether an absent keyboard is worth looking for now (see KeyboardBridgeHost). */
shouldReconnect(): boolean;
};
function loadSavedKeyboard(): SavedKeyboard | null {
try {
const raw = getStringSetting(SAVED_KEYBOARD_KEY, "");
if (!raw) return null;
const value = JSON.parse(raw);
if (typeof value?.address !== "string" || typeof value?.addressType !== "number") return null;
return { address: value.address, addressType: value.addressType, name: typeof value.name === "string" ? value.name : "" };
} catch {
return null;
}
}
/** Passkey entry only protects the pairing if the passkey can't be guessed. */
function randomPasskey(): number {
try {
if (global.isAndroid) return new java.security.SecureRandom().nextInt(1_000_000);
if (typeof arc4random_uniform === "function") return arc4random_uniform(1_000_000);
} catch {
// Fall through to the weaker source rather than failing the pairing.
}
return Math.floor(Math.random() * 1_000_000);
}
class KeyboardService {
private host: KeyboardServiceHost | null = null;
private session: unknown = null;
private offEvents: (() => void) | null = null;
private readonly listeners = new Set<() => void>();
private bindings: KeyBinding[] = parseKeyBindings(getStringSetting(KEY_BINDINGS_KEY, ""));
private bindingMap = keyBindingMap(this.bindings);
private readonly router = new KeyRouter({
ready: () => this.host?.ready() ?? false,
locked: () => this.host?.locked() ?? false,
screenOn: () => shell.isScreenOn(),
wake: () => this.host?.wake(),
offerToWindow: (event) => shell.receiveKeyEvent(event),
gesture: (gesture, origin) => this.host?.gesture(gesture, origin) ?? Promise.resolve(),
bindings: () => this.bindingMap,
noteActivity: () => shell.noteUserActivity(),
log: (line) => console.log(`[keyboard] ${line}`),
});
private readonly translator = new KeyTranslator({ emit: (event) => this.router.handleKey(event) });
/** The touchpad's buttons and wheel go in as keys; see pointer.ts. */
private readonly pointer = new PointerInput({
buttons: (events) => this.translator.handle(events),
wheel: (key) => this.translator.pulse(key),
});
readonly bridge = new KeyboardBridge({
send: (record) => {
const session = this.session;
if (!session) return false;
try {
return sendKeyboardControlRecord(session, record, `op=${record[3]}`);
} catch {
return false;
}
},
loadSaved: loadSavedKeyboard,
storeSaved: (saved) => setStringSetting(SAVED_KEYBOARD_KEY, saved ? JSON.stringify(saved) : ""),
onKeys: (events) => this.translator.handle(events),
onPointer: (sample) => this.pointer.handle(sample),
onKeyboardGone: () => {
this.translator.reset();
this.pointer.reset();
this.router.reset();
},
onChanged: () => {
for (const listener of [...this.listeners]) listener();
},
// Pairing and setup are rare and are what a failure report needs, so
// they always reach the device log. Nothing here ever includes a key.
log: (line, routine) => {
if (routine) this.host?.log(line);
else console.log(`[keyboard] ${line}`);
},
randomPasskey,
shouldReconnect: () => this.host?.shouldReconnect() ?? true,
});
setHost(host: KeyboardServiceHost): void {
this.host = host;
}
snapshot(): KeyboardBridgeSnapshot {
return this.bridge.snapshot();
}
/** Called on every bridge state change. Returns an unsubscribe function. */
subscribe(listener: () => void): () => void {
this.listeners.add(listener);
return () => this.listeners.delete(listener);
}
/** The glasses came into use (the display woke): look for an absent keyboard now. */
retryNow(): void {
this.bridge.retryNow();
}
/** A G2 session on firmware with the bridge is up. Idempotent. */
linkReady(): void {
const session = keyboardLinkSession();
if (!session) return;
if (session !== this.session) {
// The native session object is rebuilt across reconnects.
this.offEvents?.();
this.session = session;
this.offEvents = addKeyboardEventListener(session, (body) => this.bridge.accept(body));
}
this.bridge.linkReady();
}
/**
* The session can't host the keyboard any more. `release` asks the firmware
* to let the keyboard go first, for when the link itself stays up (the
* glasses went into their case, or the app is disconnecting on purpose).
* Idempotent.
*/
linkLost(release = false): void {
if (release) this.bridge.linkClosing();
this.bridge.linkLost();
this.offEvents?.();
this.offEvents = null;
this.session = null;
}
/** A key a worker's window was sent and had no use for (the "key-unhandled" reply). */
handleUnhandledKey(event: KeyEvent): void {
this.router.handleUnhandledKey(event);
}
// ---------- Custom keybindings ----------
/** The custom keybindings, in the order they were added. */
keyBindings(): readonly KeyBinding[] {
return this.bindings;
}
/** Bind a chord, replacing what it was bound to. False when the list is full. */
setKeyBinding(chord: KeyChord, action: KeyAction): boolean {
const id = chordId(chord);
const index = this.bindings.findIndex((binding) => chordId(binding.chord) === id);
if (index < 0 && this.bindings.length >= MAX_KEY_BINDINGS) return false;
const next = [...this.bindings];
if (index < 0) next.push({ chord, action });
else next[index] = { chord, action };
this.storeBindings(next);
return true;
}
removeKeyBinding(chord: KeyChord): void {
const id = chordId(chord);
this.storeBindings(this.bindings.filter((binding) => chordId(binding.chord) !== id));
}
/**
* Take the next key pressed on the keyboard instead of acting on it. Returns
* a function that calls the capture off.
*/
captureNextKey(onChord: (chord: KeyChord) => void): () => void {
return this.router.captureNextKey(onChord);
}
/**
* Take the keyboard's touchpad or mouse: every report goes to `listener`,
* movement included, and its buttons and wheel stop acting as keys until
* the returned function is called. For the cursor test page; the main UI
* has no cursor.
*/
takePointer(listener: (sample: HidPointerSample) => void): () => void {
return this.pointer.take(listener);
}
private storeBindings(bindings: KeyBinding[]): void {
this.bindings = bindings;
this.bindingMap = keyBindingMap(bindings);
setStringSetting(KEY_BINDINGS_KEY, serializeKeyBindings(bindings));
}
}
export const keyboardService = new KeyboardService();
+72
View File
@@ -0,0 +1,72 @@
import { healthStore } from "./health-store-files";
import { isFixtureData } from "./health-seed";
import { healthExportEnabledSetting, healthExportUrlSetting, healthExportKeySetting } from "../ui/dashboard-settings";
import { getStringSetting, setStringSetting } from "../native/settings-store";
import { fetchTextWithUserAgent } from "../util/http";
let running = false;
let started = false;
let status = "Not synchronized";
export function healthExportStatus(): string { return status; }
/** One bounded batch, with durable acknowledgement and no overlapping requests. */
export async function syncHealthExport(): Promise<void> {
if (running) return;
running = true;
try {
if (!healthExportEnabledSetting.get()) { status = "Export disabled"; return; }
if (isFixtureData()) { status = "Sample data is never exported"; return; }
const url = healthExportUrlSetting.get();
const key = healthExportKeySetting.get();
if (!url || healthExportUrlSetting.validationError() || !key) {
status = "Configure an HTTPS endpoint and API key";
return;
}
const generation = getStringSetting("health.export.generation", "0");
let held: { url?: string; generation?: string; offsets?: Record<string, number> } = {};
try { held = JSON.parse(getStringSetting("health.export.cursor", "{}")) ?? {}; } catch { /* First sync. */ }
const offsets = held.url === url && held.generation === generation ? held.offsets ?? {} : {};
let limit = 200;
let batch = healthStore().exportBatch(offsets, limit);
let body = JSON.stringify({ version: 1, samples: batch.samples, sleep: batch.sleep });
// Normalized records contain only ASCII keys, numbers and metric names.
// Large sleep histories must fit the server body limit as well as its record limit.
while (body.length > 240_000 && limit > 1) {
limit = Math.max(1, Math.floor(limit / 2));
batch = healthStore().exportBatch(offsets, limit);
body = JSON.stringify({ version: 1, samples: batch.samples, sleep: batch.sleep });
}
if (body.length > 240_000) { status = "A health record exceeds the export size limit"; return; }
if (!batch.scanned) { status = "Up to date"; return; }
if (batch.samples.length || batch.sleep.length) {
status = "Sending health data";
const response = await fetchTextWithUserAgent(url, {
method: "POST", headers: { "Authorization": `Bearer ${key}`, "Content-Type": "application/json" },
body,
timeoutMs: 30_000, dontFollowRedirects: true,
});
if (response.status !== 200) {
status = response.status === 401 || response.status === 403
? "API key refused; check configuration" : `Server refused batch (HTTP ${response.status}); will retry`;
return;
}
const ack = await response.json<{ accepted?: boolean }>();
if (ack.accepted !== true) { status = "Invalid server acknowledgement; will retry"; return; }
}
// An edit during the request must not acknowledge data on a different destination.
if (url !== healthExportUrlSetting.get() || generation !== getStringSetting("health.export.generation", "0")) return;
setStringSetting("health.export.cursor", JSON.stringify({ url, generation, offsets: batch.cursor }));
status = "Batch synchronized; remaining history follows automatically";
} catch {
// Errors may include request headers or data: never log or display them.
status = "Synchronization failed; will retry automatically";
} finally { running = false; }
}
/** Android app process only; no extra BLE pulls, wake locks, or background service. */
export function startHealthExport(): void {
if (started) return;
started = true;
setTimeout(() => { void syncHealthExport(); }, 10_000);
setInterval(() => { void syncHealthExport(); }, 60_000);
}
+19
View File
@@ -48,6 +48,7 @@
import {
HOUR_MS,
HEART_RATE_MEASUREMENT_SPAN_MS,
TEN_MINUTES_MS,
type HealthSample,
type SleepSession,
@@ -65,6 +66,8 @@ export type WireHourlyRecord = {
* zero bytes instead.
*/
anchorMs: number | null;
/** Latest reading, only from live pages; raw journal replay does not guess its clock. */
latest?: { timestampSeconds: number; value: number; receivedAtMs: number; clockOffsetMs: number };
groups: readonly { hourIndex: number; avg: number; max: number; min: number }[];
};
@@ -146,6 +149,22 @@ export function convertRecords(records: readonly WireRecord[]): ConversionResult
* guesses. Dropping loses a page; guessing corrupts the history silently.
*/
function convertHourly(record: WireHourlyRecord, result: ConversionResult): void {
if (record.metric === "heartRate" && record.latest) {
const { timestampSeconds, value, receivedAtMs, clockOffsetMs } = record.latest;
const startMs = timestampSeconds * 1000 - clockOffsetMs;
// Invert the existing handshake clock shift, never replace measurement time
// with reception time or snap it onto an invented ten-minute bucket.
// Older pages can mix clock domains: only accept a recent live reading.
if (Number.isSafeInteger(timestampSeconds) && Number.isSafeInteger(receivedAtMs)
&& Number.isFinite(clockOffsetMs) && Number.isInteger(value) && value > 0 && value <= 300
&& startMs >= 946684800000 && startMs <= receivedAtMs + 120_000
&& receivedAtMs - startMs <= 20 * 60_000) {
result.samples.push({ metric: "heartRate", startMs, spanMs: HEART_RATE_MEASUREMENT_SPAN_MS,
min: value, max: value, avg: value, total: value });
} else {
result.skipped.push({ what: "latest heart rate", why: "invalid, stale or unresolved measurement timestamp/value" });
}
}
if (record.anchorMs === null) {
result.skipped.push({
what: `${record.metric} page, ${record.groups.length} groups`,
+17 -14
View File
@@ -240,7 +240,7 @@ function accumulateStepBuckets(
return merged;
}
function toWire(record: any): WireRecord | null {
function toWire(record: any, includeLatest: boolean): WireRecord | null {
const cmdHi = Number(record.cmdHi);
const metric = HOURLY_METRIC[cmdHi];
@@ -256,7 +256,12 @@ function toWire(record: any): WireRecord | null {
min: Number(group.min),
});
}
return { kind: "hourly", metric, anchorMs: anchorToMs(Number(record.anchorUnixSeconds), true), groups };
const receivedAtMs = Number(record.receivedAtMs);
const latest = includeLatest && metric === "heartRate" && Number(record.tagRaw) > 0 ? {
timestampSeconds: Number(record.tagRaw), value: Number(record.current), receivedAtMs,
clockOffsetMs: ringClockOffsetMs(receivedAtMs),
} : undefined;
return { kind: "hourly", metric, anchorMs: anchorToMs(Number(record.anchorUnixSeconds), true), groups, latest };
}
if (cmdHi === CMD_STEPS) {
@@ -391,12 +396,12 @@ type StoreResult = Omit<LiveSyncResult, "seen">;
* fails, so the caller keeps whatever it would otherwise mark as consumed.
* Shared by the live batch and the journal replay.
*/
function storeRecords(records: JavaRecordList): StoreResult {
function storeRecords(records: JavaRecordList, includeLatest = true): StoreResult {
const wire: WireRecord[] = [];
const size = records.size();
for (let i = 0; i < size; i++) {
try {
const converted = toWire(records.get(i));
const converted = toWire(records.get(i), includeLatest);
if (converted) wire.push(converted);
} catch (error) {
console.warn("health live: skipped an undecodable record", error);
@@ -480,7 +485,7 @@ function replayFrameJournalOnce(): void {
const records: JavaRecordList = ringProtocol.decodeFrameJournal(File.fromPath(journalPath).readTextSync(), after);
const size = records.size();
if (size > 0) {
const result = storeRecords(records);
const result = storeRecords(records, false);
let through = after;
for (let i = 0; i < size; i++) through = Math.max(through, Number(records.get(i).receivedAtMs));
const next: ReplayCursor = { version: JOURNAL_DECODER_VERSION, throughReceivedAtMs: through };
@@ -543,7 +548,7 @@ export function syncLiveRecords(): LiveSyncResult {
/**
* Ask the ring for a fresh pull now, rather than waiting out the automatic
* 30-minute cycle. Returns immediately; the pull itself takes ~15s on the
* ten-minute cycle. Returns immediately; the pull itself takes ~15s on the
* communicator's worker thread and lands through the next `syncLiveRecords()`.
*
* Throttled communicator-side to one a minute — see
@@ -584,19 +589,17 @@ export function startLiveHealthSync(): void {
}
/**
* Pull the ring on a wall-clock-aligned cadence: :01 and :31, every hour.
* Pull the ring every ten minutes, one minute after each bucket boundary.
*
* Chris, 2026-09-12: *"I think we should be polling the ring on the 30 minute
* cycle (that was my intent, not a floor honestly)."* The 30-minute value in
* `RING_HEALTH_MIN_PULL_INTERVAL_MS` was always meant as a collection rhythm
* and got built as a throttle, so a connected, stable ring pulled never —
* nothing in the system drives a pull on a timer at all.
* This collects the latest heart-rate observation at roughly the cadence
* the stock app documents, as well as the existing historical buckets.
* It does not force the sensor to take a new measurement.
*
* Two reasons this is aligned to the wall clock rather than a plain interval:
*
* 1. **The ring's own step buckets close on 10-minute boundaries.** Pulling on
* the boundary collects a just-closed bucket instead of a half-formed one.
* 2. **An elapsed timer drifts.** A pull at :07 sets the next at :37, then :07,
* 2. **An elapsed timer drifts.** A pull at :07 sets the next at :17, then :27,
* wandering away from the boundary it is supposed to track. Re-arming
* against the clock each time cannot drift.
*
@@ -606,7 +609,7 @@ export function startLiveHealthSync(): void {
* This only *requests* a pull. The communicator still applies its own anti-spam
* floor, so an extra call here can never hammer the ring.
*/
const ALIGNED_PULL_MINUTES = [1, 31];
const ALIGNED_PULL_MINUTES = [1, 11, 21, 31, 41, 51];
let alignedPullTimer: ReturnType<typeof setTimeout> | null = null;
function msUntilNextAlignedPull(now: Date): number {
+47 -4
View File
@@ -59,6 +59,7 @@
import {
type HealthSample,
isHeartRateMeasurement,
type Rollup,
type SampleMetric,
type SleepSession,
@@ -228,12 +229,51 @@ export class HealthStore {
// -------------------------------------------------------------------------
// Reading
/** Every stored sample overlapping [startMs, endMs). Exact - reads shards. */
samplesInRange(startMs: number, endMs: number): HealthSample[] {
/** Append-journal cursor, committed by the exporter only after server acknowledgement.
* Revisions appended to an old month are exported too; a time watermark would lose them.
*/
exportBatch(cursor: Record<string, number>, limit = 200): {
cursor: Record<string, number>; samples: HealthSample[]; sleep: SleepSession[]; scanned: number;
} {
const next = { ...cursor };
const samples: HealthSample[] = [];
const sleep: SleepSession[] = [];
let scanned = 0;
for (const name of this.backend.list().filter(candidate => /^samples-\d{4}-\d{2}\.jsonl$/.test(candidate) || candidate === SLEEP_FILE).sort()) {
const text = this.backend.read(name);
if (text === null) throw new Error("Health export could not read history");
let offset = next[name] ?? 0;
if (!Number.isSafeInteger(offset) || offset < 0 || offset > text.length) offset = 0;
while (scanned < limit) {
const end = text.indexOf("\n", offset);
if (end < 0) break; // Never consume a partially written final record.
const line = text.slice(offset, end);
offset = end + 1;
scanned++;
if (name === SLEEP_FILE) {
try {
const session = JSON.parse(line) as SleepSession;
if (session?.timeResolved === true) sleep.push(session);
} catch { /* Ignore incomplete/corrupt journal lines as the local reader does. */ }
} else {
const sample = fromLine(line);
if (sample) samples.push(sample);
}
}
next[name] = offset;
if (scanned >= limit) break;
}
return { cursor: next, samples, sleep, scanned };
}
/** Bucket samples in [startMs, endMs). Point observations are opt-in so
* existing charts and daily averages never combine two overlapping series. */
samplesInRange(startMs: number, endMs: number, includeMeasurements = false): HealthSample[] {
const out: HealthSample[] = [];
for (const name of this.shardNamesInRange(startMs, endMs)) {
for (const sample of this.loadShard(name).values()) {
if (sample.startMs >= startMs && sample.startMs < endMs) out.push(sample);
if (sample.startMs >= startMs && sample.startMs < endMs
&& (includeMeasurements || !isHeartRateMeasurement(sample))) out.push(sample);
}
}
out.sort((a, b) => a.startMs - b.startMs);
@@ -297,6 +337,7 @@ export class HealthStore {
for (const name of this.backend.list()) {
if (!name.startsWith(SAMPLE_PREFIX) || !name.endsWith(SAMPLE_SUFFIX)) continue;
for (const sample of this.loadShard(name).values()) {
if (isHeartRateMeasurement(sample)) continue;
const key = dayKey(sample.startMs);
let byMetric = perDay.get(key);
if (!byMetric) {
@@ -323,7 +364,9 @@ export class HealthStore {
private updateRollupsFor(samples: readonly HealthSample[]): void {
const file = this.loadRollups();
const touched = new Set<string>();
for (const sample of samples) touched.add(dayKey(sample.startMs));
for (const sample of samples) {
if (!isHeartRateMeasurement(sample)) touched.add(dayKey(sample.startMs));
}
for (const key of touched) {
const parts = key.split("-").map(Number);
const dayStart = new Date(parts[0]!, parts[1]! - 1, parts[2]!).getTime();
+6
View File
@@ -36,6 +36,12 @@ export function isCumulative(metric: SampleMetric): boolean {
export const HOUR_MS = 3_600_000;
export const TEN_MINUTES_MS = 600_000;
/** Point observation, not a ten-minute average. Distinct from hourly buckets. */
export const HEART_RATE_MEASUREMENT_SPAN_MS = 1;
export function isHeartRateMeasurement(sample: HealthSample): boolean {
return sample.metric === "heartRate" && sample.spanMs === HEART_RATE_MEASUREMENT_SPAN_MS;
}
export const DAY_MS = 86_400_000;
/**
+49 -2
View File
@@ -9,7 +9,7 @@ declare const java: any;
* than bundled in the APK; FaceclawVoiceController reads the files from the
* directory each model's `dirName` names.
*
* Two models, same shape:
* Downloadable models, same shape:
* - "moonshine": the original on-device option (three files). The model is
* no longer bundled in the APK; it is fetched into the same filesDir
* location that earlier releases copied the bundled files to, so
@@ -29,12 +29,20 @@ declare const java: any;
* used for Moonshine below. Every hash here was computed locally from a
* freshly downloaded copy of the file, not copied from a doc page.
*
* - "whisper-base": multilingual base, int8-quantized. Used for non-English
* speech and auto-detection; English-only downloads remain untouched.
* SHA-256/size values come from the maintainer's LFS metadata, with the
* vocabulary hash computed locally. URLs pin the source revision.
*
* - "whisper-small-fr": a dedicated French fine-tune, exported and quantized
* with sherpa-onnx 1.13.0. Versioned release assets include its Apache license.
*
* Mirrors the on-phone assistant model flow in llama.ts, except each model
* here is multiple files rather than one; they download sequentially through
* FaceclawModelDownloader (resume + pinned sha256 per file).
*/
export type AsrModelId = "moonshine" | "whisper-base-en";
export type AsrModelId = "moonshine" | "whisper-base-en" | "whisper-base" | "whisper-small-fr";
type AsrModelFile = {
name: string;
@@ -105,6 +113,43 @@ export const ASR_MODELS: Record<AsrModelId, AsrModelDef> = {
],
totalBytes: 160626066,
},
"whisper-base": {
label: "Whisper (multilingual, base)",
dirName: "sherpa-onnx-whisper-base-int8",
baseUrl: "https://huggingface.co/csukuangfj/sherpa-onnx-whisper-base/resolve/bb53ee204431c90d314c1cc08d28d23e5b7927cc/",
files: [
{
name: "base-encoder.int8.onnx",
sha256: "0b8fb1304b6109976038efff5ace81720e00386f3ff6b54ee8c75291ca0a1e11",
sizeBytes: 29120534,
},
{
name: "base-decoder.int8.onnx",
sha256: "9759d217388a01b3a4c7c15533201067b48ae819c4daafc8624e64b9409dc02d",
sizeBytes: 130672026,
},
{
name: "base-tokens.txt",
sha256: "b34b360dbb493e781e479794586d661700670d65564001f23024971d1f2fa126",
sizeBytes: 816730,
},
],
totalBytes: 160609290,
},
"whisper-small-fr": {
label: "Whisper (French, small)",
dirName: "sherpa-onnx-whisper-small-fr-int8-v1",
// Pinned export of bofenghuang/whisper-small-cv11-french; see docs/models/whisper-small-fr.
baseUrl: "https://git.undefined-dev.fr/api/packages/Blendman974/generic/faceclaw-asr/whisper-small-fr-v1/",
files: [
{"name": "small-fr-encoder.int8.onnx", "sha256": "ec34fc42182f37727ad982d254b18a6001f5dcccebc5a36060031995f1782d5d", "sizeBytes": 112413414},
{"name": "small-fr-decoder.int8.onnx", "sha256": "b554e35dc8c593b550a82cd01ebb00c1347e8c2b089b780b4c4fe16450cd7dc0", "sizeBytes": 262200764},
{"name": "small-fr-tokens.txt", "sha256": "b34b360dbb493e781e479794586d661700670d65564001f23024971d1f2fa126", "sizeBytes": 816730},
{"name": "LICENSE.txt", "sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30", "sizeBytes": 11358},
],
totalBytes: 375442266,
},
};
export type AsrModelState = {
@@ -129,6 +174,8 @@ function freshRuntime(): ModelRuntime {
const runtimes: Record<AsrModelId, ModelRuntime> = {
moonshine: freshRuntime(),
"whisper-base-en": freshRuntime(),
"whisper-base": freshRuntime(),
"whisper-small-fr": freshRuntime(),
};
function modelDirPath(id: AsrModelId): string {
+19
View File
@@ -0,0 +1,19 @@
import { getStringSetting, setStringSetting } from "./settings-store";
export const CALENDAR_SELECTION_KEY = "calendar.selectedIds";
/** null means all calendars (including future ones); [] explicitly means none. */
export function readCalendarSelection(): string[] | null {
try {
const value = JSON.parse(getStringSetting(CALENDAR_SELECTION_KEY, "null"));
if (value === null) return null;
if (!Array.isArray(value) || !value.every(id => typeof id === "string" && /^\d+$/.test(id))) return null;
return Array.from(new Set<string>(value));
} catch {
return null;
}
}
export function setCalendarSelection(ids: readonly string[] | null): void {
if (ids !== null && !ids.every(id => /^\d+$/.test(id))) throw new Error("Invalid calendar ID");
setStringSetting(CALENDAR_SELECTION_KEY, JSON.stringify(ids === null ? null : Array.from(new Set(ids))));
}
+22
View File
@@ -0,0 +1,22 @@
import { Utils } from "@nativescript/core";
import { hasCalendarPermission } from "./calendar-permissions";
declare const com: any;
export type CalendarSource = { id: string; name: string; accountName: string; accountType: string };
/** Android calendar IDs survive renames and distinguish equal display names. */
export function readAvailableCalendars(): CalendarSource[] {
if (!global.isAndroid || !hasCalendarPermission()) return [];
try {
const context = Utils.android.getApplicationContext();
if (!context) return [];
const parsed = JSON.parse(String(com.faceclaw.app.FaceclawCalendarProvider.getCalendarsJson(context)));
if (!Array.isArray(parsed)) return [];
return parsed.filter(row => row && typeof row.id === "string" && /^\d+$/.test(row.id))
.map(row => ({ id: row.id, name: String(row.name ?? ""),
accountName: String(row.accountName ?? ""), accountType: String(row.accountType ?? "") }));
} catch {
return [];
}
}
+15 -3
View File
@@ -2,6 +2,8 @@ import { Utils } from "@nativescript/core";
import { hasCalendarPermission } from "../g2/android-permissions";
import { spanCurrent } from "./frame-timings";
import { type CalendarEvent, type CalendarReadState } from "./calendar-types";
import { CALENDAR_SELECTION_KEY, readCalendarSelection } from "./calendar-selection";
import { onSettingsStoreChanged } from "./settings-store";
export type { CalendarEvent } from "./calendar-types";
declare const com: any;
@@ -13,14 +15,21 @@ declare const com: any;
*/
// Android reads synchronously; iOS notifies subscribers after background reads.
export function onCalendarChanged(_listener: () => void): () => void { return () => {}; }
export function onCalendarChanged(listener: () => void): () => void {
return onSettingsStoreChanged(key => {
if (key === CALENDAR_SELECTION_KEY || key == null) {
invalidateCalendarCache();
listener();
}
});
}
export function getCalendarReadState(): CalendarReadState { return "ready"; }
const DEFAULT_MAX_EVENTS = 50;
const DEFAULT_WINDOW_MS = 14 * 24 * 60 * 60 * 1000;
const CACHE_MS = 30_000;
let cache: { events: CalendarEvent[]; atMs: number; maxEvents: number; windowMs: number } | null = null;
let cache: { events: CalendarEvent[]; atMs: number; maxEvents: number; windowMs: number; selection: string } | null = null;
/**
* Upcoming events from now through the given window, ordered by start time.
@@ -35,9 +44,11 @@ export function readUpcomingEvents(
if (!global.isAndroid || !hasCalendarPermission()) return [];
const now = Date.now();
const selection = JSON.stringify(readCalendarSelection());
if (
!forceRefresh &&
cache &&
cache.selection === selection &&
cache.maxEvents === maxEvents &&
cache.windowMs === windowMs &&
now - cache.atMs < CACHE_MS
@@ -55,6 +66,7 @@ export function readUpcomingEvents(
context,
Math.max(0, Math.round(maxEvents)),
Math.max(0, Math.round(windowMs)),
selection,
),
),
);
@@ -62,7 +74,7 @@ export function readUpcomingEvents(
const events = Array.isArray(parsed)
? parsed.map(normalizeEvent).filter((event): event is CalendarEvent => Boolean(event))
: [];
cache = { events, atMs: now, maxEvents, windowMs };
cache = { events, atMs: now, maxEvents, windowMs, selection };
return events;
} catch {
return [];
+9 -5
View File
@@ -1,6 +1,6 @@
import type { CloudSttClient, CloudSttOptions } from "./cloud-stt";
import { ElevenLabsSttClient } from "./elevenlabs-stt";
import { OpenAiRealtimeSttClient } from "./openai-stt";
import { DEFAULT_OPENAI_STT_MODEL, OpenAiRealtimeSttClient, type OpenAiSttModel } from "./openai-stt";
import { ReconnectingSttClient } from "./reconnecting-stt";
import { SonioxSttClient } from "./soniox-stt";
import type { VoiceProviderKind } from "./voice-control";
@@ -14,11 +14,12 @@ export type VoiceProviderSelection = {
provider: VoiceProviderKind;
/**
* Spoken language the provider should expect (ISO 639-1); empty lets it
* detect the language. Cloud providers only.
* detect the language. Also used by on-device Whisper.
*/
language?: string;
elevenLabsApiKey: string;
openAiApiKey: string;
openAiTranscriptionModel?: OpenAiSttModel;
sonioxApiKey: string;
/** Self-hosted Whisper ("whisper-server"); the key is optional there. */
whisperServerUrl?: string;
@@ -52,7 +53,7 @@ const CLOUD_PROVIDERS: Partial<Record<VoiceProviderKind, CloudProvider>> = {
whisper: {
name: "OpenAI",
apiKey: (selection) => selection.openAiApiKey,
create: (options) => new OpenAiRealtimeSttClient(options),
create: (options, selection) => new OpenAiRealtimeSttClient({ ...options, model: selection.openAiTranscriptionModel }),
},
"whisper-server": {
name: "Whisper server",
@@ -76,8 +77,11 @@ function isConfigured(provider: CloudProvider, selection: VoiceProviderSelection
* What a capture transcribes with: the provider and the language it is told
* to expect. A running capture is shared only by holders asking for the same.
*/
export function transcriptionKey(selection: Pick<VoiceProviderSelection, "provider" | "language">): string {
return `${selection.provider}:${(selection.language ?? "").trim().toLowerCase()}`;
export function transcriptionKey(selection: Pick<VoiceProviderSelection, "provider" | "language" | "openAiTranscriptionModel">): string {
const fallback = selection.provider === "onboard-whisper-fr" ? "fr" : selection.provider === "onboard-whisper" ? "en" : "";
const language = selection.provider === "onboard-whisper-fr" ? "fr" : (selection.language ?? fallback).trim().toLowerCase();
const model = selection.provider === "whisper" ? `:${selection.openAiTranscriptionModel ?? DEFAULT_OPENAI_STT_MODEL}` : "";
return `${selection.provider}:${language}${model}`;
}
/** Whether createCloudSttClient would return a cloud client for this selection. */
+2 -20
View File
@@ -3,18 +3,11 @@ import type { DiscoveryEvents } from './device-discovery'
import type { RawAdvertisement } from '../g2/even-advertisement'
export { buildAddressSet } from './device-discovery-common'
/**
* How often a scan re-reads which arms the system is connected to. Each read refreshes
* their rows, which age out like advertisements once the link drops and they go quiet.
*/
const CONNECTED_ARMS_POLL_MS = 2000
export class DeviceDiscoveryBridge {
private ble = iosBluetooth()
private off: (() => void) | null = null
private generation = 0
private cancelCollection: (() => void) | null = null
private connectedArmsTimer: ReturnType<typeof setInterval> | null = null
get bluetoothEnabled(): boolean { return this.ble.state === 5 }
get scanning(): boolean { return !!this.off && this.ble.scanning }
startScan(events: DiscoveryEvents): boolean {
@@ -28,27 +21,16 @@ export class DeviceDiscoveryBridge {
void this.ble.startScan(24 * 60 * 60 * 1000).catch(error => {
if (generation === this.generation) events.onScanFailed?.(-1, String(error.message ?? error))
})
// A connected arm doesn't advertise (iOS holds the right arm's ANCS link open after a
// session ends), so the scan alone would show those glasses one-armed.
const reportConnectedArms = () => { for (const device of this.ble.systemConnectedDevices()) events.onAdvertisement(device) }
reportConnectedArms()
this.connectedArmsTimer = setInterval(reportConnectedArms, CONNECTED_ARMS_POLL_MS)
return true
}
stopScan(): void {
++this.generation; this.cancelCollection?.()
if (this.connectedArmsTimer) { clearInterval(this.connectedArmsTimer); this.connectedArmsTimer = null }
if (this.off) { this.off(); this.off = null; this.ble.stopScan() }
}
// CoreBluetooth does not expose the phone's bond list. Fresh advertisements
// supply both identity and the UUID mapping that iOS needs for connections;
// startScan() reports the arms the system is connected to.
// supply both identity and the UUID mapping that iOS needs for connections.
emitBondedDevices(): void {}
async getBondedCandidates(): Promise<RawAdvertisement[]> {
const devices = new Map(this.ble.devices)
for (const device of this.ble.systemConnectedDevices()) if (!devices.has(device.address)) devices.set(device.address, device)
return [...devices.values()]
}
async getBondedCandidates(): Promise<RawAdvertisement[]> { return [...this.ble.devices.values()] }
async scanCandidates(timeoutMs = 6000): Promise<RawAdvertisement[]> {
const generation = this.generation
await this.ble.ensureReady()
-21
View File
@@ -19,7 +19,6 @@ declare const FaceclawKitFaceclawCompassListener: any;
declare const FaceclawKitFaceclawImuListener: any;
declare const FaceclawKitIosAmbientLightListener: any;
declare const FaceclawKitIosMicStatusListener: any;
declare const FaceclawKitIosKeyboardEventListener: any;
export type CommunicatorPhase =
@@ -199,13 +198,6 @@ class MicStatusListener extends NSObject {
onNsDataMicStatusBodyArm(body: NSData, arm: string): void { this.handler(fromData(body), String(arm ?? "")); }
}
@NativeClass()
class KeyboardEventListener extends NSObject {
static ObjCProtocols = [FaceclawKitIosKeyboardEventListener];
handler!: (body: Uint8Array) => void;
onNsDataKeyboardEventBody(body: NSData): void { this.handler(fromData(body)); }
}
/**
* The iOS twin of the Android communicator bridge: the same public API over
* the shared Kotlin session (FaceclawKitIosGlassesSession, CoreBluetooth
@@ -811,19 +803,6 @@ export class FaceclawCommunicatorBridge {
toData(new Uint8Array(record)), label, Boolean(rightTemple), Boolean(leftTemple)));
}
/** Raw field-131 keyboard-bridge records, in arrival order, on the main queue. */
addKeyboardEventListener(listener: (body: Uint8Array) => void): () => void {
const proxy = KeyboardEventListener.new() as KeyboardEventListener;
proxy.handler = listener;
const handle = this.communicator.addKeyboardEventListenerNsDataListener(proxy);
return () => { this.communicator.removeKeyboardEventListenerListener(handle); };
}
/** Queue a CFW keyboard-bridge control for the right temple; false when nothing was queued. */
sendFaceclawKeyboardControl(record: Uint8Array, label: string): boolean {
return Boolean(this.communicator.sendFaceclawKeyboardControlRecordLabel(toData(new Uint8Array(record)), label));
}
/**
* Every audio-characteristic packet, whatever enabled the mics, on the
* main queue. Passive: sends nothing and doesn't affect capture.
+5
View File
@@ -524,6 +524,11 @@ export class FaceclawCommunicatorBridge {
await this.enqueueJavaCall(() => this.communicator.enableWearDetectionAndRequestState());
}
/** -1 leaves a field unchanged. The native session retains explicit values across reconnects. */
async configureHeadUpWake(enabled: number, angle: number): Promise<void> {
await this.enqueueJavaCall(() => this.communicator.configureHeadUpWake(enabled, angle));
}
/** Whether the transport prelude has finished (each arm reports "connected" before it). */
isSessionReady(): boolean {
return Boolean(this.communicator.isSessionReady());
Loaded 100 of 290 files, more files were not shown because too many files have changed in this diff. Show more