Merge branch 'main' into ring-health-fixes

This commit is contained in:
James Babcock committed 2026-10-25 01:06:38 -07:00
commit 4f8c082b52
368 files changed
+26238 -3401

No files matched your search

+41
View File
@@ -0,0 +1,41 @@
# Hermes agent as the assistant
The "Hermes agent" assistant model sends what you say to a
[Hermes Agent](https://github.com/NousResearch/hermes-agent) gateway through its
OpenAI-compatible API server and streams the answer back to the glasses. Hermes
runs its own tools and keeps the conversation: each Faceclaw assistant
conversation maps to one Hermes session (`X-Hermes-Session-Id`), and only the
latest message is sent. Faceclaw's own glasses tools are not offered to Hermes.
The session ID is saved with the conversation, so a conversation restored after
an app restart continues the same Hermes session.
A Hermes turn may run for up to 20 minutes (other models get 2), because Hermes
runs whole multi-tool tasks in one request and closing the stream interrupts the
run on the server. A run that fails or is interrupted on the server shows as an
error on the glasses, with any partial answer kept.
## Enable the API server in Hermes
In the Hermes config, enable the `api_server` platform and bind it where the
phone can reach it, then set a key in Hermes's `.env` and restart the gateway:
hermes config set platforms.api_server.enabled true
hermes config set platforms.api_server.extra.host <address the phone can reach>
hermes config set platforms.api_server.extra.port 8642
echo "API_SERVER_KEY=$(openssl rand -hex 24)" >> ~/.hermes/.env
Check it with `curl -H "Authorization: Bearer <key>" http://<host>:8642/v1/models`.
## Settings (Settings > Assistant)
- **Assistant model**: Hermes agent. Disabled until the URL is set.
- **Hermes URL**: e.g. `http://host:8642/v1` (a bare host gets `/v1` added).
- **Hermes key**: the `API_SERVER_KEY` value.
On iOS, App Transport Security blocks plain `http://` to IP addresses unless
`App_Resources/iOS/Info.plist` has an `NSExceptionDomains` entry for that host
(`NSExceptionAllowsInsecureHTTPLoads`); otherwise serve the API over https, for
example with `tailscale serve`.
Actions that Hermes holds for approval (`approvals.mode manual`) cannot be
approved from the glasses; approve them from Hermes itself.
+53
View File
@@ -0,0 +1,53 @@
# Self-hosted Whisper transcription
The "Whisper server (self-hosted)" transcription provider sends voice audio to
any OpenAI-compatible Whisper server instead of a cloud service: vLLM, speaches
(faster-whisper-server), LocalAI and similar. It uses the batch endpoint
`POST {url}/audio/transcriptions` with a 16 kHz mono WAV. Push-to-talk
utterances are one request each; in Transcribe, each span ended by a pause is
one request. While a span is open its audio so far is re-sent every 1.5 s, so
text appears while you are still talking. A span that runs 25 s without a pause
(nonstop speech, music) is cut at its quietest moment in the last 5 s, so no
request outgrows Whisper's 30 s window; the pieces still arrive as one final.
On Android the WAV goes out through `FaceclawBinaryPost` (okhttp), since
NativeScript's http module copies a binary body into Java a byte at a time.
## Server
Any server exposing `/v1/models` and `/v1/audio/transcriptions` works, e.g.
vllm serve openai/whisper-large-v3 --port 8004
## Settings (Settings > Voice)
- **Transcription Provider**: Whisper server (self-hosted). Disabled until the
URL is set.
- **Whisper server URL**: the API root, e.g. `http://host:8004/v1`. A bare
`http://host:8004` gets `/v1` added.
- **Whisper server model**: empty uses the first model the server lists.
- **Whisper language**: an ISO code such as `en`, or empty to auto-detect.
A wrong fixed language makes Whisper translate rather than transcribe.
- **Whisper server key**: optional bearer token.
## iOS: plain http
iOS App Transport Security blocks plain `http://` to IP addresses (including
Tailscale's 100.x addresses) unless the app's Info.plist has an exception for
that host. Either serve the API over https (for example with `tailscale serve`)
or add your server to `NSExceptionDomains` in `App_Resources/iOS/Info.plist`
before building:
<key>192.0.2.10</key>
<dict><key>NSExceptionAllowsInsecureHTTPLoads</key><true/></dict>
## Transcribe on iOS
Transcribe needs a cloud or Whisper-server provider on iOS (Apple's recognizer
only handles short utterances). Its capture follows the glasses: a disconnect,
a glasses lock or backgrounding (for phone-mic capture) transcribes what was
already heard, and listening resumes once the glasses reconnect, unlock or the
app returns to the foreground. Without a glasses session the phone mic stands in
while Faceclaw is in the foreground. With "Save Transcribe audio" on (Voice
settings), each session's audio is kept as `transcribe-<time>.wav` in Downloads,
next to `transcript-<time>.txt` when the text is saved; a resumed capture keeps
appending to the same file.