forked from Blendman974/faceclaw
Replace Unicode property escapes, which crash the glasses' V8
The NativeScript Android runtime embeds a V8 built without ICU, so a regex
literal containing `\p{...}` cannot be parsed there. Because the app ships as a
single ES module, the three such literals added with the calculator and
microphone apps made the whole of bundle.mjs fail to compile:
com.tns.NativeScriptException: Cannot compile ES module .../app/bundle.mjs
SyntaxError: Invalid regular expression: /\p{L}/: Invalid property name
That throws inside Application.onCreate, so the app died on launch before any
of its own code ran — every screen, not just the two new ones. Neither tsc nor
the test suite caught it: the syntax is valid ES2018 and node has ICU.
app/util/unicode-class.ts replaces them with explicit ranges. The two uses pull
in opposite directions, so they are stated differently:
- isLetter (maths lexer) is a letter table. The lexer's alphabet is small and
known, and an unrecognised letter just becomes an identifier matching no
known function, which is already a handled error.
- NON_WORD_RUN (conversation search, sentiment) is a *separator* table rather
than the complement of that letter table. Complementing it would make every
letter the table omits — minority scripts, later Unicode additions — act as
a separator and silently shred that text; listing separators instead keeps
an unlisted character as word content. It also holds decomposed graphemes
together, which `\p{L}` did not.
Verified against the real properties under node's ICU: identical output on a
prose corpus except where the new split is better (a Lao vowel sign and a
combining acute now stay attached).
The new test's last case scans app/ for `\p{` outside comments — the check that
would have caught this before it reached a device.
This commit is contained in:
5 files changed
+198
-6
No files matched your search
@@ -2,6 +2,7 @@
|
||||
|
||||
import { type MathNumber, fromDecimal, integer, real } from "./math-number";
|
||||
import { type MathRelation } from "./expression";
|
||||
import { isLetter } from "../../../util/unicode-class";
|
||||
|
||||
export type MathToken =
|
||||
| { type: "number"; value: MathNumber }
|
||||
@@ -71,10 +72,6 @@ function isDigit(character: string): boolean {
|
||||
return character >= "0" && character <= "9";
|
||||
}
|
||||
|
||||
function isLetter(character: string): boolean {
|
||||
return /\p{L}/u.test(character);
|
||||
}
|
||||
|
||||
export function tokenize(input: string): MathToken[] {
|
||||
const tokens: MathToken[] = [];
|
||||
const characters = Array.from(input);
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
// scoring. No platform NLP dependency — the deterministic tiers carry the
|
||||
// whole signal.
|
||||
|
||||
import { NON_WORD_RUN } from "../../util/unicode-class";
|
||||
|
||||
export type EmotionLabel =
|
||||
| "happy"
|
||||
| "sad"
|
||||
@@ -143,7 +145,7 @@ function emotionTokens(text: string): string[] {
|
||||
.replace(/can't/g, "cannot")
|
||||
.replace(/i'm/g, "i am")
|
||||
.replace(/n't/g, " not");
|
||||
return normalized.split(/[^\p{L}\p{N}]+/u).filter((token) => token.length > 0);
|
||||
return normalized.split(NON_WORD_RUN).filter((token) => token.length > 0);
|
||||
}
|
||||
|
||||
function emotionCueIsNegated(start: number, tokens: string[]): boolean {
|
||||
|
||||
Reference in new issue
Block a user