Learn classifier weights from synthetic typing data

Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.

Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.

Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-11 08:58:08 +08:00
co-authored by Claude Opus 5.5
parent 0a7242ddb3
commit 476858d841
19 changed files with 5712 additions and 225 deletions
+24
View File
@@ -53,6 +53,30 @@ public final class EnglishModel {
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
/// Feature dict for the classifier (see Features.swift); `score` is their dot
/// product with the default weights.
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
let w = word.lowercased()
let n = Double(w.count)
var f: FeatureVector = [:]
if n <= 2 { f["lit.short"] = 1 }
if common.contains(w) {
f["lit.common"] = 1
f["lit.common_len"] = n
} else if dictionary.contains(w) {
f["lit.dict"] = 1
f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) {
f["lit.prefix_dict_len"] = n
} else {
let plaus = plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
return f
}
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
public func plausibility(_ w: String) -> Double {
let chars = ["^"] + w.map(String.init) + ["$"]