The converter's K best paths are rescored with CKIP gpt2-tiny-chinese (GPL-3.0,
character-level, ~12M parameters, 22 MB as float16):
score = lexicon log-prob + 0.6 x LM log-prob, with the recently committed text
as context. Inference is plain Accelerate (Apple Silicon and Intel).
Modes off / saver (K=3, 15 ms) / standard (K=10, 40 ms) / quality (K=30, 120 ms)
in the input menu. A rerank over its time budget falls back to the lexicon; three
in a row pause AI for a minute. With auto-pause on, AI pauses when the Mac is hot
or the CPU is busy and drops to saver in Low Power Mode. The model loads in the
background and is not loaded at all when off.
Evaluation (training/lm/eval_rerank.py): chat 96.6% -> 97.5% chars,
Wikipedia 92.9% -> 94.7% chars (about 25-30% fewer wrong characters).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THIRD_PARTY_LICENSES.txt carries McBopomofo's full MIT notice (required when the
converted lexicon is distributed) plus notes on the public-domain word list and
PIME. It goes into AutoZhuyin.app/Contents/Resources and the Windows zip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
Windows build) instead of retraining the bigram model on every launch, and
uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.
Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.
Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>