Files
AutoZhuyin/training/common.py
T
ays953andClaude Opus 5.5 476858d841 Learn classifier weights from synthetic typing data
Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.

Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.

Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:58:08 +08:00

31 lines
1.3 KiB
Python

"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
import os
import sys
import types
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
WIN = os.path.join(ROOT, "windows")
DATA = os.path.join(WIN, "autozhuyin", "data")
RESOURCES = os.path.join(ROOT, "Resources")
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
sys.path.insert(0, WIN)
from autozhuyin.core import bopomofo, features # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
def load_models():
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
sys.exit("Run windows/tools/build_data.py first.")
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
return lexicon, english