diff --git a/README.md b/README.md index 9e5c6e7..1efb5a2 100644 --- a/README.md +++ b/README.md @@ -69,12 +69,12 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安 | 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 | | 一開始稍微偏向注音 | 純數字 → 數字 | -分數高的一方勝出。按空白時,注音那邊會把空白當成一聲來評分,英文那邊當作完整單字評分。 +每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。 中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎: -- 每多切一刀扣 3 分(`segmentPenalty`) +- 每多切一刀會扣分(`seg.switch` 特徵) - 後面還接著其他段的注音段,最後一個音節必須打了聲調 -- 接在英文後面、還沒打聲調的注音段再扣 2 分(`unconfirmedSwitchPenalty`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」 +- 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」 音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。 @@ -123,6 +123,33 @@ scripts/package-windows.sh # 產生 build/AutoZhuyin-Wi > 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。 +### 訓練判斷權重 + +中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法: + +1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。 +2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。 +3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。 + +```bash +python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用) +python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘) +python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘 +python3 training/train.py --eval-only # 只評估目前的權重 +python3 training/make_parity.py # 權重或特徵改了之後都要重跑 +``` + +目前的結果(10,000 筆測試資料): + +| | 送出時整段判斷正確 | 打字過程中顯示正確 | +|---|---|---| +| 手調權重 | 97.58% | 86.15% | +| 訓練後 | **99.62%** | 87.81% | + +送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。 + +新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。 + ### 詞庫 `Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。 @@ -164,11 +191,12 @@ Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試) Lexicon.swift 詞庫 Converter.swift 音節 → 中文(Viterbi) EnglishModel.swift 英文判斷 - Classifier.swift 中英判斷 + 學習 + Features.swift 判斷用的特徵與預設權重 + Classifier.swift 中英判斷(特徵 × 權重)、切分、學習 CompositionEngine.swift 輸入法狀態機 Sources/AutoZhuyin/ macOS InputMethodKit 外殼 Tests/ Swift 單元測試 -Resources/ Info.plist、詞庫、英文常用字 +Resources/ Info.plist、詞庫、英文常用字、訓練好的權重 scripts/ 建置、安裝、打包、詞庫轉換 pkg/postinstall PKG 安裝後腳本 windows/ @@ -179,6 +207,7 @@ windows/ installer/ install.bat、uninstall.bat、給使用者的 README.txt tests/ Python 測試 tools/ 產生資料檔、圖示 +training/ 合成資料、訓練權重、Swift/Python 一致性測試資料 ``` ## 已知限制 @@ -193,7 +222,8 @@ windows/ - [x] 匯入完整詞庫 - [x] 注音順序錯誤修正 - [x] Windows 版 -- [ ] 用真實打字資料調整判斷權重 +- [x] 用資料訓練判斷權重(合成資料) +- [ ] 加入真實打字資料一起訓練 - [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」 - [ ] 用 Core ML 小模型取代手調分數,參考整句語境 - [ ] 使用者詞頻學習(選過的候選字優先) diff --git a/Resources/classifier-weights.json b/Resources/classifier-weights.json new file mode 100644 index 0000000..4fe66b3 --- /dev/null +++ b/Resources/classifier-weights.json @@ -0,0 +1,39 @@ +{ + "about": "AutoZhuyin classifier weights, learned by training/train.py", + "trained": "2026-11-11", + "examples": 40001, + "test_accuracy": { + "commit": 0.9962, + "typing": 0.8781 + }, + "weights": { + "lit.bias": 0.4276, + "lit.common": -0.3465, + "lit.common_len": 0.81, + "lit.dict": -1.7902, + "lit.dict_len": 0.6765, + "lit.digits": -2.7403, + "lit.num_digits": -0.0751, + "lit.num_others": -0.7433, + "lit.others": -1.5685, + "lit.prefix_common_len": 0.2082, + "lit.prefix_dict_len": 0.0003, + "lit.short": -0.2397, + "lit.unknown_score": 0.213, + "lit.upper": 20.0, + "seg.switch": -1.7203, + "zh.bias": 1.3522, + "zh.lexscore": 0.26, + "zh.pending_invalid": -5.0, + "zh.pending_prefix": -0.9295, + "zh.phrase": 0.4595, + "zh.reordered": -1.4348, + "zh.stray": -5.0, + "zh.toned_invalid": -5.0, + "zh.toned_valid_lex": -0.9823, + "zh.toned_valid_nolex": -1.8742, + "zh.unconfirmed": -0.7891, + "zh.untoned_mid_invalid": -5.0, + "zh.untoned_mid_valid": -2.8009 + } +} diff --git a/Sources/AutoZhuyin/InputController.swift b/Sources/AutoZhuyin/InputController.swift index 1341465..ae28167 100644 --- a/Sources/AutoZhuyin/InputController.swift +++ b/Sources/AutoZhuyin/InputController.swift @@ -26,7 +26,8 @@ enum Shared { let learned = LearnedPreferences(weights: saved) { weights in UserDefaults.standard.set(weights, forKey: learnedKey) } - return Classifier(lexicon: lexicon, english: english, learned: learned) + let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json")) + return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained) }() } diff --git a/Sources/AutoZhuyinCore/Classifier.swift b/Sources/AutoZhuyinCore/Classifier.swift index 4221625..916730b 100644 --- a/Sources/AutoZhuyinCore/Classifier.swift +++ b/Sources/AutoZhuyinCore/Classifier.swift @@ -50,48 +50,64 @@ public final class LearnedPreferences { } /// Decides whether a key sequence is Zhuyin or literal (English / numbers). +/// Every segment is scored as features · weights (see Features.swift). public final class Classifier { public let lexicon: Lexicon public let english: EnglishModel public let learned: LearnedPreferences + /// Feature weights: the hand-tuned defaults overlaid with trained ones. + public private(set) var weights: FeatureVector - /// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it. - public var zhuyinPrior = 0.5 /// How much the previous segment's mode pulls the current one. public var contextWeight = 1.5 - public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) { + public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(), + weights: FeatureVector = [:]) { self.lexicon = lexicon self.english = english self.learned = learned + self.weights = Features.defaultWeights.merging(weights) { _, new in new } } - /// - Parameters: - /// - finishing: the user pressed space. Zhuyin is scored as if the space were a - /// first-tone key; literal is scored as a completed word. - /// - previous: mode of the previously committed segment, if recent. - public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification { - var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior - var lit = literalScore(keys, complete: finishing) + /// (Zhuyin features or nil, literal features or nil) for one segment. + public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, + finishing: Bool) -> (FeatureVector?, FeatureVector?) { + (Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing), + Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing)) + } + + private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool, + previous: InputMode?, whole: Bool) -> [Double] { + let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing) + var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity + var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity switch previous { case .zhuyin?: zh += contextWeight case .literal?: lit += contextWeight case nil: break } - let learnedBias = learned.bias(for: String(keys)) - if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias } - return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit) + if whole { + let b = learned.bias(for: String(seg)) + if b > 0 { zh += b } else { lit -= b } + } + return [zh, lit] + } + + /// The whole buffer as one segment. + /// - Parameters: + /// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as + /// first tone; literal is scored as a completed word. + /// - previous: mode of the previously committed segment, if recent. + public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification { + let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true) + return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1]) } // MARK: - Segmentation - /// Cost of each switch between Zhuyin and literal inside one buffer, so that a normal - /// word or sentence is not chopped up. - public var segmentPenalty = 3.0 - /// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" → - /// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point. - /// A Zhuyin segment followed by another segment must end with a tone key. + /// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the + /// cost of each switch is the "seg.switch" feature. public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] { let n = keys.count guard n > 0 else { return [] } @@ -102,12 +118,12 @@ public final class Classifier { for i in 1...n { for j in 0.. 0, - finishing: finishing, previous: j == 0 ? previous : nil, - whole: j == 0 && i == n) - for m in 0..<2 where scores[m] > -.infinity { + let s = scores(Array(keys[j.. 0, finishing: finishing, + previous: j == 0 ? previous : nil, whole: j == 0 && i == n) + for m in 0..<2 where s[m] > -.infinity { // Adjacent segments must alternate modes; same-mode neighbours are one segment. - let total = j == 0 ? scores[m] : best[j][1 - m] + scores[m] - segmentPenalty + if j > 0 && best[j][1 - m] == -.infinity { continue } + let total = j == 0 ? s[m] : best[j][1 - m] + s[m] if total > best[i][m] { best[i][m] = total back[i][m] = j @@ -130,79 +146,4 @@ public final class Classifier { } return out.reversed() } - - /// Extra cost for a trailing Zhuyin segment after English that has no tone key yet - /// ("ji3hel" should stay 我hel, not become 我heㄠ, until a tone confirms the switch). - public var unconfirmedSwitchPenalty = 2.0 - - private func segmentScores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool, - previous: InputMode?, whole: Bool) -> [Double] { - var zh: Double - let parse = ZhuyinParser.parse(seg) - if !isLast && parse.pending != nil { - zh = -.infinity - } else { - zh = zhuyinScore(finishing && isLast ? seg + [" "] : seg) + zhuyinPrior - if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) { - zh -= unconfirmedSwitchPenalty - } - } - var lit = literalScore(seg, complete: !isLast || finishing) - switch previous { - case .zhuyin?: zh += contextWeight - case .literal?: lit += contextWeight - case nil: break - } - if whole { - let b = learned.bias(for: String(seg)) - if b > 0 { zh += b } else { lit -= b } - } - return [zh, lit] - } - - public func zhuyinScore(_ keys: [Character]) -> Double { - let parse = ZhuyinParser.parse(keys) - if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity } - var s = -5.0 * Double(parse.strayTones) - for (i, p) in parse.syllables.enumerated() { - let isLast = i == parse.syllables.count - 1 - // Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin. - if p.reordered { s -= 0.5 } - let valid = SyllableTable.isValid(p.syllable) - if p.toned { - if valid { - s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5) - } else { - s -= 5 - } - } else if isLast { - s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5 - } else { - // Untoned syllable in the middle: allowed (tone omission) but unusual. - s += valid ? -2.5 : -5 - } - } - return s - } - - public func literalScore(_ keys: [Character], complete: Bool) -> Double { - if keys.isEmpty || keys.contains(" ") { return -.infinity } - var score = 0.0 - if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English - - let letters = String(keys.filter { $0.isLetter }) - let digits = keys.filter { $0.isNumber }.count - let others = keys.count - letters.count - digits - - if letters.isEmpty { - // Pure number like "2024" (with possible "." or ","). - return score + 0.8 * Double(digits) - 1.0 * Double(others) - } - score += english.score(letters, complete: complete || digits > 0) - score -= 1.5 * Double(digits) - // `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not. - let apostrophes = keys.filter { $0 == "'" }.count - score -= 1.5 * Double(others - apostrophes) - return score - } } diff --git a/Sources/AutoZhuyinCore/EnglishModel.swift b/Sources/AutoZhuyinCore/EnglishModel.swift index 5bdd405..dc860de 100644 --- a/Sources/AutoZhuyinCore/EnglishModel.swift +++ b/Sources/AutoZhuyinCore/EnglishModel.swift @@ -53,6 +53,30 @@ public final class EnglishModel { return min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) } + /// Feature dict for the classifier (see Features.swift); `score` is their dot + /// product with the default weights. + public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector { + let w = word.lowercased() + let n = Double(w.count) + var f: FeatureVector = [:] + if n <= 2 { f["lit.short"] = 1 } + if common.contains(w) { + f["lit.common"] = 1 + f["lit.common_len"] = n + } else if dictionary.contains(w) { + f["lit.dict"] = 1 + f["lit.dict_len"] = n + } else if !complete && hasPrefix(w, in: sortedCommon) { + f["lit.prefix_common_len"] = n + } else if !complete && hasPrefix(w, in: sortedWords) { + f["lit.prefix_dict_len"] = n + } else { + let plaus = plausibility(w) + f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) + } + return f + } + /// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish). public func plausibility(_ w: String) -> Double { let chars = ["^"] + w.map(String.init) + ["$"] diff --git a/Sources/AutoZhuyinCore/Features.swift b/Sources/AutoZhuyinCore/Features.swift new file mode 100644 index 0000000..b58614d --- /dev/null +++ b/Sources/AutoZhuyinCore/Features.swift @@ -0,0 +1,138 @@ +import Foundation + +/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode +/// is the dot product of its features with the weights. `defaultWeights` reproduces the +/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json +/// (training/train.py). +/// +/// Python twin: windows/autozhuyin/core/features.py — both must produce identical +/// features (checked by ParityTests against training/parity-fixtures.json). +public typealias FeatureVector = [String: Double] + +public enum Features { + public static let defaultWeights: FeatureVector = [ + // Zhuyin + "zh.bias": 0.5, + "zh.stray": -5.0, + "zh.reordered": -0.5, + "zh.toned_valid_lex": 2.8, + "zh.toned_valid_nolex": 0.5, + "zh.toned_invalid": -5.0, + "zh.pending_prefix": 0.0, + "zh.pending_invalid": -5.0, + "zh.untoned_mid_valid": -2.5, + "zh.untoned_mid_invalid": -5.0, + "zh.unconfirmed": -2.0, + "zh.lexscore": 0.0, + "zh.phrase": 0.0, + // Literal + "lit.bias": 0.0, + "lit.upper": 20.0, + "lit.num_digits": 0.8, + "lit.num_others": -1.0, + "lit.common": 3.0, + "lit.common_len": 1.0, + "lit.dict": 1.0, + "lit.dict_len": 0.6, + "lit.prefix_common_len": 0.4, + "lit.prefix_dict_len": 0.2, + "lit.unknown_score": 1.0, + "lit.short": 0.0, + "lit.digits": -1.5, + "lit.others": -1.5, + // Segmentation + "seg.switch": -3.0, + ] + + /// Reads {"weights": {...}} from a JSON file; empty if missing or invalid. + public static func loadWeights(from url: URL?) -> FeatureVector { + guard let url, let data = try? Data(contentsOf: url), + let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let w = obj["weights"] as? [String: Double] else { return [:] } + return w + } + + public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double { + f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value } + } + + /// Features of `keys` read as Zhuyin, or nil if impossible. + public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true, + afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { + let parse = ZhuyinParser.parse(keys) + // A Zhuyin segment followed by another must end with a tone. + if !isLast && parse.pending != nil { return nil } + // Space finishes an untoned last syllable as first tone; after a toned syllable it + // just commits, so it must not count as an extra (stray) tone. + let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse + if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil } + + var f: FeatureVector = ["zh.bias": 1] + func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v } + if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) } + let last = scored.syllables.count - 1 + var tonedRun: [Syllable] = [] + for (i, p) in scored.syllables.enumerated() { + if p.reordered { add("zh.reordered") } + let valid = SyllableTable.isValid(p.syllable) + if p.toned { + if valid { + if let best = lexicon.bestScore(for: p.syllable) { + add("zh.toned_valid_lex") + add("zh.lexscore", best + 4) + } else { + add("zh.toned_valid_nolex") + } + tonedRun.append(p.syllable) + } else { + add("zh.toned_invalid") + } + } else if i == last { + add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid") + } else { + add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid") + } + } + // Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words). + var covered = [Bool](repeating: false, count: tonedRun.count) + for i in tonedRun.indices { + for n in 2...4 where i + n <= tonedRun.count { + if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty { + for k in i..<(i + n) { covered[k] = true } + } + } + } + let coveredCount = covered.filter { $0 }.count + if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) } + if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) { + add("zh.unconfirmed") + } + if afterOther { add("seg.switch") } + return f + } + + /// Features of `keys` output as typed, or nil if impossible. + public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true, + afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { + if keys.isEmpty || keys.contains(" ") { return nil } + let complete = !isLast || finishing + var f: FeatureVector = ["lit.bias": 1] + func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v } + if afterOther { add("seg.switch") } + if keys.contains(where: { $0.isUppercase }) { add("lit.upper") } + + let letters = String(keys.filter { $0.isLetter }) + let digits = keys.filter { $0.isNumber }.count + let others = keys.count - letters.count - digits + if letters.isEmpty { + if digits > 0 { add("lit.num_digits", Double(digits)) } + if others > 0 { add("lit.num_others", Double(others)) } + return f + } + for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) } + if digits > 0 { add("lit.digits", Double(digits)) } + let nonApostrophe = others - keys.filter { $0 == "'" }.count + if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) } + return f + } +} diff --git a/Tests/AutoZhuyinCoreTests/AutoZhuyinCoreTests.swift b/Tests/AutoZhuyinCoreTests/AutoZhuyinCoreTests.swift index 100fd1d..2b7112b 100644 --- a/Tests/AutoZhuyinCoreTests/AutoZhuyinCoreTests.swift +++ b/Tests/AutoZhuyinCoreTests/AutoZhuyinCoreTests.swift @@ -7,6 +7,7 @@ private let resources = URL(fileURLWithPath: #filePath) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) +private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")) final class ParserTests: XCTestCase { func testParsesNiHao() { @@ -84,7 +85,7 @@ final class ConverterTests: XCTestCase { } final class ClassifierTests: XCTestCase { - let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish) + let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights) func mode(_ s: String, finishing: Bool = false) -> InputMode { classifier.classify(Array(s), finishing: finishing).mode @@ -106,7 +107,7 @@ final class ClassifierTests: XCTestCase { func testLearningFlipsDecision() { let learned = LearnedPreferences() - let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned) + let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights) XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal) learned.record("so", as: .zhuyin) XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin) @@ -115,7 +116,7 @@ final class ClassifierTests: XCTestCase { final class EngineTests: XCTestCase { func makeEngine() -> CompositionEngine { - CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) + CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) } /// Types a string and returns everything committed. @@ -188,7 +189,7 @@ final class EngineTests: XCTestCase { final class DigitInitialTests: XCTestCase { func engine() -> CompositionEngine { - CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) + CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) } func run(_ s: String) -> (out: String, marked: String) { @@ -222,7 +223,7 @@ final class DigitInitialTests: XCTestCase { final class MixedSegmentTests: XCTestCase { func run(_ s: String) -> (out: String, marked: String) { - let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) + let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) var out = "" for c in s { let r = e.handle(c == "\n" ? .enter : .character(c)) @@ -253,3 +254,30 @@ final class MixedSegmentTests: XCTestCase { XCTAssertEqual(run("183\n").out.count, 1) } } + +/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py). +final class ParityTests: XCTestCase { + func testMatchesPythonSegmentation() throws { + let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json") + let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]] + let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights) + var mismatches: [String] = [] + for fx in fixtures { + let keys = Array(fx["keys"] as! String) + let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" } + let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map { + "\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)" + } + if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") } + } + XCTAssertGreaterThan(fixtures.count, 300) + XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n")) + } + + func testToneThenSpaceStaysChinese() { + let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) + var out = "" + for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } } + XCTAssertEqual(out.count, 1, out) + } +} diff --git a/scripts/build.sh b/scripts/build.sh index d4ddc7c..916715d 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -18,7 +18,7 @@ rm -rf "$APP" mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp Resources/Info.plist "$APP/Contents/Info.plist" -cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/" +cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/" if [ ! -f build/icon.tiff ]; then xcrun swift scripts/make-icon.swift build/icon.tiff diff --git a/training/common.py b/training/common.py new file mode 100644 index 0000000..94e6357 --- /dev/null +++ b/training/common.py @@ -0,0 +1,30 @@ +"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data.""" +import os +import sys +import types + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +WIN = os.path.join(ROOT, "windows") +DATA = os.path.join(WIN, "autozhuyin", "data") +RESOURCES = os.path.join(ROOT, "Resources") +WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json") + +sys.path.insert(0, WIN) + +from autozhuyin.core import bopomofo, features # noqa: E402 +from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402 +from autozhuyin.core.english import EnglishModel # noqa: E402 +from autozhuyin.core.lexicon import Lexicon # noqa: E402 + +core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier, + ZHUYIN=ZHUYIN, LITERAL=LITERAL) + + +def load_models(): + if not os.path.exists(os.path.join(DATA, "english-dict.txt")): + sys.exit("Run windows/tools/build_data.py first.") + lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt")) + english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"), + os.path.join(DATA, "english-dict.txt"), + os.path.join(DATA, "english-bigram.json")) + return lexicon, english diff --git a/training/make_parity.py b/training/make_parity.py new file mode 100644 index 0000000..420d587 --- /dev/null +++ b/training/make_parity.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Writes training/parity-fixtures.json: segmentations produced by the Python core +with the current weights. The Swift tests (ParityTests) must produce the same ones, +which keeps the two implementations in sync. Re-run after changing features or weights. +""" +import json +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402 +from synth import Generator # noqa: E402 + +OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json") + +HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up", + "u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"] + + +def main(): + lexicon, english = load_models() + clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH)) + gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"), + os.path.join(DATA, "english-dict.txt"), seed=7) + cases = [(k, f) for k in HAND_PICKED for f in (False, True)] + while len(cases) < 400: + ex = gen.example() + if len(ex["keys"]) <= 28: + cases.append((ex["keys"], ex["finishing"])) + cases.append((gen.prefix_of(ex)["keys"], False)) + fixtures = [{"keys": k, "finishing": f, + "segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases] + with open(OUT, "w", encoding="utf-8") as fh: + json.dump(fixtures, fh, ensure_ascii=False, indent=0) + print("wrote %d fixtures to %s" % (len(fixtures), OUT)) + + +if __name__ == "__main__": + main() diff --git a/training/parity-fixtures.json b/training/parity-fixtures.json new file mode 100644 index 0000000..253c15e --- /dev/null +++ b/training/parity-fixtures.json @@ -0,0 +1,4807 @@ +[ +{ +"keys": "ji3hello", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +8, +"literal" +] +] +}, +{ +"keys": "ji3hello", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +8, +"literal" +] +] +}, +{ +"keys": "hellosu3cl3", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +11, +"zhuyin" +] +] +}, +{ +"keys": "hellosu3cl3", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +11, +"zhuyin" +] +] +}, +{ +"keys": "su3cl3github", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +], +[ +6, +12, +"literal" +] +] +}, +{ +"keys": "su3cl3github", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +], +[ +6, +12, +"literal" +] +] +}, +{ +"keys": "ji3hel", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +6, +"literal" +] +] +}, +{ +"keys": "ji3hel", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +6, +"literal" +] +] +}, +{ +"keys": "2;3", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "2;3", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "2.5", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "2.5", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "uvl3", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "uvl3", +"finishing": true, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "so", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "so", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "up", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "up", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "u4", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "u4", +"finishing": true, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "183", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "183", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "2024", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "2024", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "Hello", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "Hello", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "don't", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "don't", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "mp3", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "mp3", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "ji3emailsu3cl3", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +8, +"literal" +], +[ +8, +14, +"zhuyin" +] +] +}, +{ +"keys": "ji3emailsu3cl3", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +8, +"literal" +], +[ +8, +14, +"zhuyin" +] +] +}, +{ +"keys": "rup wu0 wu0 fu4cp3cl3", +"finishing": false, +"segments": [ +[ +0, +21, +"zhuyin" +] +] +}, +{ +"keys": "rup wu0 wu0 fu4cp3cl3", +"finishing": true, +"segments": [ +[ +0, +21, +"zhuyin" +] +] +}, +{ +"keys": "ru3xu;6", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "ru3xu", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "qu6vul41j6eji4", +"finishing": true, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "qu", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "ru.4u.3nj4zj45. ", +"finishing": false, +"segments": [ +[ +0, +16, +"zhuyin" +] +] +}, +{ +"keys": "ru.4", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "ask", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "as", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "1j4vm ", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "1j", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "vmp ", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "vmp", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "zo4cj84", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "zo", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "y4rm,65l32l4t/6bp4", +"finishing": true, +"segments": [ +[ +0, +18, +"zhuyin" +] +] +}, +{ +"keys": "y4rm,65l32l4t/", +"finishing": false, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "5j4aj4rm4cjo4even", +"finishing": true, +"segments": [ +[ +0, +13, +"zhuyin" +], +[ +13, +17, +"literal" +] +] +}, +{ +"keys": "5j4aj4rm", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "vu;32k6j04su06fu/ ", +"finishing": false, +"segments": [ +[ +0, +18, +"zhuyin" +] +] +}, +{ +"keys": "vu;32k6j04su06f", +"finishing": false, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "zj tj ", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "zj t", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "2;3vu ", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "2", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "9143", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "914", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "d04g4ck6wu3Part", +"finishing": true, +"segments": [ +[ +0, +11, +"zhuyin" +], +[ +11, +15, +"literal" +] +] +}, +{ +"keys": "d0", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "vul3three", +"finishing": true, +"segments": [ +[ +0, +4, +"zhuyin" +], +[ +4, +9, +"literal" +] +] +}, +{ +"keys": "v", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "2842842k7x06zj4vu06", +"finishing": true, +"segments": [ +[ +0, +19, +"zhuyin" +] +] +}, +{ +"keys": "2842842k7x06zj4v", +"finishing": false, +"segments": [ +[ +0, +16, +"zhuyin" +] +] +}, +{ +"keys": "ru4su04w;6xjp6", +"finishing": true, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "ru4su04", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "numberxu.42; dj/ ", +"finishing": false, +"segments": [ +[ +0, +6, +"literal" +], +[ +6, +17, +"zhuyin" +] +] +}, +{ +"keys": "number", +"finishing": false, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "1l c06", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "1l", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "2. 1j6", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "2. 1j", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "serriped", +"finishing": true, +"segments": [ +[ +0, +8, +"literal" +] +] +}, +{ +"keys": "serri", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "rm04villainy", +"finishing": true, +"segments": [ +[ +0, +4, +"zhuyin" +], +[ +4, +12, +"literal" +] +] +}, +{ +"keys": "rm04villa", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +], +[ +4, +9, +"literal" +] +] +}, +{ +"keys": "ru3fm zp ", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "ru3fm z", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "j03", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "j", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "zo3hjo4gjo3dj4", +"finishing": false, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "zo3hjo4gj", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "t;6t;6ai4g4", +"finishing": true, +"segments": [ +[ +0, +11, +"zhuyin" +] +] +}, +{ +"keys": "t;6t;6a", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "sm3t41u4", +"finishing": true, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "s", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "messmateaul6zo ji3", +"finishing": true, +"segments": [ +[ +0, +8, +"literal" +], +[ +8, +18, +"zhuyin" +] +] +}, +{ +"keys": "messmatea", +"finishing": false, +"segments": [ +[ +0, +8, +"literal" +], +[ +8, +9, +"zhuyin" +] +] +}, +{ +"keys": "Search", +"finishing": true, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "Searc", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "284n0 g6e; fu;3", +"finishing": false, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "28", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "vup ej; bp6g.4ji3go", +"finishing": true, +"segments": [ +[ +0, +17, +"zhuyin" +], +[ +17, +19, +"literal" +] +] +}, +{ +"keys": "vup ej; bp6g.4ji3", +"finishing": false, +"segments": [ +[ +0, +17, +"zhuyin" +] +] +}, +{ +"keys": "t;6ej/ ", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "t;", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "stenosisvup 2l4", +"finishing": true, +"segments": [ +[ +0, +8, +"literal" +], +[ +8, +15, +"zhuyin" +] +] +}, +{ +"keys": "s", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "60.4", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "6", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "ru/3ru,3Hume", +"finishing": true, +"segments": [ +[ +0, +8, +"zhuyin" +], +[ +8, +12, +"literal" +] +] +}, +{ +"keys": "ru", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "d0 ", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "d0", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "Unlathed", +"finishing": true, +"segments": [ +[ +0, +8, +"literal" +] +] +}, +{ +"keys": "Unlath", +"finishing": false, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "u06xu4", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "u06x", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "fu/ fu4up", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +], +[ +7, +9, +"literal" +] +] +}, +{ +"keys": "fu/ fu4", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "better", +"finishing": true, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "be", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "9679", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "96", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "these2u653w.7", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +13, +"zhuyin" +] +] +}, +{ +"keys": "these2u653w.", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +12, +"zhuyin" +] +] +}, +{ +"keys": "coffee", +"finishing": true, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "co", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "herxk75j4yji45k3xul4", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +], +[ +3, +20, +"zhuyin" +] +] +}, +{ +"keys": "herxk", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +], +[ +3, +5, +"zhuyin" +] +] +}, +{ +"keys": "mp4h;6g.3vu0 ", +"finishing": false, +"segments": [ +[ +0, +13, +"zhuyin" +] +] +}, +{ +"keys": "mp4h;6g.3", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "neurogram", +"finishing": true, +"segments": [ +[ +0, +9, +"literal" +] +] +}, +{ +"keys": "neu", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "or", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "o", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "2ji 284cat", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +], +[ +7, +10, +"literal" +] +] +}, +{ +"keys": "2ji 28", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "54vu,4u wj3z06bj/6help", +"finishing": true, +"segments": [ +[ +0, +18, +"zhuyin" +], +[ +18, +22, +"literal" +] +] +}, +{ +"keys": "54", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "rm41ul3m06ru z83let", +"finishing": true, +"segments": [ +[ +0, +16, +"zhuyin" +], +[ +16, +19, +"literal" +] +] +}, +{ +"keys": "rm41ul3m06ru z83l", +"finishing": false, +"segments": [ +[ +0, +17, +"zhuyin" +] +] +}, +{ +"keys": "idea", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "ide", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "helloul4m06", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +11, +"zhuyin" +] +] +}, +{ +"keys": "helloul4", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +8, +"zhuyin" +] +] +}, +{ +"keys": "ru cjo4", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "ru cjo", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "wjo b;4", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "w", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "youtube2l4m3", +"finishing": false, +"segments": [ +[ +0, +7, +"literal" +], +[ +7, +12, +"zhuyin" +] +] +}, +{ +"keys": "yout", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "ru6ru; went", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +], +[ +7, +11, +"literal" +] +] +}, +{ +"keys": "ru6", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "ru4u.3rul4g cjo4co u/3Me", +"finishing": true, +"segments": [ +[ +0, +22, +"zhuyin" +], +[ +22, +24, +"literal" +] +] +}, +{ +"keys": "ru4u.3rul4g cjo4co", +"finishing": false, +"segments": [ +[ +0, +18, +"zhuyin" +] +] +}, +{ +"keys": "move", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "mov", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "z;3n ", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "z;", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "4351", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "43", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "xu;4", +"finishing": true, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "xu", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "himjo65p 2jo4bp4", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +], +[ +3, +16, +"zhuyin" +] +] +}, +{ +"keys": "h", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "ru.4283against", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +], +[ +7, +14, +"literal" +] +] +}, +{ +"keys": "ru.4283agains", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +], +[ +7, +12, +"literal" +], +[ +12, +13, +"zhuyin" +] +] +}, +{ +"keys": "j4xu3nji3a83j94vu; ", +"finishing": false, +"segments": [ +[ +0, +19, +"zhuyin" +] +] +}, +{ +"keys": "j4xu3nji3a83j94", +"finishing": false, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "rul cj04ek4tj vm,6", +"finishing": true, +"segments": [ +[ +0, +18, +"zhuyin" +] +] +}, +{ +"keys": "rul cj04", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "xu.4kindn; el ru6vu,3y94", +"finishing": true, +"segments": [ +[ +0, +4, +"zhuyin" +], +[ +4, +8, +"literal" +], +[ +8, +24, +"zhuyin" +] +] +}, +{ +"keys": "xu.4kindn; e", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +], +[ +4, +8, +"literal" +], +[ +8, +12, +"zhuyin" +] +] +}, +{ +"keys": "Annuitant56w94fup6rup g/ ", +"finishing": false, +"segments": [ +[ +0, +9, +"literal" +], +[ +9, +25, +"zhuyin" +] +] +}, +{ +"keys": "Annuitant56w94f", +"finishing": false, +"segments": [ +[ +0, +9, +"literal" +], +[ +9, +15, +"zhuyin" +] +] +}, +{ +"keys": "gk45 vu84", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "gk45 ", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "dj84bj4sitatunga", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +], +[ +7, +16, +"literal" +] +] +}, +{ +"keys": "dj8", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "are", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "a", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "ru03hl3tj6ep q04wl6", +"finishing": false, +"segments": [ +[ +0, +19, +"zhuyin" +] +] +}, +{ +"keys": "ru03hl3tj6ep ", +"finishing": false, +"segments": [ +[ +0, +13, +"zhuyin" +] +] +}, +{ +"keys": "in1j4cjo4wk4", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +], +[ +2, +12, +"zhuyin" +] +] +}, +{ +"keys": "i", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "gp 53", +"finishing": true, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "gp ", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "18 ek7", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "18 e", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "x;3", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "x", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "2l42k6ej0 ", +"finishing": false, +"segments": [ +[ +0, +10, +"zhuyin" +] +] +}, +{ +"keys": "2", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "ej84su06", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "ej84s", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "youtubecj/6t86ej/4g/ j03", +"finishing": true, +"segments": [ +[ +0, +7, +"literal" +], +[ +7, +24, +"zhuyin" +] +] +}, +{ +"keys": "youtubecj/6t86ej/4g/", +"finishing": false, +"segments": [ +[ +0, +7, +"literal" +], +[ +7, +20, +"zhuyin" +] +] +}, +{ +"keys": "youtube", +"finishing": true, +"segments": [ +[ +0, +7, +"literal" +] +] +}, +{ +"keys": "youtub", +"finishing": false, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "said", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "sai", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "u06ru.4u42u03u.3g4you", +"finishing": true, +"segments": [ +[ +0, +18, +"zhuyin" +], +[ +18, +21, +"literal" +] +] +}, +{ +"keys": "u06ru.4u", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "284tj04", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "2", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "5k4tjp fu. yji3g4", +"finishing": false, +"segments": [ +[ +0, +17, +"zhuyin" +] +] +}, +{ +"keys": "5k4tjp f", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "bad", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "b", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "h/6tj 1j4fm/6because", +"finishing": true, +"segments": [ +[ +0, +13, +"zhuyin" +], +[ +13, +20, +"literal" +] +] +}, +{ +"keys": "h/6tj 1j", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "su6g8 people", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +], +[ +6, +12, +"literal" +] +] +}, +{ +"keys": "su6g8 p", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "nj/4tj3m6t/6zp4", +"finishing": false, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "nj", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "want", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "wa", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "Put", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "Pu", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "3566", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "356", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "input", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "inpu", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "wu6n3ru.3Password", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +], +[ +9, +17, +"literal" +] +] +}, +{ +"keys": "wu6n3ru.3Passwo", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +], +[ +9, +15, +"literal" +] +] +}, +{ +"keys": "3452", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "3", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "send", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "se", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "xm3z;3zj6cp31j6", +"finishing": true, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "xm3z;3zj", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "1i wk4x06", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "1i wk4x0", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "of", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "o", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "18 n4u n dl3", +"finishing": false, +"segments": [ +[ +0, +12, +"zhuyin" +] +] +}, +{ +"keys": "18", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "on", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "o", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "u06xji6", +"finishing": true, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "u0", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "rup bj/6ru,4ej0 m06", +"finishing": false, +"segments": [ +[ +0, +19, +"zhuyin" +] +] +}, +{ +"keys": "rup bj/6ru,4ej", +"finishing": false, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "reconcede53tj y94h3q84", +"finishing": false, +"segments": [ +[ +0, +9, +"literal" +], +[ +9, +22, +"zhuyin" +] +] +}, +{ +"keys": "reconcede53tj y", +"finishing": false, +"segments": [ +[ +0, +9, +"literal" +], +[ +9, +15, +"zhuyin" +] +] +}, +{ +"keys": "input", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "in", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "dj94xk4e; wu,3", +"finishing": false, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "dj94xk4e; w", +"finishing": false, +"segments": [ +[ +0, +11, +"zhuyin" +] +] +}, +{ +"keys": "ej3", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "ej", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "ao6", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "ao", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "link", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "li", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "Have", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "Ha", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "meet", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "mee", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "n ", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "n", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "wj6ru/4xu;6g0 n4", +"finishing": true, +"segments": [ +[ +0, +16, +"zhuyin" +] +] +}, +{ +"keys": "wj6ru/4xu;", +"finishing": false, +"segments": [ +[ +0, +10, +"zhuyin" +] +] +}, +{ +"keys": "8247", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "824", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "Dicoccous", +"finishing": true, +"segments": [ +[ +0, +9, +"literal" +] +] +}, +{ +"keys": "Dico", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "ru04ejo yjo4", +"finishing": false, +"segments": [ +[ +0, +12, +"zhuyin" +] +] +}, +{ +"keys": "r", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "y3m04", +"finishing": true, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "y3m0", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "vu;4s84", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "vu;4", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "1786", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "1", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "g/ zp fu6u.3fm4", +"finishing": false, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "g/ zp fu6u.3fm", +"finishing": false, +"segments": [ +[ +0, +14, +"zhuyin" +] +] +}, +{ +"keys": "au4-6bj4check", +"finishing": true, +"segments": [ +[ +0, +8, +"zhuyin" +], +[ +8, +13, +"literal" +] +] +}, +{ +"keys": "a", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "city", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "cit", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "Came", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "Ca", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "must", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "mus", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "ej32j/ fu3cj/ a87", +"finishing": false, +"segments": [ +[ +0, +17, +"zhuyin" +] +] +}, +{ +"keys": "ej32j/ f", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "Hello", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "Hell", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "q041u,6w.6y bp6revile", +"finishing": true, +"segments": [ +[ +0, +15, +"zhuyin" +], +[ +15, +21, +"literal" +] +] +}, +{ +"keys": "q041u,6w.", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "dk3s06g/ ", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "d", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "c.4bp6ru6is", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +], +[ +9, +11, +"literal" +] +] +}, +{ +"keys": "c.4bp6ru6", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "5p ejo cj84", +"finishing": false, +"segments": [ +[ +0, +11, +"zhuyin" +] +] +}, +{ +"keys": "5p e", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "it", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "i", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "mottoless", +"finishing": true, +"segments": [ +[ +0, +9, +"literal" +] +] +}, +{ +"keys": "m", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "tj airru,42ul4", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +6, +"literal" +], +[ +6, +14, +"zhuyin" +] +] +}, +{ +"keys": "tj ", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "yjo4rup4ru3su062u4z; ", +"finishing": false, +"segments": [ +[ +0, +21, +"zhuyin" +] +] +}, +{ +"keys": "yjo4ru", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "g4u4cjo4", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "g4u4", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "up m,4j06", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "up", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "a04aul4rul4", +"finishing": false, +"segments": [ +[ +0, +11, +"zhuyin" +] +] +}, +{ +"keys": "a04aul4ru", +"finishing": false, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "5843", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "58", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "applecj84", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +9, +"zhuyin" +] +] +}, +{ +"keys": "appl", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "1u04ru, vu04g/ ", +"finishing": false, +"segments": [ +[ +0, +15, +"zhuyin" +] +] +}, +{ +"keys": "1u04ru, vu04g", +"finishing": false, +"segments": [ +[ +0, +13, +"zhuyin" +] +] +}, +{ +"keys": "Mind", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "M", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "feeld042k6xu/6cjp6xu.4", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +], +[ +4, +22, +"zhuyin" +] +] +}, +{ +"keys": "feeld042k", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +], +[ +4, +9, +"zhuyin" +] +] +}, +{ +"keys": "g6e; there", +"finishing": true, +"segments": [ +[ +0, +5, +"zhuyin" +], +[ +5, +10, +"literal" +] +] +}, +{ +"keys": "g6", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "zp ", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "zp", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "u g4ru45j/3cardona1o3", +"finishing": false, +"segments": [ +[ +0, +11, +"zhuyin" +], +[ +11, +18, +"literal" +], +[ +18, +21, +"zhuyin" +] +] +}, +{ +"keys": "u g4ru45j/", +"finishing": false, +"segments": [ +[ +0, +10, +"zhuyin" +] +] +}, +{ +"keys": "proxysmfu353u 2j04", +"finishing": false, +"segments": [ +[ +0, +7, +"literal" +], +[ +7, +18, +"zhuyin" +] +] +}, +{ +"keys": "proxysmfu353u 2j0", +"finishing": false, +"segments": [ +[ +0, +7, +"literal" +], +[ +7, +17, +"zhuyin" +] +] +}, +{ +"keys": "x06", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "x", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "do", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "d", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "1377", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "1", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "su3cl3gp6", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "su3", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "where", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "w", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "e0 5.3his", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +], +[ +6, +9, +"literal" +] +] +}, +{ +"keys": "e0 5", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "2/ rm4", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "2/ rm", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "ru6", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "r", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "place", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "p", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "up3m/4", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +}, +{ +"keys": "u", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "applew.6nj4ao6vu0654", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +20, +"zhuyin" +] +] +}, +{ +"keys": "applew.6n", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +], +[ +5, +9, +"zhuyin" +] +] +}, +{ +"keys": "z06together", +"finishing": true, +"segments": [ +[ +0, +3, +"zhuyin" +], +[ +3, +11, +"literal" +] +] +}, +{ +"keys": "z", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "1j4eji4", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "1", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "for", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "fo", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "together", +"finishing": true, +"segments": [ +[ +0, +8, +"literal" +] +] +}, +{ +"keys": "to", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "53au/62l4vu/4qu/6w96", +"finishing": true, +"segments": [ +[ +0, +20, +"zhuyin" +] +] +}, +{ +"keys": "53au/62l4vu/4", +"finishing": false, +"segments": [ +[ +0, +13, +"zhuyin" +] +] +}, +{ +"keys": "g6ej; docs", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +], +[ +6, +10, +"literal" +] +] +}, +{ +"keys": "g6ej;", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "keep", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "k", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "m3g4ru,6fu/4104wj6", +"finishing": false, +"segments": [ +[ +0, +18, +"zhuyin" +] +] +}, +{ +"keys": "m3g4ru,6fu/4", +"finishing": false, +"segments": [ +[ +0, +12, +"zhuyin" +] +] +}, +{ +"keys": "night", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "n", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "can", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "ca", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "docs", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "d", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "New", +"finishing": true, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "Ne", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "2. ao6u.3see", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +], +[ +9, +12, +"literal" +] +] +}, +{ +"keys": "2.", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "n4vu;4u3", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "n4v", +"finishing": false, +"segments": [ +[ +0, +3, +"zhuyin" +] +] +}, +{ +"keys": "yji4", +"finishing": true, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "y", +"finishing": false, +"segments": [ +[ +0, +1, +"zhuyin" +] +] +}, +{ +"keys": "between", +"finishing": true, +"segments": [ +[ +0, +7, +"literal" +] +] +}, +{ +"keys": "betwe", +"finishing": false, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "g6ru0 rad", +"finishing": true, +"segments": [ +[ +0, +6, +"zhuyin" +], +[ +6, +9, +"literal" +] +] +}, +{ +"keys": "g6ru", +"finishing": false, +"segments": [ +[ +0, +4, +"zhuyin" +] +] +}, +{ +"keys": "Git2u4cl62ji6204196", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +], +[ +3, +19, +"zhuyin" +] +] +}, +{ +"keys": "Git2u4c", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +], +[ +3, +7, +"zhuyin" +] +] +}, +{ +"keys": "u;3g/ nj/4n3gp4y;4gp zp4", +"finishing": true, +"segments": [ +[ +0, +24, +"zhuyin" +] +] +}, +{ +"keys": "u;3g/ nj/4n3g", +"finishing": false, +"segments": [ +[ +0, +13, +"zhuyin" +] +] +}, +{ +"keys": "turn", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "tur", +"finishing": false, +"segments": [ +[ +0, +3, +"literal" +] +] +}, +{ +"keys": "okay", +"finishing": true, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "ok", +"finishing": false, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "Ai", +"finishing": true, +"segments": [ +[ +0, +2, +"literal" +] +] +}, +{ +"keys": "A", +"finishing": false, +"segments": [ +[ +0, +1, +"literal" +] +] +}, +{ +"keys": "today", +"finishing": true, +"segments": [ +[ +0, +5, +"literal" +] +] +}, +{ +"keys": "toda", +"finishing": false, +"segments": [ +[ +0, +4, +"literal" +] +] +}, +{ +"keys": "junipercl3gp ", +"finishing": false, +"segments": [ +[ +0, +7, +"literal" +], +[ +7, +13, +"zhuyin" +] +] +}, +{ +"keys": "junipe", +"finishing": false, +"segments": [ +[ +0, +6, +"literal" +] +] +}, +{ +"keys": "vul3q/6u.3gk4cjo4", +"finishing": true, +"segments": [ +[ +0, +17, +"zhuyin" +] +] +}, +{ +"keys": "vul3q", +"finishing": false, +"segments": [ +[ +0, +5, +"zhuyin" +] +] +}, +{ +"keys": "-41o4vu03", +"finishing": true, +"segments": [ +[ +0, +9, +"zhuyin" +] +] +}, +{ +"keys": "-41o4vu", +"finishing": false, +"segments": [ +[ +0, +7, +"zhuyin" +] +] +}, +{ +"keys": "1u4n3time", +"finishing": true, +"segments": [ +[ +0, +5, +"zhuyin" +], +[ +5, +9, +"literal" +] +] +}, +{ +"keys": "1u", +"finishing": false, +"segments": [ +[ +0, +2, +"zhuyin" +] +] +}, +{ +"keys": "e94u4jp6", +"finishing": false, +"segments": [ +[ +0, +8, +"zhuyin" +] +] +}, +{ +"keys": "e94u4j", +"finishing": false, +"segments": [ +[ +0, +6, +"zhuyin" +] +] +} +] \ No newline at end of file diff --git a/training/synth.py b/training/synth.py new file mode 100644 index 0000000..e4b5a8a --- /dev/null +++ b/training/synth.py @@ -0,0 +1,132 @@ +"""Synthetic typing data for the Zhuyin / literal classifier. + +Each example is what sits in the input buffer when the user commits it, with the +intended mode of every key: + {"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"} +labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter. + +Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys; +English from the common-word list and the dictionary. Real typing commits English at +a space, so a buffer holds at most one English word next to Chinese. +""" +import random + +from common import core + +bopomofo = core.bopomofo + +KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()} +TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"} + + +def reading_to_keys(reading): + """'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable.""" + out = [] + for syl in reading.split("-"): + tone = " " + if syl and syl[-1] in TONE_KEY: + tone = TONE_KEY[syl[-1]] + syl = syl[:-1] + if not syl or syl not in bopomofo.VALID: + return None + out.append("".join(KEY_OF[c] for c in syl) + tone) + return "".join(out) + + +def _is_cjk(s): + return all(0x4E00 <= ord(c) <= 0x9FFF for c in s) + + +class Generator: + def __init__(self, lexicon_path, common_path, dict_path, seed=0): + self.rng = random.Random(seed) + phrases, weights = [], [] + with open(lexicon_path, encoding="utf-8") as f: + for line in f: + cols = line.split() + if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]): + continue + n = cols[0].count("-") + 1 + if n > 4: + continue + keys = reading_to_keys(cols[0]) + if keys is None: + continue + phrases.append((cols[1], keys)) + weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency + self.phrases = phrases + self.phrase_cum = _cumulative(weights) + self.common = [w.strip() for w in open(common_path, encoding="utf-8") + if w.strip() and not w.startswith("#")] + self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8") + if 3 <= len(w.strip()) <= 9] + + # --- pieces --- + + def chinese(self): + n = self.rng.choice([1, 1, 2, 2, 3]) + picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n) + return "".join(keys for _, keys in picked) + + def english(self): + w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words) + if self.rng.random() < 0.1: + w = w[:1].upper() + w[1:] + return w + + def number(self): + r = self.rng.random() + if r < 0.4: + return str(self.rng.randint(0, 9999)) + if r < 0.7: + return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30])) + return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9)) + + # --- examples --- + + def example(self): + r = self.rng.random() + if r < 0.40: + parts, kind = [("z", self.chinese())], "zh" + elif r < 0.65: + parts, kind = [("l", self.english())], "en" + elif r < 0.80: + parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en" + elif r < 0.93: + parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh" + elif r < 0.96: + parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh" + else: + parts, kind = [("l", self.number())], "num" + keys = "".join(p for _, p in parts) + labels = "".join(m * len(p) for m, p in parts) + if parts[-1][0] == "l": + finishing = True # English / numbers end with a space + else: + # Chinese ends with Enter, or space after a finished (toned) syllable. + finishing = keys[-1] != " " and self.rng.random() < 0.5 + return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind} + + def prefix_of(self, ex): + """The buffer while still typing: a random cut, not finishing.""" + cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1 + return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False, + "kind": ex["kind"] + "/prefix"} + + +def _cumulative(ws): + out, t = [], 0.0 + for w in ws: + t += w + out.append(t) + return out + + +def gold_segments(labels): + """'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]""" + segs, start = [], 0 + for i in range(1, len(labels) + 1): + if i == len(labels) or labels[i] != labels[start]: + segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL)) + start = i + return segs diff --git a/training/train.py b/training/train.py new file mode 100644 index 0000000..ce2d9b4 --- /dev/null +++ b/training/train.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Learns the classifier weights from synthetic typing data (averaged structured +perceptron over the same segmentation DP the input method runs) and reports +accuracy against the hand-tuned defaults. + +Usage (from the repo root, after windows/tools/build_data.py): + python3 training/train.py # train, evaluate, write Resources/classifier-weights.json + python3 training/train.py --eval-only # just evaluate the current weights file +""" +import argparse +import datetime +import json +import os +import random +import sys +import time +from collections import defaultdict + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402 +from synth import Generator, gold_segments # noqa: E402 + +MAX_KEYS = 28 +FIXED = {"lit.upper"} # hard rule (Shift = English), not learned + + +def make_set(gen, n): + out = [] + while len(out) < 2 * n: + ex = gen.example() + if len(ex["keys"]) > MAX_KEYS: + continue + out.append(ex) + if len(ex["keys"]) > 1: + out.append(gen.prefix_of(ex)) + return out + + +def labels_of(segs, n): + lab = [""] * n + for s, e, m in segs: + for i in range(s, e): + lab[i] = "z" if m == core.ZHUYIN else "l" + return "".join(lab) + + +def seg_features(clf, keys, segs, finishing): + """Summed features of a segmentation, or None if any segment is impossible.""" + total = defaultdict(float) + n = len(keys) + for idx, (s, e, m) in enumerate(segs): + fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing) + f = fz if m == core.ZHUYIN else fl + if f is None: + return None + for k, v in f.items(): + total[k] += v + return total + + +def evaluate(clf, weights, data): + by_kind = defaultdict(lambda: [0, 0]) + for ex in data: + segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights) + ok = labels_of(segs, len(ex["keys"])) == ex["labels"] + bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]] + bucket[0] += ok + bucket[1] += 1 + commit = [v for k, v in by_kind.items() if k != "prefix"] + summary = { + "commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)), + "prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]), + "by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())}, + } + return summary + + +def show(name, s): + kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items()) + print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds)) + + +def train(clf, data, epochs, lr, seed): + rng = random.Random(seed) + w = dict(clf.weights) + total = defaultdict(float) + steps = 0 + for epoch in range(epochs): + rng.shuffle(data) + mistakes = 0 + t0 = time.time() + for ex in data: + keys = list(ex["keys"]) + pred = clf.segment(keys, ex["finishing"], None, w) + if labels_of(pred, len(keys)) != ex["labels"]: + gold = gold_segments(ex["labels"]) + fg = seg_features(clf, keys, gold, ex["finishing"]) + fp = seg_features(clf, keys, pred, ex["finishing"]) + if fg is not None and fp is not None: + mistakes += 1 + scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0) + for k in set(fg) | set(fp): + if k not in FIXED: + w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0)) + steps += 1 + for k, v in w.items(): + total[k] += v + print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True) + return {k: round(v / steps, 4) for k, v in total.items()} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)") + ap.add_argument("--test", type=int, default=1500) + ap.add_argument("--epochs", type=int, default=6) + ap.add_argument("--lr", type=float, default=0.1) + ap.add_argument("--eval-only", action="store_true") + args = ap.parse_args() + + lexicon, english = load_models() + paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"), + os.path.join(DATA, "english-dict.txt")) + test = make_set(Generator(*paths, seed=2), args.test) + + clf = core.Classifier(lexicon, english) # default (hand-tuned) weights + show("default", evaluate(clf, clf.weights, test)) + if os.path.exists(WEIGHTS_PATH): + current = dict(clf.weights) + current.update(core.Classifier.load_weights(WEIGHTS_PATH)) + show("current", evaluate(clf, current, test)) + if args.eval_only: + return + + data = make_set(Generator(*paths, seed=1), args.train) + print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True) + learned = train(clf, data, args.epochs, args.lr, seed=3) + result = evaluate(clf, learned, test) + show("learned", result) + + with open(WEIGHTS_PATH, "w", encoding="utf-8") as f: + json.dump({ + "about": "AutoZhuyin classifier weights, learned by training/train.py", + "trained": datetime.date.today().isoformat(), + "examples": len(data), + "test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)}, + "weights": dict(sorted(learned.items())), + }, f, ensure_ascii=False, indent=2) + f.write("\n") + print("wrote", WEIGHTS_PATH) + + +if __name__ == "__main__": + main() diff --git a/windows/autozhuyin/autozhuyin_ime.py b/windows/autozhuyin/autozhuyin_ime.py index ede1619..cc485d0 100644 --- a/windows/autozhuyin/autozhuyin_ime.py +++ b/windows/autozhuyin/autozhuyin_ime.py @@ -52,7 +52,8 @@ def _shared_classifier(): weights = json.load(f) except (OSError, ValueError): pass - _classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned)) + trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json")) + _classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained) return _classifier diff --git a/windows/autozhuyin/core/classifier.py b/windows/autozhuyin/core/classifier.py index caafbd8..960ff74 100644 --- a/windows/autozhuyin/core/classifier.py +++ b/windows/autozhuyin/core/classifier.py @@ -1,5 +1,7 @@ """Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift.""" -from . import bopomofo +import json + +from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features ZHUYIN = "zhuyin" LITERAL = "literal" @@ -28,84 +30,35 @@ class LearnedPreferences: class Classifier: - zhuyin_prior = 0.5 context_weight = 1.5 - def __init__(self, lexicon, english, learned=None): + def __init__(self, lexicon, english, learned=None, weights=None): self.lexicon = lexicon self.english = english self.learned = learned or LearnedPreferences() + self.weights = dict(DEFAULT_WEIGHTS) + if weights: + self.weights.update(weights) - def classify(self, keys, finishing=False, previous=None): - """Returns (mode, zhuyin_score, literal_score).""" - keys = list(keys) - zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior - lit = self.literal_score(keys, complete=finishing) - if previous == ZHUYIN: - zh += self.context_weight - elif previous == LITERAL: - lit += self.context_weight - b = self.learned.bias("".join(keys)) - if b > 0: - zh += b - else: - lit -= b - return (ZHUYIN if zh >= lit else LITERAL), zh, lit + @classmethod + def load_weights(cls, path): + """Reads a weights JSON file ({"weights": {...}}); returns {} if missing.""" + try: + with open(path, encoding="utf-8") as f: + return json.load(f).get("weights", {}) + except (OSError, ValueError): + return {} - def mode(self, keys, finishing=False, previous=None): - return self.classify(keys, finishing, previous)[0] + def segment_features(self, seg, is_last, after_other, finishing): + """(zhuyin features or None, literal features or None) for one segment.""" + seg = list(seg) + return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing), + literal_features(seg, self.english, is_last, after_other, finishing)) - # --- segmentation --- - - segment_penalty = 3.0 - unconfirmed_switch_penalty = 2.0 - - def segment(self, keys, finishing=False, previous=None): - """Splits keys into alternating segments: [(start, end, mode)], e.g. - "ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over - split points; a Zhuyin segment followed by another must end with a tone.""" - keys = list(keys) - n = len(keys) - if n == 0: - return [] - modes = (ZHUYIN, LITERAL) - best = [[NEG_INF, NEG_INF] for _ in range(n + 1)] - back = [[-1, -1] for _ in range(n + 1)] - for i in range(1, n + 1): - for j in range(i): - scores = self._segment_scores(keys[j:i], i == n, j > 0, finishing, - previous if j == 0 else None, j == 0 and i == n) - for m in (0, 1): - if scores[m] == NEG_INF: - continue - if j == 0: - total = scores[m] - else: - if best[j][1 - m] == NEG_INF: - continue - total = best[j][1 - m] + scores[m] - self.segment_penalty - if total > best[i][m]: - best[i][m] = total - back[i][m] = j - if best[n][0] == NEG_INF and best[n][1] == NEG_INF: - return [(0, n, self.mode(keys, finishing, previous))] - m = 0 if best[n][0] >= best[n][1] else 1 - out, i = [], n - while i > 0: - j = back[i][m] - out.append((j, i, modes[m])) - i, m = j, 1 - m - return out[::-1] - - def _segment_scores(self, seg, is_last, after_other, finishing, previous, whole): - parse = bopomofo.parse(seg) - if not is_last and parse.pending is not None: - zh = NEG_INF - else: - zh = self.zhuyin_score(seg + [" "] if finishing and is_last else seg) + self.zhuyin_prior - if is_last and after_other and not finishing and not any(s.toned for s in parse.syllables): - zh -= self.unconfirmed_switch_penalty - lit = self.literal_score(seg, complete=(not is_last) or finishing) + def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights): + fz, fl = self.segment_features(seg, is_last, after_other, finishing) + zh = dot(weights, fz) if fz is not None else NEG_INF + lit = dot(weights, fl) if fl is not None else NEG_INF if previous == ZHUYIN: zh += self.context_weight elif previous == LITERAL: @@ -118,40 +71,50 @@ class Classifier: lit -= b return zh, lit - def zhuyin_score(self, keys): - p = bopomofo.parse(keys) - if p.foreign_keys > 0 or not p.syllables: - return NEG_INF - s = -5.0 * p.stray_tones - last = len(p.syllables) - 1 - for i, ps in enumerate(p.syllables): - if ps.reordered: - s -= 0.5 - valid = bopomofo.is_valid(ps.syllable) - if ps.toned: - if valid: - s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5) - else: - s -= 5 - elif i == last: - s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5 - else: - s += -2.5 if valid else -5 - return s + def classify(self, keys, finishing=False, previous=None): + """Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score).""" + zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights) + return (ZHUYIN if zh >= lit else LITERAL), zh, lit - def literal_score(self, keys, complete): - if not keys or " " in keys: - return NEG_INF - score = 0.0 - if any(c.isupper() for c in keys): - score += 20 - letters = "".join(c for c in keys if c.isalpha()) - digits = sum(1 for c in keys if c.isdigit()) - others = len(keys) - len(letters) - digits - if not letters: - return score + 0.8 * digits - 1.0 * others - score += self.english.score(letters, complete or digits > 0) - score -= 1.5 * digits - apostrophes = keys.count("'") - score -= 1.5 * (others - apostrophes) - return score + def mode(self, keys, finishing=False, previous=None): + return self.classify(keys, finishing, previous)[0] + + # --- segmentation --- + + def segment(self, keys, finishing=False, previous=None, weights=None): + """Splits keys into alternating segments: [(start, end, mode)], e.g. + "ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over + split points; each segment is scored by features x weights.""" + weights = weights or self.weights + keys = list(keys) + n = len(keys) + if n == 0: + return [] + modes = (ZHUYIN, LITERAL) + best = [[NEG_INF, NEG_INF] for _ in range(n + 1)] + back = [[-1, -1] for _ in range(n + 1)] + for i in range(1, n + 1): + for j in range(i): + scores = self._scores(keys[j:i], i == n, j > 0, finishing, + previous if j == 0 else None, j == 0 and i == n, weights) + for m in (0, 1): + if scores[m] == NEG_INF: + continue + if j == 0: + total = scores[m] + else: + if best[j][1 - m] == NEG_INF: + continue + total = best[j][1 - m] + scores[m] + if total > best[i][m]: + best[i][m] = total + back[i][m] = j + if best[n][0] == NEG_INF and best[n][1] == NEG_INF: + return [(0, n, self.mode(keys, finishing, previous))] + m = 0 if best[n][0] >= best[n][1] else 1 + out, i = [], n + while i > 0: + j = back[i][m] + out.append((j, i, modes[m])) + i, m = j, 1 - m + return out[::-1] diff --git a/windows/autozhuyin/core/english.py b/windows/autozhuyin/core/english.py index ebdeeee..7a6a4e5 100644 --- a/windows/autozhuyin/core/english.py +++ b/windows/autozhuyin/core/english.py @@ -56,6 +56,29 @@ class EnglishModel: plaus = self.plausibility(w) return min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) + def word_features(self, word, complete): + """Feature dict for the classifier (see features.py); score() is their + dot product with the default weights.""" + w = word.lower() + n = float(len(w)) + f = {} + if n <= 2: + f["lit.short"] = 1.0 + if w in self.common: + f["lit.common"] = 1.0 + f["lit.common_len"] = n + elif w in self.dictionary: + f["lit.dict"] = 1.0 + f["lit.dict_len"] = n + elif not complete and _has_prefix(w, self.sorted_common): + f["lit.prefix_common_len"] = n + elif not complete and _has_prefix(w, self.sorted_words): + f["lit.prefix_dict_len"] = n + else: + plaus = self.plausibility(w) + f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) + return f + def plausibility(self, w): chars = ["^"] + list(w) + ["$"] total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars))) diff --git a/windows/autozhuyin/core/features.py b/windows/autozhuyin/core/features.py new file mode 100644 index 0000000..b5fc3f3 --- /dev/null +++ b/windows/autozhuyin/core/features.py @@ -0,0 +1,131 @@ +"""Segment features for the Zhuyin / literal classifier. + +A segment's score in each mode is the dot product of its feature dict with a weight +dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights +(training/train.py → Resources/classifier-weights.json) replace them. + +Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical +features; Tests/.../ParityTests checks this against training/parity-fixtures.json. +""" +from . import bopomofo + +DEFAULT_WEIGHTS = { + # Zhuyin + "zh.bias": 0.5, + "zh.stray": -5.0, + "zh.reordered": -0.5, + "zh.toned_valid_lex": 2.8, + "zh.toned_valid_nolex": 0.5, + "zh.toned_invalid": -5.0, + "zh.pending_prefix": 0.0, + "zh.pending_invalid": -5.0, + "zh.untoned_mid_valid": -2.5, + "zh.untoned_mid_invalid": -5.0, + "zh.unconfirmed": -2.0, + "zh.lexscore": 0.0, + "zh.phrase": 0.0, + # Literal + "lit.bias": 0.0, + "lit.upper": 20.0, + "lit.num_digits": 0.8, + "lit.num_others": -1.0, + "lit.common": 3.0, + "lit.common_len": 1.0, + "lit.dict": 1.0, + "lit.dict_len": 0.6, + "lit.prefix_common_len": 0.4, + "lit.prefix_dict_len": 0.2, + "lit.unknown_score": 1.0, + "lit.short": 0.0, + "lit.digits": -1.5, + "lit.others": -1.5, + # Segmentation + "seg.switch": -3.0, +} + + +def dot(weights, feats): + return sum(weights.get(k, 0.0) * v for k, v in feats.items()) + + +def _add(f, k, v=1.0): + f[k] = f.get(k, 0.0) + v + + +def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False): + """Features of `keys` read as Zhuyin, or None if impossible.""" + parse = bopomofo.parse(keys) + if not is_last and parse.pending is not None: + return None # a Zhuyin segment followed by another must end with a tone + # Pressing space finishes an untoned last syllable as first tone; after a toned + # syllable it just commits, so it must not count as an extra (stray) tone. + scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse + if scored.foreign_keys > 0 or not scored.syllables: + return None + f = {"zh.bias": 1.0} + if scored.stray_tones: + _add(f, "zh.stray", scored.stray_tones) + last = len(scored.syllables) - 1 + toned_run = [] + for i, ps in enumerate(scored.syllables): + if ps.reordered: + _add(f, "zh.reordered") + valid = bopomofo.is_valid(ps.syllable) + if ps.toned: + if valid: + best = lexicon.best_score(ps.syllable) + if best is not None: + _add(f, "zh.toned_valid_lex") + _add(f, "zh.lexscore", best + 4.0) + else: + _add(f, "zh.toned_valid_nolex") + toned_run.append(ps.syllable) + else: + _add(f, "zh.toned_invalid") + elif i == last: + _add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid") + else: + _add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid") + # Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words). + covered = [False] * len(toned_run) + for i in range(len(toned_run)): + for n in (2, 3, 4): + if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]): + for k in range(i, i + n): + covered[k] = True + if any(covered): + _add(f, "zh.phrase", sum(covered)) + if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables): + _add(f, "zh.unconfirmed") + if after_other: + _add(f, "seg.switch") + return f + + +def literal_features(keys, english, is_last=True, after_other=False, finishing=False): + """Features of `keys` output as typed, or None if impossible.""" + if not keys or " " in keys: + return None + complete = (not is_last) or finishing + f = {"lit.bias": 1.0} + if after_other: + _add(f, "seg.switch") + if any(c.isupper() for c in keys): + _add(f, "lit.upper") + letters = "".join(c for c in keys if c.isalpha()) + digits = sum(1 for c in keys if c.isdigit()) + others = len(keys) - len(letters) - digits + if not letters: + if digits: + _add(f, "lit.num_digits", digits) + if others: + _add(f, "lit.num_others", others) + return f + for k, v in english.word_features(letters, complete or digits > 0).items(): + _add(f, k, v) + if digits: + _add(f, "lit.digits", digits) + non_apostrophe = others - keys.count("'") + if non_apostrophe: + _add(f, "lit.others", non_apostrophe) + return f diff --git a/windows/tests/test_core.py b/windows/tests/test_core.py index fec03ee..edf7d67 100644 --- a/windows/tests/test_core.py +++ b/windows/tests/test_core.py @@ -21,6 +21,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt")) ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"), os.path.join(DATA, "english-dict.txt"), os.path.join(DATA, "english-bigram.json")) +WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json")) LOAD_TIME = time.time() - _start @@ -34,7 +35,7 @@ def convert(keys): def make_engine(): - return CompositionEngine(Classifier(LEX, ENG)) + return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS)) def type_keys(s, e): @@ -86,7 +87,7 @@ class ConverterTests(unittest.TestCase): class ClassifierTests(unittest.TestCase): - c = Classifier(LEX, ENG) + c = Classifier(LEX, ENG, weights=WEIGHTS) def test_zhuyin(self): for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]: @@ -101,7 +102,7 @@ class ClassifierTests(unittest.TestCase): def test_learning(self): learned = LearnedPreferences() - c = Classifier(LEX, ENG, learned) + c = Classifier(LEX, ENG, learned, WEIGHTS) self.assertEqual(c.mode(list("so"), finishing=True), LITERAL) learned.record("so", ZHUYIN) self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN) @@ -150,6 +151,9 @@ class EngineTests(unittest.TestCase): self.assertEqual(type_keys("2j/ \n", make_engine()), "東") self.assertEqual(type_keys("2.5\n", make_engine()), "2.5") + def test_tone_then_space_stays_chinese(self): + self.assertEqual(len(type_keys("u4 ", make_engine())), 1) + def test_out_of_order_converts(self): self.assertEqual(type_keys("uvl3\n", make_engine()), "小") diff --git a/windows/tools/build_data.py b/windows/tools/build_data.py index 41c5e6c..8d9fbde 100755 --- a/windows/tools/build_data.py +++ b/windows/tools/build_data.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Fills windows/autozhuyin/data/ from the shared Mac resources. -- lexicon.txt, english-common.txt: copied from Resources/ +- lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/ - english-dict.txt: filtered from a system word list (default /usr/share/dict/words, Webster's 2nd edition, public domain), since Windows has no such file - english-bigram.json: precomputed letter-bigram table @@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402 data = WIN / "autozhuyin" / "data" data.mkdir(parents=True, exist_ok=True) -for name in ("lexicon.txt", "english-common.txt"): +for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"): shutil.copy(ROOT / "Resources" / name, data / name) word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")