Learn classifier weights from synthetic typing data
Scoring is now features x weights (Features.swift / features.py), with the old hand-tuned values as defaults. training/ generates synthetic mixed Chinese/English buffers from the lexicon and word lists and trains the weights with an averaged structured perceptron over the same segmentation DP the input method runs. On 10,000 test buffers commit-time accuracy goes from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%. Both apps load Resources/classifier-weights.json. A parity test checks the Swift segmentation matches the Python one on 400 fixtures. Also fixes space after a toned syllable being scored as an extra stray tone, which could turn short Chinese like "u4 " into English. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -69,12 +69,12 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
|
||||
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
|
||||
| 一開始稍微偏向注音 | 純數字 → 數字 |
|
||||
|
||||
分數高的一方勝出。按空白時,注音那邊會把空白當成一聲來評分,英文那邊當作完整單字評分。
|
||||
每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。
|
||||
|
||||
中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎:
|
||||
- 每多切一刀扣 3 分(`segmentPenalty`)
|
||||
- 每多切一刀會扣分(`seg.switch` 特徵)
|
||||
- 後面還接著其他段的注音段,最後一個音節必須打了聲調
|
||||
- 接在英文後面、還沒打聲調的注音段再扣 2 分(`unconfirmedSwitchPenalty`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
|
||||
- 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
|
||||
|
||||
音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。
|
||||
|
||||
@@ -123,6 +123,33 @@ scripts/package-windows.sh # 產生 build/AutoZhuyin-Wi
|
||||
|
||||
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
|
||||
|
||||
### 訓練判斷權重
|
||||
|
||||
中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法:
|
||||
|
||||
1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。
|
||||
2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。
|
||||
3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。
|
||||
|
||||
```bash
|
||||
python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用)
|
||||
python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘)
|
||||
python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘
|
||||
python3 training/train.py --eval-only # 只評估目前的權重
|
||||
python3 training/make_parity.py # 權重或特徵改了之後都要重跑
|
||||
```
|
||||
|
||||
目前的結果(10,000 筆測試資料):
|
||||
|
||||
| | 送出時整段判斷正確 | 打字過程中顯示正確 |
|
||||
|---|---|---|
|
||||
| 手調權重 | 97.58% | 86.15% |
|
||||
| 訓練後 | **99.62%** | 87.81% |
|
||||
|
||||
送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
|
||||
|
||||
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
|
||||
|
||||
### 詞庫
|
||||
|
||||
`Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。
|
||||
@@ -164,11 +191,12 @@ Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試)
|
||||
Lexicon.swift 詞庫
|
||||
Converter.swift 音節 → 中文(Viterbi)
|
||||
EnglishModel.swift 英文判斷
|
||||
Classifier.swift 中英判斷 + 學習
|
||||
Features.swift 判斷用的特徵與預設權重
|
||||
Classifier.swift 中英判斷(特徵 × 權重)、切分、學習
|
||||
CompositionEngine.swift 輸入法狀態機
|
||||
Sources/AutoZhuyin/ macOS InputMethodKit 外殼
|
||||
Tests/ Swift 單元測試
|
||||
Resources/ Info.plist、詞庫、英文常用字
|
||||
Resources/ Info.plist、詞庫、英文常用字、訓練好的權重
|
||||
scripts/ 建置、安裝、打包、詞庫轉換
|
||||
pkg/postinstall PKG 安裝後腳本
|
||||
windows/
|
||||
@@ -179,6 +207,7 @@ windows/
|
||||
installer/ install.bat、uninstall.bat、給使用者的 README.txt
|
||||
tests/ Python 測試
|
||||
tools/ 產生資料檔、圖示
|
||||
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
|
||||
```
|
||||
|
||||
## 已知限制
|
||||
@@ -193,7 +222,8 @@ windows/
|
||||
- [x] 匯入完整詞庫
|
||||
- [x] 注音順序錯誤修正
|
||||
- [x] Windows 版
|
||||
- [ ] 用真實打字資料調整判斷權重
|
||||
- [x] 用資料訓練判斷權重(合成資料)
|
||||
- [ ] 加入真實打字資料一起訓練
|
||||
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
|
||||
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境
|
||||
- [ ] 使用者詞頻學習(選過的候選字優先)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"about": "AutoZhuyin classifier weights, learned by training/train.py",
|
||||
"trained": "2026-11-11",
|
||||
"examples": 40001,
|
||||
"test_accuracy": {
|
||||
"commit": 0.9962,
|
||||
"typing": 0.8781
|
||||
},
|
||||
"weights": {
|
||||
"lit.bias": 0.4276,
|
||||
"lit.common": -0.3465,
|
||||
"lit.common_len": 0.81,
|
||||
"lit.dict": -1.7902,
|
||||
"lit.dict_len": 0.6765,
|
||||
"lit.digits": -2.7403,
|
||||
"lit.num_digits": -0.0751,
|
||||
"lit.num_others": -0.7433,
|
||||
"lit.others": -1.5685,
|
||||
"lit.prefix_common_len": 0.2082,
|
||||
"lit.prefix_dict_len": 0.0003,
|
||||
"lit.short": -0.2397,
|
||||
"lit.unknown_score": 0.213,
|
||||
"lit.upper": 20.0,
|
||||
"seg.switch": -1.7203,
|
||||
"zh.bias": 1.3522,
|
||||
"zh.lexscore": 0.26,
|
||||
"zh.pending_invalid": -5.0,
|
||||
"zh.pending_prefix": -0.9295,
|
||||
"zh.phrase": 0.4595,
|
||||
"zh.reordered": -1.4348,
|
||||
"zh.stray": -5.0,
|
||||
"zh.toned_invalid": -5.0,
|
||||
"zh.toned_valid_lex": -0.9823,
|
||||
"zh.toned_valid_nolex": -1.8742,
|
||||
"zh.unconfirmed": -0.7891,
|
||||
"zh.untoned_mid_invalid": -5.0,
|
||||
"zh.untoned_mid_valid": -2.8009
|
||||
}
|
||||
}
|
||||
@@ -26,7 +26,8 @@ enum Shared {
|
||||
let learned = LearnedPreferences(weights: saved) { weights in
|
||||
UserDefaults.standard.set(weights, forKey: learnedKey)
|
||||
}
|
||||
return Classifier(lexicon: lexicon, english: english, learned: learned)
|
||||
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
|
||||
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
|
||||
}()
|
||||
}
|
||||
|
||||
|
||||
@@ -50,48 +50,64 @@ public final class LearnedPreferences {
|
||||
}
|
||||
|
||||
/// Decides whether a key sequence is Zhuyin or literal (English / numbers).
|
||||
/// Every segment is scored as features · weights (see Features.swift).
|
||||
public final class Classifier {
|
||||
public let lexicon: Lexicon
|
||||
public let english: EnglishModel
|
||||
public let learned: LearnedPreferences
|
||||
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
|
||||
public private(set) var weights: FeatureVector
|
||||
|
||||
/// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it.
|
||||
public var zhuyinPrior = 0.5
|
||||
/// How much the previous segment's mode pulls the current one.
|
||||
public var contextWeight = 1.5
|
||||
|
||||
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) {
|
||||
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
|
||||
weights: FeatureVector = [:]) {
|
||||
self.lexicon = lexicon
|
||||
self.english = english
|
||||
self.learned = learned
|
||||
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
|
||||
}
|
||||
|
||||
/// - Parameters:
|
||||
/// - finishing: the user pressed space. Zhuyin is scored as if the space were a
|
||||
/// first-tone key; literal is scored as a completed word.
|
||||
/// - previous: mode of the previously committed segment, if recent.
|
||||
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
|
||||
var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior
|
||||
var lit = literalScore(keys, complete: finishing)
|
||||
/// (Zhuyin features or nil, literal features or nil) for one segment.
|
||||
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool,
|
||||
finishing: Bool) -> (FeatureVector?, FeatureVector?) {
|
||||
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing),
|
||||
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing))
|
||||
}
|
||||
|
||||
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
|
||||
previous: InputMode?, whole: Bool) -> [Double] {
|
||||
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing)
|
||||
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
|
||||
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
|
||||
switch previous {
|
||||
case .zhuyin?: zh += contextWeight
|
||||
case .literal?: lit += contextWeight
|
||||
case nil: break
|
||||
}
|
||||
let learnedBias = learned.bias(for: String(keys))
|
||||
if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias }
|
||||
return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit)
|
||||
if whole {
|
||||
let b = learned.bias(for: String(seg))
|
||||
if b > 0 { zh += b } else { lit -= b }
|
||||
}
|
||||
return [zh, lit]
|
||||
}
|
||||
|
||||
/// The whole buffer as one segment.
|
||||
/// - Parameters:
|
||||
/// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as
|
||||
/// first tone; literal is scored as a completed word.
|
||||
/// - previous: mode of the previously committed segment, if recent.
|
||||
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
|
||||
let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true)
|
||||
return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1])
|
||||
}
|
||||
|
||||
// MARK: - Segmentation
|
||||
|
||||
/// Cost of each switch between Zhuyin and literal inside one buffer, so that a normal
|
||||
/// word or sentence is not chopped up.
|
||||
public var segmentPenalty = 3.0
|
||||
|
||||
/// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" →
|
||||
/// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point.
|
||||
/// A Zhuyin segment followed by another segment must end with a tone key.
|
||||
/// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the
|
||||
/// cost of each switch is the "seg.switch" feature.
|
||||
public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] {
|
||||
let n = keys.count
|
||||
guard n > 0 else { return [] }
|
||||
@@ -102,12 +118,12 @@ public final class Classifier {
|
||||
|
||||
for i in 1...n {
|
||||
for j in 0..<i {
|
||||
let scores = segmentScores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0,
|
||||
finishing: finishing, previous: j == 0 ? previous : nil,
|
||||
whole: j == 0 && i == n)
|
||||
for m in 0..<2 where scores[m] > -.infinity {
|
||||
let s = scores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, finishing: finishing,
|
||||
previous: j == 0 ? previous : nil, whole: j == 0 && i == n)
|
||||
for m in 0..<2 where s[m] > -.infinity {
|
||||
// Adjacent segments must alternate modes; same-mode neighbours are one segment.
|
||||
let total = j == 0 ? scores[m] : best[j][1 - m] + scores[m] - segmentPenalty
|
||||
if j > 0 && best[j][1 - m] == -.infinity { continue }
|
||||
let total = j == 0 ? s[m] : best[j][1 - m] + s[m]
|
||||
if total > best[i][m] {
|
||||
best[i][m] = total
|
||||
back[i][m] = j
|
||||
@@ -130,79 +146,4 @@ public final class Classifier {
|
||||
}
|
||||
return out.reversed()
|
||||
}
|
||||
|
||||
/// Extra cost for a trailing Zhuyin segment after English that has no tone key yet
|
||||
/// ("ji3hel" should stay 我hel, not become 我heㄠ, until a tone confirms the switch).
|
||||
public var unconfirmedSwitchPenalty = 2.0
|
||||
|
||||
private func segmentScores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
|
||||
previous: InputMode?, whole: Bool) -> [Double] {
|
||||
var zh: Double
|
||||
let parse = ZhuyinParser.parse(seg)
|
||||
if !isLast && parse.pending != nil {
|
||||
zh = -.infinity
|
||||
} else {
|
||||
zh = zhuyinScore(finishing && isLast ? seg + [" "] : seg) + zhuyinPrior
|
||||
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
|
||||
zh -= unconfirmedSwitchPenalty
|
||||
}
|
||||
}
|
||||
var lit = literalScore(seg, complete: !isLast || finishing)
|
||||
switch previous {
|
||||
case .zhuyin?: zh += contextWeight
|
||||
case .literal?: lit += contextWeight
|
||||
case nil: break
|
||||
}
|
||||
if whole {
|
||||
let b = learned.bias(for: String(seg))
|
||||
if b > 0 { zh += b } else { lit -= b }
|
||||
}
|
||||
return [zh, lit]
|
||||
}
|
||||
|
||||
public func zhuyinScore(_ keys: [Character]) -> Double {
|
||||
let parse = ZhuyinParser.parse(keys)
|
||||
if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity }
|
||||
var s = -5.0 * Double(parse.strayTones)
|
||||
for (i, p) in parse.syllables.enumerated() {
|
||||
let isLast = i == parse.syllables.count - 1
|
||||
// Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin.
|
||||
if p.reordered { s -= 0.5 }
|
||||
let valid = SyllableTable.isValid(p.syllable)
|
||||
if p.toned {
|
||||
if valid {
|
||||
s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5)
|
||||
} else {
|
||||
s -= 5
|
||||
}
|
||||
} else if isLast {
|
||||
s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5
|
||||
} else {
|
||||
// Untoned syllable in the middle: allowed (tone omission) but unusual.
|
||||
s += valid ? -2.5 : -5
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
public func literalScore(_ keys: [Character], complete: Bool) -> Double {
|
||||
if keys.isEmpty || keys.contains(" ") { return -.infinity }
|
||||
var score = 0.0
|
||||
if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English
|
||||
|
||||
let letters = String(keys.filter { $0.isLetter })
|
||||
let digits = keys.filter { $0.isNumber }.count
|
||||
let others = keys.count - letters.count - digits
|
||||
|
||||
if letters.isEmpty {
|
||||
// Pure number like "2024" (with possible "." or ",").
|
||||
return score + 0.8 * Double(digits) - 1.0 * Double(others)
|
||||
}
|
||||
score += english.score(letters, complete: complete || digits > 0)
|
||||
score -= 1.5 * Double(digits)
|
||||
// `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not.
|
||||
let apostrophes = keys.filter { $0 == "'" }.count
|
||||
score -= 1.5 * Double(others - apostrophes)
|
||||
return score
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,6 +53,30 @@ public final class EnglishModel {
|
||||
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||
}
|
||||
|
||||
/// Feature dict for the classifier (see Features.swift); `score` is their dot
|
||||
/// product with the default weights.
|
||||
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
|
||||
let w = word.lowercased()
|
||||
let n = Double(w.count)
|
||||
var f: FeatureVector = [:]
|
||||
if n <= 2 { f["lit.short"] = 1 }
|
||||
if common.contains(w) {
|
||||
f["lit.common"] = 1
|
||||
f["lit.common_len"] = n
|
||||
} else if dictionary.contains(w) {
|
||||
f["lit.dict"] = 1
|
||||
f["lit.dict_len"] = n
|
||||
} else if !complete && hasPrefix(w, in: sortedCommon) {
|
||||
f["lit.prefix_common_len"] = n
|
||||
} else if !complete && hasPrefix(w, in: sortedWords) {
|
||||
f["lit.prefix_dict_len"] = n
|
||||
} else {
|
||||
let plaus = plausibility(w)
|
||||
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
|
||||
public func plausibility(_ w: String) -> Double {
|
||||
let chars = ["^"] + w.map(String.init) + ["$"]
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
import Foundation
|
||||
|
||||
/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode
|
||||
/// is the dot product of its features with the weights. `defaultWeights` reproduces the
|
||||
/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json
|
||||
/// (training/train.py).
|
||||
///
|
||||
/// Python twin: windows/autozhuyin/core/features.py — both must produce identical
|
||||
/// features (checked by ParityTests against training/parity-fixtures.json).
|
||||
public typealias FeatureVector = [String: Double]
|
||||
|
||||
public enum Features {
|
||||
public static let defaultWeights: FeatureVector = [
|
||||
// Zhuyin
|
||||
"zh.bias": 0.5,
|
||||
"zh.stray": -5.0,
|
||||
"zh.reordered": -0.5,
|
||||
"zh.toned_valid_lex": 2.8,
|
||||
"zh.toned_valid_nolex": 0.5,
|
||||
"zh.toned_invalid": -5.0,
|
||||
"zh.pending_prefix": 0.0,
|
||||
"zh.pending_invalid": -5.0,
|
||||
"zh.untoned_mid_valid": -2.5,
|
||||
"zh.untoned_mid_invalid": -5.0,
|
||||
"zh.unconfirmed": -2.0,
|
||||
"zh.lexscore": 0.0,
|
||||
"zh.phrase": 0.0,
|
||||
// Literal
|
||||
"lit.bias": 0.0,
|
||||
"lit.upper": 20.0,
|
||||
"lit.num_digits": 0.8,
|
||||
"lit.num_others": -1.0,
|
||||
"lit.common": 3.0,
|
||||
"lit.common_len": 1.0,
|
||||
"lit.dict": 1.0,
|
||||
"lit.dict_len": 0.6,
|
||||
"lit.prefix_common_len": 0.4,
|
||||
"lit.prefix_dict_len": 0.2,
|
||||
"lit.unknown_score": 1.0,
|
||||
"lit.short": 0.0,
|
||||
"lit.digits": -1.5,
|
||||
"lit.others": -1.5,
|
||||
// Segmentation
|
||||
"seg.switch": -3.0,
|
||||
]
|
||||
|
||||
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
|
||||
public static func loadWeights(from url: URL?) -> FeatureVector {
|
||||
guard let url, let data = try? Data(contentsOf: url),
|
||||
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let w = obj["weights"] as? [String: Double] else { return [:] }
|
||||
return w
|
||||
}
|
||||
|
||||
public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double {
|
||||
f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value }
|
||||
}
|
||||
|
||||
/// Features of `keys` read as Zhuyin, or nil if impossible.
|
||||
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
|
||||
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? {
|
||||
let parse = ZhuyinParser.parse(keys)
|
||||
// A Zhuyin segment followed by another must end with a tone.
|
||||
if !isLast && parse.pending != nil { return nil }
|
||||
// Space finishes an untoned last syllable as first tone; after a toned syllable it
|
||||
// just commits, so it must not count as an extra (stray) tone.
|
||||
let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse
|
||||
if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil }
|
||||
|
||||
var f: FeatureVector = ["zh.bias": 1]
|
||||
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
|
||||
if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) }
|
||||
let last = scored.syllables.count - 1
|
||||
var tonedRun: [Syllable] = []
|
||||
for (i, p) in scored.syllables.enumerated() {
|
||||
if p.reordered { add("zh.reordered") }
|
||||
let valid = SyllableTable.isValid(p.syllable)
|
||||
if p.toned {
|
||||
if valid {
|
||||
if let best = lexicon.bestScore(for: p.syllable) {
|
||||
add("zh.toned_valid_lex")
|
||||
add("zh.lexscore", best + 4)
|
||||
} else {
|
||||
add("zh.toned_valid_nolex")
|
||||
}
|
||||
tonedRun.append(p.syllable)
|
||||
} else {
|
||||
add("zh.toned_invalid")
|
||||
}
|
||||
} else if i == last {
|
||||
add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid")
|
||||
} else {
|
||||
add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid")
|
||||
}
|
||||
}
|
||||
// Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
|
||||
var covered = [Bool](repeating: false, count: tonedRun.count)
|
||||
for i in tonedRun.indices {
|
||||
for n in 2...4 where i + n <= tonedRun.count {
|
||||
if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty {
|
||||
for k in i..<(i + n) { covered[k] = true }
|
||||
}
|
||||
}
|
||||
}
|
||||
let coveredCount = covered.filter { $0 }.count
|
||||
if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) }
|
||||
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
|
||||
add("zh.unconfirmed")
|
||||
}
|
||||
if afterOther { add("seg.switch") }
|
||||
return f
|
||||
}
|
||||
|
||||
/// Features of `keys` output as typed, or nil if impossible.
|
||||
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
|
||||
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? {
|
||||
if keys.isEmpty || keys.contains(" ") { return nil }
|
||||
let complete = !isLast || finishing
|
||||
var f: FeatureVector = ["lit.bias": 1]
|
||||
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
|
||||
if afterOther { add("seg.switch") }
|
||||
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
|
||||
|
||||
let letters = String(keys.filter { $0.isLetter })
|
||||
let digits = keys.filter { $0.isNumber }.count
|
||||
let others = keys.count - letters.count - digits
|
||||
if letters.isEmpty {
|
||||
if digits > 0 { add("lit.num_digits", Double(digits)) }
|
||||
if others > 0 { add("lit.num_others", Double(others)) }
|
||||
return f
|
||||
}
|
||||
for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) }
|
||||
if digits > 0 { add("lit.digits", Double(digits)) }
|
||||
let nonApostrophe = others - keys.filter { $0 == "'" }.count
|
||||
if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) }
|
||||
return f
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ private let resources = URL(fileURLWithPath: #filePath)
|
||||
|
||||
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
|
||||
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"))
|
||||
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
|
||||
|
||||
final class ParserTests: XCTestCase {
|
||||
func testParsesNiHao() {
|
||||
@@ -84,7 +85,7 @@ final class ConverterTests: XCTestCase {
|
||||
}
|
||||
|
||||
final class ClassifierTests: XCTestCase {
|
||||
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish)
|
||||
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
|
||||
|
||||
func mode(_ s: String, finishing: Bool = false) -> InputMode {
|
||||
classifier.classify(Array(s), finishing: finishing).mode
|
||||
@@ -106,7 +107,7 @@ final class ClassifierTests: XCTestCase {
|
||||
|
||||
func testLearningFlipsDecision() {
|
||||
let learned = LearnedPreferences()
|
||||
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned)
|
||||
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights)
|
||||
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
|
||||
learned.record("so", as: .zhuyin)
|
||||
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
|
||||
@@ -115,7 +116,7 @@ final class ClassifierTests: XCTestCase {
|
||||
|
||||
final class EngineTests: XCTestCase {
|
||||
func makeEngine() -> CompositionEngine {
|
||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
|
||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||
}
|
||||
|
||||
/// Types a string and returns everything committed.
|
||||
@@ -188,7 +189,7 @@ final class EngineTests: XCTestCase {
|
||||
|
||||
final class DigitInitialTests: XCTestCase {
|
||||
func engine() -> CompositionEngine {
|
||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
|
||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||
}
|
||||
|
||||
func run(_ s: String) -> (out: String, marked: String) {
|
||||
@@ -222,7 +223,7 @@ final class DigitInitialTests: XCTestCase {
|
||||
|
||||
final class MixedSegmentTests: XCTestCase {
|
||||
func run(_ s: String) -> (out: String, marked: String) {
|
||||
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
|
||||
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||
var out = ""
|
||||
for c in s {
|
||||
let r = e.handle(c == "\n" ? .enter : .character(c))
|
||||
@@ -253,3 +254,30 @@ final class MixedSegmentTests: XCTestCase {
|
||||
XCTAssertEqual(run("183\n").out.count, 1)
|
||||
}
|
||||
}
|
||||
|
||||
/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py).
|
||||
final class ParityTests: XCTestCase {
|
||||
func testMatchesPythonSegmentation() throws {
|
||||
let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json")
|
||||
let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]]
|
||||
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
|
||||
var mismatches: [String] = []
|
||||
for fx in fixtures {
|
||||
let keys = Array(fx["keys"] as! String)
|
||||
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
|
||||
let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map {
|
||||
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
|
||||
}
|
||||
if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") }
|
||||
}
|
||||
XCTAssertGreaterThan(fixtures.count, 300)
|
||||
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
|
||||
}
|
||||
|
||||
func testToneThenSpaceStaysChinese() {
|
||||
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||
var out = ""
|
||||
for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } }
|
||||
XCTAssertEqual(out.count, 1, out)
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ rm -rf "$APP"
|
||||
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
||||
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
|
||||
cp Resources/Info.plist "$APP/Contents/Info.plist"
|
||||
cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/"
|
||||
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
|
||||
|
||||
if [ ! -f build/icon.tiff ]; then
|
||||
xcrun swift scripts/make-icon.swift build/icon.tiff
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
|
||||
import os
|
||||
import sys
|
||||
import types
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
WIN = os.path.join(ROOT, "windows")
|
||||
DATA = os.path.join(WIN, "autozhuyin", "data")
|
||||
RESOURCES = os.path.join(ROOT, "Resources")
|
||||
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
|
||||
|
||||
sys.path.insert(0, WIN)
|
||||
|
||||
from autozhuyin.core import bopomofo, features # noqa: E402
|
||||
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
|
||||
from autozhuyin.core.english import EnglishModel # noqa: E402
|
||||
from autozhuyin.core.lexicon import Lexicon # noqa: E402
|
||||
|
||||
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
|
||||
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
|
||||
|
||||
|
||||
def load_models():
|
||||
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
|
||||
sys.exit("Run windows/tools/build_data.py first.")
|
||||
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
|
||||
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
|
||||
os.path.join(DATA, "english-dict.txt"),
|
||||
os.path.join(DATA, "english-bigram.json"))
|
||||
return lexicon, english
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Writes training/parity-fixtures.json: segmentations produced by the Python core
|
||||
with the current weights. The Swift tests (ParityTests) must produce the same ones,
|
||||
which keeps the two implementations in sync. Re-run after changing features or weights.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
|
||||
from synth import Generator # noqa: E402
|
||||
|
||||
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json")
|
||||
|
||||
HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up",
|
||||
"u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"]
|
||||
|
||||
|
||||
def main():
|
||||
lexicon, english = load_models()
|
||||
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
|
||||
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
|
||||
os.path.join(DATA, "english-dict.txt"), seed=7)
|
||||
cases = [(k, f) for k in HAND_PICKED for f in (False, True)]
|
||||
while len(cases) < 400:
|
||||
ex = gen.example()
|
||||
if len(ex["keys"]) <= 28:
|
||||
cases.append((ex["keys"], ex["finishing"]))
|
||||
cases.append((gen.prefix_of(ex)["keys"], False))
|
||||
fixtures = [{"keys": k, "finishing": f,
|
||||
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases]
|
||||
with open(OUT, "w", encoding="utf-8") as fh:
|
||||
json.dump(fixtures, fh, ensure_ascii=False, indent=0)
|
||||
print("wrote %d fixtures to %s" % (len(fixtures), OUT))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,132 @@
|
||||
"""Synthetic typing data for the Zhuyin / literal classifier.
|
||||
|
||||
Each example is what sits in the input buffer when the user commits it, with the
|
||||
intended mode of every key:
|
||||
{"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"}
|
||||
labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter.
|
||||
|
||||
Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys;
|
||||
English from the common-word list and the dictionary. Real typing commits English at
|
||||
a space, so a buffer holds at most one English word next to Chinese.
|
||||
"""
|
||||
import random
|
||||
|
||||
from common import core
|
||||
|
||||
bopomofo = core.bopomofo
|
||||
|
||||
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
|
||||
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
|
||||
|
||||
|
||||
def reading_to_keys(reading):
|
||||
"""'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable."""
|
||||
out = []
|
||||
for syl in reading.split("-"):
|
||||
tone = " "
|
||||
if syl and syl[-1] in TONE_KEY:
|
||||
tone = TONE_KEY[syl[-1]]
|
||||
syl = syl[:-1]
|
||||
if not syl or syl not in bopomofo.VALID:
|
||||
return None
|
||||
out.append("".join(KEY_OF[c] for c in syl) + tone)
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _is_cjk(s):
|
||||
return all(0x4E00 <= ord(c) <= 0x9FFF for c in s)
|
||||
|
||||
|
||||
class Generator:
|
||||
def __init__(self, lexicon_path, common_path, dict_path, seed=0):
|
||||
self.rng = random.Random(seed)
|
||||
phrases, weights = [], []
|
||||
with open(lexicon_path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
cols = line.split()
|
||||
if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]):
|
||||
continue
|
||||
n = cols[0].count("-") + 1
|
||||
if n > 4:
|
||||
continue
|
||||
keys = reading_to_keys(cols[0])
|
||||
if keys is None:
|
||||
continue
|
||||
phrases.append((cols[1], keys))
|
||||
weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency
|
||||
self.phrases = phrases
|
||||
self.phrase_cum = _cumulative(weights)
|
||||
self.common = [w.strip() for w in open(common_path, encoding="utf-8")
|
||||
if w.strip() and not w.startswith("#")]
|
||||
self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8")
|
||||
if 3 <= len(w.strip()) <= 9]
|
||||
|
||||
# --- pieces ---
|
||||
|
||||
def chinese(self):
|
||||
n = self.rng.choice([1, 1, 2, 2, 3])
|
||||
picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n)
|
||||
return "".join(keys for _, keys in picked)
|
||||
|
||||
def english(self):
|
||||
w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words)
|
||||
if self.rng.random() < 0.1:
|
||||
w = w[:1].upper() + w[1:]
|
||||
return w
|
||||
|
||||
def number(self):
|
||||
r = self.rng.random()
|
||||
if r < 0.4:
|
||||
return str(self.rng.randint(0, 9999))
|
||||
if r < 0.7:
|
||||
return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30]))
|
||||
return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9))
|
||||
|
||||
# --- examples ---
|
||||
|
||||
def example(self):
|
||||
r = self.rng.random()
|
||||
if r < 0.40:
|
||||
parts, kind = [("z", self.chinese())], "zh"
|
||||
elif r < 0.65:
|
||||
parts, kind = [("l", self.english())], "en"
|
||||
elif r < 0.80:
|
||||
parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en"
|
||||
elif r < 0.93:
|
||||
parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh"
|
||||
elif r < 0.96:
|
||||
parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh"
|
||||
else:
|
||||
parts, kind = [("l", self.number())], "num"
|
||||
keys = "".join(p for _, p in parts)
|
||||
labels = "".join(m * len(p) for m, p in parts)
|
||||
if parts[-1][0] == "l":
|
||||
finishing = True # English / numbers end with a space
|
||||
else:
|
||||
# Chinese ends with Enter, or space after a finished (toned) syllable.
|
||||
finishing = keys[-1] != " " and self.rng.random() < 0.5
|
||||
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind}
|
||||
|
||||
def prefix_of(self, ex):
|
||||
"""The buffer while still typing: a random cut, not finishing."""
|
||||
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
|
||||
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
|
||||
"kind": ex["kind"] + "/prefix"}
|
||||
|
||||
|
||||
def _cumulative(ws):
|
||||
out, t = [], 0.0
|
||||
for w in ws:
|
||||
t += w
|
||||
out.append(t)
|
||||
return out
|
||||
|
||||
|
||||
def gold_segments(labels):
|
||||
"""'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]"""
|
||||
segs, start = [], 0
|
||||
for i in range(1, len(labels) + 1):
|
||||
if i == len(labels) or labels[i] != labels[start]:
|
||||
segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL))
|
||||
start = i
|
||||
return segs
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Learns the classifier weights from synthetic typing data (averaged structured
|
||||
perceptron over the same segmentation DP the input method runs) and reports
|
||||
accuracy against the hand-tuned defaults.
|
||||
|
||||
Usage (from the repo root, after windows/tools/build_data.py):
|
||||
python3 training/train.py # train, evaluate, write Resources/classifier-weights.json
|
||||
python3 training/train.py --eval-only # just evaluate the current weights file
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
from collections import defaultdict
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
|
||||
from synth import Generator, gold_segments # noqa: E402
|
||||
|
||||
MAX_KEYS = 28
|
||||
FIXED = {"lit.upper"} # hard rule (Shift = English), not learned
|
||||
|
||||
|
||||
def make_set(gen, n):
|
||||
out = []
|
||||
while len(out) < 2 * n:
|
||||
ex = gen.example()
|
||||
if len(ex["keys"]) > MAX_KEYS:
|
||||
continue
|
||||
out.append(ex)
|
||||
if len(ex["keys"]) > 1:
|
||||
out.append(gen.prefix_of(ex))
|
||||
return out
|
||||
|
||||
|
||||
def labels_of(segs, n):
|
||||
lab = [""] * n
|
||||
for s, e, m in segs:
|
||||
for i in range(s, e):
|
||||
lab[i] = "z" if m == core.ZHUYIN else "l"
|
||||
return "".join(lab)
|
||||
|
||||
|
||||
def seg_features(clf, keys, segs, finishing):
|
||||
"""Summed features of a segmentation, or None if any segment is impossible."""
|
||||
total = defaultdict(float)
|
||||
n = len(keys)
|
||||
for idx, (s, e, m) in enumerate(segs):
|
||||
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing)
|
||||
f = fz if m == core.ZHUYIN else fl
|
||||
if f is None:
|
||||
return None
|
||||
for k, v in f.items():
|
||||
total[k] += v
|
||||
return total
|
||||
|
||||
|
||||
def evaluate(clf, weights, data):
|
||||
by_kind = defaultdict(lambda: [0, 0])
|
||||
for ex in data:
|
||||
segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights)
|
||||
ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
|
||||
bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]]
|
||||
bucket[0] += ok
|
||||
bucket[1] += 1
|
||||
commit = [v for k, v in by_kind.items() if k != "prefix"]
|
||||
summary = {
|
||||
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
|
||||
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
|
||||
"by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())},
|
||||
}
|
||||
return summary
|
||||
|
||||
|
||||
def show(name, s):
|
||||
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
|
||||
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
|
||||
|
||||
|
||||
def train(clf, data, epochs, lr, seed):
|
||||
rng = random.Random(seed)
|
||||
w = dict(clf.weights)
|
||||
total = defaultdict(float)
|
||||
steps = 0
|
||||
for epoch in range(epochs):
|
||||
rng.shuffle(data)
|
||||
mistakes = 0
|
||||
t0 = time.time()
|
||||
for ex in data:
|
||||
keys = list(ex["keys"])
|
||||
pred = clf.segment(keys, ex["finishing"], None, w)
|
||||
if labels_of(pred, len(keys)) != ex["labels"]:
|
||||
gold = gold_segments(ex["labels"])
|
||||
fg = seg_features(clf, keys, gold, ex["finishing"])
|
||||
fp = seg_features(clf, keys, pred, ex["finishing"])
|
||||
if fg is not None and fp is not None:
|
||||
mistakes += 1
|
||||
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
|
||||
for k in set(fg) | set(fp):
|
||||
if k not in FIXED:
|
||||
w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0))
|
||||
steps += 1
|
||||
for k, v in w.items():
|
||||
total[k] += v
|
||||
print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True)
|
||||
return {k: round(v / steps, 4) for k, v in total.items()}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)")
|
||||
ap.add_argument("--test", type=int, default=1500)
|
||||
ap.add_argument("--epochs", type=int, default=6)
|
||||
ap.add_argument("--lr", type=float, default=0.1)
|
||||
ap.add_argument("--eval-only", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
lexicon, english = load_models()
|
||||
paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
|
||||
os.path.join(DATA, "english-dict.txt"))
|
||||
test = make_set(Generator(*paths, seed=2), args.test)
|
||||
|
||||
clf = core.Classifier(lexicon, english) # default (hand-tuned) weights
|
||||
show("default", evaluate(clf, clf.weights, test))
|
||||
if os.path.exists(WEIGHTS_PATH):
|
||||
current = dict(clf.weights)
|
||||
current.update(core.Classifier.load_weights(WEIGHTS_PATH))
|
||||
show("current", evaluate(clf, current, test))
|
||||
if args.eval_only:
|
||||
return
|
||||
|
||||
data = make_set(Generator(*paths, seed=1), args.train)
|
||||
print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True)
|
||||
learned = train(clf, data, args.epochs, args.lr, seed=3)
|
||||
result = evaluate(clf, learned, test)
|
||||
show("learned", result)
|
||||
|
||||
with open(WEIGHTS_PATH, "w", encoding="utf-8") as f:
|
||||
json.dump({
|
||||
"about": "AutoZhuyin classifier weights, learned by training/train.py",
|
||||
"trained": datetime.date.today().isoformat(),
|
||||
"examples": len(data),
|
||||
"test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)},
|
||||
"weights": dict(sorted(learned.items())),
|
||||
}, f, ensure_ascii=False, indent=2)
|
||||
f.write("\n")
|
||||
print("wrote", WEIGHTS_PATH)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -52,7 +52,8 @@ def _shared_classifier():
|
||||
weights = json.load(f)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned))
|
||||
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
|
||||
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained)
|
||||
return _classifier
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
|
||||
from . import bopomofo
|
||||
import json
|
||||
|
||||
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
|
||||
|
||||
ZHUYIN = "zhuyin"
|
||||
LITERAL = "literal"
|
||||
@@ -28,84 +30,35 @@ class LearnedPreferences:
|
||||
|
||||
|
||||
class Classifier:
|
||||
zhuyin_prior = 0.5
|
||||
context_weight = 1.5
|
||||
|
||||
def __init__(self, lexicon, english, learned=None):
|
||||
def __init__(self, lexicon, english, learned=None, weights=None):
|
||||
self.lexicon = lexicon
|
||||
self.english = english
|
||||
self.learned = learned or LearnedPreferences()
|
||||
self.weights = dict(DEFAULT_WEIGHTS)
|
||||
if weights:
|
||||
self.weights.update(weights)
|
||||
|
||||
def classify(self, keys, finishing=False, previous=None):
|
||||
"""Returns (mode, zhuyin_score, literal_score)."""
|
||||
keys = list(keys)
|
||||
zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior
|
||||
lit = self.literal_score(keys, complete=finishing)
|
||||
if previous == ZHUYIN:
|
||||
zh += self.context_weight
|
||||
elif previous == LITERAL:
|
||||
lit += self.context_weight
|
||||
b = self.learned.bias("".join(keys))
|
||||
if b > 0:
|
||||
zh += b
|
||||
else:
|
||||
lit -= b
|
||||
return (ZHUYIN if zh >= lit else LITERAL), zh, lit
|
||||
@classmethod
|
||||
def load_weights(cls, path):
|
||||
"""Reads a weights JSON file ({"weights": {...}}); returns {} if missing."""
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f).get("weights", {})
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
def mode(self, keys, finishing=False, previous=None):
|
||||
return self.classify(keys, finishing, previous)[0]
|
||||
def segment_features(self, seg, is_last, after_other, finishing):
|
||||
"""(zhuyin features or None, literal features or None) for one segment."""
|
||||
seg = list(seg)
|
||||
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing),
|
||||
literal_features(seg, self.english, is_last, after_other, finishing))
|
||||
|
||||
# --- segmentation ---
|
||||
|
||||
segment_penalty = 3.0
|
||||
unconfirmed_switch_penalty = 2.0
|
||||
|
||||
def segment(self, keys, finishing=False, previous=None):
|
||||
"""Splits keys into alternating segments: [(start, end, mode)], e.g.
|
||||
"ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
|
||||
split points; a Zhuyin segment followed by another must end with a tone."""
|
||||
keys = list(keys)
|
||||
n = len(keys)
|
||||
if n == 0:
|
||||
return []
|
||||
modes = (ZHUYIN, LITERAL)
|
||||
best = [[NEG_INF, NEG_INF] for _ in range(n + 1)]
|
||||
back = [[-1, -1] for _ in range(n + 1)]
|
||||
for i in range(1, n + 1):
|
||||
for j in range(i):
|
||||
scores = self._segment_scores(keys[j:i], i == n, j > 0, finishing,
|
||||
previous if j == 0 else None, j == 0 and i == n)
|
||||
for m in (0, 1):
|
||||
if scores[m] == NEG_INF:
|
||||
continue
|
||||
if j == 0:
|
||||
total = scores[m]
|
||||
else:
|
||||
if best[j][1 - m] == NEG_INF:
|
||||
continue
|
||||
total = best[j][1 - m] + scores[m] - self.segment_penalty
|
||||
if total > best[i][m]:
|
||||
best[i][m] = total
|
||||
back[i][m] = j
|
||||
if best[n][0] == NEG_INF and best[n][1] == NEG_INF:
|
||||
return [(0, n, self.mode(keys, finishing, previous))]
|
||||
m = 0 if best[n][0] >= best[n][1] else 1
|
||||
out, i = [], n
|
||||
while i > 0:
|
||||
j = back[i][m]
|
||||
out.append((j, i, modes[m]))
|
||||
i, m = j, 1 - m
|
||||
return out[::-1]
|
||||
|
||||
def _segment_scores(self, seg, is_last, after_other, finishing, previous, whole):
|
||||
parse = bopomofo.parse(seg)
|
||||
if not is_last and parse.pending is not None:
|
||||
zh = NEG_INF
|
||||
else:
|
||||
zh = self.zhuyin_score(seg + [" "] if finishing and is_last else seg) + self.zhuyin_prior
|
||||
if is_last and after_other and not finishing and not any(s.toned for s in parse.syllables):
|
||||
zh -= self.unconfirmed_switch_penalty
|
||||
lit = self.literal_score(seg, complete=(not is_last) or finishing)
|
||||
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
|
||||
fz, fl = self.segment_features(seg, is_last, after_other, finishing)
|
||||
zh = dot(weights, fz) if fz is not None else NEG_INF
|
||||
lit = dot(weights, fl) if fl is not None else NEG_INF
|
||||
if previous == ZHUYIN:
|
||||
zh += self.context_weight
|
||||
elif previous == LITERAL:
|
||||
@@ -118,40 +71,50 @@ class Classifier:
|
||||
lit -= b
|
||||
return zh, lit
|
||||
|
||||
def zhuyin_score(self, keys):
|
||||
p = bopomofo.parse(keys)
|
||||
if p.foreign_keys > 0 or not p.syllables:
|
||||
return NEG_INF
|
||||
s = -5.0 * p.stray_tones
|
||||
last = len(p.syllables) - 1
|
||||
for i, ps in enumerate(p.syllables):
|
||||
if ps.reordered:
|
||||
s -= 0.5
|
||||
valid = bopomofo.is_valid(ps.syllable)
|
||||
if ps.toned:
|
||||
if valid:
|
||||
s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5)
|
||||
else:
|
||||
s -= 5
|
||||
elif i == last:
|
||||
s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5
|
||||
else:
|
||||
s += -2.5 if valid else -5
|
||||
return s
|
||||
def classify(self, keys, finishing=False, previous=None):
|
||||
"""Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score)."""
|
||||
zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights)
|
||||
return (ZHUYIN if zh >= lit else LITERAL), zh, lit
|
||||
|
||||
def literal_score(self, keys, complete):
|
||||
if not keys or " " in keys:
|
||||
return NEG_INF
|
||||
score = 0.0
|
||||
if any(c.isupper() for c in keys):
|
||||
score += 20
|
||||
letters = "".join(c for c in keys if c.isalpha())
|
||||
digits = sum(1 for c in keys if c.isdigit())
|
||||
others = len(keys) - len(letters) - digits
|
||||
if not letters:
|
||||
return score + 0.8 * digits - 1.0 * others
|
||||
score += self.english.score(letters, complete or digits > 0)
|
||||
score -= 1.5 * digits
|
||||
apostrophes = keys.count("'")
|
||||
score -= 1.5 * (others - apostrophes)
|
||||
return score
|
||||
def mode(self, keys, finishing=False, previous=None):
|
||||
return self.classify(keys, finishing, previous)[0]
|
||||
|
||||
# --- segmentation ---
|
||||
|
||||
def segment(self, keys, finishing=False, previous=None, weights=None):
|
||||
"""Splits keys into alternating segments: [(start, end, mode)], e.g.
|
||||
"ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
|
||||
split points; each segment is scored by features x weights."""
|
||||
weights = weights or self.weights
|
||||
keys = list(keys)
|
||||
n = len(keys)
|
||||
if n == 0:
|
||||
return []
|
||||
modes = (ZHUYIN, LITERAL)
|
||||
best = [[NEG_INF, NEG_INF] for _ in range(n + 1)]
|
||||
back = [[-1, -1] for _ in range(n + 1)]
|
||||
for i in range(1, n + 1):
|
||||
for j in range(i):
|
||||
scores = self._scores(keys[j:i], i == n, j > 0, finishing,
|
||||
previous if j == 0 else None, j == 0 and i == n, weights)
|
||||
for m in (0, 1):
|
||||
if scores[m] == NEG_INF:
|
||||
continue
|
||||
if j == 0:
|
||||
total = scores[m]
|
||||
else:
|
||||
if best[j][1 - m] == NEG_INF:
|
||||
continue
|
||||
total = best[j][1 - m] + scores[m]
|
||||
if total > best[i][m]:
|
||||
best[i][m] = total
|
||||
back[i][m] = j
|
||||
if best[n][0] == NEG_INF and best[n][1] == NEG_INF:
|
||||
return [(0, n, self.mode(keys, finishing, previous))]
|
||||
m = 0 if best[n][0] >= best[n][1] else 1
|
||||
out, i = [], n
|
||||
while i > 0:
|
||||
j = back[i][m]
|
||||
out.append((j, i, modes[m]))
|
||||
i, m = j, 1 - m
|
||||
return out[::-1]
|
||||
|
||||
@@ -56,6 +56,29 @@ class EnglishModel:
|
||||
plaus = self.plausibility(w)
|
||||
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||
|
||||
def word_features(self, word, complete):
|
||||
"""Feature dict for the classifier (see features.py); score() is their
|
||||
dot product with the default weights."""
|
||||
w = word.lower()
|
||||
n = float(len(w))
|
||||
f = {}
|
||||
if n <= 2:
|
||||
f["lit.short"] = 1.0
|
||||
if w in self.common:
|
||||
f["lit.common"] = 1.0
|
||||
f["lit.common_len"] = n
|
||||
elif w in self.dictionary:
|
||||
f["lit.dict"] = 1.0
|
||||
f["lit.dict_len"] = n
|
||||
elif not complete and _has_prefix(w, self.sorted_common):
|
||||
f["lit.prefix_common_len"] = n
|
||||
elif not complete and _has_prefix(w, self.sorted_words):
|
||||
f["lit.prefix_dict_len"] = n
|
||||
else:
|
||||
plaus = self.plausibility(w)
|
||||
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||
return f
|
||||
|
||||
def plausibility(self, w):
|
||||
chars = ["^"] + list(w) + ["$"]
|
||||
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
"""Segment features for the Zhuyin / literal classifier.
|
||||
|
||||
A segment's score in each mode is the dot product of its feature dict with a weight
|
||||
dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights
|
||||
(training/train.py → Resources/classifier-weights.json) replace them.
|
||||
|
||||
Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical
|
||||
features; Tests/.../ParityTests checks this against training/parity-fixtures.json.
|
||||
"""
|
||||
from . import bopomofo
|
||||
|
||||
DEFAULT_WEIGHTS = {
|
||||
# Zhuyin
|
||||
"zh.bias": 0.5,
|
||||
"zh.stray": -5.0,
|
||||
"zh.reordered": -0.5,
|
||||
"zh.toned_valid_lex": 2.8,
|
||||
"zh.toned_valid_nolex": 0.5,
|
||||
"zh.toned_invalid": -5.0,
|
||||
"zh.pending_prefix": 0.0,
|
||||
"zh.pending_invalid": -5.0,
|
||||
"zh.untoned_mid_valid": -2.5,
|
||||
"zh.untoned_mid_invalid": -5.0,
|
||||
"zh.unconfirmed": -2.0,
|
||||
"zh.lexscore": 0.0,
|
||||
"zh.phrase": 0.0,
|
||||
# Literal
|
||||
"lit.bias": 0.0,
|
||||
"lit.upper": 20.0,
|
||||
"lit.num_digits": 0.8,
|
||||
"lit.num_others": -1.0,
|
||||
"lit.common": 3.0,
|
||||
"lit.common_len": 1.0,
|
||||
"lit.dict": 1.0,
|
||||
"lit.dict_len": 0.6,
|
||||
"lit.prefix_common_len": 0.4,
|
||||
"lit.prefix_dict_len": 0.2,
|
||||
"lit.unknown_score": 1.0,
|
||||
"lit.short": 0.0,
|
||||
"lit.digits": -1.5,
|
||||
"lit.others": -1.5,
|
||||
# Segmentation
|
||||
"seg.switch": -3.0,
|
||||
}
|
||||
|
||||
|
||||
def dot(weights, feats):
|
||||
return sum(weights.get(k, 0.0) * v for k, v in feats.items())
|
||||
|
||||
|
||||
def _add(f, k, v=1.0):
|
||||
f[k] = f.get(k, 0.0) + v
|
||||
|
||||
|
||||
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False):
|
||||
"""Features of `keys` read as Zhuyin, or None if impossible."""
|
||||
parse = bopomofo.parse(keys)
|
||||
if not is_last and parse.pending is not None:
|
||||
return None # a Zhuyin segment followed by another must end with a tone
|
||||
# Pressing space finishes an untoned last syllable as first tone; after a toned
|
||||
# syllable it just commits, so it must not count as an extra (stray) tone.
|
||||
scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse
|
||||
if scored.foreign_keys > 0 or not scored.syllables:
|
||||
return None
|
||||
f = {"zh.bias": 1.0}
|
||||
if scored.stray_tones:
|
||||
_add(f, "zh.stray", scored.stray_tones)
|
||||
last = len(scored.syllables) - 1
|
||||
toned_run = []
|
||||
for i, ps in enumerate(scored.syllables):
|
||||
if ps.reordered:
|
||||
_add(f, "zh.reordered")
|
||||
valid = bopomofo.is_valid(ps.syllable)
|
||||
if ps.toned:
|
||||
if valid:
|
||||
best = lexicon.best_score(ps.syllable)
|
||||
if best is not None:
|
||||
_add(f, "zh.toned_valid_lex")
|
||||
_add(f, "zh.lexscore", best + 4.0)
|
||||
else:
|
||||
_add(f, "zh.toned_valid_nolex")
|
||||
toned_run.append(ps.syllable)
|
||||
else:
|
||||
_add(f, "zh.toned_invalid")
|
||||
elif i == last:
|
||||
_add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid")
|
||||
else:
|
||||
_add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid")
|
||||
# Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
|
||||
covered = [False] * len(toned_run)
|
||||
for i in range(len(toned_run)):
|
||||
for n in (2, 3, 4):
|
||||
if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]):
|
||||
for k in range(i, i + n):
|
||||
covered[k] = True
|
||||
if any(covered):
|
||||
_add(f, "zh.phrase", sum(covered))
|
||||
if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables):
|
||||
_add(f, "zh.unconfirmed")
|
||||
if after_other:
|
||||
_add(f, "seg.switch")
|
||||
return f
|
||||
|
||||
|
||||
def literal_features(keys, english, is_last=True, after_other=False, finishing=False):
|
||||
"""Features of `keys` output as typed, or None if impossible."""
|
||||
if not keys or " " in keys:
|
||||
return None
|
||||
complete = (not is_last) or finishing
|
||||
f = {"lit.bias": 1.0}
|
||||
if after_other:
|
||||
_add(f, "seg.switch")
|
||||
if any(c.isupper() for c in keys):
|
||||
_add(f, "lit.upper")
|
||||
letters = "".join(c for c in keys if c.isalpha())
|
||||
digits = sum(1 for c in keys if c.isdigit())
|
||||
others = len(keys) - len(letters) - digits
|
||||
if not letters:
|
||||
if digits:
|
||||
_add(f, "lit.num_digits", digits)
|
||||
if others:
|
||||
_add(f, "lit.num_others", others)
|
||||
return f
|
||||
for k, v in english.word_features(letters, complete or digits > 0).items():
|
||||
_add(f, k, v)
|
||||
if digits:
|
||||
_add(f, "lit.digits", digits)
|
||||
non_apostrophe = others - keys.count("'")
|
||||
if non_apostrophe:
|
||||
_add(f, "lit.others", non_apostrophe)
|
||||
return f
|
||||
@@ -21,6 +21,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
|
||||
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
|
||||
os.path.join(DATA, "english-dict.txt"),
|
||||
os.path.join(DATA, "english-bigram.json"))
|
||||
WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json"))
|
||||
LOAD_TIME = time.time() - _start
|
||||
|
||||
|
||||
@@ -34,7 +35,7 @@ def convert(keys):
|
||||
|
||||
|
||||
def make_engine():
|
||||
return CompositionEngine(Classifier(LEX, ENG))
|
||||
return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS))
|
||||
|
||||
|
||||
def type_keys(s, e):
|
||||
@@ -86,7 +87,7 @@ class ConverterTests(unittest.TestCase):
|
||||
|
||||
|
||||
class ClassifierTests(unittest.TestCase):
|
||||
c = Classifier(LEX, ENG)
|
||||
c = Classifier(LEX, ENG, weights=WEIGHTS)
|
||||
|
||||
def test_zhuyin(self):
|
||||
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
|
||||
@@ -101,7 +102,7 @@ class ClassifierTests(unittest.TestCase):
|
||||
|
||||
def test_learning(self):
|
||||
learned = LearnedPreferences()
|
||||
c = Classifier(LEX, ENG, learned)
|
||||
c = Classifier(LEX, ENG, learned, WEIGHTS)
|
||||
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
|
||||
learned.record("so", ZHUYIN)
|
||||
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
|
||||
@@ -150,6 +151,9 @@ class EngineTests(unittest.TestCase):
|
||||
self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
|
||||
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
|
||||
|
||||
def test_tone_then_space_stays_chinese(self):
|
||||
self.assertEqual(len(type_keys("u4 ", make_engine())), 1)
|
||||
|
||||
def test_out_of_order_converts(self):
|
||||
self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fills windows/autozhuyin/data/ from the shared Mac resources.
|
||||
|
||||
- lexicon.txt, english-common.txt: copied from Resources/
|
||||
- lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/
|
||||
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
|
||||
Webster's 2nd edition, public domain), since Windows has no such file
|
||||
- english-bigram.json: precomputed letter-bigram table
|
||||
@@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402
|
||||
|
||||
data = WIN / "autozhuyin" / "data"
|
||||
data.mkdir(parents=True, exist_ok=True)
|
||||
for name in ("lexicon.txt", "english-common.txt"):
|
||||
for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"):
|
||||
shutil.copy(ROOT / "Resources" / name, data / name)
|
||||
|
||||
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")
|
||||
|
||||
Reference in New Issue
Block a user