Compare commits

..
8 Commits
Author SHA1 Message Date
ays953 251a16fa08 Release v0.7.0 2026-11-11 16:29:06 +08:00
ays953andClaude Opus 5.5 7cb1f62dfa Backtick + punctuation key forces a full-width symbol
` then , . / ; : ' [ ] ( ) ! ~ - = and their shifted keys always give the
full-width symbol (,。?;:、「」()!~——=), whatever the context. Backtick
followed by any other key is a plain backtick, so ``` still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:28:54 +08:00
ays953 3670305f0f Release v0.6.3 2026-11-11 16:10:59 +08:00
ays953andClaude Opus 5.5 ef9f91a6b7 Hold ambiguous English words (up = 因) until the next word decides
"up " also reads as ㄧㄣ (因). Instead of committing English on space, keep the
word composing: "up jo4" becomes 因為, "up the" stays English. Only applies when
the reading is a common single syllable (lexicon score >= -3.5): up, url.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:10:47 +08:00
ays953 1dacb8f164 Release v0.6.2 2026-11-11 12:13:15 +08:00
ays953andClaude Opus 5.5 bfeed22916 Cut input method startup from ~0.95 s to ~0.15 s
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
  parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
  Windows build) instead of retraining the bigram model on every launch, and
  uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:13:03 +08:00
ays953 bb9cb0c4e0 Release v0.6.1 2026-11-11 12:00:10 +08:00
ays953andClaude Opus 5.5 2c5c70c066 Learn the previous-word context; fix first-tone characters after English
After an English word, 23 of 50 common first-tone characters (機, 吃, 書…)
were read as English when space was pressed: the fixed +1.5 context bonus
was far too strong for the trained weight scale, and the training data never
contained the "space just pressed on a first-tone syllable" decision.

Context is now the ctx.zh / ctx.lit features, trained with the rest; the
synthetic data includes the previous mode and the first-tone space decision.
Test accuracy: commit 98.22% -> 99.52%, typing 88.73% -> 91.49%, and when the
language differs from the previous word 92.0% -> 99.3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:59:57 +08:00
22 changed files with 5752 additions and 3526 deletions
+8 -7
View File
@@ -16,7 +16,7 @@
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小) - **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」 - **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來 - **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,` - **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權) - **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝 ## 下載與安裝
@@ -58,6 +58,7 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| Shift + 字母 | 打大寫字母(也會強制英文) | | Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` | | `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) | | `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
| `` ` `` + 標點鍵 | **強制全形標點**,不管前後是中文還是英文:`` `, `` `` `< `` = `,`、`` `. `` `` `> `` = `。`、`` `/ `` = `?`、`` `; `` = `;`、`` `: `` = `:`、`` `' `` `` `\ `` = `、`、`` `[ `` `` `] `` = `「」`、`` `{ `` `` `} `` = `『』`、`` `( `` `` `) `` = `()`、`` `! `` = `!`、`` `~ `` = `~`、`` `- `` = `——`、`` `= `` = `=`。`` ` `` 後面接其他鍵就是普通的反引號(` ``` ` 照常可用) |
## 運作方式 ## 運作方式
@@ -144,12 +145,12 @@ python3 training/make_parity.py # 權重或特徵改了之後都要重
目前的結果(10,000 筆測試資料): 目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 | | | 送出時整段判斷正確 | 打字過程中顯示正確 | 跟前一個字語言不同時 |
|---|---|---| |---|---|---|---|
| 手調權重 | 97.58% | 86.15% | | 手調權重 | 97.84% | 88.19% | 96.2% |
| 訓練後 | **99.62%** | 87.81% | | 訓練後 | **99.52%** | **91.49%** | **99.3%** |
送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。 訓練資料也模擬了「前一個字是中文或英文」和「按下空白(一聲)的那一刻」,所以「前一個字」的影響(`ctx.zh` / `ctx.lit`)也是訓練出來的,不是手調的。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。 新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
@@ -217,7 +218,7 @@ training/ 合成資料、訓練權重、Swift/Python 一致
- Windows 版還沒在實機上測試過。 - Windows 版還沒在實機上測試過。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。 - 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。 - 很短的字串本質上就有歧義。像 `up`(也是 ㄧㄣ 因)這種常用的,按空白後會先保留,看下一個詞再決定(`up jo4` = 因為、`up the` = up the);其他判斷錯的可以按 Tab 修正,修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。 - 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap) ## 下一步(Roadmap)
+2 -2
View File
@@ -15,9 +15,9 @@
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.6.0</string> <string>0.7.0</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>8</string> <string>12</string>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
+28 -26
View File
@@ -1,39 +1,41 @@
{ {
"about": "AutoZhuyin classifier weights, learned by training/train.py", "about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11", "trained": "2026-11-11",
"examples": 40001, "examples": 40000,
"test_accuracy": { "test_accuracy": {
"commit": 0.9962, "commit": 0.9952,
"typing": 0.8781 "typing": 0.9149
}, },
"weights": { "weights": {
"lit.bias": 0.4276, "ctx.lit": 0.0916,
"lit.common": -0.3465, "ctx.zh": 0.1349,
"lit.common_len": 0.81, "lit.bias": 0.3982,
"lit.dict": -1.7902, "lit.common": -0.2906,
"lit.dict_len": 0.6765, "lit.common_len": 0.8046,
"lit.digits": -2.7403, "lit.dict": -1.6662,
"lit.num_digits": -0.0751, "lit.dict_len": 0.6441,
"lit.num_others": -0.7433, "lit.digits": -2.9689,
"lit.others": -1.5685, "lit.num_digits": -0.018,
"lit.prefix_common_len": 0.2082, "lit.num_others": -0.5554,
"lit.prefix_dict_len": 0.0003, "lit.others": -2.0002,
"lit.short": -0.2397, "lit.prefix_common_len": 0.2831,
"lit.unknown_score": 0.213, "lit.prefix_dict_len": 0.0299,
"lit.short": -0.2482,
"lit.unknown_score": 0.2479,
"lit.upper": 20.0, "lit.upper": 20.0,
"seg.switch": -1.7203, "seg.switch": -1.7596,
"zh.bias": 1.3522, "zh.bias": 1.3423,
"zh.lexscore": 0.26, "zh.lexscore": 0.2917,
"zh.pending_invalid": -5.0, "zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9295, "zh.pending_prefix": -0.9162,
"zh.phrase": 0.4595, "zh.phrase": 0.4499,
"zh.reordered": -1.4348, "zh.reordered": -1.3079,
"zh.stray": -5.0, "zh.stray": -5.0,
"zh.toned_invalid": -5.0, "zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9823, "zh.toned_valid_lex": -0.9018,
"zh.toned_valid_nolex": -1.8742, "zh.toned_valid_nolex": -1.6341,
"zh.unconfirmed": -0.7891, "zh.unconfirmed": -0.7952,
"zh.untoned_mid_invalid": -5.0, "zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.8009 "zh.untoned_mid_valid": -2.7479
} }
} }
+9 -3
View File
@@ -11,22 +11,28 @@ enum Shared {
static let classifier: Classifier = { static let classifier: Classifier = {
let bundle = Bundle.main let bundle = Bundle.main
let start = Date()
let lexicon = Lexicon() let lexicon = Lexicon()
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"), if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
let text = try? String(contentsOf: url, encoding: .utf8) { let data = try? Data(contentsOf: url) {
lexicon.load(text) lexicon.load(data: data)
} }
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt // Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first { if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt") let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) } if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
} }
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt")) let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:] let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json")) let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained) return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
}() }()
} }
+10 -14
View File
@@ -58,9 +58,6 @@ public final class Classifier {
/// Feature weights: the hand-tuned defaults overlaid with trained ones. /// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector public private(set) var weights: FeatureVector
/// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(), public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) { weights: FeatureVector = [:]) {
self.lexicon = lexicon self.lexicon = lexicon
@@ -69,23 +66,22 @@ public final class Classifier {
self.weights = Features.defaultWeights.merging(weights) { _, new in new } self.weights = Features.defaultWeights.merging(weights) { _, new in new }
} }
/// (Zhuyin features or nil, literal features or nil) for one segment. /// (Zhuyin features or nil, literal features or nil) for one segment. `previous` (mode of
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, /// the text committed just before) only affects the first segment.
finishing: Bool) -> (FeatureVector?, FeatureVector?) { public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing), previous: InputMode? = nil) -> (FeatureVector?, FeatureVector?) {
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing)) (Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous),
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous))
} }
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool, private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] { previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing) let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
if whole { if whole {
let b = learned.bias(for: String(seg)) let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b } if b > 0 { zh += b } else { lit -= b }
+69 -1
View File
@@ -51,6 +51,15 @@ public final class CompositionEngine {
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
/// Index of a held space: an English word that is also a common one-syllable Chinese
/// reading ("up " = ㄧㄣ 因) is not committed on space; the next word decides
/// ("up jo4" = 因為, "up the" = up the).
private var held: Int?
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
public private(set) var pendingBacktick = false
static let holdMinScore = -3.5
private var previousMode: InputMode? private var previousMode: InputMode?
private var previousCommitTime = Date.distantPast private var previousCommitTime = Date.distantPast
/// How long the previous segment's mode keeps influencing the next one. /// How long the previous segment's mode keeps influencing the next one.
@@ -71,9 +80,33 @@ public final class CompositionEngine {
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] { private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
guard !keys.isEmpty else { return [] } guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] } if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
if let h = held, h < keys.count, keys[h] == " " {
return heldSegments(keys, held: h, finishing: finishing)
}
return classifier.segment(keys, finishing: finishing, previous: context) return classifier.segment(keys, finishing: finishing, previous: context)
} }
/// The held word is Chinese if the words typed after it start with Chinese.
private func heldSegments(_ keys: [Character], held h: Int, finishing: Bool) -> [Segment] {
let base = h + 1
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
}
if let first = rest.first, first.mode == .zhuyin {
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
}
return [Segment(range: 0..<base, mode: .literal)] + rest
}
/// Whether keys + space also read as a common Chinese syllable (see `held`).
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
let p = ZhuyinParser.parse(keys + [" "])
guard p.pending == nil, p.syllables.count == 1 else { return false }
guard let best = classifier.lexicon.entries(for: p.syllables.map(\.syllable)).first else { return false }
return best.score >= CompositionEngine.holdMinScore
}
/// Current split of the buffer into Zhuyin / literal runs. /// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) } public var segments: [Segment] { segments(of: keys) }
@@ -85,9 +118,11 @@ public final class CompositionEngine {
public var markedPieces: [MarkedPiece] { public var markedPieces: [MarkedPiece] {
let segs = segments let segs = segments
return segs.enumerated().map { i, seg in var pieces = segs.enumerated().map { i, seg in
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode) MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
} }
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
return pieces
} }
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo. /// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
@@ -156,6 +191,7 @@ public final class CompositionEngine {
public func handle(_ key: KeyInput) -> KeyResult { public func handle(_ key: KeyInput) -> KeyResult {
if isShowingCandidates { return handleCandidateKey(key) } if isShowingCandidates { return handleCandidateKey(key) }
if pendingBacktick { return afterBacktick(key) }
switch key { switch key {
case .character(let c): case .character(let c):
@@ -164,6 +200,7 @@ public final class CompositionEngine {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
cursor = nil cursor = nil
keys.removeLast() keys.removeLast()
if let h = held, keys.count <= h { held = nil }
fixed = fixed.filter { $0.key < keys.count } fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] } if keys.isEmpty { forcedMode = nil; fixed = [:] }
return .consumed return .consumed
@@ -214,10 +251,19 @@ public final class CompositionEngine {
cursor = nil // typing always continues at the end cursor = nil // typing always continues at the end
// English mode: every key goes to the app as typed. // English mode: every key goes to the app as typed.
if isEnglishMode && keys.isEmpty { return .passThrough } if isEnglishMode && keys.isEmpty { return .passThrough }
if c == "`" {
pendingBacktick = true
return .consumed
}
if c == " " { if c == " " {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
let segs = segments(of: keys, finishing: true) let segs = segments(of: keys, finishing: true)
if segs.last?.mode == .literal { if segs.last?.mode == .literal {
if held == nil, segs.count == 1, forcedMode == nil, isAmbiguousWord(keys) {
held = keys.count
keys.append(" ")
return .consumed
}
return KeyResult(handled: true, commit: commitCurrent(segs) + " ") return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
} }
if parse(ofLast: segs)?.pending != nil { if parse(ofLast: segs)?.pending != nil {
@@ -254,6 +300,26 @@ public final class CompositionEngine {
return ZhuyinParser.parse(Array(keys[last.range])) return ZhuyinParser.parse(Array(keys[last.range]))
} }
private func afterBacktick(_ key: KeyInput) -> KeyResult {
pendingBacktick = false
if key == .escape || key == .backspace { return .consumed }
let committed = keys.isEmpty ? "" : commitCurrent()
if case .character(let c) = key, let symbol = CompositionEngine.symbols[c] {
return KeyResult(handled: true, commit: committed + symbol)
}
// Not a symbol: the backtick was meant literally; then handle the key as usual.
var r = handle(key)
if !r.handled, case .character(let c) = key { r = KeyResult(handled: true, commit: String(c)) }
return KeyResult(handled: r.handled, commit: committed + "`" + (r.commit ?? ""))
}
/// "`" + key → full-width symbol.
static let symbols: [Character: String] = [
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", "\"": "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
]
static let fullwidth: [Character: String] = [ static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
@@ -326,6 +392,8 @@ public final class CompositionEngine {
public func reset() { public func reset() {
keys = [] keys = []
held = nil
pendingBacktick = false
cursor = nil cursor = nil
forcedMode = nil forcedMode = nil
fixed = [:] fixed = [:]
+51 -16
View File
@@ -8,33 +8,63 @@ import Foundation
/// - a letter-bigram model trained on those words, for words not in either list. /// - a letter-bigram model trained on those words, for words not in either list.
public final class EnglishModel { public final class EnglishModel {
private var common = Set<String>() private var common = Set<String>()
private var dictionary = Set<String>()
private var sortedWords: [String] = []
private var sortedCommon: [String] = [] private var sortedCommon: [String] = []
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
/// binary searches, so no 230k-entry Set has to be built at startup.
private var sortedDict: [String] = []
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
private var unseenBigram = -12.0 private var unseenBigram = -12.0
public init(commonWords: [String], dictionaryWords: [String] = []) { /// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
public init(commonWords: [String], dictionaryWords: [String] = [],
bigram: (table: [String: Double], unseen: Double)? = nil) {
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty }) common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
sortedCommon = common.sorted() sortedCommon = common.sorted()
sortedWords = common.union(dictionary).sorted() var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
dict = Array(Set(dict)).sorted()
}
sortedDict = dict
if let bigram {
self.bigram = bigram.table
unseenBigram = bigram.unseen
} else {
trainBigram() trainBigram()
} }
}
/// Loads common words from a bundle resource file and the system dictionary if present. /// Loads the bundled data files. Falls back to the system word list and to training
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) { /// the bigram table when the precomputed files are missing.
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
func lines(_ url: URL?) -> [String] { func lines(_ url: URL?) -> [String] {
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] } guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
return text.split(whereSeparator: \.isNewline) return text.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) } .map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.hasPrefix("#") } .filter { !$0.isEmpty && !$0.hasPrefix("#") }
} }
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary)) var bigram: (table: [String: Double], unseen: Double)?
if let url = bigramURL, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
bigram = (table, unseen)
}
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
?? URL(fileURLWithPath: "/usr/share/dict/words")
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
} }
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) } public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) } public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
private func inDictionary(_ w: String) -> Bool {
let i = lowerBound(w, in: sortedDict)
return i < sortedDict.count && sortedDict[i] == w
}
/// Some word strictly longer than `p` starts with it (common words or dictionary).
private func isPrefixOfWord(_ p: String) -> Bool {
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
}
/// - Parameter complete: true when the user ended the word (space / punctuation), /// - Parameter complete: true when the user ended the word (space / punctuation),
/// false while still typing (prefixes of words get partial credit). /// false while still typing (prefixes of words get partial credit).
@@ -43,10 +73,10 @@ public final class EnglishModel {
let n = Double(w.count) let n = Double(w.count)
guard !w.isEmpty else { return 0 } guard !w.isEmpty else { return 0 }
if common.contains(w) { return 3 + n } if common.contains(w) { return 3 + n }
if dictionary.contains(w) { return 1 + 0.6 * n } if inDictionary(w) { return 1 + 0.6 * n }
if !complete { if !complete {
if hasPrefix(w, in: sortedCommon) { return 0.4 * n } if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
if hasPrefix(w, in: sortedWords) { return 0.2 * n } if isPrefixOfWord(w) { return 0.2 * n }
} }
// Unknown word: judge by how English-like its letter sequence is. // Unknown word: judge by how English-like its letter sequence is.
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -63,12 +93,12 @@ public final class EnglishModel {
if common.contains(w) { if common.contains(w) {
f["lit.common"] = 1 f["lit.common"] = 1
f["lit.common_len"] = n f["lit.common_len"] = n
} else if dictionary.contains(w) { } else if inDictionary(w) {
f["lit.dict"] = 1 f["lit.dict"] = 1
f["lit.dict_len"] = n f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) { } else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) { } else if !complete && isPrefixOfWord(w) {
f["lit.prefix_dict_len"] = n f["lit.prefix_dict_len"] = n
} else { } else {
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -85,12 +115,17 @@ public final class EnglishModel {
return total / Double(chars.count - 1) return total / Double(chars.count - 1)
} }
private func hasPrefix(_ p: String, in words: [String]) -> Bool { private func lowerBound(_ p: String, in words: [String]) -> Int {
var lo = 0, hi = words.count var lo = 0, hi = words.count
while lo < hi { while lo < hi {
let mid = (lo + hi) / 2 let mid = (lo + hi) / 2
if words[mid] < p { lo = mid + 1 } else { hi = mid } if words[mid] < p { lo = mid + 1 } else { hi = mid }
} }
return lo
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
var lo = lowerBound(p, in: words)
// words[lo] is the first word >= p; skip an exact match, then test the next one. // words[lo] is the first word >= p; skip an exact match, then test the next one.
if lo < words.count && words[lo] == p { lo += 1 } if lo < words.count && words[lo] == p { lo += 1 }
return lo < words.count && words[lo].hasPrefix(p) return lo < words.count && words[lo].hasPrefix(p)
@@ -99,7 +134,7 @@ public final class EnglishModel {
private func trainBigram() { private func trainBigram() {
var pair: [String: Double] = [:] var pair: [String: Double] = [:]
var first: [String: Double] = [:] var first: [String: Double] = [:]
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) { for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
let weight = common.contains(w) ? 5.0 : 1.0 let weight = common.contains(w) ? 5.0 : 1.0
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
for i in 1..<chars.count { for i in 1..<chars.count {
+17 -4
View File
@@ -42,6 +42,9 @@ public enum Features {
"lit.others": -1.5, "lit.others": -1.5,
// Segmentation // Segmentation
"seg.switch": -3.0, "seg.switch": -3.0,
// Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
] ]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid. /// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
@@ -58,7 +61,8 @@ public enum Features {
/// Features of `keys` read as Zhuyin, or nil if impossible. /// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true, public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys) let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone. // A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil } if !isLast && parse.pending != nil { return nil }
@@ -107,18 +111,27 @@ public enum Features {
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) { if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed") add("zh.unconfirmed")
} }
if afterOther { add("seg.switch") } if afterOther {
add("seg.switch")
} else if previous == .zhuyin {
add("ctx.zh")
}
return f return f
} }
/// Features of `keys` output as typed, or nil if impossible. /// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true, public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil } if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1] var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v } func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther { add("seg.switch") } if afterOther {
add("seg.switch")
} else if previous == .literal {
add("ctx.lit")
}
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") } if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter }) let letters = String(keys.filter { $0.isLetter })
+138 -28
View File
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
/// File format, one entry per line (`#` starts a comment): /// File format, one entry per line (`#` starts a comment):
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2 /// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
/// The phrase/reading columns may be in either order; the score is optional. /// The phrase/reading columns may be in either order; the score is optional.
///
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
/// call `prepareTonelessIndex()` from a background queue to have it ready.
public final class Lexicon { public final class Lexicon {
private var exact: [String: [LexiconEntry]] = [:] private var exact: [String: [LexiconEntry]] = [:]
private var toneless: [String: [LexiconEntry]] = [:] private var tonelessIndex: [String: [LexiconEntry]]?
private let lock = NSLock()
public private(set) var maxPhraseLength = 1 public private(set) var maxPhraseLength = 1
public init() {} public init() {}
public convenience init(contentsOf url: URL) throws { public convenience init(contentsOf url: URL) throws {
self.init() self.init()
load(try String(contentsOf: url, encoding: .utf8)) load(data: try Data(contentsOf: url))
} }
public func load(_ text: String) { public func load(_ text: String) {
for line in text.split(whereSeparator: \.isNewline) { load(data: Data(text.utf8))
let trimmed = line.trimmingCharacters(in: .whitespaces)
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
guard cols.count >= 2 else { continue }
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
guard Lexicon.isBopomofo(reading) else { continue }
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
add(reading: reading, phrase: phrase, score: score)
}
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
} }
public func add(reading rawReading: String, phrase: String, score: Double) { public func load(data: Data) {
let entry = LexiconEntry(phrase: phrase, score: score) lock.lock()
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix. defer { lock.unlock() }
let syllables = rawReading.split(separator: "-").map { s -> String in exact.reserveCapacity(exact.count + data.count / 30)
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s) data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
let bytes = raw.bindMemory(to: UInt8.self)
let n = bytes.count
var i = 0
while i < n {
var end = i
while end < n && bytes[end] != 0x0A { end += 1 }
parseLine(bytes, i, end)
i = end + 1
} }
let reading = syllables.joined(separator: "-")
maxPhraseLength = max(maxPhraseLength, syllables.count)
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) {
exact[reading, default: []].append(entry)
} }
let key = Lexicon.stripTones(reading) for (key, entries) in exact where entries.count > 1 {
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) { exact[key] = entries.sorted { $0.score > $1.score }
toneless[key]![i].score = max(toneless[key]![i].score, score) }
tonelessIndex = nil // rebuilt on demand
}
/// Splits one line into up to three space/tab separated fields.
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
var fields: [Range<Int>] = []
var i = start
while i < end && fields.count < 3 {
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
if i >= end { break }
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
let s = i
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
fields.append(s..<i)
}
guard fields.count >= 2 else { return }
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
guard Lexicon.isBopomofo(b, r) else { return }
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
var prefixNeutral = b[r.lowerBound] == 0xCB
var k = r.lowerBound
while !prefixNeutral && k + 2 < r.upperBound {
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
k += 1
}
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
}
/// Parses "-3.9151"-style decimals without creating a String.
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
var i = r.lowerBound
var sign = 1.0
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
var value = 0.0, scale = 1.0, digits = 0, fraction = false
while i < r.upperBound {
let c = b[i]
if c >= 0x30 && c <= 0x39 {
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
digits += 1
} else if c == 0x2E && !fraction {
fraction = true
} else { } else {
toneless[key, default: []].append(entry) return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
} }
i += 1
}
return digits > 0 ? sign * value : nil
}
/// Adds one entry (call `load` for bulk data; this does not re-sort).
public func add(reading: String, phrase: String, score: Double) {
lock.lock()
defer { lock.unlock() }
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
exact[key]?.sort { $0.score > $1.score }
tonelessIndex = nil
}
@discardableResult
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
var reading = rawReading
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
if normalise {
reading = reading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
}.joined(separator: "-")
}
maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
if exact[reading] == nil {
exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
} else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
}
return reading
}
/// Builds the index that ignores tones (for untoned syllables). Safe to call from a
/// background queue; lookups build it themselves if it is not ready yet.
public func prepareTonelessIndex() {
lock.lock()
defer { lock.unlock() }
_ = buildTonelessIfNeeded()
}
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
if let index = tonelessIndex { return index }
var best: [String: [String: Double]] = [:]
for (reading, entries) in exact {
let key = Lexicon.stripTones(reading)
for e in entries {
if let s = best[key]?[e.phrase], s >= e.score { continue }
best[key, default: [:]][e.phrase] = e.score
}
}
var index: [String: [LexiconEntry]] = [:]
index.reserveCapacity(best.count)
for (key, phrases) in best {
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
}
tonelessIndex = index
return index
} }
/// Entries for a sequence of readings. Untoned syllables match any tone. /// Entries for a sequence of readings. Untoned syllables match any tone.
public func entries(for syllables: [Syllable]) -> [LexiconEntry] { public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
lock.lock()
defer { lock.unlock() }
if syllables.allSatisfy({ $0.tone != nil }) { if syllables.allSatisfy({ $0.tone != nil }) {
return exact[syllables.map(\.reading).joined(separator: "-")] ?? [] return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
} }
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? [] return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
} }
public func bestScore(for syllable: Syllable) -> Double? { public func bestScore(for syllable: Syllable) -> Double? {
@@ -81,4 +182,13 @@ public final class Lexicon {
guard let first = s.unicodeScalars.first else { return false } guard let first = s.unicodeScalars.first else { return false }
return (0x3105...0x312F).contains(first.value) || first == "˙" return (0x3105...0x312F).contains(first.value) || first == "˙"
} }
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
guard r.count >= 2 else { return false }
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
let c = b[r.lowerBound + 2]
return c >= 0x85 && c <= 0xAF
}
} }
@@ -6,7 +6,10 @@ private let resources = URL(fileURLWithPath: #filePath)
.appendingPathComponent("Resources") .appendingPathComponent("Resources")
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")) private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase { final class ParserTests: XCTestCase {
@@ -265,10 +268,13 @@ final class ParityTests: XCTestCase {
for fx in fixtures { for fx in fixtures {
let keys = Array(fx["keys"] as! String) let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" } let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map { let previous = (fx["previous"] as? String).flatMap(InputMode.init(rawValue:))
let got = c.segment(keys, finishing: fx["finishing"] as! Bool, previous: previous).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)" "\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
} }
if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") } if got != expected {
mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!) previous=\(String(describing: previous)): \(got) vs \(expected)")
}
} }
XCTAssertGreaterThan(fixtures.count, 300) XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n")) XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
@@ -408,3 +414,100 @@ final class CursorTests: XCTestCase {
XCTAssertTrue(e.isShowingCandidates) XCTAssertTrue(e.isShowingCandidates)
} }
} }
final class FirstToneAfterEnglishTests: XCTestCase {
/// After an English word, a first-tone syllable + space must stay Chinese (機, 吃, 書…).
func testFirstToneAfterEnglish() {
for keys in ["ru", "t", "gj", "vup", "tj", "el"] {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out, "hello ", keys)
XCTAssertEqual(e.markedText.count, 1, keys) // still composing one Chinese character
}
}
}
final class HeldWordTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// "up " is also ㄧㄣ (因): the next word decides.
func testNextWordDecides() {
let cases = ["up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testHeldWordStaysComposing() {
let e = makeEngine()
XCTAssertEqual(type("up ", into: e), "")
XCTAssertEqual(e.markedText, "up ")
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "up")
XCTAssertEqual(type(" the ", into: e), "up the ")
}
}
final class BacktickSymbolTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testForcedSymbols() {
let cases = ["`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testPlainBacktick() {
let e = makeEngine()
XCTAssertEqual(type("`", into: e), "")
XCTAssertEqual(e.markedText, "`")
XCTAssertEqual(type("ls` ", into: e), "`ls` ")
}
func testBackspaceCancels() {
let e = makeEngine()
_ = type("su3`", into: e)
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "你")
}
}
final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws {
var t = Date()
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
}
}
+3
View File
@@ -19,6 +19,9 @@ mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
+6 -6
View File
@@ -23,14 +23,14 @@ def main():
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH)) clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"), gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7) os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f) for k in HAND_PICKED for f in (False, True)] cases = [(k, f, p) for k in HAND_PICKED for f in (False, True) for p in (None, "literal")]
while len(cases) < 400: while len(cases) < 500:
ex = gen.example() ex = gen.example()
if len(ex["keys"]) <= 28: if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"])) cases.append((ex["keys"], ex["finishing"], ex["previous"]))
cases.append((gen.prefix_of(ex)["keys"], False)) cases.append((gen.prefix_of(ex)["keys"], False, ex["previous"]))
fixtures = [{"keys": k, "finishing": f, fixtures = [{"keys": k, "finishing": f, "previous": p,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases] "segments": [[s, e, m] for s, e, m in clf.segment(list(k), f, p)]} for k, f, p in cases]
with open(OUT, "w", encoding="utf-8") as fh: with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0) json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT)) print("wrote %d fixtures to %s" % (len(fixtures), OUT))
+4971 -3231
View File
File diff suppressed because it is too large Load Diff
+17 -2
View File
@@ -102,16 +102,31 @@ class Generator:
labels = "".join(m * len(p) for m, p in parts) labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l": if parts[-1][0] == "l":
finishing = True # English / numbers end with a space finishing = True # English / numbers end with a space
elif keys[-1] == " " and self.rng.random() < 0.6:
# The moment the space (first tone) is pressed: the buffer has no space yet and
# the input method decides "first tone" vs "English word + space".
keys, labels, finishing = keys[:-1], labels[:-1], True
kind += "/tone1"
else: else:
# Chinese ends with Enter, or space after a finished (toned) syllable. # Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5 finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind} return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind,
"previous": self.previous_mode(labels[0])}
def previous_mode(self, first_label):
"""Mode of the text committed just before: none, mostly the same, sometimes the other."""
r = self.rng.random()
if r < 0.35:
return None
same = core.ZHUYIN if first_label == "z" else core.LITERAL
other = core.LITERAL if first_label == "z" else core.ZHUYIN
return same if r < 0.80 else other
def prefix_of(self, ex): def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing.""" """The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1 cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False, return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix"} "kind": ex["kind"] + "/prefix", "previous": ex["previous"]}
def _cumulative(ws): def _cumulative(ws):
+19 -10
View File
@@ -45,12 +45,12 @@ def labels_of(segs, n):
return "".join(lab) return "".join(lab)
def seg_features(clf, keys, segs, finishing): def seg_features(clf, keys, segs, finishing, previous):
"""Summed features of a segmentation, or None if any segment is impossible.""" """Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float) total = defaultdict(float)
n = len(keys) n = len(keys)
for idx, (s, e, m) in enumerate(segs): for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing) fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing, previous if idx == 0 else None)
f = fz if m == core.ZHUYIN else fl f = fz if m == core.ZHUYIN else fl
if f is None: if f is None:
return None return None
@@ -62,12 +62,13 @@ def seg_features(clf, keys, segs, finishing):
def evaluate(clf, weights, data): def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0]) by_kind = defaultdict(lambda: [0, 0])
for ex in data: for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights) segs = clf.segment(list(ex["keys"]), ex["finishing"], ex["previous"], weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"] ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]] kind = "prefix" if ex["kind"].endswith("/prefix") else ex["kind"]
bucket[0] += ok for k in (kind, "after-other") if _after_other(ex) else (kind,):
bucket[1] += 1 by_kind[k][0] += ok
commit = [v for k, v in by_kind.items() if k != "prefix"] by_kind[k][1] += 1
commit = [v for k, v in by_kind.items() if k not in ("prefix", "after-other")]
summary = { summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)), "commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]), "prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
@@ -76,6 +77,14 @@ def evaluate(clf, weights, data):
return summary return summary
def _after_other(ex):
"""Committed buffer whose first segment differs from the text typed just before."""
if ex["kind"].endswith("/prefix") or ex["previous"] is None:
return False
first = core.ZHUYIN if ex["labels"][0] == "z" else core.LITERAL
return ex["previous"] != first
def show(name, s): def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items()) kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds)) print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
@@ -92,11 +101,11 @@ def train(clf, data, epochs, lr, seed):
t0 = time.time() t0 = time.time()
for ex in data: for ex in data:
keys = list(ex["keys"]) keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], None, w) pred = clf.segment(keys, ex["finishing"], ex["previous"], w)
if labels_of(pred, len(keys)) != ex["labels"]: if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"]) gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"]) fg = seg_features(clf, keys, gold, ex["finishing"], ex["previous"])
fp = seg_features(clf, keys, pred, ex["finishing"]) fp = seg_features(clf, keys, pred, ex["finishing"], ex["previous"])
if fg is not None and fp is not None: if fg is not None and fp is not None:
mistakes += 1 mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0) scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
+6 -10
View File
@@ -30,7 +30,6 @@ class LearnedPreferences:
class Classifier: class Classifier:
context_weight = 1.5
def __init__(self, lexicon, english, learned=None, weights=None): def __init__(self, lexicon, english, learned=None, weights=None):
self.lexicon = lexicon self.lexicon = lexicon
@@ -49,20 +48,17 @@ class Classifier:
except (OSError, ValueError): except (OSError, ValueError):
return {} return {}
def segment_features(self, seg, is_last, after_other, finishing): def segment_features(self, seg, is_last, after_other, finishing, previous=None):
"""(zhuyin features or None, literal features or None) for one segment.""" """(zhuyin features or None, literal features or None) for one segment.
`previous` (mode of the text committed just before) only affects the first segment."""
seg = list(seg) seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing), return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing, previous),
literal_features(seg, self.english, is_last, after_other, finishing)) literal_features(seg, self.english, is_last, after_other, finishing, previous))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights): def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing) fz, fl = self.segment_features(seg, is_last, after_other, finishing, previous)
zh = dot(weights, fz) if fz is not None else NEG_INF zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF lit = dot(weights, fl) if fl is not None else NEG_INF
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
if whole: if whole:
b = self.learned.bias("".join(seg)) b = self.learned.bias("".join(seg))
if b > 0: if b > 0:
+72 -3
View File
@@ -20,6 +20,19 @@ FULLWIDTH = {
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
} }
# An English word that is also a common one-syllable Chinese reading ("up " = ㄧㄣ 因)
# is not committed on space; the next word decides ("up jo4" = 因為, "up the" = up the).
HOLD_MIN_SCORE = -3.5
# "`" then one of these keys always gives the full-width symbol, whatever the context.
# "`" followed by anything else is just a backtick (so ``` still works).
BACKTICK = "`"
SYMBOLS = {
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", '"': "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
}
PASS = (False, None) PASS = (False, None)
CONSUMED = (True, None) CONSUMED = (True, None)
@@ -37,12 +50,15 @@ class CompositionEngine:
# English mode, toggled by a Shift tap when nothing is being typed: keys go straight # English mode, toggled by a Shift tap when nothing is being typed: keys go straight
# to the app (no composition, no underline) until Shift is tapped again. # to the app (no composition, no underline) until Shift is tapped again.
self.english_mode = False self.english_mode = False
self.backtick = False # "`" typed; the next key may be a forced symbol
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text} self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = [] self.candidates = []
self.candidate_index = 0 self.candidate_index = 0
self._candidate_segment_start = 0 self._candidate_segment_start = 0
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end. # Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
self.cursor = None self.cursor = None
# Index of a held space (see HOLD_MIN_SCORE), or None.
self.held = None
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None self._cache_key = None
@@ -60,12 +76,35 @@ class CompositionEngine:
if self.forced_mode: if self.forced_mode:
return [(0, len(keys), self.forced_mode)] return [(0, len(keys), self.forced_mode)]
ctx = self.context ctx = self.context
cache_key = (tuple(keys), finishing, ctx) held = self.held if self.held is not None and self.held < len(keys) and keys[self.held] == " " else None
cache_key = (tuple(keys), finishing, ctx, held)
if cache_key != self._cache_key: if cache_key != self._cache_key:
self._cache_key = cache_key self._cache_key = cache_key
if held is None:
self._cache_value = self.classifier.segment(keys, finishing, ctx) self._cache_value = self.classifier.segment(keys, finishing, ctx)
else:
self._cache_value = self._held_segments(keys, held, finishing)
return self._cache_value return self._cache_value
def _held_segments(self, keys, held, finishing):
"""The held word is Chinese if the words typed after it start with Chinese."""
rest = keys[held + 1:]
base = held + 1
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
if rest_segs and rest_segs[0][2] == ZHUYIN:
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
return [(0, base, LITERAL)] + rest_segs
def _is_ambiguous_word(self, keys):
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
if not all(k.isascii() and k.isalpha() for k in keys):
return False
p = bopomofo.parse(list(keys) + [" "])
if p.pending is not None or len(p.syllables) != 1:
return False
entries = self.classifier.lexicon.entries([s.syllable for s in p.syllables])
return bool(entries) and entries[0][1] >= HOLD_MIN_SCORE
@property @property
def segments(self): def segments(self):
return self._segments(self.keys) return self._segments(self.keys)
@@ -142,7 +181,10 @@ class CompositionEngine:
@property @property
def marked_pieces(self): def marked_pieces(self):
segs = self.segments segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)] pieces = [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
if self.backtick:
pieces.append((BACKTICK, LITERAL))
return pieces
@property @property
def marked_text(self): def marked_text(self):
@@ -156,7 +198,7 @@ class CompositionEngine:
def wants(self, key): def wants(self, key):
"""Whether handle(key) would consume the key, without changing state.""" """Whether handle(key) would consume the key, without changing state."""
if self.candidates or key == SHIFT_TAP: if self.candidates or key == SHIFT_TAP or self.backtick:
return True return True
if self.english_mode and not self.keys: if self.english_mode and not self.keys:
return False # English mode: everything goes to the app as typed return False # English mode: everything goes to the app as typed
@@ -173,11 +215,15 @@ class CompositionEngine:
def handle(self, key): def handle(self, key):
if self.candidates: if self.candidates:
return self._handle_candidate_key(key) return self._handle_candidate_key(key)
if self.backtick:
return self._after_backtick(key)
if key == BACKSPACE: if key == BACKSPACE:
if not self.keys: if not self.keys:
return PASS return PASS
self.cursor = None self.cursor = None
self.keys.pop() self.keys.pop()
if self.held is not None and len(self.keys) <= self.held:
self.held = None
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)} self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys: if not self.keys:
self.forced_mode = None self.forced_mode = None
@@ -220,6 +266,19 @@ class CompositionEngine:
return CONSUMED if self.keys else PASS return CONSUMED if self.keys else PASS
return self._handle_character(key) return self._handle_character(key)
def _after_backtick(self, key):
self.backtick = False
if key in (ESCAPE, BACKSPACE):
return CONSUMED
committed = self._commit() if self.keys else ""
if key in SYMBOLS:
return True, committed + SYMBOLS[key]
# Not a symbol: the backtick was meant literally; then handle the key as usual.
handled, more = self.handle(key)
if not handled and len(key) == 1:
handled, more = True, key
return handled, committed + BACKTICK + (more or "")
def _last_parse(self, segs): def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN: if not segs or segs[-1][2] != ZHUYIN:
return None return None
@@ -230,11 +289,19 @@ class CompositionEngine:
self.cursor = None # typing always continues at the end self.cursor = None # typing always continues at the end
if self.english_mode and not self.keys: if self.english_mode and not self.keys:
return PASS # English mode: every key goes to the app as typed return PASS # English mode: every key goes to the app as typed
if c == BACKTICK:
self.backtick = True
return CONSUMED
if c == " ": if c == " ":
if not self.keys: if not self.keys:
return PASS return PASS
segs = self._segments(self.keys, finishing=True) segs = self._segments(self.keys, finishing=True)
if segs[-1][2] == LITERAL: if segs[-1][2] == LITERAL:
if (self.held is None and len(segs) == 1 and not self.forced_mode
and self._is_ambiguous_word(self.keys)):
self.held = len(self.keys)
self.keys.append(" ")
return CONSUMED
return True, self._commit(segs) + " " return True, self._commit(segs) + " "
p = self._last_parse(segs) p = self._last_parse(segs)
if p is not None and p.pending is not None: if p is not None and p.pending is not None:
@@ -319,6 +386,8 @@ class CompositionEngine:
def reset(self): def reset(self):
self.keys = [] self.keys = []
self.held = None
self.backtick = False
self.cursor = None self.cursor = None
self.forced_mode = None self.forced_mode = None
self.fixed = {} self.fixed = {}
+9 -2
View File
@@ -41,6 +41,9 @@ DEFAULT_WEIGHTS = {
"lit.others": -1.5, "lit.others": -1.5,
# Segmentation # Segmentation
"seg.switch": -3.0, "seg.switch": -3.0,
# Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
} }
@@ -52,7 +55,7 @@ def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False): def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` read as Zhuyin, or None if impossible.""" """Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys) parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None: if not is_last and parse.pending is not None:
@@ -99,10 +102,12 @@ def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=Fa
_add(f, "zh.unconfirmed") _add(f, "zh.unconfirmed")
if after_other: if after_other:
_add(f, "seg.switch") _add(f, "seg.switch")
elif previous == "zhuyin":
_add(f, "ctx.zh")
return f return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False): def literal_features(keys, english, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` output as typed, or None if impossible.""" """Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys: if not keys or " " in keys:
return None return None
@@ -110,6 +115,8 @@ def literal_features(keys, english, is_last=True, after_other=False, finishing=F
f = {"lit.bias": 1.0} f = {"lit.bias": 1.0}
if after_other: if after_other:
_add(f, "seg.switch") _add(f, "seg.switch")
elif previous == "literal":
_add(f, "ctx.lit")
if any(c.isupper() for c in keys): if any(c.isupper() for c in keys):
_add(f, "lit.upper") _add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha()) letters = "".join(c for c in keys if c.isalpha())
+18 -12
View File
@@ -16,7 +16,7 @@ class Lexicon:
def __init__(self): def __init__(self):
self.exact = {} self.exact = {}
self.toneless = {} self._toneless = None # built on first use (only untoned syllables need it)
self.max_phrase_length = 1 self.max_phrase_length = 1
@classmethod @classmethod
@@ -42,9 +42,10 @@ class Lexicon:
except ValueError: except ValueError:
score = -6.0 score = -6.0
self.add(reading, phrase, score) self.add(reading, phrase, score)
for table in (self.exact, self.toneless): for v in self.exact.values():
for v in table.values(): if len(v) > 1:
v.sort(key=lambda e: -e[1]) v.sort(key=lambda e: -e[1])
self._toneless = None
def add(self, reading, phrase, score): def add(self, reading, phrase, score):
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")] syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
@@ -53,15 +54,20 @@ class Lexicon:
entries = self.exact.setdefault(reading, []) entries = self.exact.setdefault(reading, [])
if not any(p == phrase for p, _ in entries): if not any(p == phrase for p, _ in entries):
entries.append((phrase, score)) entries.append((phrase, score))
key = _strip_tones(reading) self._toneless = None
tl = self.toneless.setdefault(key, [])
for i, (p, s) in enumerate(tl): @property
if p == phrase: def toneless(self):
if score > s: """Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
tl[i] = (p, score) if self._toneless is None:
break best = {}
else: for reading, entries in self.exact.items():
tl.append((phrase, score)) bucket = best.setdefault(_strip_tones(reading), {})
for phrase, score in entries:
if score > bucket.get(phrase, float("-inf")):
bucket[phrase] = score
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
return self._toneless
def entries(self, syllables): def entries(self, syllables):
if all(s.tone is not None for s in syllables): if all(s.tone is not None for s in syllables):
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.6.0", "version": "0.7.0",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+3
View File
@@ -34,6 +34,9 @@
打到一半時按,同 Tab 整段翻轉 打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文) Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!: < > ? ! : 中文語境下自動變成全形 ,。?!:
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
`' = 、 `[ `] = 「」 `( `) = () `! = !
` 後面接其他鍵就是普通的反引號
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
+44
View File
@@ -271,5 +271,49 @@ class CursorTests(unittest.TestCase):
self.assertTrue(e.showing_candidates) self.assertTrue(e.showing_candidates)
class FirstToneAfterEnglishTests(unittest.TestCase):
def test_first_tone_after_english(self):
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
e = make_engine()
self.assertEqual(type_keys("hello " + keys + " ", e), "hello ", keys)
self.assertEqual(len(e.marked_text), 1, keys)
class HeldWordTests(unittest.TestCase):
def test_next_word_decides(self):
cases = {"up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_held_word_stays_composing(self):
e = make_engine()
self.assertEqual(type_keys("up ", e), "")
self.assertEqual(e.marked_text, "up ")
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "up")
self.assertEqual(type_keys(" the ", e), "up the ")
class BacktickSymbolTests(unittest.TestCase):
def test_forced_symbols(self):
cases = {"`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_plain_backtick(self):
e = make_engine()
self.assertEqual(type_keys("`", e), "")
self.assertEqual(e.marked_text, "`")
self.assertEqual(type_keys("ls` ", e), "`ls` ")
def test_escape_cancels(self):
e = make_engine()
type_keys("su3`", e)
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "你")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()