Compare commits

..
10 Commits
Author SHA1 Message Date
ays953 1dacb8f164 Release v0.6.2 2026-11-11 12:13:15 +08:00
ays953andClaude Opus 5.5 bfeed22916 Cut input method startup from ~0.95 s to ~0.15 s
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
  parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
  Windows build) instead of retraining the bigram model on every launch, and
  uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:13:03 +08:00
ays953 bb9cb0c4e0 Release v0.6.1 2026-11-11 12:00:10 +08:00
ays953andClaude Opus 5.5 2c5c70c066 Learn the previous-word context; fix first-tone characters after English
After an English word, 23 of 50 common first-tone characters (機, 吃, 書…)
were read as English when space was pressed: the fixed +1.5 context bonus
was far too strong for the trained weight scale, and the training data never
contained the "space just pressed on a first-tone syllable" decision.

Context is now the ctx.zh / ctx.lit features, trained with the rest; the
synthetic data includes the previous mode and the first-tone space decision.
Test accuracy: commit 98.22% -> 99.52%, typing 88.73% -> 91.49%, and when the
language differs from the previous word 92.0% -> 99.3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:59:57 +08:00
ays953 17f7d6f50d Release v0.6.0 2026-11-11 11:44:38 +08:00
ays953andClaude Opus 5.5 7e173a1da8 Make the Shift tap a persistent English-mode toggle
A Shift tap with nothing composed now switches English mode on until the
next tap, instead of forcing only the next word: space, Enter and other keys
no longer drop back to automatic detection. In English mode every key goes
straight to the app, so there is no composition or underline. A Shift tap
while typing still flips the buffer like Tab. Windows shows "英文" /
"中英自動" when toggling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:44:25 +08:00
ays953 fa0b78e5a4 Release v0.5.1 2026-11-11 11:36:12 +08:00
ays953andClaude Opus 5.5 e928d08fdc Type forced English directly, without an underlined composition
After a Shift tap, letters, digits and word punctuation go straight to the
app instead of into marked text, so nothing is underlined. Space, Enter, Tab,
Escape or another symbol ends the forced word and automatic detection
resumes; backspace is handled by the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:35:59 +08:00
ays953 82f20a6551 Release v0.5.0 2026-11-11 10:37:37 +08:00
ays953andClaude Opus 5.5 e6e5004630 Add a composition cursor so candidates can be chosen for any character
Left/Right move a cursor between Zhuyin syllables in the composition; Down
opens candidates for the phrase ending at the cursor instead of always the
last syllable. Typing or backspace returns the cursor to the end. Candidates
also work for Chinese that is followed by English in the same buffer. The Mac
caret and PIME composition cursor follow it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:37:24 +08:00
24 changed files with 5817 additions and 3631 deletions
+10 -9
View File
@@ -12,10 +12,10 @@
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示) - **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分 - **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對 - **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **單按 Shift 強制英文**:要打英文時先單按一下 Shift,接下來這個字照打、保持小寫 - **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小) - **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」 - **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:按 ↓ 選最後一個字或詞,選過的會固定下來 - **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,` - **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權) - **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
@@ -51,9 +51,10 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 | | 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 | | Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 | | **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字,Esc 關閉 | | ← → | 在組字區移動游標,選要改哪個字 |
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 | | Esc | 取消組字 |
| **單按 Shift** | 沒在打字時:下一個字強制英文(保持小寫),送出後恢復自動判斷,再按一次取消。打到一半時:同 Tab,整段翻轉 | | **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
| Shift + 字母 | 打大寫字母(也會強制英文) | | Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` | | `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) | | `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
@@ -143,12 +144,12 @@ python3 training/make_parity.py # 權重或特徵改了之後都要重
目前的結果(10,000 筆測試資料): 目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 | | | 送出時整段判斷正確 | 打字過程中顯示正確 | 跟前一個字語言不同時 |
|---|---|---| |---|---|---|---|
| 手調權重 | 97.58% | 86.15% | | 手調權重 | 97.84% | 88.19% | 96.2% |
| 訓練後 | **99.62%** | 87.81% | | 訓練後 | **99.52%** | **91.49%** | **99.3%** |
送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。 訓練資料也模擬了「前一個字是中文或英文」和「按下空白(一聲)的那一刻」,所以「前一個字」的影響(`ctx.zh` / `ctx.lit`)也是訓練出來的,不是手調的。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。 新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
+2 -2
View File
@@ -15,9 +15,9 @@
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.4.1</string> <string>0.6.2</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>5</string> <string>10</string>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
+28 -26
View File
@@ -1,39 +1,41 @@
{ {
"about": "AutoZhuyin classifier weights, learned by training/train.py", "about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11", "trained": "2026-11-11",
"examples": 40001, "examples": 40000,
"test_accuracy": { "test_accuracy": {
"commit": 0.9962, "commit": 0.9952,
"typing": 0.8781 "typing": 0.9149
}, },
"weights": { "weights": {
"lit.bias": 0.4276, "ctx.lit": 0.0916,
"lit.common": -0.3465, "ctx.zh": 0.1349,
"lit.common_len": 0.81, "lit.bias": 0.3982,
"lit.dict": -1.7902, "lit.common": -0.2906,
"lit.dict_len": 0.6765, "lit.common_len": 0.8046,
"lit.digits": -2.7403, "lit.dict": -1.6662,
"lit.num_digits": -0.0751, "lit.dict_len": 0.6441,
"lit.num_others": -0.7433, "lit.digits": -2.9689,
"lit.others": -1.5685, "lit.num_digits": -0.018,
"lit.prefix_common_len": 0.2082, "lit.num_others": -0.5554,
"lit.prefix_dict_len": 0.0003, "lit.others": -2.0002,
"lit.short": -0.2397, "lit.prefix_common_len": 0.2831,
"lit.unknown_score": 0.213, "lit.prefix_dict_len": 0.0299,
"lit.short": -0.2482,
"lit.unknown_score": 0.2479,
"lit.upper": 20.0, "lit.upper": 20.0,
"seg.switch": -1.7203, "seg.switch": -1.7596,
"zh.bias": 1.3522, "zh.bias": 1.3423,
"zh.lexscore": 0.26, "zh.lexscore": 0.2917,
"zh.pending_invalid": -5.0, "zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9295, "zh.pending_prefix": -0.9162,
"zh.phrase": 0.4595, "zh.phrase": 0.4499,
"zh.reordered": -1.4348, "zh.reordered": -1.3079,
"zh.stray": -5.0, "zh.stray": -5.0,
"zh.toned_invalid": -5.0, "zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9823, "zh.toned_valid_lex": -0.9018,
"zh.toned_valid_nolex": -1.8742, "zh.toned_valid_nolex": -1.6341,
"zh.unconfirmed": -0.7891, "zh.unconfirmed": -0.7952,
"zh.untoned_mid_invalid": -5.0, "zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.8009 "zh.untoned_mid_valid": -2.7479
} }
} }
+12 -4
View File
@@ -11,22 +11,28 @@ enum Shared {
static let classifier: Classifier = { static let classifier: Classifier = {
let bundle = Bundle.main let bundle = Bundle.main
let start = Date()
let lexicon = Lexicon() let lexicon = Lexicon()
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"), if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
let text = try? String(contentsOf: url, encoding: .utf8) { let data = try? Data(contentsOf: url) {
lexicon.load(text) lexicon.load(data: data)
} }
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt // Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first { if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt") let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) } if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
} }
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt")) let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:] let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json")) let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained) return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
}() }()
} }
@@ -117,8 +123,10 @@ final class InputController: IMKInputController {
])) ]))
} }
let text = marked.string let text = marked.string
// Caret at the candidate cursor (UTF-16 offset of the first `cursorOffset` characters).
let caret = String(text.prefix(engine.cursorOffset)).utf16.count
client.setMarkedText(marked, client.setMarkedText(marked,
selectionRange: NSRange(location: (text as NSString).length, length: 0), selectionRange: NSRange(location: caret, length: 0),
replacementRange: NSRange(location: NSNotFound, length: 0)) replacementRange: NSRange(location: NSNotFound, length: 0))
if engine.isShowingCandidates { if engine.isShowingCandidates {
+10 -14
View File
@@ -58,9 +58,6 @@ public final class Classifier {
/// Feature weights: the hand-tuned defaults overlaid with trained ones. /// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector public private(set) var weights: FeatureVector
/// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(), public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) { weights: FeatureVector = [:]) {
self.lexicon = lexicon self.lexicon = lexicon
@@ -69,23 +66,22 @@ public final class Classifier {
self.weights = Features.defaultWeights.merging(weights) { _, new in new } self.weights = Features.defaultWeights.merging(weights) { _, new in new }
} }
/// (Zhuyin features or nil, literal features or nil) for one segment. /// (Zhuyin features or nil, literal features or nil) for one segment. `previous` (mode of
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, /// the text committed just before) only affects the first segment.
finishing: Bool) -> (FeatureVector?, FeatureVector?) { public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing), previous: InputMode? = nil) -> (FeatureVector?, FeatureVector?) {
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing)) (Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous),
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous))
} }
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool, private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] { previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing) let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
if whole { if whole {
let b = learned.bias(for: String(seg)) let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b } if b > 0 { zh += b } else { lit -= b }
+93 -35
View File
@@ -35,16 +35,19 @@ public final class CompositionEngine {
public let converter: Converter public let converter: Converter
public private(set) var keys: [Character] = [] public private(set) var keys: [Character] = []
/// Set by Tab / Shift tap: overrides the classifier for the whole buffer. /// Set by Tab / a Shift tap while typing: overrides the classifier for the whole
/// buffer. It is a correction, learned on commit.
public private(set) var forcedMode: InputMode? public private(set) var forcedMode: InputMode?
/// True when forcedMode corrects the classifier (learned on commit); false when the /// English mode, toggled by a Shift tap when nothing is being typed. Keys go straight
/// user forced English before typing (Shift tap on an empty buffer), which is not a /// to the app (no composition, no underline) until Shift is tapped again.
/// correction and is not learned. public private(set) var isEnglishMode = false
private var forcedIsCorrection = false
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text. /// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
private var fixed: [Int: [Range<Int>: String]] = [:] private var fixed: [Int: [Range<Int>: String]] = [:]
public private(set) var candidates: [Candidate] = [] public private(set) var candidates: [Candidate] = []
private var candidateSegmentStart = 0 private var candidateSegmentStart = 0
/// Composition cursor for choosing candidates: index into `cursorStops`, nil = at the end.
/// ← → move it; ↓ shows candidates for the phrase ending at the cursor.
private var cursor: Int?
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
@@ -89,17 +92,64 @@ public final class CompositionEngine {
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo. /// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
private func render(_ seg: Segment, final: Bool) -> String { private func render(_ seg: Segment, final: Bool) -> String {
let segKeys = Array(keys[seg.range]) guard seg.mode == .zhuyin else { return String(keys[seg.range]) }
guard seg.mode == .zhuyin else { return String(segKeys) } let (converted, pending) = conversion(of: seg, final: final)
let p = ZhuyinParser.parse(segKeys) return converted.map(\.text).joined() + pending
let fixedHere = (fixed[seg.range.lowerBound] ?? [:])
if final {
let all = p.syllables.map(\.syllable)
return converter.convert(all, fixed: fixedHere.filter { $0.key.upperBound <= all.count }).map(\.text).joined()
} }
let done = p.completed.map(\.syllable)
let text = converter.convert(done, fixed: fixedHere.filter { $0.key.upperBound <= done.count }).map(\.text).joined() /// Syllables of a Zhuyin segment that are converted to Chinese (all of them when
return text + (p.pending?.syllable.reading ?? "") /// final, otherwise the finished ones).
private func convertibleSyllables(_ seg: Segment, final: Bool) -> (syllables: [Syllable], pending: String) {
let p = ZhuyinParser.parse(Array(keys[seg.range]))
if final { return (p.syllables.map(\.syllable), "") }
return (p.completed.map(\.syllable), p.pending?.syllable.reading ?? "")
}
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
let (syl, pending) = convertibleSyllables(seg, final: final)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
return (converter.convert(syl, fixed: fixedHere), pending)
}
// MARK: - Cursor
/// A place the cursor can stop: after the `local`-th syllable of Zhuyin segment `segIndex`.
private struct CursorStop { let segIndex: Int; let local: Int }
private var cursorStops: [CursorStop] {
let segs = segments
var out: [CursorStop] = []
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin {
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
for k in stride(from: 1, through: n, by: 1) { out.append(CursorStop(segIndex: i, local: k)) }
}
return out
}
/// Cursor position in `markedText`, in Characters.
public var cursorOffset: Int {
let pieces = markedPieces
let total = pieces.reduce(0) { $0 + $1.text.count }
let stops = cursorStops
guard let c = cursor, stops.indices.contains(c) else { return total }
let stop = stops[c]
var offset = pieces[..<stop.segIndex].reduce(0) { $0 + $1.text.count }
let segs = segments
for conv in conversion(of: segs[stop.segIndex], final: stop.segIndex < segs.count - 1).segments {
if conv.range.upperBound <= stop.local {
offset += conv.text.count
} else if conv.range.lowerBound < stop.local {
offset += min(stop.local - conv.range.lowerBound, conv.text.count)
}
}
return offset
}
private func moveCursor(by delta: Int) {
let stops = cursorStops
guard !stops.isEmpty else { return }
let next = (cursor ?? stops.count - 1) + delta
cursor = next >= stops.count - 1 ? nil : max(0, next)
} }
// MARK: - Key handling // MARK: - Key handling
@@ -112,13 +162,10 @@ public final class CompositionEngine {
return handleCharacter(c) return handleCharacter(c)
case .backspace: case .backspace:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
cursor = nil
keys.removeLast() keys.removeLast()
fixed = fixed.filter { $0.key < keys.count } fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { if keys.isEmpty { forcedMode = nil; fixed = [:] }
// A Shift-tap "next word is English" stays armed; a correction is dropped.
if forcedIsCorrection { forcedMode = nil; forcedIsCorrection = false }
fixed = [:]
}
return .consumed return .consumed
case .enter: case .enter:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
@@ -133,9 +180,10 @@ public final class CompositionEngine {
return .consumed return .consumed
case .shiftTap: case .shiftTap:
if keys.isEmpty { if keys.isEmpty {
// Next word is English, typed as-is (lowercase stays lowercase). Tap again to cancel. // Toggle English mode; it stays on until the next Shift tap.
forcedMode = forcedMode == .literal ? nil : .literal isEnglishMode.toggle()
forcedIsCorrection = false previousMode = isEnglishMode ? .literal : nil
previousCommitTime = now()
} else { } else {
flip() flip()
} }
@@ -144,7 +192,15 @@ public final class CompositionEngine {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
openCandidates() openCandidates()
return .consumed return .consumed
case .up, .left, .right: case .left:
guard !keys.isEmpty else { return .passThrough }
moveCursor(by: -1)
return .consumed
case .right:
guard !keys.isEmpty else { return .passThrough }
moveCursor(by: 1)
return .consumed
case .up:
return keys.isEmpty ? .passThrough : .consumed return keys.isEmpty ? .passThrough : .consumed
} }
} }
@@ -152,13 +208,12 @@ public final class CompositionEngine {
/// Forces the whole buffer into the other mode (a correction that is learned). /// Forces the whole buffer into the other mode (a correction that is learned).
private func flip() { private func flip() {
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
forcedIsCorrection = true
} }
/// True while the user has forced English before typing anything (Shift tap).
public var isEnglishArmed: Bool { keys.isEmpty && forcedMode == .literal }
private func handleCharacter(_ c: Character) -> KeyResult { private func handleCharacter(_ c: Character) -> KeyResult {
cursor = nil // typing always continues at the end
// English mode: every key goes to the app as typed.
if isEnglishMode && keys.isEmpty { return .passThrough }
if c == " " { if c == " " {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
let segs = segments(of: keys, finishing: true) let segs = segments(of: keys, finishing: true)
@@ -206,13 +261,16 @@ public final class CompositionEngine {
// MARK: - Candidates (for the last Zhuyin segment) // MARK: - Candidates (for the last Zhuyin segment)
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
private func openCandidates() { private func openCandidates() {
let stops = cursorStops
guard !stops.isEmpty else { return }
let stop = stops[min(cursor ?? stops.count - 1, stops.count - 1)]
let segs = segments let segs = segments
guard let last = segs.last, last.mode == .zhuyin else { return } let seg = segs[stop.segIndex]
let done = ZhuyinParser.parse(Array(keys[last.range])).completed.map(\.syllable) let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
guard !done.isEmpty else { return } candidates = converter.candidates(syl, endingAt: stop.local)
candidates = converter.candidates(done, endingAt: done.count) candidateSegmentStart = seg.range.lowerBound
candidateSegmentStart = last.range.lowerBound
candidateIndex = 0 candidateIndex = 0
} }
@@ -252,7 +310,7 @@ public final class CompositionEngine {
private func commitCurrent(_ given: [Segment]? = nil) -> String { private func commitCurrent(_ given: [Segment]? = nil) -> String {
let segs = given ?? segments let segs = given ?? segments
let text = segs.map { render($0, final: true) }.joined() let text = segs.map { render($0, final: true) }.joined()
if let forced = forcedMode, forcedIsCorrection { if let forced = forcedMode {
classifier.learned.record(String(keys), as: forced) classifier.learned.record(String(keys), as: forced)
} }
previousMode = segs.last?.mode previousMode = segs.last?.mode
@@ -268,8 +326,8 @@ public final class CompositionEngine {
public func reset() { public func reset() {
keys = [] keys = []
cursor = nil
forcedMode = nil forcedMode = nil
forcedIsCorrection = false
fixed = [:] fixed = [:]
candidates = [] candidates = []
} }
+51 -16
View File
@@ -8,33 +8,63 @@ import Foundation
/// - a letter-bigram model trained on those words, for words not in either list. /// - a letter-bigram model trained on those words, for words not in either list.
public final class EnglishModel { public final class EnglishModel {
private var common = Set<String>() private var common = Set<String>()
private var dictionary = Set<String>()
private var sortedWords: [String] = []
private var sortedCommon: [String] = [] private var sortedCommon: [String] = []
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
/// binary searches, so no 230k-entry Set has to be built at startup.
private var sortedDict: [String] = []
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
private var unseenBigram = -12.0 private var unseenBigram = -12.0
public init(commonWords: [String], dictionaryWords: [String] = []) { /// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
public init(commonWords: [String], dictionaryWords: [String] = [],
bigram: (table: [String: Double], unseen: Double)? = nil) {
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty }) common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
sortedCommon = common.sorted() sortedCommon = common.sorted()
sortedWords = common.union(dictionary).sorted() var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
dict = Array(Set(dict)).sorted()
}
sortedDict = dict
if let bigram {
self.bigram = bigram.table
unseenBigram = bigram.unseen
} else {
trainBigram() trainBigram()
} }
}
/// Loads common words from a bundle resource file and the system dictionary if present. /// Loads the bundled data files. Falls back to the system word list and to training
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) { /// the bigram table when the precomputed files are missing.
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
func lines(_ url: URL?) -> [String] { func lines(_ url: URL?) -> [String] {
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] } guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
return text.split(whereSeparator: \.isNewline) return text.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) } .map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.hasPrefix("#") } .filter { !$0.isEmpty && !$0.hasPrefix("#") }
} }
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary)) var bigram: (table: [String: Double], unseen: Double)?
if let url = bigramURL, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
bigram = (table, unseen)
}
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
?? URL(fileURLWithPath: "/usr/share/dict/words")
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
} }
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) } public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) } public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
private func inDictionary(_ w: String) -> Bool {
let i = lowerBound(w, in: sortedDict)
return i < sortedDict.count && sortedDict[i] == w
}
/// Some word strictly longer than `p` starts with it (common words or dictionary).
private func isPrefixOfWord(_ p: String) -> Bool {
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
}
/// - Parameter complete: true when the user ended the word (space / punctuation), /// - Parameter complete: true when the user ended the word (space / punctuation),
/// false while still typing (prefixes of words get partial credit). /// false while still typing (prefixes of words get partial credit).
@@ -43,10 +73,10 @@ public final class EnglishModel {
let n = Double(w.count) let n = Double(w.count)
guard !w.isEmpty else { return 0 } guard !w.isEmpty else { return 0 }
if common.contains(w) { return 3 + n } if common.contains(w) { return 3 + n }
if dictionary.contains(w) { return 1 + 0.6 * n } if inDictionary(w) { return 1 + 0.6 * n }
if !complete { if !complete {
if hasPrefix(w, in: sortedCommon) { return 0.4 * n } if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
if hasPrefix(w, in: sortedWords) { return 0.2 * n } if isPrefixOfWord(w) { return 0.2 * n }
} }
// Unknown word: judge by how English-like its letter sequence is. // Unknown word: judge by how English-like its letter sequence is.
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -63,12 +93,12 @@ public final class EnglishModel {
if common.contains(w) { if common.contains(w) {
f["lit.common"] = 1 f["lit.common"] = 1
f["lit.common_len"] = n f["lit.common_len"] = n
} else if dictionary.contains(w) { } else if inDictionary(w) {
f["lit.dict"] = 1 f["lit.dict"] = 1
f["lit.dict_len"] = n f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) { } else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) { } else if !complete && isPrefixOfWord(w) {
f["lit.prefix_dict_len"] = n f["lit.prefix_dict_len"] = n
} else { } else {
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -85,12 +115,17 @@ public final class EnglishModel {
return total / Double(chars.count - 1) return total / Double(chars.count - 1)
} }
private func hasPrefix(_ p: String, in words: [String]) -> Bool { private func lowerBound(_ p: String, in words: [String]) -> Int {
var lo = 0, hi = words.count var lo = 0, hi = words.count
while lo < hi { while lo < hi {
let mid = (lo + hi) / 2 let mid = (lo + hi) / 2
if words[mid] < p { lo = mid + 1 } else { hi = mid } if words[mid] < p { lo = mid + 1 } else { hi = mid }
} }
return lo
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
var lo = lowerBound(p, in: words)
// words[lo] is the first word >= p; skip an exact match, then test the next one. // words[lo] is the first word >= p; skip an exact match, then test the next one.
if lo < words.count && words[lo] == p { lo += 1 } if lo < words.count && words[lo] == p { lo += 1 }
return lo < words.count && words[lo].hasPrefix(p) return lo < words.count && words[lo].hasPrefix(p)
@@ -99,7 +134,7 @@ public final class EnglishModel {
private func trainBigram() { private func trainBigram() {
var pair: [String: Double] = [:] var pair: [String: Double] = [:]
var first: [String: Double] = [:] var first: [String: Double] = [:]
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) { for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
let weight = common.contains(w) ? 5.0 : 1.0 let weight = common.contains(w) ? 5.0 : 1.0
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
for i in 1..<chars.count { for i in 1..<chars.count {
+17 -4
View File
@@ -42,6 +42,9 @@ public enum Features {
"lit.others": -1.5, "lit.others": -1.5,
// Segmentation // Segmentation
"seg.switch": -3.0, "seg.switch": -3.0,
// Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
] ]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid. /// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
@@ -58,7 +61,8 @@ public enum Features {
/// Features of `keys` read as Zhuyin, or nil if impossible. /// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true, public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys) let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone. // A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil } if !isLast && parse.pending != nil { return nil }
@@ -107,18 +111,27 @@ public enum Features {
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) { if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed") add("zh.unconfirmed")
} }
if afterOther { add("seg.switch") } if afterOther {
add("seg.switch")
} else if previous == .zhuyin {
add("ctx.zh")
}
return f return f
} }
/// Features of `keys` output as typed, or nil if impossible. /// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true, public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil } if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1] var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v } func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther { add("seg.switch") } if afterOther {
add("seg.switch")
} else if previous == .literal {
add("ctx.lit")
}
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") } if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter }) let letters = String(keys.filter { $0.isLetter })
+138 -28
View File
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
/// File format, one entry per line (`#` starts a comment): /// File format, one entry per line (`#` starts a comment):
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2 /// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
/// The phrase/reading columns may be in either order; the score is optional. /// The phrase/reading columns may be in either order; the score is optional.
///
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
/// call `prepareTonelessIndex()` from a background queue to have it ready.
public final class Lexicon { public final class Lexicon {
private var exact: [String: [LexiconEntry]] = [:] private var exact: [String: [LexiconEntry]] = [:]
private var toneless: [String: [LexiconEntry]] = [:] private var tonelessIndex: [String: [LexiconEntry]]?
private let lock = NSLock()
public private(set) var maxPhraseLength = 1 public private(set) var maxPhraseLength = 1
public init() {} public init() {}
public convenience init(contentsOf url: URL) throws { public convenience init(contentsOf url: URL) throws {
self.init() self.init()
load(try String(contentsOf: url, encoding: .utf8)) load(data: try Data(contentsOf: url))
} }
public func load(_ text: String) { public func load(_ text: String) {
for line in text.split(whereSeparator: \.isNewline) { load(data: Data(text.utf8))
let trimmed = line.trimmingCharacters(in: .whitespaces)
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
guard cols.count >= 2 else { continue }
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
guard Lexicon.isBopomofo(reading) else { continue }
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
add(reading: reading, phrase: phrase, score: score)
}
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
} }
public func add(reading rawReading: String, phrase: String, score: Double) { public func load(data: Data) {
let entry = LexiconEntry(phrase: phrase, score: score) lock.lock()
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix. defer { lock.unlock() }
let syllables = rawReading.split(separator: "-").map { s -> String in exact.reserveCapacity(exact.count + data.count / 30)
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s) data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
let bytes = raw.bindMemory(to: UInt8.self)
let n = bytes.count
var i = 0
while i < n {
var end = i
while end < n && bytes[end] != 0x0A { end += 1 }
parseLine(bytes, i, end)
i = end + 1
} }
let reading = syllables.joined(separator: "-")
maxPhraseLength = max(maxPhraseLength, syllables.count)
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) {
exact[reading, default: []].append(entry)
} }
let key = Lexicon.stripTones(reading) for (key, entries) in exact where entries.count > 1 {
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) { exact[key] = entries.sorted { $0.score > $1.score }
toneless[key]![i].score = max(toneless[key]![i].score, score) }
tonelessIndex = nil // rebuilt on demand
}
/// Splits one line into up to three space/tab separated fields.
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
var fields: [Range<Int>] = []
var i = start
while i < end && fields.count < 3 {
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
if i >= end { break }
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
let s = i
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
fields.append(s..<i)
}
guard fields.count >= 2 else { return }
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
guard Lexicon.isBopomofo(b, r) else { return }
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
var prefixNeutral = b[r.lowerBound] == 0xCB
var k = r.lowerBound
while !prefixNeutral && k + 2 < r.upperBound {
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
k += 1
}
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
}
/// Parses "-3.9151"-style decimals without creating a String.
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
var i = r.lowerBound
var sign = 1.0
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
var value = 0.0, scale = 1.0, digits = 0, fraction = false
while i < r.upperBound {
let c = b[i]
if c >= 0x30 && c <= 0x39 {
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
digits += 1
} else if c == 0x2E && !fraction {
fraction = true
} else { } else {
toneless[key, default: []].append(entry) return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
} }
i += 1
}
return digits > 0 ? sign * value : nil
}
/// Adds one entry (call `load` for bulk data; this does not re-sort).
public func add(reading: String, phrase: String, score: Double) {
lock.lock()
defer { lock.unlock() }
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
exact[key]?.sort { $0.score > $1.score }
tonelessIndex = nil
}
@discardableResult
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
var reading = rawReading
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
if normalise {
reading = reading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
}.joined(separator: "-")
}
maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
if exact[reading] == nil {
exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
} else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
}
return reading
}
/// Builds the index that ignores tones (for untoned syllables). Safe to call from a
/// background queue; lookups build it themselves if it is not ready yet.
public func prepareTonelessIndex() {
lock.lock()
defer { lock.unlock() }
_ = buildTonelessIfNeeded()
}
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
if let index = tonelessIndex { return index }
var best: [String: [String: Double]] = [:]
for (reading, entries) in exact {
let key = Lexicon.stripTones(reading)
for e in entries {
if let s = best[key]?[e.phrase], s >= e.score { continue }
best[key, default: [:]][e.phrase] = e.score
}
}
var index: [String: [LexiconEntry]] = [:]
index.reserveCapacity(best.count)
for (key, phrases) in best {
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
}
tonelessIndex = index
return index
} }
/// Entries for a sequence of readings. Untoned syllables match any tone. /// Entries for a sequence of readings. Untoned syllables match any tone.
public func entries(for syllables: [Syllable]) -> [LexiconEntry] { public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
lock.lock()
defer { lock.unlock() }
if syllables.allSatisfy({ $0.tone != nil }) { if syllables.allSatisfy({ $0.tone != nil }) {
return exact[syllables.map(\.reading).joined(separator: "-")] ?? [] return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
} }
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? [] return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
} }
public func bestScore(for syllable: Syllable) -> Double? { public func bestScore(for syllable: Syllable) -> Double? {
@@ -81,4 +182,13 @@ public final class Lexicon {
guard let first = s.unicodeScalars.first else { return false } guard let first = s.unicodeScalars.first else { return false }
return (0x3105...0x312F).contains(first.value) || first == "˙" return (0x3105...0x312F).contains(first.value) || first == "˙"
} }
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
guard r.count >= 2 else { return false }
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
let c = b[r.lowerBound + 2]
return c >= 0x85 && c <= 0xAF
}
} }
@@ -6,7 +6,10 @@ private let resources = URL(fileURLWithPath: #filePath)
.appendingPathComponent("Resources") .appendingPathComponent("Resources")
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")) private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase { final class ParserTests: XCTestCase {
@@ -265,10 +268,13 @@ final class ParityTests: XCTestCase {
for fx in fixtures { for fx in fixtures {
let keys = Array(fx["keys"] as! String) let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" } let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map { let previous = (fx["previous"] as? String).flatMap(InputMode.init(rawValue:))
let got = c.segment(keys, finishing: fx["finishing"] as! Bool, previous: previous).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)" "\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
} }
if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") } if got != expected {
mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!) previous=\(String(describing: previous)): \(got) vs \(expected)")
}
} }
XCTAssertGreaterThan(fixtures.count, 300) XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n")) XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
@@ -297,13 +303,16 @@ final class ShiftTapTests: XCTestCase {
return out return out
} }
func testShiftTapForcesNextWordEnglishLowercase() { func testShiftTapTogglesEnglishMode() {
let e = engine() let e = engine()
// "ru" alone would be ㄐㄧ (機); after a Shift tap it stays English, lowercase. // "ru" alone would be ㄐㄧ (機); in English mode keys pass straight through, lowercase.
XCTAssertEqual(type("⇧ru ", into: e), "ru ") XCTAssertEqual(type("⇧ru su ", into: e), "ru su ")
// Automatic detection resumes after the word. XCTAssertTrue(e.isEnglishMode) // still English after spaces
XCTAssertFalse(e.isEnglishArmed) XCTAssertEqual(e.markedText, "") // never composed: no underline
XCTAssertEqual(type("su3cl3\n", into: e), "你好") XCTAssertEqual(type("\n", into: e), "\n") // Enter goes to the app, mode stays
XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(type("⇧su3cl3\n", into: e), "你好") // tap again: automatic again
XCTAssertFalse(e.isEnglishMode)
} }
func testShiftTapTwiceCancels() { func testShiftTapTwiceCancels() {
@@ -316,22 +325,22 @@ final class ShiftTapTests: XCTestCase {
XCTAssertEqual(e.mode, .literal) XCTAssertEqual(e.mode, .literal)
_ = e.handle(.shiftTap) _ = e.handle(.shiftTap)
XCTAssertEqual(e.mode, .zhuyin) XCTAssertEqual(e.mode, .zhuyin)
XCTAssertFalse(e.isEnglishMode) // a flip, not English mode
_ = type("\n", into: e) _ = type("\n", into: e)
XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned
} }
func testForcingBeforeTypingIsNotLearned() { func testEnglishModeIsNotLearned() {
let e = engine() let e = engine()
_ = type("⇧hello ", into: e) _ = type("⇧hello ", into: e)
XCTAssertNil(e.classifier.learned.weights["hello"]) XCTAssertTrue(e.classifier.learned.weights.isEmpty)
} }
func testArmingSurvivesBackspace() { func testBackspaceInEnglishModeGoesToApp() {
let e = engine() let e = engine()
_ = type("⇧r", into: e) _ = type("⇧", into: e)
_ = e.handle(.backspace) XCTAssertFalse(e.handle(.backspace).handled)
XCTAssertTrue(e.isEnglishArmed) XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(type("ru ", into: e), "ru ")
} }
func testDetector() { func testDetector() {
@@ -351,3 +360,82 @@ final class ShiftTapTests: XCTestCase {
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 4.2)) XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 4.2))
} }
} }
final class CursorTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testCursorMovesAndClamps() {
let e = engine()
_ = type("rup wu0 wu0 fu4", into: e) // 今天天氣
XCTAssertEqual(e.cursorOffset, 4)
_ = type("←", into: e)
XCTAssertEqual(e.cursorOffset, 3)
_ = type("←←←←", into: e)
XCTAssertEqual(e.cursorOffset, 1) // stops after the first character
_ = type("→→→→", into: e)
XCTAssertEqual(e.cursorOffset, 4)
}
func testChooseCandidateInTheMiddle() {
let e = engine()
_ = type("rup wu0 wu0 fu4", into: e)
_ = type("←←↓", into: e) // cursor after the 2nd syllable (天)
XCTAssertTrue(e.isShowingCandidates)
XCTAssertTrue(e.candidates.allSatisfy { $0.range.upperBound == 2 })
let i = e.candidates.firstIndex { $0.text == "添" }!
e.selectCandidate(i)
XCTAssertEqual(type("\n", into: e), "今添天氣")
}
func testTypingReturnsCursorToEnd() {
let e = engine()
_ = type("rup wu0 ←", into: e)
XCTAssertEqual(e.cursorOffset, 1)
_ = type("wu0 ", into: e)
XCTAssertEqual(e.cursorOffset, e.markedText.count)
}
func testCandidatesForChineseBeforeEnglish() {
let e = engine()
_ = type("su3cl3hello", into: e) // 你好hello
_ = type("↓", into: e) // last Chinese syllable, not the English word
XCTAssertTrue(e.isShowingCandidates)
}
}
final class FirstToneAfterEnglishTests: XCTestCase {
/// After an English word, a first-tone syllable + space must stay Chinese (機, 吃, 書…).
func testFirstToneAfterEnglish() {
for keys in ["ru", "t", "gj", "vup", "tj", "el"] {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out, "hello ", keys)
XCTAssertEqual(e.markedText.count, 1, keys) // still composing one Chinese character
}
}
}
final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws {
var t = Date()
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
}
}
+3
View File
@@ -19,6 +19,9 @@ mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
+6 -6
View File
@@ -23,14 +23,14 @@ def main():
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH)) clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"), gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7) os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f) for k in HAND_PICKED for f in (False, True)] cases = [(k, f, p) for k in HAND_PICKED for f in (False, True) for p in (None, "literal")]
while len(cases) < 400: while len(cases) < 500:
ex = gen.example() ex = gen.example()
if len(ex["keys"]) <= 28: if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"])) cases.append((ex["keys"], ex["finishing"], ex["previous"]))
cases.append((gen.prefix_of(ex)["keys"], False)) cases.append((gen.prefix_of(ex)["keys"], False, ex["previous"]))
fixtures = [{"keys": k, "finishing": f, fixtures = [{"keys": k, "finishing": f, "previous": p,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases] "segments": [[s, e, m] for s, e, m in clf.segment(list(k), f, p)]} for k, f, p in cases]
with open(OUT, "w", encoding="utf-8") as fh: with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0) json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT)) print("wrote %d fixtures to %s" % (len(fixtures), OUT))
+4971 -3231
View File
File diff suppressed because it is too large Load Diff
+17 -2
View File
@@ -102,16 +102,31 @@ class Generator:
labels = "".join(m * len(p) for m, p in parts) labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l": if parts[-1][0] == "l":
finishing = True # English / numbers end with a space finishing = True # English / numbers end with a space
elif keys[-1] == " " and self.rng.random() < 0.6:
# The moment the space (first tone) is pressed: the buffer has no space yet and
# the input method decides "first tone" vs "English word + space".
keys, labels, finishing = keys[:-1], labels[:-1], True
kind += "/tone1"
else: else:
# Chinese ends with Enter, or space after a finished (toned) syllable. # Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5 finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind} return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind,
"previous": self.previous_mode(labels[0])}
def previous_mode(self, first_label):
"""Mode of the text committed just before: none, mostly the same, sometimes the other."""
r = self.rng.random()
if r < 0.35:
return None
same = core.ZHUYIN if first_label == "z" else core.LITERAL
other = core.LITERAL if first_label == "z" else core.ZHUYIN
return same if r < 0.80 else other
def prefix_of(self, ex): def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing.""" """The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1 cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False, return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix"} "kind": ex["kind"] + "/prefix", "previous": ex["previous"]}
def _cumulative(ws): def _cumulative(ws):
+19 -10
View File
@@ -45,12 +45,12 @@ def labels_of(segs, n):
return "".join(lab) return "".join(lab)
def seg_features(clf, keys, segs, finishing): def seg_features(clf, keys, segs, finishing, previous):
"""Summed features of a segmentation, or None if any segment is impossible.""" """Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float) total = defaultdict(float)
n = len(keys) n = len(keys)
for idx, (s, e, m) in enumerate(segs): for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing) fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing, previous if idx == 0 else None)
f = fz if m == core.ZHUYIN else fl f = fz if m == core.ZHUYIN else fl
if f is None: if f is None:
return None return None
@@ -62,12 +62,13 @@ def seg_features(clf, keys, segs, finishing):
def evaluate(clf, weights, data): def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0]) by_kind = defaultdict(lambda: [0, 0])
for ex in data: for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights) segs = clf.segment(list(ex["keys"]), ex["finishing"], ex["previous"], weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"] ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]] kind = "prefix" if ex["kind"].endswith("/prefix") else ex["kind"]
bucket[0] += ok for k in (kind, "after-other") if _after_other(ex) else (kind,):
bucket[1] += 1 by_kind[k][0] += ok
commit = [v for k, v in by_kind.items() if k != "prefix"] by_kind[k][1] += 1
commit = [v for k, v in by_kind.items() if k not in ("prefix", "after-other")]
summary = { summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)), "commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]), "prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
@@ -76,6 +77,14 @@ def evaluate(clf, weights, data):
return summary return summary
def _after_other(ex):
"""Committed buffer whose first segment differs from the text typed just before."""
if ex["kind"].endswith("/prefix") or ex["previous"] is None:
return False
first = core.ZHUYIN if ex["labels"][0] == "z" else core.LITERAL
return ex["previous"] != first
def show(name, s): def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items()) kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds)) print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
@@ -92,11 +101,11 @@ def train(clf, data, epochs, lr, seed):
t0 = time.time() t0 = time.time()
for ex in data: for ex in data:
keys = list(ex["keys"]) keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], None, w) pred = clf.segment(keys, ex["finishing"], ex["previous"], w)
if labels_of(pred, len(keys)) != ex["labels"]: if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"]) gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"]) fg = seg_features(clf, keys, gold, ex["finishing"], ex["previous"])
fp = seg_features(clf, keys, pred, ex["finishing"]) fp = seg_features(clf, keys, pred, ex["finishing"], ex["previous"])
if fg is not None and fp is not None: if fg is not None and fp is not None:
mistakes += 1 mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0) scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
+3 -3
View File
@@ -142,8 +142,8 @@ class AutoZhuyinTextService(TextService):
return False return False
self.engine.handle(SHIFT_TAP) self.engine.handle(SHIFT_TAP)
self._update(None) self._update(None)
if self.engine.english_armed: if not self.engine.keys:
self.showMessage("英文", 1) self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
return True return True
# --- replies --- # --- replies ---
@@ -153,7 +153,7 @@ class AutoZhuyinTextService(TextService):
self.setCommitString(commit) self.setCommitString(commit)
marked = self.engine.marked_text marked = self.engine.marked_text
self.setCompositionString(marked) self.setCompositionString(marked)
self.setCompositionCursor(len(marked)) self.setCompositionCursor(self.engine.cursor_offset)
if self.engine.showing_candidates: if self.engine.showing_candidates:
page, cursor = self.engine.page() page, cursor = self.engine.page()
self.setCandidateList(page) self.setCandidateList(page)
+6 -10
View File
@@ -30,7 +30,6 @@ class LearnedPreferences:
class Classifier: class Classifier:
context_weight = 1.5
def __init__(self, lexicon, english, learned=None, weights=None): def __init__(self, lexicon, english, learned=None, weights=None):
self.lexicon = lexicon self.lexicon = lexicon
@@ -49,20 +48,17 @@ class Classifier:
except (OSError, ValueError): except (OSError, ValueError):
return {} return {}
def segment_features(self, seg, is_last, after_other, finishing): def segment_features(self, seg, is_last, after_other, finishing, previous=None):
"""(zhuyin features or None, literal features or None) for one segment.""" """(zhuyin features or None, literal features or None) for one segment.
`previous` (mode of the text committed just before) only affects the first segment."""
seg = list(seg) seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing), return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing, previous),
literal_features(seg, self.english, is_last, after_other, finishing)) literal_features(seg, self.english, is_last, after_other, finishing, previous))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights): def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing) fz, fl = self.segment_features(seg, is_last, after_other, finishing, previous)
zh = dot(weights, fz) if fz is not None else NEG_INF zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF lit = dot(weights, fl) if fl is not None else NEG_INF
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
if whole: if whole:
b = self.learned.bias("".join(seg)) b = self.learned.bias("".join(seg))
if b > 0: if b > 0:
+88 -37
View File
@@ -34,13 +34,15 @@ class CompositionEngine:
self.now = now self.now = now
self.keys = [] self.keys = []
self.forced_mode = None self.forced_mode = None
# True when forced_mode corrects the classifier (learned on commit); False when the # English mode, toggled by a Shift tap when nothing is being typed: keys go straight
# user forced English before typing (Shift tap on an empty buffer). # to the app (no composition, no underline) until Shift is tapped again.
self._forced_is_correction = False self.english_mode = False
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text} self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = [] self.candidates = []
self.candidate_index = 0 self.candidate_index = 0
self._candidate_segment_start = 0 self._candidate_segment_start = 0
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
self.cursor = None
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None self._cache_key = None
@@ -81,19 +83,61 @@ class CompositionEngine:
def page_start(self): def page_start(self):
return (self.candidate_index // self.page_size) * self.page_size return (self.candidate_index // self.page_size) * self.page_size
def _convertible(self, seg, final):
"""(syllables converted to Chinese, pending Bopomofo) for a Zhuyin segment."""
start, end, _ = seg
p = bopomofo.parse(self.keys[start:end])
if final:
return [s.syllable for s in p.syllables], ""
return [s.syllable for s in p.completed], (p.pending.syllable.reading if p.pending else "")
def _conversion(self, seg, final):
syl, pending = self._convertible(seg, final)
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
return self.converter.convert(syl, fx), pending
def _render(self, seg, final): def _render(self, seg, final):
start, end, mode = seg start, end, mode = seg
seg_keys = self.keys[start:end]
if mode == LITERAL: if mode == LITERAL:
return "".join(seg_keys) return "".join(self.keys[start:end])
p = bopomofo.parse(seg_keys) converted, pending = self._conversion(seg, final)
fixed_here = self.fixed.get(start, {}) return "".join(t for _, t in converted) + pending
syl = [s.syllable for s in (p.syllables if final else p.completed)]
fx = {k: v for k, v in fixed_here.items() if k[1] <= len(syl)} # --- cursor ---
text = "".join(t for _, t in self.converter.convert(syl, fx))
if not final and p.pending: def _cursor_stops(self):
text += p.pending.syllable.reading """[(segment index, syllables before the stop)] for every Zhuyin syllable."""
return text segs = self.segments
out = []
for i, seg in enumerate(segs):
if seg[2] == ZHUYIN:
n = len(self._convertible(seg, i < len(segs) - 1)[0])
out.extend((i, k) for k in range(1, n + 1))
return out
@property
def cursor_offset(self):
"""Cursor position in marked_text (characters)."""
pieces = self.marked_pieces
stops = self._cursor_stops()
if self.cursor is None or not 0 <= self.cursor < len(stops):
return sum(len(t) for t, _ in pieces)
seg_index, local = stops[self.cursor]
offset = sum(len(t) for t, _ in pieces[:seg_index])
segs = self.segments
for (a, b), text in self._conversion(segs[seg_index], seg_index < len(segs) - 1)[0]:
if b <= local:
offset += len(text)
elif a < local:
offset += min(local - a, len(text))
return offset
def _move_cursor(self, delta):
stops = self._cursor_stops()
if not stops:
return
nxt = (len(stops) - 1 if self.cursor is None else self.cursor) + delta
self.cursor = None if nxt >= len(stops) - 1 else max(0, nxt)
@property @property
def marked_pieces(self): def marked_pieces(self):
@@ -106,19 +150,16 @@ class CompositionEngine:
# --- key handling --- # --- key handling ---
@property
def english_armed(self):
"""True while the user has forced English before typing anything (Shift tap)."""
return not self.keys and self.forced_mode == LITERAL
def _flip(self): def _flip(self):
"""Forces the whole buffer into the other mode (a correction that is learned)."""
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
self._forced_is_correction = True
def wants(self, key): def wants(self, key):
"""Whether handle(key) would consume the key, without changing state.""" """Whether handle(key) would consume the key, without changing state."""
if self.candidates or key == SHIFT_TAP: if self.candidates or key == SHIFT_TAP:
return True return True
if self.english_mode and not self.keys:
return False # English mode: everything goes to the app as typed
if key in SPECIAL_KEYS: if key in SPECIAL_KEYS:
return bool(self.keys) return bool(self.keys)
if self.keys: if self.keys:
@@ -135,17 +176,17 @@ class CompositionEngine:
if key == BACKSPACE: if key == BACKSPACE:
if not self.keys: if not self.keys:
return PASS return PASS
self.cursor = None
self.keys.pop() self.keys.pop()
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)} self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys: if not self.keys:
# A Shift-tap "next word is English" stays armed; a correction is dropped.
if self._forced_is_correction:
self.forced_mode = None self.forced_mode = None
self._forced_is_correction = False
self.fixed = {} self.fixed = {}
return CONSUMED return CONSUMED
if key == ENTER: if key == ENTER:
return (True, self._commit()) if self.keys else PASS if not self.keys:
return PASS
return True, self._commit()
if key == ESCAPE: if key == ESCAPE:
if not self.keys: if not self.keys:
return PASS return PASS
@@ -158,9 +199,10 @@ class CompositionEngine:
return CONSUMED return CONSUMED
if key == SHIFT_TAP: if key == SHIFT_TAP:
if not self.keys: if not self.keys:
# Next word is English, typed as-is. Tap again to cancel. # Toggle English mode; it stays on until the next Shift tap.
self.forced_mode = None if self.forced_mode == LITERAL else LITERAL self.english_mode = not self.english_mode
self._forced_is_correction = False self._previous_mode = LITERAL if self.english_mode else None
self._previous_time = self.now()
else: else:
self._flip() self._flip()
return CONSUMED return CONSUMED
@@ -169,7 +211,12 @@ class CompositionEngine:
return PASS return PASS
self._open_candidates() self._open_candidates()
return CONSUMED return CONSUMED
if key in (UP, LEFT, RIGHT): if key in (LEFT, RIGHT):
if not self.keys:
return PASS
self._move_cursor(-1 if key == LEFT else 1)
return CONSUMED
if key == UP:
return CONSUMED if self.keys else PASS return CONSUMED if self.keys else PASS
return self._handle_character(key) return self._handle_character(key)
@@ -180,6 +227,9 @@ class CompositionEngine:
return bopomofo.parse(self.keys[start:end]) return bopomofo.parse(self.keys[start:end])
def _handle_character(self, c): def _handle_character(self, c):
self.cursor = None # typing always continues at the end
if self.english_mode and not self.keys:
return PASS # English mode: every key goes to the app as typed
if c == " ": if c == " ":
if not self.keys: if not self.keys:
return PASS return PASS
@@ -212,15 +262,16 @@ class CompositionEngine:
# --- candidates (for the last Zhuyin segment) --- # --- candidates (for the last Zhuyin segment) ---
def _open_candidates(self): def _open_candidates(self):
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
stops = self._cursor_stops()
if not stops:
return
seg_index, local = stops[min(len(stops) - 1 if self.cursor is None else self.cursor, len(stops) - 1)]
segs = self.segments segs = self.segments
if not segs or segs[-1][2] != ZHUYIN: seg = segs[seg_index]
return syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
start, end, _ = segs[-1] self.candidates = self.converter.candidates(syl, local)
syl = [s.syllable for s in bopomofo.parse(self.keys[start:end]).completed] self._candidate_segment_start = seg[0]
if not syl:
return
self.candidates = self.converter.candidates(syl, len(syl))
self._candidate_segment_start = start
self.candidate_index = 0 self.candidate_index = 0
def page(self): def page(self):
@@ -256,7 +307,7 @@ class CompositionEngine:
def _commit(self, segs=None): def _commit(self, segs=None):
segs = segs if segs is not None else self.segments segs = segs if segs is not None else self.segments
text = "".join(self._render(s, True) for s in segs) text = "".join(self._render(s, True) for s in segs)
if self.forced_mode and self._forced_is_correction: if self.forced_mode:
self.classifier.learned.record("".join(self.keys), self.forced_mode) self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = segs[-1][2] if segs else None self._previous_mode = segs[-1][2] if segs else None
self._previous_time = self.now() self._previous_time = self.now()
@@ -268,7 +319,7 @@ class CompositionEngine:
def reset(self): def reset(self):
self.keys = [] self.keys = []
self.cursor = None
self.forced_mode = None self.forced_mode = None
self._forced_is_correction = False
self.fixed = {} self.fixed = {}
self.candidates = [] self.candidates = []
+9 -2
View File
@@ -41,6 +41,9 @@ DEFAULT_WEIGHTS = {
"lit.others": -1.5, "lit.others": -1.5,
# Segmentation # Segmentation
"seg.switch": -3.0, "seg.switch": -3.0,
# Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
} }
@@ -52,7 +55,7 @@ def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False): def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` read as Zhuyin, or None if impossible.""" """Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys) parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None: if not is_last and parse.pending is not None:
@@ -99,10 +102,12 @@ def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=Fa
_add(f, "zh.unconfirmed") _add(f, "zh.unconfirmed")
if after_other: if after_other:
_add(f, "seg.switch") _add(f, "seg.switch")
elif previous == "zhuyin":
_add(f, "ctx.zh")
return f return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False): def literal_features(keys, english, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` output as typed, or None if impossible.""" """Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys: if not keys or " " in keys:
return None return None
@@ -110,6 +115,8 @@ def literal_features(keys, english, is_last=True, after_other=False, finishing=F
f = {"lit.bias": 1.0} f = {"lit.bias": 1.0}
if after_other: if after_other:
_add(f, "seg.switch") _add(f, "seg.switch")
elif previous == "literal":
_add(f, "ctx.lit")
if any(c.isupper() for c in keys): if any(c.isupper() for c in keys):
_add(f, "lit.upper") _add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha()) letters = "".join(c for c in keys if c.isalpha())
+18 -12
View File
@@ -16,7 +16,7 @@ class Lexicon:
def __init__(self): def __init__(self):
self.exact = {} self.exact = {}
self.toneless = {} self._toneless = None # built on first use (only untoned syllables need it)
self.max_phrase_length = 1 self.max_phrase_length = 1
@classmethod @classmethod
@@ -42,9 +42,10 @@ class Lexicon:
except ValueError: except ValueError:
score = -6.0 score = -6.0
self.add(reading, phrase, score) self.add(reading, phrase, score)
for table in (self.exact, self.toneless): for v in self.exact.values():
for v in table.values(): if len(v) > 1:
v.sort(key=lambda e: -e[1]) v.sort(key=lambda e: -e[1])
self._toneless = None
def add(self, reading, phrase, score): def add(self, reading, phrase, score):
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")] syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
@@ -53,15 +54,20 @@ class Lexicon:
entries = self.exact.setdefault(reading, []) entries = self.exact.setdefault(reading, [])
if not any(p == phrase for p, _ in entries): if not any(p == phrase for p, _ in entries):
entries.append((phrase, score)) entries.append((phrase, score))
key = _strip_tones(reading) self._toneless = None
tl = self.toneless.setdefault(key, [])
for i, (p, s) in enumerate(tl): @property
if p == phrase: def toneless(self):
if score > s: """Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
tl[i] = (p, score) if self._toneless is None:
break best = {}
else: for reading, entries in self.exact.items():
tl.append((phrase, score)) bucket = best.setdefault(_strip_tones(reading), {})
for phrase, score in entries:
if score > bucket.get(phrase, float("-inf")):
bucket[phrase] = score
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
return self._toneless
def entries(self, syllables): def entries(self, syllables):
if all(s.tone is not None for s in syllables): if all(s.tone is not None for s in syllables):
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.4.1", "version": "0.6.2",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+5 -3
View File
@@ -26,10 +26,12 @@
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白 空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
Enter 送出目前組字 Enter 送出目前組字
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正 Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
↓ 開啟候選字,按數字鍵 1–9 選字 ← → 在組字區移動游標,選要改哪個字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字
Esc 取消組字 Esc 取消組字
單按 Shift 下一個字強制英文(保持小寫),送出後恢復自動判斷; 單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
打到一半時按,同 Tab 整段翻轉。會出現「英文」小提示 再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文) Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!: < > ? ! : 中文語境下自動變成全形 ,。?!:
+58 -13
View File
@@ -11,7 +11,7 @@ sys.path.insert(0, os.path.dirname(HERE))
from autozhuyin.core import bopomofo # noqa: E402 from autozhuyin.core import bopomofo # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402 from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import DOWN, ENTER, SHIFT_TAP, TAB, BACKSPACE, CompositionEngine # noqa: E402 from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BACKSPACE, CompositionEngine # noqa: E402
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402 from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402 from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402 from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
@@ -42,7 +42,7 @@ def make_engine():
def type_keys(s, e): def type_keys(s, e):
out = "" out = ""
for c in s: for c in s:
key = ENTER if c == "\n" else TAB if c == "\t" else SHIFT_TAP if c == "⇧" else c key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN}.get(c, c)
handled, commit = e.handle(key) handled, commit = e.handle(key)
if commit is not None: if commit is not None:
out += commit out += commit
@@ -194,11 +194,15 @@ class MixedSegmentTests(unittest.TestCase):
class ShiftTapTests(unittest.TestCase): class ShiftTapTests(unittest.TestCase):
def test_forces_next_word_english_lowercase(self): def test_toggles_english_mode(self):
e = make_engine() e = make_engine()
self.assertEqual(type_keys("⇧ru ", e), "ru ") self.assertEqual(type_keys("⇧ru su ", e), "ru su ")
self.assertFalse(e.english_armed) self.assertTrue(e.english_mode)
self.assertEqual(type_keys("su3cl3\n", e), "你好") self.assertEqual(e.marked_text, "")
self.assertEqual(e.handle(ENTER), (False, None))
self.assertTrue(e.english_mode)
self.assertEqual(type_keys("⇧su3cl3\n", e), "你好")
self.assertFalse(e.english_mode)
def test_tap_twice_cancels(self): def test_tap_twice_cancels(self):
self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好") self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好")
@@ -208,20 +212,20 @@ class ShiftTapTests(unittest.TestCase):
type_keys("so", e) type_keys("so", e)
e.handle(SHIFT_TAP) e.handle(SHIFT_TAP)
self.assertEqual(e.mode, ZHUYIN) self.assertEqual(e.mode, ZHUYIN)
self.assertFalse(e.english_mode)
type_keys("\n", e) type_keys("\n", e)
self.assertEqual(e.classifier.learned.weights["so"], 1) self.assertEqual(e.classifier.learned.weights["so"], 1)
def test_forcing_before_typing_not_learned(self): def test_english_mode_not_learned(self):
e = make_engine() e = make_engine()
type_keys("⇧hello ", e) type_keys("⇧hello ", e)
self.assertNotIn("hello", e.classifier.learned.weights) self.assertEqual(e.classifier.learned.weights, {})
def test_arming_survives_backspace(self): def test_backspace_goes_to_app(self):
e = make_engine() e = make_engine()
type_keys("⇧r", e) e.handle(SHIFT_TAP)
e.handle(BACKSPACE) self.assertEqual(e.handle(BACKSPACE), (False, None))
self.assertTrue(e.english_armed) self.assertTrue(e.english_mode)
self.assertEqual(type_keys("ru ", e), "ru ")
def test_detector(self): def test_detector(self):
d = ShiftTapDetector() d = ShiftTapDetector()
@@ -234,5 +238,46 @@ class ShiftTapTests(unittest.TestCase):
self.assertFalse(d.shift_up(3)) self.assertFalse(d.shift_up(3))
class CursorTests(unittest.TestCase):
def test_moves_and_clamps(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4", e)
self.assertEqual(e.cursor_offset, 4)
type_keys("←", e)
self.assertEqual(e.cursor_offset, 3)
type_keys("←←←←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("→→→→", e)
self.assertEqual(e.cursor_offset, 4)
def test_choose_candidate_in_the_middle(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4←←↓", e)
self.assertTrue(e.showing_candidates)
self.assertTrue(all(r[1] == 2 for r, _ in e.candidates))
e.select_candidate([t for _, t in e.candidates].index("添"))
self.assertEqual(type_keys("\n", e), "今添天氣")
def test_typing_returns_cursor_to_end(self):
e = make_engine()
type_keys("rup wu0 ←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("wu0 ", e)
self.assertEqual(e.cursor_offset, len(e.marked_text))
def test_candidates_for_chinese_before_english(self):
e = make_engine()
type_keys("su3cl3hello↓", e)
self.assertTrue(e.showing_candidates)
class FirstToneAfterEnglishTests(unittest.TestCase):
def test_first_tone_after_english(self):
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
e = make_engine()
self.assertEqual(type_keys("hello " + keys + " ", e), "hello ", keys)
self.assertEqual(len(e.marked_text), 1, keys)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+3 -2
View File
@@ -144,14 +144,15 @@ class ShiftTapWrapperTests(unittest.TestCase):
def test_tap_forces_english_lowercase(self): def test_tap_forces_english_lowercase(self):
reply = tap_shift(self.svc) reply = tap_shift(self.svc)
self.assertEqual(reply.get("showMessage"), "英文") self.assertEqual(reply.get("showMessage"), "英文")
self.assertEqual(type_text(self.svc, "ru "), "ru ") self.assertEqual(type_text(self.svc, "ru su "), "ru su ")
self.assertEqual(tap_shift(self.svc).get("showMessage"), "中英自動")
def test_shift_with_letter_is_not_a_tap(self): def test_shift_with_letter_is_not_a_tap(self):
self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True)) self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True))
press(self.svc, "H", shift=True) press(self.svc, "H", shift=True)
up = FakeKeyEvent("⇧") up = FakeKeyEvent("⇧")
self.assertFalse(self.svc.filterKeyUp(up)) self.assertFalse(self.svc.filterKeyUp(up))
self.assertFalse(self.svc.engine.english_armed) self.assertFalse(self.svc.engine.english_mode)
if __name__ == "__main__": if __name__ == "__main__":