Compare commits

..
8 Commits
Author SHA1 Message Date
ays953 6b4ec39dc1 Release v0.7.1 2026-11-11 17:01:37 +08:00
ays953andClaude Opus 5.5 955b0ca5c3 Ship third-party notices with both installers
THIRD_PARTY_LICENSES.txt carries McBopomofo's full MIT notice (required when the
converted lexicon is distributed) plus notes on the public-domain word list and
PIME. It goes into AutoZhuyin.app/Contents/Resources and the Windows zip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:01:26 +08:00
ays953 251a16fa08 Release v0.7.0 2026-11-11 16:29:06 +08:00
ays953andClaude Opus 5.5 7cb1f62dfa Backtick + punctuation key forces a full-width symbol
` then , . / ; : ' [ ] ( ) ! ~ - = and their shifted keys always give the
full-width symbol (,。?;:、「」()!~——=), whatever the context. Backtick
followed by any other key is a plain backtick, so ``` still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:28:54 +08:00
ays953 3670305f0f Release v0.6.3 2026-11-11 16:10:59 +08:00
ays953andClaude Opus 5.5 ef9f91a6b7 Hold ambiguous English words (up = 因) until the next word decides
"up " also reads as ㄧㄣ (因). Instead of committing English on space, keep the
word composing: "up jo4" becomes 因為, "up the" stays English. Only applies when
the reading is a common single syllable (lexicon score >= -3.5): up, url.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:10:47 +08:00
ays953 1dacb8f164 Release v0.6.2 2026-11-11 12:13:15 +08:00
ays953andClaude Opus 5.5 bfeed22916 Cut input method startup from ~0.95 s to ~0.15 s
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
  parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
  Windows build) instead of retraining the bigram model on every launch, and
  uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:13:03 +08:00
15 changed files with 552 additions and 71 deletions
+6 -2
View File
@@ -16,7 +16,7 @@
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小) - **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」 - **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來 - **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,` - **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權) - **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝 ## 下載與安裝
@@ -58,6 +58,7 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| Shift + 字母 | 打大寫字母(也會強制英文) | | Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` | | `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) | | `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
| `` ` `` + 標點鍵 | **強制全形標點**,不管前後是中文還是英文:`` `, `` `` `< `` = `,`、`` `. `` `` `> `` = `。`、`` `/ `` = `?`、`` `; `` = `;`、`` `: `` = `:`、`` `' `` `` `\ `` = `、`、`` `[ `` `` `] `` = `「」`、`` `{ `` `` `} `` = `『』`、`` `( `` `` `) `` = `()`、`` `! `` = `!`、`` `~ `` = `~`、`` `- `` = `——`、`` `= `` = `=`。`` ` `` 後面接其他鍵就是普通的反引號(` ``` ` 照常可用) |
## 運作方式 ## 運作方式
@@ -211,13 +212,14 @@ windows/
tests/ Python 測試 tests/ Python 測試
tools/ 產生資料檔、圖示 tools/ 產生資料檔、圖示
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料 training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝檔)
``` ```
## 已知限制 ## 已知限制
- Windows 版還沒在實機上測試過。 - Windows 版還沒在實機上測試過。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。 - 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。 - 很短的字串本質上就有歧義。像 `up`(也是 ㄧㄣ 因)這種常用的,按空白後會先保留,看下一個詞再決定(`up jo4` = 因為、`up the` = up the);其他判斷錯的可以按 Tab 修正,修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。 - 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap) ## 下一步(Roadmap)
@@ -237,3 +239,5 @@ training/ 合成資料、訓練權重、Swift/Python 一致
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License) - 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼) - Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`) - 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
+2 -2
View File
@@ -15,9 +15,9 @@
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.6.1</string> <string>0.7.1</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>9</string> <string>13</string>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
+9 -3
View File
@@ -11,22 +11,28 @@ enum Shared {
static let classifier: Classifier = { static let classifier: Classifier = {
let bundle = Bundle.main let bundle = Bundle.main
let start = Date()
let lexicon = Lexicon() let lexicon = Lexicon()
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"), if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
let text = try? String(contentsOf: url, encoding: .utf8) { let data = try? Data(contentsOf: url) {
lexicon.load(text) lexicon.load(data: data)
} }
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt // Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first { if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt") let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) } if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
} }
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt")) let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:] let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json")) let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained) return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
}() }()
} }
+69 -1
View File
@@ -51,6 +51,15 @@ public final class CompositionEngine {
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
/// Index of a held space: an English word that is also a common one-syllable Chinese
/// reading ("up " = ㄧㄣ 因) is not committed on space; the next word decides
/// ("up jo4" = 因為, "up the" = up the).
private var held: Int?
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
public private(set) var pendingBacktick = false
static let holdMinScore = -3.5
private var previousMode: InputMode? private var previousMode: InputMode?
private var previousCommitTime = Date.distantPast private var previousCommitTime = Date.distantPast
/// How long the previous segment's mode keeps influencing the next one. /// How long the previous segment's mode keeps influencing the next one.
@@ -71,9 +80,33 @@ public final class CompositionEngine {
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] { private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
guard !keys.isEmpty else { return [] } guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] } if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
if let h = held, h < keys.count, keys[h] == " " {
return heldSegments(keys, held: h, finishing: finishing)
}
return classifier.segment(keys, finishing: finishing, previous: context) return classifier.segment(keys, finishing: finishing, previous: context)
} }
/// The held word is Chinese if the words typed after it start with Chinese.
private func heldSegments(_ keys: [Character], held h: Int, finishing: Bool) -> [Segment] {
let base = h + 1
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
}
if let first = rest.first, first.mode == .zhuyin {
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
}
return [Segment(range: 0..<base, mode: .literal)] + rest
}
/// Whether keys + space also read as a common Chinese syllable (see `held`).
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
let p = ZhuyinParser.parse(keys + [" "])
guard p.pending == nil, p.syllables.count == 1 else { return false }
guard let best = classifier.lexicon.entries(for: p.syllables.map(\.syllable)).first else { return false }
return best.score >= CompositionEngine.holdMinScore
}
/// Current split of the buffer into Zhuyin / literal runs. /// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) } public var segments: [Segment] { segments(of: keys) }
@@ -85,9 +118,11 @@ public final class CompositionEngine {
public var markedPieces: [MarkedPiece] { public var markedPieces: [MarkedPiece] {
let segs = segments let segs = segments
return segs.enumerated().map { i, seg in var pieces = segs.enumerated().map { i, seg in
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode) MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
} }
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
return pieces
} }
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo. /// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
@@ -156,6 +191,7 @@ public final class CompositionEngine {
public func handle(_ key: KeyInput) -> KeyResult { public func handle(_ key: KeyInput) -> KeyResult {
if isShowingCandidates { return handleCandidateKey(key) } if isShowingCandidates { return handleCandidateKey(key) }
if pendingBacktick { return afterBacktick(key) }
switch key { switch key {
case .character(let c): case .character(let c):
@@ -164,6 +200,7 @@ public final class CompositionEngine {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
cursor = nil cursor = nil
keys.removeLast() keys.removeLast()
if let h = held, keys.count <= h { held = nil }
fixed = fixed.filter { $0.key < keys.count } fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] } if keys.isEmpty { forcedMode = nil; fixed = [:] }
return .consumed return .consumed
@@ -214,10 +251,19 @@ public final class CompositionEngine {
cursor = nil // typing always continues at the end cursor = nil // typing always continues at the end
// English mode: every key goes to the app as typed. // English mode: every key goes to the app as typed.
if isEnglishMode && keys.isEmpty { return .passThrough } if isEnglishMode && keys.isEmpty { return .passThrough }
if c == "`" {
pendingBacktick = true
return .consumed
}
if c == " " { if c == " " {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
let segs = segments(of: keys, finishing: true) let segs = segments(of: keys, finishing: true)
if segs.last?.mode == .literal { if segs.last?.mode == .literal {
if held == nil, segs.count == 1, forcedMode == nil, isAmbiguousWord(keys) {
held = keys.count
keys.append(" ")
return .consumed
}
return KeyResult(handled: true, commit: commitCurrent(segs) + " ") return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
} }
if parse(ofLast: segs)?.pending != nil { if parse(ofLast: segs)?.pending != nil {
@@ -254,6 +300,26 @@ public final class CompositionEngine {
return ZhuyinParser.parse(Array(keys[last.range])) return ZhuyinParser.parse(Array(keys[last.range]))
} }
private func afterBacktick(_ key: KeyInput) -> KeyResult {
pendingBacktick = false
if key == .escape || key == .backspace { return .consumed }
let committed = keys.isEmpty ? "" : commitCurrent()
if case .character(let c) = key, let symbol = CompositionEngine.symbols[c] {
return KeyResult(handled: true, commit: committed + symbol)
}
// Not a symbol: the backtick was meant literally; then handle the key as usual.
var r = handle(key)
if !r.handled, case .character(let c) = key { r = KeyResult(handled: true, commit: String(c)) }
return KeyResult(handled: r.handled, commit: committed + "`" + (r.commit ?? ""))
}
/// "`" + key → full-width symbol.
static let symbols: [Character: String] = [
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", "\"": "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
]
static let fullwidth: [Character: String] = [ static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
@@ -326,6 +392,8 @@ public final class CompositionEngine {
public func reset() { public func reset() {
keys = [] keys = []
held = nil
pendingBacktick = false
cursor = nil cursor = nil
forcedMode = nil forcedMode = nil
fixed = [:] fixed = [:]
+52 -17
View File
@@ -8,33 +8,63 @@ import Foundation
/// - a letter-bigram model trained on those words, for words not in either list. /// - a letter-bigram model trained on those words, for words not in either list.
public final class EnglishModel { public final class EnglishModel {
private var common = Set<String>() private var common = Set<String>()
private var dictionary = Set<String>()
private var sortedWords: [String] = []
private var sortedCommon: [String] = [] private var sortedCommon: [String] = []
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
/// binary searches, so no 230k-entry Set has to be built at startup.
private var sortedDict: [String] = []
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
private var unseenBigram = -12.0 private var unseenBigram = -12.0
public init(commonWords: [String], dictionaryWords: [String] = []) { /// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
public init(commonWords: [String], dictionaryWords: [String] = [],
bigram: (table: [String: Double], unseen: Double)? = nil) {
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty }) common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
sortedCommon = common.sorted() sortedCommon = common.sorted()
sortedWords = common.union(dictionary).sorted() var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
trainBigram() if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
dict = Array(Set(dict)).sorted()
}
sortedDict = dict
if let bigram {
self.bigram = bigram.table
unseenBigram = bigram.unseen
} else {
trainBigram()
}
} }
/// Loads common words from a bundle resource file and the system dictionary if present. /// Loads the bundled data files. Falls back to the system word list and to training
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) { /// the bigram table when the precomputed files are missing.
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
func lines(_ url: URL?) -> [String] { func lines(_ url: URL?) -> [String] {
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] } guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
return text.split(whereSeparator: \.isNewline) return text.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) } .map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.hasPrefix("#") } .filter { !$0.isEmpty && !$0.hasPrefix("#") }
} }
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary)) var bigram: (table: [String: Double], unseen: Double)?
if let url = bigramURL, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
bigram = (table, unseen)
}
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
?? URL(fileURLWithPath: "/usr/share/dict/words")
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
} }
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) } public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) } public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
private func inDictionary(_ w: String) -> Bool {
let i = lowerBound(w, in: sortedDict)
return i < sortedDict.count && sortedDict[i] == w
}
/// Some word strictly longer than `p` starts with it (common words or dictionary).
private func isPrefixOfWord(_ p: String) -> Bool {
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
}
/// - Parameter complete: true when the user ended the word (space / punctuation), /// - Parameter complete: true when the user ended the word (space / punctuation),
/// false while still typing (prefixes of words get partial credit). /// false while still typing (prefixes of words get partial credit).
@@ -43,10 +73,10 @@ public final class EnglishModel {
let n = Double(w.count) let n = Double(w.count)
guard !w.isEmpty else { return 0 } guard !w.isEmpty else { return 0 }
if common.contains(w) { return 3 + n } if common.contains(w) { return 3 + n }
if dictionary.contains(w) { return 1 + 0.6 * n } if inDictionary(w) { return 1 + 0.6 * n }
if !complete { if !complete {
if hasPrefix(w, in: sortedCommon) { return 0.4 * n } if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
if hasPrefix(w, in: sortedWords) { return 0.2 * n } if isPrefixOfWord(w) { return 0.2 * n }
} }
// Unknown word: judge by how English-like its letter sequence is. // Unknown word: judge by how English-like its letter sequence is.
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -63,12 +93,12 @@ public final class EnglishModel {
if common.contains(w) { if common.contains(w) {
f["lit.common"] = 1 f["lit.common"] = 1
f["lit.common_len"] = n f["lit.common_len"] = n
} else if dictionary.contains(w) { } else if inDictionary(w) {
f["lit.dict"] = 1 f["lit.dict"] = 1
f["lit.dict_len"] = n f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) { } else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) { } else if !complete && isPrefixOfWord(w) {
f["lit.prefix_dict_len"] = n f["lit.prefix_dict_len"] = n
} else { } else {
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -85,12 +115,17 @@ public final class EnglishModel {
return total / Double(chars.count - 1) return total / Double(chars.count - 1)
} }
private func hasPrefix(_ p: String, in words: [String]) -> Bool { private func lowerBound(_ p: String, in words: [String]) -> Int {
var lo = 0, hi = words.count var lo = 0, hi = words.count
while lo < hi { while lo < hi {
let mid = (lo + hi) / 2 let mid = (lo + hi) / 2
if words[mid] < p { lo = mid + 1 } else { hi = mid } if words[mid] < p { lo = mid + 1 } else { hi = mid }
} }
return lo
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
var lo = lowerBound(p, in: words)
// words[lo] is the first word >= p; skip an exact match, then test the next one. // words[lo] is the first word >= p; skip an exact match, then test the next one.
if lo < words.count && words[lo] == p { lo += 1 } if lo < words.count && words[lo] == p { lo += 1 }
return lo < words.count && words[lo].hasPrefix(p) return lo < words.count && words[lo].hasPrefix(p)
@@ -99,7 +134,7 @@ public final class EnglishModel {
private func trainBigram() { private func trainBigram() {
var pair: [String: Double] = [:] var pair: [String: Double] = [:]
var first: [String: Double] = [:] var first: [String: Double] = [:]
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) { for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
let weight = common.contains(w) ? 5.0 : 1.0 let weight = common.contains(w) ? 5.0 : 1.0
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
for i in 1..<chars.count { for i in 1..<chars.count {
+138 -28
View File
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
/// File format, one entry per line (`#` starts a comment): /// File format, one entry per line (`#` starts a comment):
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2 /// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
/// The phrase/reading columns may be in either order; the score is optional. /// The phrase/reading columns may be in either order; the score is optional.
///
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
/// call `prepareTonelessIndex()` from a background queue to have it ready.
public final class Lexicon { public final class Lexicon {
private var exact: [String: [LexiconEntry]] = [:] private var exact: [String: [LexiconEntry]] = [:]
private var toneless: [String: [LexiconEntry]] = [:] private var tonelessIndex: [String: [LexiconEntry]]?
private let lock = NSLock()
public private(set) var maxPhraseLength = 1 public private(set) var maxPhraseLength = 1
public init() {} public init() {}
public convenience init(contentsOf url: URL) throws { public convenience init(contentsOf url: URL) throws {
self.init() self.init()
load(try String(contentsOf: url, encoding: .utf8)) load(data: try Data(contentsOf: url))
} }
public func load(_ text: String) { public func load(_ text: String) {
for line in text.split(whereSeparator: \.isNewline) { load(data: Data(text.utf8))
let trimmed = line.trimmingCharacters(in: .whitespaces)
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
guard cols.count >= 2 else { continue }
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
guard Lexicon.isBopomofo(reading) else { continue }
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
add(reading: reading, phrase: phrase, score: score)
}
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
} }
public func add(reading rawReading: String, phrase: String, score: Double) { public func load(data: Data) {
let entry = LexiconEntry(phrase: phrase, score: score) lock.lock()
defer { lock.unlock() }
exact.reserveCapacity(exact.count + data.count / 30)
data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
let bytes = raw.bindMemory(to: UInt8.self)
let n = bytes.count
var i = 0
while i < n {
var end = i
while end < n && bytes[end] != 0x0A { end += 1 }
parseLine(bytes, i, end)
i = end + 1
}
}
for (key, entries) in exact where entries.count > 1 {
exact[key] = entries.sorted { $0.score > $1.score }
}
tonelessIndex = nil // rebuilt on demand
}
/// Splits one line into up to three space/tab separated fields.
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
var fields: [Range<Int>] = []
var i = start
while i < end && fields.count < 3 {
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
if i >= end { break }
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
let s = i
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
fields.append(s..<i)
}
guard fields.count >= 2 else { return }
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
guard Lexicon.isBopomofo(b, r) else { return }
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
var prefixNeutral = b[r.lowerBound] == 0xCB
var k = r.lowerBound
while !prefixNeutral && k + 2 < r.upperBound {
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
k += 1
}
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
}
/// Parses "-3.9151"-style decimals without creating a String.
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
var i = r.lowerBound
var sign = 1.0
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
var value = 0.0, scale = 1.0, digits = 0, fraction = false
while i < r.upperBound {
let c = b[i]
if c >= 0x30 && c <= 0x39 {
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
digits += 1
} else if c == 0x2E && !fraction {
fraction = true
} else {
return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
}
i += 1
}
return digits > 0 ? sign * value : nil
}
/// Adds one entry (call `load` for bulk data; this does not re-sort).
public func add(reading: String, phrase: String, score: Double) {
lock.lock()
defer { lock.unlock() }
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
exact[key]?.sort { $0.score > $1.score }
tonelessIndex = nil
}
@discardableResult
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
var reading = rawReading
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix. // Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
let syllables = rawReading.split(separator: "-").map { s -> String in if normalise {
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s) reading = reading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
}.joined(separator: "-")
} }
let reading = syllables.joined(separator: "-") maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
maxPhraseLength = max(maxPhraseLength, syllables.count) if exact[reading] == nil {
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) { exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
exact[reading, default: []].append(entry) } else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
} }
let key = Lexicon.stripTones(reading) return reading
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) { }
toneless[key]![i].score = max(toneless[key]![i].score, score)
} else { /// Builds the index that ignores tones (for untoned syllables). Safe to call from a
toneless[key, default: []].append(entry) /// background queue; lookups build it themselves if it is not ready yet.
public func prepareTonelessIndex() {
lock.lock()
defer { lock.unlock() }
_ = buildTonelessIfNeeded()
}
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
if let index = tonelessIndex { return index }
var best: [String: [String: Double]] = [:]
for (reading, entries) in exact {
let key = Lexicon.stripTones(reading)
for e in entries {
if let s = best[key]?[e.phrase], s >= e.score { continue }
best[key, default: [:]][e.phrase] = e.score
}
} }
var index: [String: [LexiconEntry]] = [:]
index.reserveCapacity(best.count)
for (key, phrases) in best {
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
}
tonelessIndex = index
return index
} }
/// Entries for a sequence of readings. Untoned syllables match any tone. /// Entries for a sequence of readings. Untoned syllables match any tone.
public func entries(for syllables: [Syllable]) -> [LexiconEntry] { public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
lock.lock()
defer { lock.unlock() }
if syllables.allSatisfy({ $0.tone != nil }) { if syllables.allSatisfy({ $0.tone != nil }) {
return exact[syllables.map(\.reading).joined(separator: "-")] ?? [] return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
} }
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? [] return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
} }
public func bestScore(for syllable: Syllable) -> Double? { public func bestScore(for syllable: Syllable) -> Double? {
@@ -81,4 +182,13 @@ public final class Lexicon {
guard let first = s.unicodeScalars.first else { return false } guard let first = s.unicodeScalars.first else { return false }
return (0x3105...0x312F).contains(first.value) || first == "˙" return (0x3105...0x312F).contains(first.value) || first == "˙"
} }
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
guard r.count >= 2 else { return false }
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
let c = b[r.lowerBound + 2]
return c >= 0x85 && c <= 0xAF
}
} }
+50
View File
@@ -0,0 +1,50 @@
自動注音 AutoZhuyin — 第三方授權聲明 / Third-party notices
==========================================================
本軟體包含或使用下列第三方資料。
This software includes or uses the following third-party material.
1. 小麥注音 McBopomofo — 詞庫資料 (lexicon data)
------------------------------------------------
https://github.com/openvanilla/McBopomofo
lexicon.txt is converted from McBopomofo's data files (BPMFBase.txt,
BPMFMappings.txt, phrase.occ) by scripts/convert-mcbopomofo.py. No McBopomofo
program code is included.
MIT License
Copyright (c) 2011-2026 Mengjuei Hsieh et al.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
2. Webster's Second International Dictionary — 英文字典 (English word list)
--------------------------------------------------------------------------
english-dict.txt is filtered from /usr/share/dict/words (Webster's Second
International Dictionary, 1934), which is in the public domain.
3. PIME — Windows 輸入法框架 (input method framework)
----------------------------------------------------
https://github.com/EasyIME/PIME — GNU LGPL 2.1
The Windows version runs as a module inside PIME, which users install
separately. No PIME code is included in this software.
@@ -6,7 +6,10 @@ private let resources = URL(fileURLWithPath: #filePath)
.appendingPathComponent("Resources") .appendingPathComponent("Resources")
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")) private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase { final class ParserTests: XCTestCase {
@@ -424,3 +427,87 @@ final class FirstToneAfterEnglishTests: XCTestCase {
} }
} }
} }
final class HeldWordTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// "up " is also ㄧㄣ (因): the next word decides.
func testNextWordDecides() {
let cases = ["up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testHeldWordStaysComposing() {
let e = makeEngine()
XCTAssertEqual(type("up ", into: e), "")
XCTAssertEqual(e.markedText, "up ")
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "up")
XCTAssertEqual(type(" the ", into: e), "up the ")
}
}
final class BacktickSymbolTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testForcedSymbols() {
let cases = ["`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testPlainBacktick() {
let e = makeEngine()
XCTAssertEqual(type("`", into: e), "")
XCTAssertEqual(e.markedText, "`")
XCTAssertEqual(type("ls` ", into: e), "`ls` ")
}
func testBackspaceCancels() {
let e = makeEngine()
_ = type("su3`", into: e)
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "你")
}
}
final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws {
var t = Date()
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
}
}
+4
View File
@@ -19,6 +19,10 @@ mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
+3
View File
@@ -26,6 +26,9 @@ rsync -a --exclude __pycache__ --exclude '*.pyc' windows/autozhuyin "$OUT/"
for f in install.bat uninstall.bat README.txt; do for f in install.bat uninstall.bat README.txt; do
sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f" sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f"
done done
# Third-party notices: next to the installer and inside the installed folder.
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
(cd build && zip -qr -X "$NAME.zip" "$NAME") (cd build && zip -qr -X "$NAME.zip" "$NAME")
echo "Built build/$NAME.zip" echo "Built build/$NAME.zip"
+73 -4
View File
@@ -20,6 +20,19 @@ FULLWIDTH = {
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
} }
# An English word that is also a common one-syllable Chinese reading ("up " = ㄧㄣ 因)
# is not committed on space; the next word decides ("up jo4" = 因為, "up the" = up the).
HOLD_MIN_SCORE = -3.5
# "`" then one of these keys always gives the full-width symbol, whatever the context.
# "`" followed by anything else is just a backtick (so ``` still works).
BACKTICK = "`"
SYMBOLS = {
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", '"': "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
}
PASS = (False, None) PASS = (False, None)
CONSUMED = (True, None) CONSUMED = (True, None)
@@ -37,12 +50,15 @@ class CompositionEngine:
# English mode, toggled by a Shift tap when nothing is being typed: keys go straight # English mode, toggled by a Shift tap when nothing is being typed: keys go straight
# to the app (no composition, no underline) until Shift is tapped again. # to the app (no composition, no underline) until Shift is tapped again.
self.english_mode = False self.english_mode = False
self.backtick = False # "`" typed; the next key may be a forced symbol
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text} self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = [] self.candidates = []
self.candidate_index = 0 self.candidate_index = 0
self._candidate_segment_start = 0 self._candidate_segment_start = 0
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end. # Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
self.cursor = None self.cursor = None
# Index of a held space (see HOLD_MIN_SCORE), or None.
self.held = None
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None self._cache_key = None
@@ -60,12 +76,35 @@ class CompositionEngine:
if self.forced_mode: if self.forced_mode:
return [(0, len(keys), self.forced_mode)] return [(0, len(keys), self.forced_mode)]
ctx = self.context ctx = self.context
cache_key = (tuple(keys), finishing, ctx) held = self.held if self.held is not None and self.held < len(keys) and keys[self.held] == " " else None
cache_key = (tuple(keys), finishing, ctx, held)
if cache_key != self._cache_key: if cache_key != self._cache_key:
self._cache_key = cache_key self._cache_key = cache_key
self._cache_value = self.classifier.segment(keys, finishing, ctx) if held is None:
self._cache_value = self.classifier.segment(keys, finishing, ctx)
else:
self._cache_value = self._held_segments(keys, held, finishing)
return self._cache_value return self._cache_value
def _held_segments(self, keys, held, finishing):
"""The held word is Chinese if the words typed after it start with Chinese."""
rest = keys[held + 1:]
base = held + 1
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
if rest_segs and rest_segs[0][2] == ZHUYIN:
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
return [(0, base, LITERAL)] + rest_segs
def _is_ambiguous_word(self, keys):
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
if not all(k.isascii() and k.isalpha() for k in keys):
return False
p = bopomofo.parse(list(keys) + [" "])
if p.pending is not None or len(p.syllables) != 1:
return False
entries = self.classifier.lexicon.entries([s.syllable for s in p.syllables])
return bool(entries) and entries[0][1] >= HOLD_MIN_SCORE
@property @property
def segments(self): def segments(self):
return self._segments(self.keys) return self._segments(self.keys)
@@ -142,7 +181,10 @@ class CompositionEngine:
@property @property
def marked_pieces(self): def marked_pieces(self):
segs = self.segments segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)] pieces = [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
if self.backtick:
pieces.append((BACKTICK, LITERAL))
return pieces
@property @property
def marked_text(self): def marked_text(self):
@@ -156,7 +198,7 @@ class CompositionEngine:
def wants(self, key): def wants(self, key):
"""Whether handle(key) would consume the key, without changing state.""" """Whether handle(key) would consume the key, without changing state."""
if self.candidates or key == SHIFT_TAP: if self.candidates or key == SHIFT_TAP or self.backtick:
return True return True
if self.english_mode and not self.keys: if self.english_mode and not self.keys:
return False # English mode: everything goes to the app as typed return False # English mode: everything goes to the app as typed
@@ -173,11 +215,15 @@ class CompositionEngine:
def handle(self, key): def handle(self, key):
if self.candidates: if self.candidates:
return self._handle_candidate_key(key) return self._handle_candidate_key(key)
if self.backtick:
return self._after_backtick(key)
if key == BACKSPACE: if key == BACKSPACE:
if not self.keys: if not self.keys:
return PASS return PASS
self.cursor = None self.cursor = None
self.keys.pop() self.keys.pop()
if self.held is not None and len(self.keys) <= self.held:
self.held = None
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)} self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys: if not self.keys:
self.forced_mode = None self.forced_mode = None
@@ -220,6 +266,19 @@ class CompositionEngine:
return CONSUMED if self.keys else PASS return CONSUMED if self.keys else PASS
return self._handle_character(key) return self._handle_character(key)
def _after_backtick(self, key):
self.backtick = False
if key in (ESCAPE, BACKSPACE):
return CONSUMED
committed = self._commit() if self.keys else ""
if key in SYMBOLS:
return True, committed + SYMBOLS[key]
# Not a symbol: the backtick was meant literally; then handle the key as usual.
handled, more = self.handle(key)
if not handled and len(key) == 1:
handled, more = True, key
return handled, committed + BACKTICK + (more or "")
def _last_parse(self, segs): def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN: if not segs or segs[-1][2] != ZHUYIN:
return None return None
@@ -230,11 +289,19 @@ class CompositionEngine:
self.cursor = None # typing always continues at the end self.cursor = None # typing always continues at the end
if self.english_mode and not self.keys: if self.english_mode and not self.keys:
return PASS # English mode: every key goes to the app as typed return PASS # English mode: every key goes to the app as typed
if c == BACKTICK:
self.backtick = True
return CONSUMED
if c == " ": if c == " ":
if not self.keys: if not self.keys:
return PASS return PASS
segs = self._segments(self.keys, finishing=True) segs = self._segments(self.keys, finishing=True)
if segs[-1][2] == LITERAL: if segs[-1][2] == LITERAL:
if (self.held is None and len(segs) == 1 and not self.forced_mode
and self._is_ambiguous_word(self.keys)):
self.held = len(self.keys)
self.keys.append(" ")
return CONSUMED
return True, self._commit(segs) + " " return True, self._commit(segs) + " "
p = self._last_parse(segs) p = self._last_parse(segs)
if p is not None and p.pending is not None: if p is not None and p.pending is not None:
@@ -319,6 +386,8 @@ class CompositionEngine:
def reset(self): def reset(self):
self.keys = [] self.keys = []
self.held = None
self.backtick = False
self.cursor = None self.cursor = None
self.forced_mode = None self.forced_mode = None
self.fixed = {} self.fixed = {}
+18 -12
View File
@@ -16,7 +16,7 @@ class Lexicon:
def __init__(self): def __init__(self):
self.exact = {} self.exact = {}
self.toneless = {} self._toneless = None # built on first use (only untoned syllables need it)
self.max_phrase_length = 1 self.max_phrase_length = 1
@classmethod @classmethod
@@ -42,9 +42,10 @@ class Lexicon:
except ValueError: except ValueError:
score = -6.0 score = -6.0
self.add(reading, phrase, score) self.add(reading, phrase, score)
for table in (self.exact, self.toneless): for v in self.exact.values():
for v in table.values(): if len(v) > 1:
v.sort(key=lambda e: -e[1]) v.sort(key=lambda e: -e[1])
self._toneless = None
def add(self, reading, phrase, score): def add(self, reading, phrase, score):
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")] syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
@@ -53,15 +54,20 @@ class Lexicon:
entries = self.exact.setdefault(reading, []) entries = self.exact.setdefault(reading, [])
if not any(p == phrase for p, _ in entries): if not any(p == phrase for p, _ in entries):
entries.append((phrase, score)) entries.append((phrase, score))
key = _strip_tones(reading) self._toneless = None
tl = self.toneless.setdefault(key, [])
for i, (p, s) in enumerate(tl): @property
if p == phrase: def toneless(self):
if score > s: """Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
tl[i] = (p, score) if self._toneless is None:
break best = {}
else: for reading, entries in self.exact.items():
tl.append((phrase, score)) bucket = best.setdefault(_strip_tones(reading), {})
for phrase, score in entries:
if score > bucket.get(phrase, float("-inf")):
bucket[phrase] = score
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
return self._toneless
def entries(self, syllables): def entries(self, syllables):
if all(s.tone is not None for s in syllables): if all(s.tone is not None for s in syllables):
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.6.1", "version": "0.7.1",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+3
View File
@@ -34,6 +34,9 @@
打到一半時按,同 Tab 整段翻轉 打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文) Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!: < > ? ! : 中文語境下自動變成全形 ,。?!:
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
`' = 、 `[ `] = 「」 `( `) = () `! = !
` 後面接其他鍵就是普通的反引號
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
+36
View File
@@ -279,5 +279,41 @@ class FirstToneAfterEnglishTests(unittest.TestCase):
self.assertEqual(len(e.marked_text), 1, keys) self.assertEqual(len(e.marked_text), 1, keys)
class HeldWordTests(unittest.TestCase):
def test_next_word_decides(self):
cases = {"up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_held_word_stays_composing(self):
e = make_engine()
self.assertEqual(type_keys("up ", e), "")
self.assertEqual(e.marked_text, "up ")
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "up")
self.assertEqual(type_keys(" the ", e), "up the ")
class BacktickSymbolTests(unittest.TestCase):
def test_forced_symbols(self):
cases = {"`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_plain_backtick(self):
e = make_engine()
self.assertEqual(type_keys("`", e), "")
self.assertEqual(e.marked_text, "`")
self.assertEqual(type_keys("ls` ", e), "`ls` ")
def test_escape_cancels(self):
e = make_engine()
type_keys("su3`", e)
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "你")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()