Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fcc0f9d2db | ||
|
|
31d9991e16 | ||
|
|
6b4ec39dc1 | ||
|
|
955b0ca5c3 | ||
|
|
251a16fa08 | ||
|
|
7cb1f62dfa | ||
|
|
3670305f0f | ||
|
|
ef9f91a6b7 | ||
|
|
1dacb8f164 | ||
|
|
bfeed22916 |
@@ -16,7 +16,8 @@
|
||||
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
|
||||
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
|
||||
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
|
||||
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`
|
||||
- **中間補字、刪字**:游標移到前面後直接打注音,會插在游標處;Backspace 刪掉游標前的那個字
|
||||
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
|
||||
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
|
||||
|
||||
## 下載與安裝
|
||||
@@ -51,13 +52,14 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
|
||||
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
|
||||
| Enter | 送出目前組字 |
|
||||
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
|
||||
| ← → | 在組字區移動游標,選要改哪個字 |
|
||||
| ← → | 在組字區移動游標(可移到最前面)。游標在中間時:直接打注音會**插入**新字,Backspace 刪掉游標前的字,↓ 選游標前的字 |
|
||||
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
|
||||
| Esc | 取消組字 |
|
||||
| **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
|
||||
| Shift + 字母 | 打大寫字母(也會強制英文) |
|
||||
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
|
||||
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
|
||||
| `` ` `` + 標點鍵 | **強制全形標點**,不管前後是中文還是英文:`` `, `` `` `< `` = `,`、`` `. `` `` `> `` = `。`、`` `/ `` = `?`、`` `; `` = `;`、`` `: `` = `:`、`` `' `` `` `\ `` = `、`、`` `[ `` `` `] `` = `「」`、`` `{ `` `` `} `` = `『』`、`` `( `` `` `) `` = `()`、`` `! `` = `!`、`` `~ `` = `~`、`` `- `` = `——`、`` `= `` = `=`。`` ` `` 後面接其他鍵就是普通的反引號(` ``` ` 照常可用) |
|
||||
|
||||
## 運作方式
|
||||
|
||||
@@ -211,13 +213,14 @@ windows/
|
||||
tests/ Python 測試
|
||||
tools/ 產生資料檔、圖示
|
||||
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
|
||||
THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝檔)
|
||||
```
|
||||
|
||||
## 已知限制
|
||||
|
||||
- Windows 版還沒在實機上測試過。
|
||||
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
|
||||
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。
|
||||
- 很短的字串本質上就有歧義。像 `up`(也是 ㄧㄣ 因)這種常用的,按空白後會先保留,看下一個詞再決定(`up jo4` = 因為、`up the` = up the);其他判斷錯的可以按 Tab 修正,修正後會記住。
|
||||
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
|
||||
|
||||
## 下一步(Roadmap)
|
||||
@@ -237,3 +240,5 @@ training/ 合成資料、訓練權重、Swift/Python 一致
|
||||
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
|
||||
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
|
||||
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
|
||||
|
||||
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
|
||||
|
||||
@@ -15,9 +15,9 @@
|
||||
<key>CFBundlePackageType</key>
|
||||
<string>APPL</string>
|
||||
<key>CFBundleShortVersionString</key>
|
||||
<string>0.6.1</string>
|
||||
<string>0.8.0</string>
|
||||
<key>CFBundleVersion</key>
|
||||
<string>9</string>
|
||||
<string>14</string>
|
||||
<key>InputMethodConnectionName</key>
|
||||
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
|
||||
<key>InputMethodServerControllerClass</key>
|
||||
|
||||
@@ -11,22 +11,28 @@ enum Shared {
|
||||
|
||||
static let classifier: Classifier = {
|
||||
let bundle = Bundle.main
|
||||
let start = Date()
|
||||
let lexicon = Lexicon()
|
||||
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
|
||||
let text = try? String(contentsOf: url, encoding: .utf8) {
|
||||
lexicon.load(text)
|
||||
let data = try? Data(contentsOf: url) {
|
||||
lexicon.load(data: data)
|
||||
}
|
||||
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
|
||||
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
|
||||
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
|
||||
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
|
||||
}
|
||||
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"))
|
||||
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
|
||||
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
|
||||
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
|
||||
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
|
||||
let learned = LearnedPreferences(weights: saved) { weights in
|
||||
UserDefaults.standard.set(weights, forKey: learnedKey)
|
||||
}
|
||||
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
|
||||
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
|
||||
// Only needed for untoned syllables; build it off the startup path.
|
||||
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
|
||||
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
|
||||
}()
|
||||
}
|
||||
|
||||
@@ -45,12 +45,23 @@ public final class CompositionEngine {
|
||||
private var fixed: [Int: [Range<Int>: String]] = [:]
|
||||
public private(set) var candidates: [Candidate] = []
|
||||
private var candidateSegmentStart = 0
|
||||
/// Composition cursor for choosing candidates: index into `cursorStops`, nil = at the end.
|
||||
/// ← → move it; ↓ shows candidates for the phrase ending at the cursor.
|
||||
/// Composition cursor: a key index into `keys` (nil = at the end). ← → move it between
|
||||
/// syllables; ↓ shows candidates for the phrase before it; typing there inserts a
|
||||
/// syllable (kept in `inserting` until its tone key, then spliced into `keys`).
|
||||
private var cursor: Int?
|
||||
private var inserting: [Character] = []
|
||||
public var candidateIndex = 0
|
||||
public var isShowingCandidates: Bool { !candidates.isEmpty }
|
||||
|
||||
/// Index of a held space: an English word that is also a common one-syllable Chinese
|
||||
/// reading ("up " = ㄧㄣ 因) is not committed on space; the next word decides
|
||||
/// ("up jo4" = 因為, "up the" = up the).
|
||||
private var held: Int?
|
||||
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
|
||||
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
|
||||
public private(set) var pendingBacktick = false
|
||||
static let holdMinScore = -3.5
|
||||
|
||||
private var previousMode: InputMode?
|
||||
private var previousCommitTime = Date.distantPast
|
||||
/// How long the previous segment's mode keeps influencing the next one.
|
||||
@@ -71,9 +82,33 @@ public final class CompositionEngine {
|
||||
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
|
||||
guard !keys.isEmpty else { return [] }
|
||||
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
|
||||
if let h = held, h < keys.count, keys[h] == " " {
|
||||
return heldSegments(keys, held: h, finishing: finishing)
|
||||
}
|
||||
return classifier.segment(keys, finishing: finishing, previous: context)
|
||||
}
|
||||
|
||||
/// The held word is Chinese if the words typed after it start with Chinese.
|
||||
private func heldSegments(_ keys: [Character], held h: Int, finishing: Bool) -> [Segment] {
|
||||
let base = h + 1
|
||||
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
|
||||
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
|
||||
}
|
||||
if let first = rest.first, first.mode == .zhuyin {
|
||||
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
|
||||
}
|
||||
return [Segment(range: 0..<base, mode: .literal)] + rest
|
||||
}
|
||||
|
||||
/// Whether keys + space also read as a common Chinese syllable (see `held`).
|
||||
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
|
||||
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
|
||||
let p = ZhuyinParser.parse(keys + [" "])
|
||||
guard p.pending == nil, p.syllables.count == 1 else { return false }
|
||||
guard let best = classifier.lexicon.entries(for: p.syllables.map(\.syllable)).first else { return false }
|
||||
return best.score >= CompositionEngine.holdMinScore
|
||||
}
|
||||
|
||||
/// Current split of the buffer into Zhuyin / literal runs.
|
||||
public var segments: [Segment] { segments(of: keys) }
|
||||
|
||||
@@ -84,6 +119,29 @@ public final class CompositionEngine {
|
||||
public var markedText: String { markedPieces.map(\.text).joined() }
|
||||
|
||||
public var markedPieces: [MarkedPiece] {
|
||||
var pieces = segmentPieces
|
||||
if !inserting.isEmpty {
|
||||
// Show the syllable being inserted at the cursor.
|
||||
var at = baseCursorOffset, out: [MarkedPiece] = [], done = false
|
||||
for piece in pieces {
|
||||
if !done && at <= piece.text.count {
|
||||
let i = piece.text.index(piece.text.startIndex, offsetBy: at)
|
||||
out.append(MarkedPiece(text: String(piece.text[..<i]), mode: piece.mode))
|
||||
out.append(MarkedPiece(text: insertingText, mode: .zhuyin))
|
||||
out.append(MarkedPiece(text: String(piece.text[i...]), mode: piece.mode))
|
||||
done = true
|
||||
} else {
|
||||
out.append(piece)
|
||||
at -= piece.text.count
|
||||
}
|
||||
}
|
||||
pieces = out.filter { !$0.text.isEmpty }
|
||||
}
|
||||
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
|
||||
return pieces
|
||||
}
|
||||
|
||||
private var segmentPieces: [MarkedPiece] {
|
||||
let segs = segments
|
||||
return segs.enumerated().map { i, seg in
|
||||
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
|
||||
@@ -113,62 +171,180 @@ public final class CompositionEngine {
|
||||
|
||||
// MARK: - Cursor
|
||||
|
||||
/// A place the cursor can stop: after the `local`-th syllable of Zhuyin segment `segIndex`.
|
||||
/// Absolute key index after each syllable of a Zhuyin segment.
|
||||
private func syllableEnds(_ seg: Segment) -> [Int] {
|
||||
ZhuyinParser.parse(Array(keys[seg.range])).syllables.map { seg.range.lowerBound + $0.keyRange.upperBound }
|
||||
}
|
||||
|
||||
/// Key indices the cursor can stop at: the start, after every Zhuyin syllable, and at
|
||||
/// every segment boundary.
|
||||
private var cursorPositions: [Int] {
|
||||
var out: Set<Int> = [0, keys.count]
|
||||
for seg in segments {
|
||||
out.insert(seg.range.upperBound)
|
||||
if seg.mode == .zhuyin { out.formUnion(syllableEnds(seg)) }
|
||||
}
|
||||
return out.sorted()
|
||||
}
|
||||
|
||||
/// A place to open candidates: after the `local`-th syllable of Zhuyin segment `segIndex`.
|
||||
private struct CursorStop { let segIndex: Int; let local: Int }
|
||||
|
||||
private var cursorStops: [CursorStop] {
|
||||
private func stop(at pos: Int) -> CursorStop? {
|
||||
let segs = segments
|
||||
var out: [CursorStop] = []
|
||||
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin {
|
||||
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin
|
||||
&& seg.range.lowerBound < pos && pos <= seg.range.upperBound {
|
||||
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
|
||||
for k in stride(from: 1, through: n, by: 1) { out.append(CursorStop(segIndex: i, local: k)) }
|
||||
let local = syllableEnds(seg).filter { $0 <= pos }.count
|
||||
return local > 0 ? CursorStop(segIndex: i, local: min(local, n)) : nil
|
||||
}
|
||||
return out
|
||||
return nil
|
||||
}
|
||||
|
||||
/// Where ↓ opens candidates: before the cursor, or the last syllable.
|
||||
private var candidateStop: CursorStop? {
|
||||
if let c = cursor { return stop(at: c) }
|
||||
let segs = segments
|
||||
for i in segs.indices.reversed() where segs[i].mode == .zhuyin {
|
||||
let n = convertibleSyllables(segs[i], final: i < segs.count - 1).syllables.count
|
||||
if n > 0 { return CursorStop(segIndex: i, local: n) }
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
/// Cursor position in the rendered segments (without the syllable being inserted).
|
||||
private var baseCursorOffset: Int {
|
||||
let pieces = segmentPieces
|
||||
guard let c = cursor else { return pieces.reduce(0) { $0 + $1.text.count } }
|
||||
let segs = segments
|
||||
for (i, seg) in segs.enumerated() where seg.range.lowerBound < c && c <= seg.range.upperBound {
|
||||
var offset = pieces[..<i].reduce(0) { $0 + $1.text.count }
|
||||
if seg.mode == .literal { return offset + c - seg.range.lowerBound }
|
||||
let local = stop(at: c)?.local ?? 0
|
||||
for conv in conversion(of: seg, final: i < segs.count - 1).segments {
|
||||
if conv.range.upperBound <= local {
|
||||
offset += conv.text.count
|
||||
} else if conv.range.lowerBound < local {
|
||||
offset += min(local - conv.range.lowerBound, conv.text.count)
|
||||
}
|
||||
}
|
||||
return offset
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
/// Cursor position in `markedText`, in Characters.
|
||||
public var cursorOffset: Int {
|
||||
let pieces = markedPieces
|
||||
let total = pieces.reduce(0) { $0 + $1.text.count }
|
||||
let stops = cursorStops
|
||||
guard let c = cursor, stops.indices.contains(c) else { return total }
|
||||
let stop = stops[c]
|
||||
var offset = pieces[..<stop.segIndex].reduce(0) { $0 + $1.text.count }
|
||||
let segs = segments
|
||||
for conv in conversion(of: segs[stop.segIndex], final: stop.segIndex < segs.count - 1).segments {
|
||||
if conv.range.upperBound <= stop.local {
|
||||
offset += conv.text.count
|
||||
} else if conv.range.lowerBound < stop.local {
|
||||
offset += min(stop.local - conv.range.lowerBound, conv.text.count)
|
||||
}
|
||||
}
|
||||
return offset
|
||||
public var cursorOffset: Int { baseCursorOffset + insertingText.count }
|
||||
|
||||
private var insertingText: String {
|
||||
inserting.isEmpty ? "" : ZhuyinParser.parse(inserting).syllables.map(\.syllable.reading).joined()
|
||||
}
|
||||
|
||||
private func moveCursor(by delta: Int) {
|
||||
let stops = cursorStops
|
||||
guard !stops.isEmpty else { return }
|
||||
let next = (cursor ?? stops.count - 1) + delta
|
||||
cursor = next >= stops.count - 1 ? nil : max(0, next)
|
||||
inserting = []
|
||||
let stops = cursorPositions
|
||||
let here = cursor ?? keys.count
|
||||
let next = delta < 0 ? (stops.last { $0 < here } ?? here) : (stops.first { $0 > here } ?? here)
|
||||
cursor = next >= keys.count ? nil : next
|
||||
}
|
||||
|
||||
// MARK: - Editing in the middle
|
||||
|
||||
/// Keeps picked phrases that end at or before key index `pos` (they don't move).
|
||||
private func trimFixed(at pos: Int) {
|
||||
var kept: [Int: [Range<Int>: String]] = [:]
|
||||
for seg in segments where seg.mode == .zhuyin {
|
||||
guard let picks = fixed[seg.range.lowerBound] else { continue }
|
||||
if seg.range.upperBound <= pos {
|
||||
kept[seg.range.lowerBound] = picks
|
||||
} else if seg.range.lowerBound < pos {
|
||||
let ends = syllableEnds(seg)
|
||||
let here = picks.filter { $0.key.upperBound <= ends.count && ends[$0.key.upperBound - 1] <= pos }
|
||||
if !here.isEmpty { kept[seg.range.lowerBound] = here }
|
||||
}
|
||||
}
|
||||
fixed = kept
|
||||
}
|
||||
|
||||
/// Inserts keys at the cursor and moves the cursor after them.
|
||||
private func splice(_ newKeys: [Character]) {
|
||||
guard let pos = cursor else { keys += newKeys; return }
|
||||
trimFixed(at: pos)
|
||||
keys.insert(contentsOf: newKeys, at: pos)
|
||||
if let h = held, h >= pos { held = h + newKeys.count }
|
||||
let next = pos + newKeys.count
|
||||
cursor = next >= keys.count ? nil : next
|
||||
}
|
||||
|
||||
/// Backspace in the middle: removes the syllable (or literal key) before the cursor.
|
||||
private func deleteBeforeCursor(_ pos: Int) {
|
||||
var start = pos - 1
|
||||
for seg in segments where seg.mode == .zhuyin && seg.range.lowerBound < pos && pos <= seg.range.upperBound {
|
||||
for ps in ZhuyinParser.parse(Array(keys[seg.range])).syllables
|
||||
where seg.range.lowerBound + ps.keyRange.upperBound == pos {
|
||||
start = seg.range.lowerBound + ps.keyRange.lowerBound
|
||||
}
|
||||
}
|
||||
trimFixed(at: start)
|
||||
keys.removeSubrange(start..<pos)
|
||||
if let h = held {
|
||||
if h >= start && h < pos { held = nil } else if h >= pos { held = h - (pos - start) }
|
||||
}
|
||||
cursor = start
|
||||
if keys.isEmpty { reset() }
|
||||
}
|
||||
|
||||
/// Typing with the cursor in the middle: compose one syllable, splice it in on its tone.
|
||||
private func insertKey(_ c: Character) -> KeyResult {
|
||||
if c == " " {
|
||||
if inserting.isEmpty { return KeyResult(handled: true, commit: commitCurrent()) }
|
||||
splice(inserting + [" "])
|
||||
inserting = []
|
||||
return .consumed
|
||||
}
|
||||
inserting.append(c)
|
||||
let p = ZhuyinParser.parse(inserting)
|
||||
if p.strayTones > 0 || p.foreignKeys > 0 {
|
||||
inserting.removeLast() // a tone with nothing to attach to
|
||||
} else if p.syllables.count == 1, p.syllables[0].toned {
|
||||
splice(inserting)
|
||||
inserting = []
|
||||
} else if p.syllables.count > 1 {
|
||||
// A new syllable started without a tone: splice the first one as it is.
|
||||
let first = p.syllables[0].keyRange.upperBound
|
||||
splice(Array(inserting[..<first]))
|
||||
inserting = Array(inserting[first...])
|
||||
}
|
||||
return .consumed
|
||||
}
|
||||
|
||||
// MARK: - Key handling
|
||||
|
||||
public func handle(_ key: KeyInput) -> KeyResult {
|
||||
if isShowingCandidates { return handleCandidateKey(key) }
|
||||
if pendingBacktick { return afterBacktick(key) }
|
||||
if !inserting.isEmpty && (key == .backspace || key == .escape) {
|
||||
if key == .backspace { inserting.removeLast() } else { inserting = [] }
|
||||
return .consumed
|
||||
}
|
||||
|
||||
switch key {
|
||||
case .character(let c):
|
||||
return handleCharacter(c)
|
||||
case .backspace:
|
||||
guard !keys.isEmpty else { return .passThrough }
|
||||
cursor = nil
|
||||
if let c = cursor {
|
||||
if c > 0 { deleteBeforeCursor(c) }
|
||||
return .consumed
|
||||
}
|
||||
keys.removeLast()
|
||||
if let h = held, keys.count <= h { held = nil }
|
||||
fixed = fixed.filter { $0.key < keys.count }
|
||||
if keys.isEmpty { forcedMode = nil; fixed = [:] }
|
||||
return .consumed
|
||||
case .enter:
|
||||
guard !keys.isEmpty else { return .passThrough }
|
||||
if !inserting.isEmpty { splice(inserting); inserting = [] }
|
||||
return KeyResult(handled: true, commit: commitCurrent())
|
||||
case .escape:
|
||||
guard !keys.isEmpty else { return .passThrough }
|
||||
@@ -190,7 +366,7 @@ public final class CompositionEngine {
|
||||
return .consumed
|
||||
case .down:
|
||||
guard !keys.isEmpty else { return .passThrough }
|
||||
openCandidates()
|
||||
if inserting.isEmpty { openCandidates() }
|
||||
return .consumed
|
||||
case .left:
|
||||
guard !keys.isEmpty else { return .passThrough }
|
||||
@@ -211,13 +387,26 @@ public final class CompositionEngine {
|
||||
}
|
||||
|
||||
private func handleCharacter(_ c: Character) -> KeyResult {
|
||||
cursor = nil // typing always continues at the end
|
||||
if cursor != nil, !keys.isEmpty, c == " " || DachenLayout.isZhuyinKey(c) {
|
||||
return insertKey(c)
|
||||
}
|
||||
if !inserting.isEmpty { splice(inserting); inserting = [] }
|
||||
cursor = nil // anything else continues at the end
|
||||
// English mode: every key goes to the app as typed.
|
||||
if isEnglishMode && keys.isEmpty { return .passThrough }
|
||||
if c == "`" {
|
||||
pendingBacktick = true
|
||||
return .consumed
|
||||
}
|
||||
if c == " " {
|
||||
guard !keys.isEmpty else { return .passThrough }
|
||||
let segs = segments(of: keys, finishing: true)
|
||||
if segs.last?.mode == .literal {
|
||||
if held == nil, segs.count == 1, forcedMode == nil, isAmbiguousWord(keys) {
|
||||
held = keys.count
|
||||
keys.append(" ")
|
||||
return .consumed
|
||||
}
|
||||
return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
|
||||
}
|
||||
if parse(ofLast: segs)?.pending != nil {
|
||||
@@ -254,6 +443,26 @@ public final class CompositionEngine {
|
||||
return ZhuyinParser.parse(Array(keys[last.range]))
|
||||
}
|
||||
|
||||
private func afterBacktick(_ key: KeyInput) -> KeyResult {
|
||||
pendingBacktick = false
|
||||
if key == .escape || key == .backspace { return .consumed }
|
||||
let committed = keys.isEmpty ? "" : commitCurrent()
|
||||
if case .character(let c) = key, let symbol = CompositionEngine.symbols[c] {
|
||||
return KeyResult(handled: true, commit: committed + symbol)
|
||||
}
|
||||
// Not a symbol: the backtick was meant literally; then handle the key as usual.
|
||||
var r = handle(key)
|
||||
if !r.handled, case .character(let c) = key { r = KeyResult(handled: true, commit: String(c)) }
|
||||
return KeyResult(handled: r.handled, commit: committed + "`" + (r.commit ?? ""))
|
||||
}
|
||||
|
||||
/// "`" + key → full-width symbol.
|
||||
static let symbols: [Character: String] = [
|
||||
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
|
||||
"'": "、", "\\": "、", "\"": "「", "[": "「", "]": "」", "{": "『", "}": "』",
|
||||
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
|
||||
]
|
||||
|
||||
static let fullwidth: [Character: String] = [
|
||||
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
|
||||
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
||||
@@ -263,9 +472,7 @@ public final class CompositionEngine {
|
||||
|
||||
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
|
||||
private func openCandidates() {
|
||||
let stops = cursorStops
|
||||
guard !stops.isEmpty else { return }
|
||||
let stop = stops[min(cursor ?? stops.count - 1, stops.count - 1)]
|
||||
guard let stop = candidateStop else { return }
|
||||
let segs = segments
|
||||
let seg = segs[stop.segIndex]
|
||||
let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
|
||||
@@ -326,6 +533,9 @@ public final class CompositionEngine {
|
||||
|
||||
public func reset() {
|
||||
keys = []
|
||||
inserting = []
|
||||
held = nil
|
||||
pendingBacktick = false
|
||||
cursor = nil
|
||||
forcedMode = nil
|
||||
fixed = [:]
|
||||
|
||||
@@ -8,33 +8,63 @@ import Foundation
|
||||
/// - a letter-bigram model trained on those words, for words not in either list.
|
||||
public final class EnglishModel {
|
||||
private var common = Set<String>()
|
||||
private var dictionary = Set<String>()
|
||||
private var sortedWords: [String] = []
|
||||
private var sortedCommon: [String] = []
|
||||
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
|
||||
/// binary searches, so no 230k-entry Set has to be built at startup.
|
||||
private var sortedDict: [String] = []
|
||||
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
|
||||
private var unseenBigram = -12.0
|
||||
|
||||
public init(commonWords: [String], dictionaryWords: [String] = []) {
|
||||
/// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
|
||||
public init(commonWords: [String], dictionaryWords: [String] = [],
|
||||
bigram: (table: [String: Double], unseen: Double)? = nil) {
|
||||
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
|
||||
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
|
||||
sortedCommon = common.sorted()
|
||||
sortedWords = common.union(dictionary).sorted()
|
||||
trainBigram()
|
||||
var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
|
||||
if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
|
||||
dict = Array(Set(dict)).sorted()
|
||||
}
|
||||
sortedDict = dict
|
||||
if let bigram {
|
||||
self.bigram = bigram.table
|
||||
unseenBigram = bigram.unseen
|
||||
} else {
|
||||
trainBigram()
|
||||
}
|
||||
}
|
||||
|
||||
/// Loads common words from a bundle resource file and the system dictionary if present.
|
||||
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) {
|
||||
/// Loads the bundled data files. Falls back to the system word list and to training
|
||||
/// the bigram table when the precomputed files are missing.
|
||||
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
|
||||
func lines(_ url: URL?) -> [String] {
|
||||
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
|
||||
return text.split(whereSeparator: \.isNewline)
|
||||
.map { $0.trimmingCharacters(in: .whitespaces) }
|
||||
.filter { !$0.hasPrefix("#") }
|
||||
.filter { !$0.isEmpty && !$0.hasPrefix("#") }
|
||||
}
|
||||
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary))
|
||||
var bigram: (table: [String: Double], unseen: Double)?
|
||||
if let url = bigramURL, let data = try? Data(contentsOf: url),
|
||||
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
|
||||
bigram = (table, unseen)
|
||||
}
|
||||
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
|
||||
?? URL(fileURLWithPath: "/usr/share/dict/words")
|
||||
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
|
||||
}
|
||||
|
||||
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
|
||||
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) }
|
||||
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
|
||||
|
||||
private func inDictionary(_ w: String) -> Bool {
|
||||
let i = lowerBound(w, in: sortedDict)
|
||||
return i < sortedDict.count && sortedDict[i] == w
|
||||
}
|
||||
|
||||
/// Some word strictly longer than `p` starts with it (common words or dictionary).
|
||||
private func isPrefixOfWord(_ p: String) -> Bool {
|
||||
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
|
||||
}
|
||||
|
||||
/// - Parameter complete: true when the user ended the word (space / punctuation),
|
||||
/// false while still typing (prefixes of words get partial credit).
|
||||
@@ -43,10 +73,10 @@ public final class EnglishModel {
|
||||
let n = Double(w.count)
|
||||
guard !w.isEmpty else { return 0 }
|
||||
if common.contains(w) { return 3 + n }
|
||||
if dictionary.contains(w) { return 1 + 0.6 * n }
|
||||
if inDictionary(w) { return 1 + 0.6 * n }
|
||||
if !complete {
|
||||
if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
|
||||
if hasPrefix(w, in: sortedWords) { return 0.2 * n }
|
||||
if isPrefixOfWord(w) { return 0.2 * n }
|
||||
}
|
||||
// Unknown word: judge by how English-like its letter sequence is.
|
||||
let plaus = plausibility(w)
|
||||
@@ -63,12 +93,12 @@ public final class EnglishModel {
|
||||
if common.contains(w) {
|
||||
f["lit.common"] = 1
|
||||
f["lit.common_len"] = n
|
||||
} else if dictionary.contains(w) {
|
||||
} else if inDictionary(w) {
|
||||
f["lit.dict"] = 1
|
||||
f["lit.dict_len"] = n
|
||||
} else if !complete && hasPrefix(w, in: sortedCommon) {
|
||||
f["lit.prefix_common_len"] = n
|
||||
} else if !complete && hasPrefix(w, in: sortedWords) {
|
||||
} else if !complete && isPrefixOfWord(w) {
|
||||
f["lit.prefix_dict_len"] = n
|
||||
} else {
|
||||
let plaus = plausibility(w)
|
||||
@@ -85,12 +115,17 @@ public final class EnglishModel {
|
||||
return total / Double(chars.count - 1)
|
||||
}
|
||||
|
||||
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
|
||||
private func lowerBound(_ p: String, in words: [String]) -> Int {
|
||||
var lo = 0, hi = words.count
|
||||
while lo < hi {
|
||||
let mid = (lo + hi) / 2
|
||||
if words[mid] < p { lo = mid + 1 } else { hi = mid }
|
||||
}
|
||||
return lo
|
||||
}
|
||||
|
||||
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
|
||||
var lo = lowerBound(p, in: words)
|
||||
// words[lo] is the first word >= p; skip an exact match, then test the next one.
|
||||
if lo < words.count && words[lo] == p { lo += 1 }
|
||||
return lo < words.count && words[lo].hasPrefix(p)
|
||||
@@ -99,7 +134,7 @@ public final class EnglishModel {
|
||||
private func trainBigram() {
|
||||
var pair: [String: Double] = [:]
|
||||
var first: [String: Double] = [:]
|
||||
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
|
||||
for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
|
||||
let weight = common.contains(w) ? 5.0 : 1.0
|
||||
let chars = ["^"] + w.map(String.init) + ["$"]
|
||||
for i in 1..<chars.count {
|
||||
|
||||
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
|
||||
/// File format, one entry per line (`#` starts a comment):
|
||||
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
|
||||
/// The phrase/reading columns may be in either order; the score is optional.
|
||||
///
|
||||
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
|
||||
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
|
||||
/// call `prepareTonelessIndex()` from a background queue to have it ready.
|
||||
public final class Lexicon {
|
||||
private var exact: [String: [LexiconEntry]] = [:]
|
||||
private var toneless: [String: [LexiconEntry]] = [:]
|
||||
private var tonelessIndex: [String: [LexiconEntry]]?
|
||||
private let lock = NSLock()
|
||||
public private(set) var maxPhraseLength = 1
|
||||
|
||||
public init() {}
|
||||
|
||||
public convenience init(contentsOf url: URL) throws {
|
||||
self.init()
|
||||
load(try String(contentsOf: url, encoding: .utf8))
|
||||
load(data: try Data(contentsOf: url))
|
||||
}
|
||||
|
||||
public func load(_ text: String) {
|
||||
for line in text.split(whereSeparator: \.isNewline) {
|
||||
let trimmed = line.trimmingCharacters(in: .whitespaces)
|
||||
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
|
||||
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
|
||||
guard cols.count >= 2 else { continue }
|
||||
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
|
||||
guard Lexicon.isBopomofo(reading) else { continue }
|
||||
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
|
||||
add(reading: reading, phrase: phrase, score: score)
|
||||
}
|
||||
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
|
||||
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
|
||||
load(data: Data(text.utf8))
|
||||
}
|
||||
|
||||
public func add(reading rawReading: String, phrase: String, score: Double) {
|
||||
let entry = LexiconEntry(phrase: phrase, score: score)
|
||||
public func load(data: Data) {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
exact.reserveCapacity(exact.count + data.count / 30)
|
||||
data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
|
||||
let bytes = raw.bindMemory(to: UInt8.self)
|
||||
let n = bytes.count
|
||||
var i = 0
|
||||
while i < n {
|
||||
var end = i
|
||||
while end < n && bytes[end] != 0x0A { end += 1 }
|
||||
parseLine(bytes, i, end)
|
||||
i = end + 1
|
||||
}
|
||||
}
|
||||
for (key, entries) in exact where entries.count > 1 {
|
||||
exact[key] = entries.sorted { $0.score > $1.score }
|
||||
}
|
||||
tonelessIndex = nil // rebuilt on demand
|
||||
}
|
||||
|
||||
/// Splits one line into up to three space/tab separated fields.
|
||||
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
|
||||
var fields: [Range<Int>] = []
|
||||
var i = start
|
||||
while i < end && fields.count < 3 {
|
||||
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
|
||||
if i >= end { break }
|
||||
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
|
||||
let s = i
|
||||
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
|
||||
fields.append(s..<i)
|
||||
}
|
||||
guard fields.count >= 2 else { return }
|
||||
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
|
||||
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
|
||||
guard Lexicon.isBopomofo(b, r) else { return }
|
||||
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
|
||||
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
|
||||
var prefixNeutral = b[r.lowerBound] == 0xCB
|
||||
var k = r.lowerBound
|
||||
while !prefixNeutral && k + 2 < r.upperBound {
|
||||
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
|
||||
k += 1
|
||||
}
|
||||
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
|
||||
}
|
||||
|
||||
/// Parses "-3.9151"-style decimals without creating a String.
|
||||
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
|
||||
var i = r.lowerBound
|
||||
var sign = 1.0
|
||||
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
|
||||
var value = 0.0, scale = 1.0, digits = 0, fraction = false
|
||||
while i < r.upperBound {
|
||||
let c = b[i]
|
||||
if c >= 0x30 && c <= 0x39 {
|
||||
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
|
||||
digits += 1
|
||||
} else if c == 0x2E && !fraction {
|
||||
fraction = true
|
||||
} else {
|
||||
return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
return digits > 0 ? sign * value : nil
|
||||
}
|
||||
|
||||
/// Adds one entry (call `load` for bulk data; this does not re-sort).
|
||||
public func add(reading: String, phrase: String, score: Double) {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
|
||||
exact[key]?.sort { $0.score > $1.score }
|
||||
tonelessIndex = nil
|
||||
}
|
||||
|
||||
@discardableResult
|
||||
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
|
||||
var reading = rawReading
|
||||
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
|
||||
let syllables = rawReading.split(separator: "-").map { s -> String in
|
||||
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
|
||||
if normalise {
|
||||
reading = reading.split(separator: "-").map { s -> String in
|
||||
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
|
||||
}.joined(separator: "-")
|
||||
}
|
||||
let reading = syllables.joined(separator: "-")
|
||||
maxPhraseLength = max(maxPhraseLength, syllables.count)
|
||||
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) {
|
||||
exact[reading, default: []].append(entry)
|
||||
maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
|
||||
if exact[reading] == nil {
|
||||
exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
|
||||
} else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
|
||||
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
|
||||
}
|
||||
let key = Lexicon.stripTones(reading)
|
||||
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) {
|
||||
toneless[key]![i].score = max(toneless[key]![i].score, score)
|
||||
} else {
|
||||
toneless[key, default: []].append(entry)
|
||||
return reading
|
||||
}
|
||||
|
||||
/// Builds the index that ignores tones (for untoned syllables). Safe to call from a
|
||||
/// background queue; lookups build it themselves if it is not ready yet.
|
||||
public func prepareTonelessIndex() {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
_ = buildTonelessIfNeeded()
|
||||
}
|
||||
|
||||
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
|
||||
if let index = tonelessIndex { return index }
|
||||
var best: [String: [String: Double]] = [:]
|
||||
for (reading, entries) in exact {
|
||||
let key = Lexicon.stripTones(reading)
|
||||
for e in entries {
|
||||
if let s = best[key]?[e.phrase], s >= e.score { continue }
|
||||
best[key, default: [:]][e.phrase] = e.score
|
||||
}
|
||||
}
|
||||
var index: [String: [LexiconEntry]] = [:]
|
||||
index.reserveCapacity(best.count)
|
||||
for (key, phrases) in best {
|
||||
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
|
||||
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
|
||||
}
|
||||
tonelessIndex = index
|
||||
return index
|
||||
}
|
||||
|
||||
/// Entries for a sequence of readings. Untoned syllables match any tone.
|
||||
public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
if syllables.allSatisfy({ $0.tone != nil }) {
|
||||
return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
|
||||
}
|
||||
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? []
|
||||
return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
|
||||
}
|
||||
|
||||
public func bestScore(for syllable: Syllable) -> Double? {
|
||||
@@ -81,4 +182,13 @@ public final class Lexicon {
|
||||
guard let first = s.unicodeScalars.first else { return false }
|
||||
return (0x3105...0x312F).contains(first.value) || first == "˙"
|
||||
}
|
||||
|
||||
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
|
||||
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
|
||||
guard r.count >= 2 else { return false }
|
||||
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
|
||||
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
|
||||
let c = b[r.lowerBound + 2]
|
||||
return c >= 0x85 && c <= 0xAF
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
自動注音 AutoZhuyin — 第三方授權聲明 / Third-party notices
|
||||
==========================================================
|
||||
|
||||
本軟體包含或使用下列第三方資料。
|
||||
This software includes or uses the following third-party material.
|
||||
|
||||
|
||||
1. 小麥注音 McBopomofo — 詞庫資料 (lexicon data)
|
||||
------------------------------------------------
|
||||
https://github.com/openvanilla/McBopomofo
|
||||
|
||||
lexicon.txt is converted from McBopomofo's data files (BPMFBase.txt,
|
||||
BPMFMappings.txt, phrase.occ) by scripts/convert-mcbopomofo.py. No McBopomofo
|
||||
program code is included.
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2011-2026 Mengjuei Hsieh et al.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
|
||||
2. Webster's Second International Dictionary — 英文字典 (English word list)
|
||||
--------------------------------------------------------------------------
|
||||
english-dict.txt is filtered from /usr/share/dict/words (Webster's Second
|
||||
International Dictionary, 1934), which is in the public domain.
|
||||
|
||||
|
||||
3. PIME — Windows 輸入法框架 (input method framework)
|
||||
----------------------------------------------------
|
||||
https://github.com/EasyIME/PIME — GNU LGPL 2.1
|
||||
|
||||
The Windows version runs as a module inside PIME, which users install
|
||||
separately. No PIME code is included in this software.
|
||||
@@ -6,7 +6,10 @@ private let resources = URL(fileURLWithPath: #filePath)
|
||||
.appendingPathComponent("Resources")
|
||||
|
||||
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
|
||||
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"))
|
||||
private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
|
||||
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
|
||||
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
|
||||
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
|
||||
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
|
||||
|
||||
final class ParserTests: XCTestCase {
|
||||
@@ -366,7 +369,8 @@ final class CursorTests: XCTestCase {
|
||||
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||
var out = ""
|
||||
for c in s {
|
||||
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter : .character(c)
|
||||
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter
|
||||
: c == "\u{8}" ? .backspace : .character(c)
|
||||
let r = e.handle(key)
|
||||
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||
}
|
||||
@@ -380,7 +384,7 @@ final class CursorTests: XCTestCase {
|
||||
_ = type("←", into: e)
|
||||
XCTAssertEqual(e.cursorOffset, 3)
|
||||
_ = type("←←←←", into: e)
|
||||
XCTAssertEqual(e.cursorOffset, 1) // stops after the first character
|
||||
XCTAssertEqual(e.cursorOffset, 0) // stops at the start
|
||||
_ = type("→→→→", into: e)
|
||||
XCTAssertEqual(e.cursorOffset, 4)
|
||||
}
|
||||
@@ -396,12 +400,48 @@ final class CursorTests: XCTestCase {
|
||||
XCTAssertEqual(type("\n", into: e), "今添天氣")
|
||||
}
|
||||
|
||||
func testTypingReturnsCursorToEnd() {
|
||||
func testPunctuationCommitsFromTheMiddle() {
|
||||
let e = engine()
|
||||
_ = type("rup wu0 ←", into: e)
|
||||
XCTAssertEqual(e.cursorOffset, 1)
|
||||
_ = type("wu0 ", into: e)
|
||||
XCTAssertEqual(e.cursorOffset, e.markedText.count)
|
||||
_ = type("?", into: e)
|
||||
XCTAssertEqual(e.markedText, "")
|
||||
}
|
||||
|
||||
func testInsertSyllable() {
|
||||
let e = engine()
|
||||
_ = type("rup fu4←wu", into: e)
|
||||
XCTAssertEqual(e.markedText, "今ㄊㄧ氣")
|
||||
XCTAssertEqual(e.cursorOffset, 3)
|
||||
_ = type("0 ", into: e)
|
||||
XCTAssertEqual(e.markedText, "今天氣")
|
||||
XCTAssertEqual(e.cursorOffset, 2)
|
||||
XCTAssertEqual(type("\n", into: e), "今天氣")
|
||||
}
|
||||
|
||||
func testInsertAtStart() {
|
||||
XCTAssertEqual(type("su3cl3←←ji3\n", into: engine()), "我你好")
|
||||
}
|
||||
|
||||
func testBackspaceDeletesCharacterBeforeCursor() {
|
||||
let e = engine()
|
||||
_ = type("rup wu0 wu0 fu4←←\u{8}", into: e)
|
||||
XCTAssertEqual(e.markedText, "今天氣")
|
||||
XCTAssertEqual(e.cursorOffset, 1)
|
||||
}
|
||||
|
||||
func testBackspaceInsideInsertedSyllable() {
|
||||
let e = engine()
|
||||
_ = type("rup fu4←wu\u{8}", into: e)
|
||||
XCTAssertEqual(e.markedText, "今ㄊ氣")
|
||||
}
|
||||
|
||||
func testKeepsPickedPhraseBeforeCursor() {
|
||||
let e = engine()
|
||||
_ = type("rup wu0 ↓", into: e)
|
||||
e.selectCandidate(e.candidates.firstIndex { $0.text == "添" }!)
|
||||
_ = type("fu4←wu0 ", into: e) // insert 天 between 添 and 氣
|
||||
XCTAssertEqual(String(e.markedText.prefix(3)), "今添天")
|
||||
}
|
||||
|
||||
func testCandidatesForChineseBeforeEnglish() {
|
||||
@@ -424,3 +464,87 @@ final class FirstToneAfterEnglishTests: XCTestCase {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
final class HeldWordTests: XCTestCase {
|
||||
func makeEngine() -> CompositionEngine {
|
||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||
}
|
||||
|
||||
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||
var out = ""
|
||||
for c in s {
|
||||
let key: KeyInput = c == "\n" ? .enter : .character(c)
|
||||
let r = e.handle(key)
|
||||
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/// "up " is also ㄧㄣ (因): the next word decides.
|
||||
func testNextWordDecides() {
|
||||
let cases = ["up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
|
||||
"up\n": "up", "i love ": "i love "]
|
||||
for (keys, want) in cases {
|
||||
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
|
||||
}
|
||||
}
|
||||
|
||||
func testHeldWordStaysComposing() {
|
||||
let e = makeEngine()
|
||||
XCTAssertEqual(type("up ", into: e), "")
|
||||
XCTAssertEqual(e.markedText, "up ")
|
||||
_ = e.handle(.backspace)
|
||||
XCTAssertEqual(e.markedText, "up")
|
||||
XCTAssertEqual(type(" the ", into: e), "up the ")
|
||||
}
|
||||
}
|
||||
|
||||
final class BacktickSymbolTests: XCTestCase {
|
||||
func makeEngine() -> CompositionEngine {
|
||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||
}
|
||||
|
||||
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||
var out = ""
|
||||
for c in s {
|
||||
let key: KeyInput = c == "\n" ? .enter : .character(c)
|
||||
let r = e.handle(key)
|
||||
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func testForcedSymbols() {
|
||||
let cases = ["`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
|
||||
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"]
|
||||
for (keys, want) in cases {
|
||||
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
|
||||
}
|
||||
}
|
||||
|
||||
func testPlainBacktick() {
|
||||
let e = makeEngine()
|
||||
XCTAssertEqual(type("`", into: e), "")
|
||||
XCTAssertEqual(e.markedText, "`")
|
||||
XCTAssertEqual(type("ls` ", into: e), "`ls` ")
|
||||
}
|
||||
|
||||
func testBackspaceCancels() {
|
||||
let e = makeEngine()
|
||||
_ = type("su3`", into: e)
|
||||
_ = e.handle(.backspace)
|
||||
XCTAssertEqual(e.markedText, "你")
|
||||
}
|
||||
}
|
||||
|
||||
final class StartupTimingTests: XCTestCase {
|
||||
func testStartupTiming() throws {
|
||||
var t = Date()
|
||||
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
|
||||
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
|
||||
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
|
||||
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
|
||||
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
|
||||
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,6 +19,10 @@ mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
||||
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
|
||||
cp Resources/Info.plist "$APP/Contents/Info.plist"
|
||||
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
|
||||
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
|
||||
# Precomputed English dictionary and bigram table (shared with the Windows build).
|
||||
python3 -I windows/tools/build_data.py >/dev/null
|
||||
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
|
||||
|
||||
if [ ! -f build/icon.tiff ]; then
|
||||
xcrun swift scripts/make-icon.swift build/icon.tiff
|
||||
|
||||
@@ -26,6 +26,9 @@ rsync -a --exclude __pycache__ --exclude '*.pyc' windows/autozhuyin "$OUT/"
|
||||
for f in install.bat uninstall.bat README.txt; do
|
||||
sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f"
|
||||
done
|
||||
# Third-party notices: next to the installer and inside the installed folder.
|
||||
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
|
||||
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
|
||||
|
||||
(cd build && zip -qr -X "$NAME.zip" "$NAME")
|
||||
echo "Built build/$NAME.zip"
|
||||
|
||||
@@ -20,6 +20,19 @@ FULLWIDTH = {
|
||||
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
||||
}
|
||||
|
||||
# An English word that is also a common one-syllable Chinese reading ("up " = ㄧㄣ 因)
|
||||
# is not committed on space; the next word decides ("up jo4" = 因為, "up the" = up the).
|
||||
HOLD_MIN_SCORE = -3.5
|
||||
|
||||
# "`" then one of these keys always gives the full-width symbol, whatever the context.
|
||||
# "`" followed by anything else is just a backtick (so ``` still works).
|
||||
BACKTICK = "`"
|
||||
SYMBOLS = {
|
||||
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
|
||||
"'": "、", "\\": "、", '"': "「", "[": "「", "]": "」", "{": "『", "}": "』",
|
||||
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
|
||||
}
|
||||
|
||||
PASS = (False, None)
|
||||
CONSUMED = (True, None)
|
||||
|
||||
@@ -37,12 +50,18 @@ class CompositionEngine:
|
||||
# English mode, toggled by a Shift tap when nothing is being typed: keys go straight
|
||||
# to the app (no composition, no underline) until Shift is tapped again.
|
||||
self.english_mode = False
|
||||
self.backtick = False # "`" typed; the next key may be a forced symbol
|
||||
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
|
||||
self.candidates = []
|
||||
self.candidate_index = 0
|
||||
self._candidate_segment_start = 0
|
||||
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
|
||||
# Composition cursor: a key index into self.keys (None = at the end). ← → move it
|
||||
# between syllables; ↓ opens candidates for the phrase before it; typing there
|
||||
# inserts a syllable (kept in self.inserting until its tone key, then spliced in).
|
||||
self.cursor = None
|
||||
self.inserting = []
|
||||
# Index of a held space (see HOLD_MIN_SCORE), or None.
|
||||
self.held = None
|
||||
self._previous_mode = None
|
||||
self._previous_time = float("-inf")
|
||||
self._cache_key = None
|
||||
@@ -60,12 +79,35 @@ class CompositionEngine:
|
||||
if self.forced_mode:
|
||||
return [(0, len(keys), self.forced_mode)]
|
||||
ctx = self.context
|
||||
cache_key = (tuple(keys), finishing, ctx)
|
||||
held = self.held if self.held is not None and self.held < len(keys) and keys[self.held] == " " else None
|
||||
cache_key = (tuple(keys), finishing, ctx, held)
|
||||
if cache_key != self._cache_key:
|
||||
self._cache_key = cache_key
|
||||
self._cache_value = self.classifier.segment(keys, finishing, ctx)
|
||||
if held is None:
|
||||
self._cache_value = self.classifier.segment(keys, finishing, ctx)
|
||||
else:
|
||||
self._cache_value = self._held_segments(keys, held, finishing)
|
||||
return self._cache_value
|
||||
|
||||
def _held_segments(self, keys, held, finishing):
|
||||
"""The held word is Chinese if the words typed after it start with Chinese."""
|
||||
rest = keys[held + 1:]
|
||||
base = held + 1
|
||||
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
|
||||
if rest_segs and rest_segs[0][2] == ZHUYIN:
|
||||
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
|
||||
return [(0, base, LITERAL)] + rest_segs
|
||||
|
||||
def _is_ambiguous_word(self, keys):
|
||||
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
|
||||
if not all(k.isascii() and k.isalpha() for k in keys):
|
||||
return False
|
||||
p = bopomofo.parse(list(keys) + [" "])
|
||||
if p.pending is not None or len(p.syllables) != 1:
|
||||
return False
|
||||
entries = self.classifier.lexicon.entries([s.syllable for s in p.syllables])
|
||||
return bool(entries) and entries[0][1] >= HOLD_MIN_SCORE
|
||||
|
||||
@property
|
||||
def segments(self):
|
||||
return self._segments(self.keys)
|
||||
@@ -105,42 +147,177 @@ class CompositionEngine:
|
||||
|
||||
# --- cursor ---
|
||||
|
||||
def _cursor_stops(self):
|
||||
"""[(segment index, syllables before the stop)] for every Zhuyin syllable."""
|
||||
segs = self.segments
|
||||
out = []
|
||||
for i, seg in enumerate(segs):
|
||||
def _syllable_ends(self, seg):
|
||||
"""Absolute key index after each syllable of a Zhuyin segment."""
|
||||
return [seg[0] + ps.end for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables]
|
||||
|
||||
def _cursor_positions(self):
|
||||
"""Key indices the cursor can stop at: the start, after every Zhuyin syllable,
|
||||
and at every segment boundary."""
|
||||
out = {0, len(self.keys)}
|
||||
for seg in self.segments:
|
||||
out.add(seg[1])
|
||||
if seg[2] == ZHUYIN:
|
||||
out.update(self._syllable_ends(seg))
|
||||
return sorted(out)
|
||||
|
||||
def _stop_at(self, pos):
|
||||
"""(segment index, syllables before pos) for the Zhuyin segment pos is in."""
|
||||
segs = self.segments
|
||||
for i, seg in enumerate(segs):
|
||||
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
|
||||
n = len(self._convertible(seg, i < len(segs) - 1)[0])
|
||||
out.extend((i, k) for k in range(1, n + 1))
|
||||
return out
|
||||
local = sum(1 for e in self._syllable_ends(seg) if e <= pos)
|
||||
return (i, min(local, n)) if local else None
|
||||
return None
|
||||
|
||||
def _candidate_stop(self):
|
||||
"""Where ↓ opens candidates: before the cursor, or the last syllable."""
|
||||
if self.cursor is not None:
|
||||
return self._stop_at(self.cursor)
|
||||
segs = self.segments
|
||||
for i in range(len(segs) - 1, -1, -1):
|
||||
if segs[i][2] == ZHUYIN:
|
||||
n = len(self._convertible(segs[i], i < len(segs) - 1)[0])
|
||||
if n:
|
||||
return i, n
|
||||
return None
|
||||
|
||||
def _base_cursor_offset(self):
|
||||
"""Cursor position in the rendered segments (without the syllable being inserted)."""
|
||||
pieces = self._segment_pieces()
|
||||
if self.cursor is None:
|
||||
return sum(len(t) for t, _ in pieces)
|
||||
segs = self.segments
|
||||
for i, seg in enumerate(segs):
|
||||
if not seg[0] < self.cursor <= seg[1]:
|
||||
continue
|
||||
offset = sum(len(t) for t, _ in pieces[:i])
|
||||
if seg[2] == LITERAL:
|
||||
return offset + self.cursor - seg[0]
|
||||
stop = self._stop_at(self.cursor)
|
||||
local = stop[1] if stop else 0
|
||||
for (a, b), text in self._conversion(seg, i < len(segs) - 1)[0]:
|
||||
if b <= local:
|
||||
offset += len(text)
|
||||
elif a < local:
|
||||
offset += min(local - a, len(text))
|
||||
return offset
|
||||
return 0
|
||||
|
||||
@property
|
||||
def cursor_offset(self):
|
||||
"""Cursor position in marked_text (characters)."""
|
||||
pieces = self.marked_pieces
|
||||
stops = self._cursor_stops()
|
||||
if self.cursor is None or not 0 <= self.cursor < len(stops):
|
||||
return sum(len(t) for t, _ in pieces)
|
||||
seg_index, local = stops[self.cursor]
|
||||
offset = sum(len(t) for t, _ in pieces[:seg_index])
|
||||
segs = self.segments
|
||||
for (a, b), text in self._conversion(segs[seg_index], seg_index < len(segs) - 1)[0]:
|
||||
if b <= local:
|
||||
offset += len(text)
|
||||
elif a < local:
|
||||
offset += min(local - a, len(text))
|
||||
return offset
|
||||
return self._base_cursor_offset() + len(self._inserting_text())
|
||||
|
||||
def _move_cursor(self, delta):
|
||||
stops = self._cursor_stops()
|
||||
if not stops:
|
||||
return
|
||||
nxt = (len(stops) - 1 if self.cursor is None else self.cursor) + delta
|
||||
self.cursor = None if nxt >= len(stops) - 1 else max(0, nxt)
|
||||
self.inserting = []
|
||||
stops = self._cursor_positions()
|
||||
here = len(self.keys) if self.cursor is None else self.cursor
|
||||
if delta < 0:
|
||||
before = [p for p in stops if p < here]
|
||||
nxt = before[-1] if before else here
|
||||
else:
|
||||
after = [p for p in stops if p > here]
|
||||
nxt = after[0] if after else here
|
||||
self.cursor = None if nxt >= len(self.keys) else nxt
|
||||
|
||||
def _inserting_text(self):
|
||||
if not self.inserting:
|
||||
return ""
|
||||
return "".join(ps.syllable.reading for ps in bopomofo.parse(self.inserting).syllables)
|
||||
|
||||
# --- editing in the middle ---
|
||||
|
||||
def _trim_fixed(self, pos):
|
||||
"""Keeps picked phrases that end at or before key index pos (they don't move)."""
|
||||
kept = {}
|
||||
for seg in self.segments:
|
||||
picks = self.fixed.get(seg[0])
|
||||
if not picks or seg[2] != ZHUYIN:
|
||||
continue
|
||||
if seg[1] <= pos:
|
||||
kept[seg[0]] = picks
|
||||
elif seg[0] < pos:
|
||||
ends = self._syllable_ends(seg)
|
||||
here = {r: t for r, t in picks.items() if r[1] <= len(ends) and ends[r[1] - 1] <= pos}
|
||||
if here:
|
||||
kept[seg[0]] = here
|
||||
self.fixed = kept
|
||||
|
||||
def _splice(self, new_keys):
|
||||
"""Inserts keys at the cursor and moves the cursor after them."""
|
||||
pos = self.cursor
|
||||
self._trim_fixed(pos)
|
||||
self.keys[pos:pos] = new_keys
|
||||
if self.held is not None and self.held >= pos:
|
||||
self.held += len(new_keys)
|
||||
self.cursor = pos + len(new_keys)
|
||||
if self.cursor >= len(self.keys):
|
||||
self.cursor = None
|
||||
|
||||
def _delete_before_cursor(self):
|
||||
"""Backspace in the middle: removes the syllable (or literal key) before the cursor."""
|
||||
pos = self.cursor
|
||||
start = pos - 1
|
||||
for seg in self.segments:
|
||||
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
|
||||
for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables:
|
||||
if seg[0] + ps.end == pos:
|
||||
start = seg[0] + ps.start
|
||||
self._trim_fixed(start)
|
||||
del self.keys[start:pos]
|
||||
if self.held is not None:
|
||||
if start <= self.held < pos:
|
||||
self.held = None
|
||||
elif self.held >= pos:
|
||||
self.held -= pos - start
|
||||
self.cursor = start
|
||||
if not self.keys:
|
||||
self.reset()
|
||||
|
||||
def _insert_key(self, c):
|
||||
"""Typing with the cursor in the middle: compose one syllable, splice it in on its tone."""
|
||||
if c == " ":
|
||||
if not self.inserting:
|
||||
return True, self._commit()
|
||||
self._splice(self.inserting + [" "])
|
||||
self.inserting = []
|
||||
return CONSUMED
|
||||
self.inserting.append(c)
|
||||
p = bopomofo.parse(self.inserting)
|
||||
if p.stray_tones or p.foreign_keys:
|
||||
self.inserting.pop() # a tone with nothing to attach to
|
||||
return CONSUMED
|
||||
if p.syllables and p.syllables[-1].toned and len(p.syllables) == 1:
|
||||
self._splice(self.inserting)
|
||||
self.inserting = []
|
||||
elif len(p.syllables) > 1:
|
||||
# A new syllable started without a tone: splice the first one as it is.
|
||||
first = p.syllables[0].end
|
||||
self._splice(self.inserting[:first])
|
||||
self.inserting = self.inserting[first:]
|
||||
return CONSUMED
|
||||
|
||||
@property
|
||||
def marked_pieces(self):
|
||||
pieces = self._segment_pieces()
|
||||
if self.inserting:
|
||||
# Show the syllable being inserted at the cursor.
|
||||
at, out, done = self._base_cursor_offset(), [], False
|
||||
for text, mode in pieces:
|
||||
if not done and at <= len(text):
|
||||
out += [(text[:at], mode), (self._inserting_text(), ZHUYIN), (text[at:], mode)]
|
||||
done = True
|
||||
else:
|
||||
out.append((text, mode))
|
||||
at -= len(text)
|
||||
pieces = [(t, m) for t, m in out if t]
|
||||
if self.backtick:
|
||||
pieces.append((BACKTICK, LITERAL))
|
||||
return pieces
|
||||
|
||||
def _segment_pieces(self):
|
||||
segs = self.segments
|
||||
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
|
||||
|
||||
@@ -156,7 +333,7 @@ class CompositionEngine:
|
||||
|
||||
def wants(self, key):
|
||||
"""Whether handle(key) would consume the key, without changing state."""
|
||||
if self.candidates or key == SHIFT_TAP:
|
||||
if self.candidates or key == SHIFT_TAP or self.backtick or self.inserting:
|
||||
return True
|
||||
if self.english_mode and not self.keys:
|
||||
return False # English mode: everything goes to the app as typed
|
||||
@@ -173,11 +350,24 @@ class CompositionEngine:
|
||||
def handle(self, key):
|
||||
if self.candidates:
|
||||
return self._handle_candidate_key(key)
|
||||
if self.backtick:
|
||||
return self._after_backtick(key)
|
||||
if self.inserting and key in (BACKSPACE, ESCAPE):
|
||||
if key == BACKSPACE:
|
||||
self.inserting.pop()
|
||||
else:
|
||||
self.inserting = []
|
||||
return CONSUMED
|
||||
if key == BACKSPACE:
|
||||
if not self.keys:
|
||||
return PASS
|
||||
self.cursor = None
|
||||
if self.cursor is not None:
|
||||
if self.cursor > 0:
|
||||
self._delete_before_cursor()
|
||||
return CONSUMED
|
||||
self.keys.pop()
|
||||
if self.held is not None and len(self.keys) <= self.held:
|
||||
self.held = None
|
||||
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
|
||||
if not self.keys:
|
||||
self.forced_mode = None
|
||||
@@ -186,6 +376,9 @@ class CompositionEngine:
|
||||
if key == ENTER:
|
||||
if not self.keys:
|
||||
return PASS
|
||||
if self.inserting:
|
||||
self._splice(self.inserting)
|
||||
self.inserting = []
|
||||
return True, self._commit()
|
||||
if key == ESCAPE:
|
||||
if not self.keys:
|
||||
@@ -209,7 +402,8 @@ class CompositionEngine:
|
||||
if key == DOWN:
|
||||
if not self.keys:
|
||||
return PASS
|
||||
self._open_candidates()
|
||||
if not self.inserting:
|
||||
self._open_candidates()
|
||||
return CONSUMED
|
||||
if key in (LEFT, RIGHT):
|
||||
if not self.keys:
|
||||
@@ -220,6 +414,19 @@ class CompositionEngine:
|
||||
return CONSUMED if self.keys else PASS
|
||||
return self._handle_character(key)
|
||||
|
||||
def _after_backtick(self, key):
|
||||
self.backtick = False
|
||||
if key in (ESCAPE, BACKSPACE):
|
||||
return CONSUMED
|
||||
committed = self._commit() if self.keys else ""
|
||||
if key in SYMBOLS:
|
||||
return True, committed + SYMBOLS[key]
|
||||
# Not a symbol: the backtick was meant literally; then handle the key as usual.
|
||||
handled, more = self.handle(key)
|
||||
if not handled and len(key) == 1:
|
||||
handled, more = True, key
|
||||
return handled, committed + BACKTICK + (more or "")
|
||||
|
||||
def _last_parse(self, segs):
|
||||
if not segs or segs[-1][2] != ZHUYIN:
|
||||
return None
|
||||
@@ -227,14 +434,27 @@ class CompositionEngine:
|
||||
return bopomofo.parse(self.keys[start:end])
|
||||
|
||||
def _handle_character(self, c):
|
||||
self.cursor = None # typing always continues at the end
|
||||
if self.cursor is not None and self.keys and (c == " " or bopomofo.is_zhuyin_key(c)):
|
||||
return self._insert_key(c)
|
||||
if self.inserting:
|
||||
self._splice(self.inserting)
|
||||
self.inserting = []
|
||||
self.cursor = None # anything else continues at the end
|
||||
if self.english_mode and not self.keys:
|
||||
return PASS # English mode: every key goes to the app as typed
|
||||
if c == BACKTICK:
|
||||
self.backtick = True
|
||||
return CONSUMED
|
||||
if c == " ":
|
||||
if not self.keys:
|
||||
return PASS
|
||||
segs = self._segments(self.keys, finishing=True)
|
||||
if segs[-1][2] == LITERAL:
|
||||
if (self.held is None and len(segs) == 1 and not self.forced_mode
|
||||
and self._is_ambiguous_word(self.keys)):
|
||||
self.held = len(self.keys)
|
||||
self.keys.append(" ")
|
||||
return CONSUMED
|
||||
return True, self._commit(segs) + " "
|
||||
p = self._last_parse(segs)
|
||||
if p is not None and p.pending is not None:
|
||||
@@ -263,10 +483,10 @@ class CompositionEngine:
|
||||
|
||||
def _open_candidates(self):
|
||||
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
|
||||
stops = self._cursor_stops()
|
||||
if not stops:
|
||||
stop = self._candidate_stop()
|
||||
if stop is None:
|
||||
return
|
||||
seg_index, local = stops[min(len(stops) - 1 if self.cursor is None else self.cursor, len(stops) - 1)]
|
||||
seg_index, local = stop
|
||||
segs = self.segments
|
||||
seg = segs[seg_index]
|
||||
syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
|
||||
@@ -319,6 +539,9 @@ class CompositionEngine:
|
||||
|
||||
def reset(self):
|
||||
self.keys = []
|
||||
self.inserting = []
|
||||
self.held = None
|
||||
self.backtick = False
|
||||
self.cursor = None
|
||||
self.forced_mode = None
|
||||
self.fixed = {}
|
||||
|
||||
@@ -16,7 +16,7 @@ class Lexicon:
|
||||
|
||||
def __init__(self):
|
||||
self.exact = {}
|
||||
self.toneless = {}
|
||||
self._toneless = None # built on first use (only untoned syllables need it)
|
||||
self.max_phrase_length = 1
|
||||
|
||||
@classmethod
|
||||
@@ -42,9 +42,10 @@ class Lexicon:
|
||||
except ValueError:
|
||||
score = -6.0
|
||||
self.add(reading, phrase, score)
|
||||
for table in (self.exact, self.toneless):
|
||||
for v in table.values():
|
||||
for v in self.exact.values():
|
||||
if len(v) > 1:
|
||||
v.sort(key=lambda e: -e[1])
|
||||
self._toneless = None
|
||||
|
||||
def add(self, reading, phrase, score):
|
||||
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
|
||||
@@ -53,15 +54,20 @@ class Lexicon:
|
||||
entries = self.exact.setdefault(reading, [])
|
||||
if not any(p == phrase for p, _ in entries):
|
||||
entries.append((phrase, score))
|
||||
key = _strip_tones(reading)
|
||||
tl = self.toneless.setdefault(key, [])
|
||||
for i, (p, s) in enumerate(tl):
|
||||
if p == phrase:
|
||||
if score > s:
|
||||
tl[i] = (p, score)
|
||||
break
|
||||
else:
|
||||
tl.append((phrase, score))
|
||||
self._toneless = None
|
||||
|
||||
@property
|
||||
def toneless(self):
|
||||
"""Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
|
||||
if self._toneless is None:
|
||||
best = {}
|
||||
for reading, entries in self.exact.items():
|
||||
bucket = best.setdefault(_strip_tones(reading), {})
|
||||
for phrase, score in entries:
|
||||
if score > bucket.get(phrase, float("-inf")):
|
||||
bucket[phrase] = score
|
||||
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
|
||||
return self._toneless
|
||||
|
||||
def entries(self, syllables):
|
||||
if all(s.tone is not None for s in syllables):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "自動注音 AutoZhuyin",
|
||||
"version": "0.6.1",
|
||||
"version": "0.8.0",
|
||||
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
|
||||
"locale": "zh-Hant-TW",
|
||||
"fallbackLocale": "zh-TW",
|
||||
|
||||
@@ -26,7 +26,8 @@
|
||||
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
|
||||
Enter 送出目前組字
|
||||
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
|
||||
← → 在組字區移動游標,選要改哪個字
|
||||
← → 在組字區移動游標,選要改哪個字;游標在中間時直接打注音
|
||||
會插入新字,Backspace 刪掉游標前的字
|
||||
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字
|
||||
Esc 取消組字
|
||||
單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
|
||||
@@ -34,6 +35,9 @@
|
||||
打到一半時按,同 Tab 整段翻轉
|
||||
Shift+字母 打大寫字母(也會強制英文)
|
||||
< > ? ! : 中文語境下自動變成全形 ,。?!:
|
||||
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
|
||||
`' = 、 `[ `] = 「」 `( `) = () `! = !
|
||||
` 後面接其他鍵就是普通的反引號
|
||||
|
||||
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ def make_engine():
|
||||
def type_keys(s, e):
|
||||
out = ""
|
||||
for c in s:
|
||||
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN}.get(c, c)
|
||||
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN, "\b": BACKSPACE}.get(c, c)
|
||||
handled, commit = e.handle(key)
|
||||
if commit is not None:
|
||||
out += commit
|
||||
@@ -246,7 +246,7 @@ class CursorTests(unittest.TestCase):
|
||||
type_keys("←", e)
|
||||
self.assertEqual(e.cursor_offset, 3)
|
||||
type_keys("←←←←", e)
|
||||
self.assertEqual(e.cursor_offset, 1)
|
||||
self.assertEqual(e.cursor_offset, 0)
|
||||
type_keys("→→→→", e)
|
||||
self.assertEqual(e.cursor_offset, 4)
|
||||
|
||||
@@ -258,12 +258,45 @@ class CursorTests(unittest.TestCase):
|
||||
e.select_candidate([t for _, t in e.candidates].index("添"))
|
||||
self.assertEqual(type_keys("\n", e), "今添天氣")
|
||||
|
||||
def test_typing_returns_cursor_to_end(self):
|
||||
def test_english_returns_cursor_to_end(self):
|
||||
e = make_engine()
|
||||
type_keys("rup wu0 ←", e)
|
||||
self.assertEqual(e.cursor_offset, 1)
|
||||
type_keys("wu0 ", e)
|
||||
self.assertEqual(e.cursor_offset, len(e.marked_text))
|
||||
type_keys("?", e)
|
||||
self.assertEqual(e.marked_text, "")
|
||||
|
||||
|
||||
class InsertInMiddleTests(unittest.TestCase):
|
||||
def test_insert_syllable(self):
|
||||
e = make_engine()
|
||||
type_keys("rup fu4←wu", e)
|
||||
self.assertEqual(e.marked_text, "今ㄊㄧ氣")
|
||||
self.assertEqual(e.cursor_offset, 3)
|
||||
type_keys("0 ", e)
|
||||
self.assertEqual(e.marked_text, "今天氣")
|
||||
self.assertEqual(e.cursor_offset, 2)
|
||||
self.assertEqual(type_keys("\n", e), "今天氣")
|
||||
|
||||
def test_insert_at_start(self):
|
||||
self.assertEqual(type_keys("su3cl3←←ji3\n", make_engine()), "我你好")
|
||||
|
||||
def test_backspace_deletes_character_before_cursor(self):
|
||||
e = make_engine()
|
||||
type_keys("rup wu0 wu0 fu4←←\b", e)
|
||||
self.assertEqual(e.marked_text, "今天氣")
|
||||
self.assertEqual(e.cursor_offset, 1)
|
||||
|
||||
def test_backspace_inside_inserted_syllable(self):
|
||||
e = make_engine()
|
||||
type_keys("rup fu4←wu\b", e)
|
||||
self.assertEqual(e.marked_text, "今ㄊ氣")
|
||||
|
||||
def test_keeps_picked_phrase_before_cursor(self):
|
||||
e = make_engine()
|
||||
type_keys("rup wu0 ↓", e)
|
||||
e.select_candidate([t for _, t in e.candidates].index("添"))
|
||||
type_keys("fu4←wu0 ", e) # insert 天 between 添 and 氣
|
||||
self.assertEqual(e.marked_text[:3], "今添天")
|
||||
|
||||
def test_candidates_for_chinese_before_english(self):
|
||||
e = make_engine()
|
||||
@@ -279,5 +312,41 @@ class FirstToneAfterEnglishTests(unittest.TestCase):
|
||||
self.assertEqual(len(e.marked_text), 1, keys)
|
||||
|
||||
|
||||
class HeldWordTests(unittest.TestCase):
|
||||
def test_next_word_decides(self):
|
||||
cases = {"up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
|
||||
"up\n": "up", "i love ": "i love "}
|
||||
for keys, want in cases.items():
|
||||
self.assertEqual(type_keys(keys, make_engine()), want, keys)
|
||||
|
||||
def test_held_word_stays_composing(self):
|
||||
e = make_engine()
|
||||
self.assertEqual(type_keys("up ", e), "")
|
||||
self.assertEqual(e.marked_text, "up ")
|
||||
e.handle(BACKSPACE)
|
||||
self.assertEqual(e.marked_text, "up")
|
||||
self.assertEqual(type_keys(" the ", e), "up the ")
|
||||
|
||||
|
||||
class BacktickSymbolTests(unittest.TestCase):
|
||||
def test_forced_symbols(self):
|
||||
cases = {"`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
|
||||
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"}
|
||||
for keys, want in cases.items():
|
||||
self.assertEqual(type_keys(keys, make_engine()), want, keys)
|
||||
|
||||
def test_plain_backtick(self):
|
||||
e = make_engine()
|
||||
self.assertEqual(type_keys("`", e), "")
|
||||
self.assertEqual(e.marked_text, "`")
|
||||
self.assertEqual(type_keys("ls` ", e), "`ls` ")
|
||||
|
||||
def test_escape_cancels(self):
|
||||
e = make_engine()
|
||||
type_keys("su3`", e)
|
||||
e.handle(BACKSPACE)
|
||||
self.assertEqual(e.marked_text, "你")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user