Compare commits

...
12 Commits
Author SHA1 Message Date
ays953 c440cd7177 Release v0.13.0 2026-11-13 16:56:27 +08:00
ays953andClaude Opus 5.5 6443f771a0 Show English mode in the input method icon
macOS: the input method now has two input modes in Info.plist, 中英自動 ("注")
and 英文 ("A"), each with its own menu-bar icon. A Shift tap with nothing typed
switches the mode (TISSelectInputSource), and choosing a mode in the menu bar
switches the engine (setValue:forTag:). --register enables both modes.
Windows: the taskbar IME icon switches between icon.ico and icon-en.ico.
Linux: the IBus mode property also switches its icon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:56:08 +08:00
ays953 0155ddd16e Release v0.12.0 2026-11-13 16:52:01 +08:00
ays953andClaude Opus 5.5 c8b3c4ae6f AI candidate selection on Windows and Linux
Python port of the reranker: core/lm.py runs the same CKIP gpt2-tiny-chinese
model with numpy (batched per candidate length, scores match the Swift and
reference implementations), Converter.kbest, core/reranker.py (modes, time
budget, pause after overruns, cache), core/system_monitor.py (Windows: CPU usage
and battery saver; Linux: load average, thermal zones, low-power profile and low
battery) and core/ai.py (settings.json, background model loading, status line).

Windows: numpy wheels for PIME's 32-bit Python 3.8 (PIME 1.3) and 3.15 (PIME 2)
are bundled under autozhuyin/vendor and picked by Python version; a settings menu
on the taskbar IME icon and the language bar (left click on the icon toggles
English mode). Linux: install.sh uses the system numpy or downloads the matching
wheel from PyPI (tools/fetch_numpy.py, SHA-256 checked); an IBus settings menu.
Without numpy or the model the input method works as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:51:41 +08:00
ays953 38e0758a13 Release v0.11.0 2026-11-13 16:34:15 +08:00
ays953andClaude Opus 5.5 fd87da8061 AI candidate selection: rerank with a Traditional Chinese language model (macOS)
The converter's K best paths are rescored with CKIP gpt2-tiny-chinese (GPL-3.0,
character-level, ~12M parameters, 22 MB as float16):
score = lexicon log-prob + 0.6 x LM log-prob, with the recently committed text
as context. Inference is plain Accelerate (Apple Silicon and Intel).

Modes off / saver (K=3, 15 ms) / standard (K=10, 40 ms) / quality (K=30, 120 ms)
in the input menu. A rerank over its time budget falls back to the lexicon; three
in a row pause AI for a minute. With auto-pause on, AI pauses when the Mac is hot
or the CPU is busy and drops to saver in Low Power Mode. The model loads in the
background and is not loaded at all when off.

Evaluation (training/lm/eval_rerank.py): chat 96.6% -> 97.5% chars,
Wikipedia 92.9% -> 94.7% chars (about 25-30% fewer wrong characters).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:33:58 +08:00
ays953 acafd8f96e Release v0.10.1 2026-11-13 15:30:18 +08:00
ays953andClaude Opus 5.5 fa5b266ae4 Stop the cursor after the first Chinese character
Moving left no longer reaches the start of the buffer (or any position before the
first Zhuyin syllable), where there is nothing to pick candidates for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:30:03 +08:00
ays953 6448bf4ca2 Release v0.10.0 2026-11-13 14:31:10 +08:00
ays953andClaude Opus 5.5 2cebf0e700 Remember picked candidates (user history)
Candidates chosen with the arrow-down list are recorded on commit as
toneless reading -> phrase with a count. The converter lifts a picked phrase just
above the best lexicon entry for that reading (more for more picks), so it
comes first next time without overriding longer phrases. Up to 2000 pairs,
least recently used evicted. Stored as user-history.json on every platform; the
Mac menu's clear item clears it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:30:55 +08:00
ays953 815dcf979d Release v0.9.1 2026-11-13 12:06:24 +08:00
ays953andClaude Opus 5.5 8e82105af9 Hold the first word on space when it also reads as Zhuyin
At the start of a composition (not right after English), a word that reads as
valid Zhuyin is no longer committed as English on space. The next word decides,
by comparing "Chinese all the way" with "English word + Chinese" using the
classifier's features: el ji3 = 高我, no su3cl3 = no 你好, up the = up the.
After English text only common one-syllable readings (up, url) are held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:10 +08:00
44 changed files with 3293 additions and 102 deletions
+73 -7
View File
@@ -12,10 +12,12 @@
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示) - **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分 - **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對 - **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷 - **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷;英文模式時輸入法圖示會從「注」變成「A」
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小) - **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」 - **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來 - **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **記住選字**:從候選字選過的字詞會被記住,下次打同樣的讀音會優先出現(選越多次越優先)
- **AI 選字**:用中研院 CKIP 的繁中小型語言模型看前後文挑同音字(他跑「得」很快、請你「再」說一次),完全在本機運算;可調整或關閉,系統忙碌時自動暫停
- **中間補字、刪字**:游標移到前面後直接打注音,會插在游標處;Backspace 刪掉游標前的那個字 - **中間補字、刪字**:游標移到前面後直接打注音,會插在游標處;Backspace 刪掉游標前的那個字
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`) - **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權) - **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
@@ -32,6 +34,8 @@
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。 2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
3. 清單裡沒出現的話,登出再登入一次。 3. 清單裡沒出現的話,登出再登入一次。
從 0.12 以前的版本更新:輸入法現在分成「中英自動」和「英文」兩個模式(各有圖示),安裝時會自動加入。如果輸入法清單裡還留著舊的「自動注音」項目,把它移除即可。
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。 PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
### Windows ### Windows
@@ -62,10 +66,10 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 | | 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 | | Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 | | **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ← → | 在組字區移動游標(可移到最前面)。游標在中間時:直接打注音會**插入**新字,Backspace 刪掉游標前的字,↓ 選游標前的字 | | ← → | 在組字區移動游標(最左停在第一個中文字後面)。游標在中間時:直接打注音會**插入**新字,Backspace 刪掉游標前的字,↓ 選游標前的字 |
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 | | ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 | | Esc | 取消組字 |
| **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 | | **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷,圖示會變成「A」(Mac 的選單列、Windows 的工作列、Linux 的 IBus 狀態)。Mac 也可以直接在選單列選「英文」模式。打到一半時:同 Tab,整段翻轉 |
| Shift + 字母 | 打大寫字母(也會強制英文) | | Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` | | `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) | | `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
@@ -95,15 +99,59 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。 轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
### AI 選字
詞庫只知道每個詞本身有多常見,不看前後文。AI 選字讓語言模型當裁判:
1. 詞庫先找出前 K 名的整句組合(例如「請你在說一次」「請你再說一次」)。
2. 模型 [CKIP GPT2-tiny](https://huggingface.co/ckiplab/gpt2-tiny-chinese)(中研院詞庫小組,繁體中文,約 1200 萬參數,22 MB)計算每個組合出現的機率,並參考剛送出的前文。
3. 分數 = 詞庫分數 + 0.6 × 模型分數,取最高的。選過的候選字仍然固定不變。
輸入法選單可以調整(Mac:選單列的輸入法選單;Windows:工作列輸入法圖示按右鍵或語言列的設定按鈕;Linux:IBus 選單的「設定」):
| 模式 | 比較幾個組合 | 每次運算上限 |
|---|---|---|
| 關閉 | — | 不載入模型、不佔記憶體 |
| 省電 | 3 | 15 毫秒 |
| 標準(預設) | 10 | 40 毫秒 |
| 高品質 | 30 | 120 毫秒 |
不會拖慢打字的機制:
- 每次運算超過上限就直接放棄,改用詞庫結果;連續 3 次超時就暫停一分鐘。
- 「系統忙碌時自動暫停 AI」(預設開啟),選單裡會顯示目前狀態:
| | 暫停 AI | 降到省電模式 |
|---|---|---|
| macOS | 過熱,或 CPU 負載(每核心 1 分鐘平均)超過 0.9 | 低耗電模式 |
| Windows | CPU 使用率超過 90% | 省電模式(電池保護) |
| Linux | CPU 負載超過 0.9,或溫度達 95 °C | 低耗電設定檔,或用電池且電量 ≤ 20% |
- 模型在背景載入,載入完成前照常用詞庫。
實測(`training/lm/eval_rerank.py`,標準模式):
| 測試資料 | 只用詞庫 | AI 選字 |
|---|---|---|
| 日常對話 60 句 | 字正確率 96.6%,整句全對 73% | 97.5%,80% |
| 維基百科 714 句 | 92.9%,31% | 94.7%,44% |
錯字大約少了 25–30%。
Windows、Linux 版用 numpy 執行同一個模型(`windows/autozhuyin/core/lm.py`),所以時間上限放寬為省電 40、標準 80、高品質 200 毫秒,超時的那次結果照用但不快取,連續 3 次超時暫停一分鐘。
- Windows:安裝包附上 PIME 兩種 Python(1.3 版的 3.8、2.x 版的 3.15,都是 32 位元)用的 numpy,安裝檔約 40 MB。
- Linux:系統沒有 numpy 時,`install.sh` 會從 PyPI 下載符合的版本到 `~/.local/share/autozhuyin/vendor/`。
- 設定存在 `%APPDATA%\PIME\autozhuyin\settings.json`、`~/.config/autozhuyin/settings.json`。
## 個人資料 ## 個人資料
| | macOS | Windows | Linux | | | macOS | Windows | Linux |
|---|---|---|---| |---|---|---|---|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | `~/.config/autozhuyin/learned.json` | | 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | `~/.config/autozhuyin/learned.json` |
| 選字紀錄 | `~/Library/Application Support/AutoZhuyin/user-history.json` | `%APPDATA%\PIME\autozhuyin\user-history.json` | `~/.config/autozhuyin/user-history.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | `~/.config/autozhuyin/user-phrases.txt` | | 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | `~/.config/autozhuyin/user-phrases.txt` |
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。 自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
Mac 版可以在輸入法選單選「清除學習紀錄」。 選字紀錄最多保留 2000 筆,太久沒用的會先淘汰。被選過的字只會排到同讀音第一順位,不會蓋過整個詞組(例如選過「添」之後,ㄊㄧㄢ 單打會出「添」,但「今天」還是「今天」)。
Mac 版可以在輸入法選單選「清除學習紀錄」(中英判斷和選字紀錄一起清除);Windows、Linux 刪除上面的檔案即可。
## 開發 ## 開發
@@ -187,6 +235,22 @@ python3 training/make_parity.py # 權重或特徵改了之後都要重
scripts/import-mcbopomofo.sh scripts/import-mcbopomofo.sh
``` ```
### AI 選字模型
```bash
scripts/fetch-lm.sh # 下載 CKIP gpt2-tiny-chinese,轉成 build/lm/lm.bin(float16)+ lm-vocab.txt
```
`build.sh` 會自動執行並把模型打包進 app。模型推論寫在 `Sources/AutoZhuyinCore/LanguageModel.swift`(Accelerate,Apple Silicon 和 Intel 都可用),重排在 `Reranker.swift`。
評估效果(需要 numpy;`build/lm-eval/` 放測試句子,一行一句):
```bash
python training/lm/eval_rerank.py build/lm-eval/chat.txt build/lm-eval/wiki.txt
```
`training/lm/gpt2.py` 是 numpy 版的參考實作,Swift 版的測試會比對它算出的分數。
### 發布新版本 ### 發布新版本
先 commit 所有修改,再執行: 先 commit 所有修改,再執行:
@@ -247,7 +311,7 @@ THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝
- Windows 版、Linux 版還沒在實機上測試過。 - Windows 版、Linux 版還沒在實機上測試過。
- Linux 版只支援 IBus。KDE 上有些 Flatpak 程式(例如 Flatpak 版 Chrome)在 IBus Wayland 下可能打不出字。 - Linux 版只支援 IBus。KDE 上有些 Flatpak 程式(例如 Flatpak 版 Chrome)在 IBus Wayland 下可能打不出字。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。 - 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串本質上就有歧義。像 `up`(也是 ㄧㄣ 因)這種常用的,按空白後會先保留,看下一個詞再決定(`up jo4` = 因為、`up the` = up the);其他判斷錯的可以按 Tab 修正,修正後會記住。 - 很短的字串本質上就有歧義。開頭第一個詞如果也能讀成注音,按空白後會先保留(顯示成英文),看下一個詞再決定:`up jo4` = 因為、`el ji3` = 高我、`no su3cl3` = no 你好、`up the` = up the。接在英文後面時只保留 `up`(因)這種很常見的讀音,不會拖慢打英文。還是判斷錯的話按 Tab 修正,修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。 - 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap) ## 下一步(Roadmap)
@@ -260,8 +324,8 @@ THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝
- [x] 用資料訓練判斷權重(合成資料) - [x] 用資料訓練判斷權重(合成資料)
- [ ] 加入真實打字資料一起訓練 - [ ] 加入真實打字資料一起訓練
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」 - [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境 - [x] AI 選字:語言模型參考前後文挑同音字(macOS、Windows、Linux)
- [ ] 使用者詞頻學習(選過的候選字優先) - [x] 使用者詞頻學習(選過的候選字優先)
- [ ] 偏好設定視窗 - [ ] 偏好設定視窗
## 致謝與授權 ## 致謝與授權
@@ -269,5 +333,7 @@ THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License) - 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼) - Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`) - 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
- AI 選字語言模型:[CKIP GPT2 Tiny Chinese](https://github.com/ckiplab/ckip-transformers)(中研院詞庫小組,GPL-3.0,全文在 `licenses/GPL-3.0.txt`,並一起打包進三個平台的安裝檔)
- 數值運算:[NumPy](https://numpy.org)(BSD 3-Clause,Windows 版附上、Linux 版安裝時下載)
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。 完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
+63 -2
View File
@@ -15,9 +15,70 @@
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.9.0</string> <string>0.13.0</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>15</string> <string>21</string>
<key>ComponentInputModeDict</key>
<dict>
<key>tsInputModeListKey</key>
<dict>
<key>com.autozhuyin.inputmethod.AutoZhuyin.Auto</key>
<dict>
<key>TISInputSourceID</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin.Auto</string>
<key>TISIntendedLanguage</key>
<string>zh-Hant</string>
<key>tsInputModeAlternateMenuIconFileKey</key>
<string>icon.tiff</string>
<key>tsInputModeCharacterRepertoireKey</key>
<array>
<string>Hant</string>
</array>
<key>tsInputModeDefaultStateKey</key>
<true/>
<key>tsInputModeIsVisibleKey</key>
<true/>
<key>tsInputModeMenuIconFileKey</key>
<string>icon.tiff</string>
<key>tsInputModePaletteIconFileKey</key>
<string>icon.tiff</string>
<key>tsInputModePrimaryInScriptKey</key>
<true/>
<key>tsInputModeScriptKey</key>
<string>smTradChinese</string>
</dict>
<key>com.autozhuyin.inputmethod.AutoZhuyin.English</key>
<dict>
<key>TISInputSourceID</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin.English</string>
<key>TISIntendedLanguage</key>
<string>zh-Hant</string>
<key>tsInputModeAlternateMenuIconFileKey</key>
<string>icon-en.tiff</string>
<key>tsInputModeCharacterRepertoireKey</key>
<array>
<string>Hant</string>
</array>
<key>tsInputModeDefaultStateKey</key>
<true/>
<key>tsInputModeIsVisibleKey</key>
<true/>
<key>tsInputModeMenuIconFileKey</key>
<string>icon-en.tiff</string>
<key>tsInputModePaletteIconFileKey</key>
<string>icon-en.tiff</string>
<key>tsInputModePrimaryInScriptKey</key>
<false/>
<key>tsInputModeScriptKey</key>
<string>smTradChinese</string>
</dict>
</dict>
<key>tsVisibleInputModeOrderedArrayKey</key>
<array>
<string>com.autozhuyin.inputmethod.AutoZhuyin.Auto</string>
<string>com.autozhuyin.inputmethod.AutoZhuyin.English</string>
</array>
</dict>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
+97
View File
@@ -0,0 +1,97 @@
import Foundation
import AutoZhuyinCore
/// "AI 選字" settings, model loading and system-load detection for the Mac app.
enum AI {
static let modeKey = "AIMode"
static let autoPauseKey = "AIAutoPause"
static let reranker: Reranker = {
let saved = UserDefaults.standard.string(forKey: modeKey).flatMap(Reranker.Mode.init(rawValue:))
let r = Reranker(mode: saved ?? .standard)
r.systemState = { SystemMonitor.shared.state() }
return r
}()
/// The model file is missing from the app bundle (built without scripts/fetch-lm.sh).
private(set) static var modelMissing = false
private static var loading = false
static var autoPause: Bool {
get { UserDefaults.standard.object(forKey: autoPauseKey) as? Bool ?? true }
set { UserDefaults.standard.set(newValue, forKey: autoPauseKey); SystemMonitor.shared.invalidate() }
}
static func setMode(_ mode: Reranker.Mode) {
reranker.mode = mode
UserDefaults.standard.set(mode.rawValue, forKey: modeKey)
if mode == .off {
reranker.model = nil // free the memory
} else {
loadModelIfNeeded()
}
}
/// Loads the model in the background (only when AI is on).
static func loadModelIfNeeded() {
guard reranker.mode != .off, reranker.model == nil, !loading, !modelMissing else { return }
guard let bin = Bundle.main.url(forResource: "lm", withExtension: "bin"),
let vocab = Bundle.main.url(forResource: "lm-vocab", withExtension: "txt") else {
modelMissing = true
return
}
loading = true
DispatchQueue.global(qos: .utility).async {
let start = Date()
let model = try? GPT2LanguageModel(modelURL: bin, vocabURL: vocab)
NSLog("AutoZhuyin: language model %@ in %.0f ms", model == nil ? "failed" : "loaded",
Date().timeIntervalSince(start) * 1000)
DispatchQueue.main.async {
loading = false
if model == nil { modelMissing = true }
if reranker.mode != .off { reranker.model = model }
}
}
}
static var statusText: String {
if modelMissing { return "AI 模型未安裝" }
switch reranker.status {
case .off: return "AI 選字已關閉"
case .loading: return "AI 模型載入中…"
case .active:
return SystemMonitor.shared.state() == .reduced ? "AI 使用中(省電:低耗電模式)" : "AI 使用中"
case .pausedBusy: return "AI 已暫停:系統忙碌或過熱"
case .pausedSlow: return "AI 已暫停:運算太慢,一分鐘後再試"
}
}
}
/// Decides when AI should step back: paused when the Mac is hot or the CPU is busy,
/// reduced (fewest candidates) in Low Power Mode. Checked at most every 2 seconds.
final class SystemMonitor {
static let shared = SystemMonitor()
private var cached: (time: Date, state: Reranker.SystemState) = (.distantPast, .normal)
/// CPU load (1-minute average per core) above which AI pauses.
var busyLoad = 0.9
func invalidate() { cached.time = .distantPast }
func state() -> Reranker.SystemState {
if Date().timeIntervalSince(cached.time) < 2 { return cached.state }
cached = (Date(), compute())
return cached.state
}
private func compute() -> Reranker.SystemState {
guard AI.autoPause else { return .normal }
let info = ProcessInfo.processInfo
if info.thermalState == .serious || info.thermalState == .critical { return .paused }
var load = [Double](repeating: 0, count: 3)
if getloadavg(&load, 3) > 0, load[0] / Double(max(1, info.activeProcessorCount)) > busyLoad {
return .paused
}
if info.isLowPowerModeEnabled { return .reduced }
return .normal
}
}
+58 -5
View File
@@ -1,3 +1,4 @@
import Carbon
import Cocoa import Cocoa
import InputMethodKit import InputMethodKit
import AutoZhuyinCore import AutoZhuyinCore
@@ -9,6 +10,11 @@ enum Shared {
static let learnedKey = "LearnedPreferences" static let learnedKey = "LearnedPreferences"
/// ~/Library/Application Support/AutoZhuyin (user phrases, picked-candidate history).
static let supportDir: URL? = FileManager.default
.urls(for: .applicationSupportDirectory, in: .userDomainMask).first?
.appendingPathComponent("AutoZhuyin")
static let classifier: Classifier = { static let classifier: Classifier = {
let bundle = Bundle.main let bundle = Bundle.main
let start = Date() let start = Date()
@@ -18,9 +24,9 @@ enum Shared {
lexicon.load(data: data) lexicon.load(data: data)
} }
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt // Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first { if let userFile = supportDir?.appendingPathComponent("user-phrases.txt"),
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt") let text = try? String(contentsOf: userFile, encoding: .utf8) {
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) } lexicon.load(text)
} }
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"), let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"), dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
@@ -29,17 +35,27 @@ enum Shared {
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
let historyFile = supportDir?.appendingPathComponent("user-history.json")
let history = UserHistory(data: historyFile.map(UserHistory.load(from:)) ?? [:]) { data in
guard let url = historyFile, let json = try? JSONEncoder().encode(data) else { return }
try? FileManager.default.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
try? json.write(to: url, options: .atomic)
}
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json")) let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000) NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path. // Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() } DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained) return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained, history: history)
}() }()
} }
@objc(AutoZhuyinInputController) @objc(AutoZhuyinInputController)
final class InputController: IMKInputController { final class InputController: IMKInputController {
private let engine = CompositionEngine(classifier: Shared.classifier) private lazy var engine: CompositionEngine = {
let e = CompositionEngine(classifier: Shared.classifier)
e.reranker = AI.reranker
return e
}()
private var shiftTap = ShiftTapDetector() private var shiftTap = ShiftTapDetector()
// MARK: - Key events // MARK: - Key events
@@ -59,8 +75,11 @@ final class InputController: IMKInputController {
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty, otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
time: event.timestamp) time: event.timestamp)
if tapped { if tapped {
let idle = engine.keys.isEmpty
_ = engine.handle(.shiftTap) _ = engine.handle(.shiftTap)
updateComposition(client) updateComposition(client)
// English mode on/off: switch input mode so the menu-bar icon shows it.
if idle { InputModes.select(english: engine.isEnglishMode) }
} }
return false return false
} }
@@ -137,6 +156,18 @@ final class InputController: IMKInputController {
} }
} }
/// The input mode changed (menu bar, keyboard shortcut, or our own Shift tap).
override func setValue(_ value: Any!, forTag tag: Int, client sender: Any!) {
if tag == kTextServiceInputModePropertyTag, let id = value as? String {
let english = InputModes.isEnglish(id)
if english != engine.isEnglishMode {
commitComposition(sender)
engine.setEnglishMode(english)
}
}
super.setValue(value, forTag: tag, client: sender)
}
override func commitComposition(_ sender: Any!) { override func commitComposition(_ sender: Any!) {
guard let client = sender as? IMKTextInput else { return } guard let client = sender as? IMKTextInput else { return }
commit(engine.commitAll(), to: client) commit(engine.commitAll(), to: client)
@@ -171,11 +202,33 @@ final class InputController: IMKInputController {
override func menu() -> NSMenu! { override func menu() -> NSMenu! {
let menu = NSMenu() let menu = NSMenu()
let modes: [(Reranker.Mode, String, Selector)] = [
(.off, "AI 選字:關閉", #selector(aiOff(_:))),
(.saver, "AI 選字:省電", #selector(aiSaver(_:))),
(.standard, "AI 選字:標準", #selector(aiStandard(_:))),
(.quality, "AI 選字:高品質", #selector(aiQuality(_:))),
]
for (mode, title, action) in modes {
let item = menu.addItem(withTitle: title, action: action, keyEquivalent: "")
item.state = AI.reranker.mode == mode ? .on : .off
}
let pause = menu.addItem(withTitle: "系統忙碌時自動暫停 AI", action: #selector(toggleAutoPause(_:)), keyEquivalent: "")
pause.state = AI.autoPause ? .on : .off
let status = menu.addItem(withTitle: AI.statusText, action: nil, keyEquivalent: "")
status.isEnabled = false
menu.addItem(.separator())
menu.addItem(withTitle: "清除學習紀錄", action: #selector(clearLearned(_:)), keyEquivalent: "") menu.addItem(withTitle: "清除學習紀錄", action: #selector(clearLearned(_:)), keyEquivalent: "")
return menu return menu
} }
@objc private func aiOff(_ sender: Any?) { AI.setMode(.off) }
@objc private func aiSaver(_ sender: Any?) { AI.setMode(.saver) }
@objc private func aiStandard(_ sender: Any?) { AI.setMode(.standard) }
@objc private func aiQuality(_ sender: Any?) { AI.setMode(.quality) }
@objc private func toggleAutoPause(_ sender: Any?) { AI.autoPause.toggle() }
@objc private func clearLearned(_ sender: Any?) { @objc private func clearLearned(_ sender: Any?) {
Shared.classifier.learned.clear() Shared.classifier.learned.clear()
Shared.classifier.history.clear()
} }
} }
+32
View File
@@ -0,0 +1,32 @@
import Carbon
import Foundation
/// The two input modes declared in Info.plist (ComponentInputModeDict). Each has its own
/// menu-bar icon, so switching modes is how the icon shows "中英自動" vs "英文".
enum InputModes {
static let auto = "com.autozhuyin.inputmethod.AutoZhuyin.Auto"
static let english = "com.autozhuyin.inputmethod.AutoZhuyin.English"
static func isEnglish(_ modeID: String) -> Bool { modeID == english }
private static func source(_ id: String) -> TISInputSource? {
let filter = [kTISPropertyInputSourceID as String: id] as CFDictionary
guard let list = TISCreateInputSourceList(filter, true)?.takeRetainedValue() as? [TISInputSource] else { return nil }
return list.first
}
/// Makes the given mode current (enabling it first if needed). Returns false on failure.
@discardableResult
static func select(english: Bool) -> Bool {
guard let src = source(english ? InputModes.english : InputModes.auto) else { return false }
TISEnableInputSource(src)
return TISSelectInputSource(src) == noErr
}
/// Called by `--register`: enable both modes so they are in the input menu.
static func enableAll() {
for id in [auto, english] {
if let src = source(id) { TISEnableInputSource(src) }
}
}
}
+2
View File
@@ -6,6 +6,7 @@ import InputMethodKit
// in System Settings without logging out (used by the installer's postinstall script). // in System Settings without logging out (used by the installer's postinstall script).
if CommandLine.arguments.contains("--register") { if CommandLine.arguments.contains("--register") {
let status = TISRegisterInputSource(Bundle.main.bundleURL as CFURL) let status = TISRegisterInputSource(Bundle.main.bundleURL as CFURL)
if status == noErr { InputModes.enableAll() } // "中英自動" and "英文" (menu-bar icons)
print(status == noErr ? "Registered \(Bundle.main.bundlePath)" : "TISRegisterInputSource failed: \(status)") print(status == noErr ? "Registered \(Bundle.main.bundlePath)" : "TISRegisterInputSource failed: \(status)")
exit(status == noErr ? 0 : 1) exit(status == noErr ? 0 : 1)
} }
@@ -22,5 +23,6 @@ else {
Shared.server = server Shared.server = server
Shared.candidates = IMKCandidates(server: server, panelType: kIMKSingleColumnScrollingCandidatePanel) Shared.candidates = IMKCandidates(server: server, panelType: kIMKSingleColumnScrollingCandidatePanel)
_ = Shared.classifier // load the lexicon now rather than on the first keystroke _ = Shared.classifier // load the lexicon now rather than on the first keystroke
AI.loadModelIfNeeded() // "AI 選字" language model, in the background
NSApplication.shared.run() NSApplication.shared.run()
+4 -1
View File
@@ -55,14 +55,17 @@ public final class Classifier {
public let lexicon: Lexicon public let lexicon: Lexicon
public let english: EnglishModel public let english: EnglishModel
public let learned: LearnedPreferences public let learned: LearnedPreferences
/// Picked candidates (shared by every engine using this classifier).
public let history: UserHistory
/// Feature weights: the hand-tuned defaults overlaid with trained ones. /// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector public private(set) var weights: FeatureVector
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(), public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) { weights: FeatureVector = [:], history: UserHistory = UserHistory()) {
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned self.learned = learned
self.history = history
self.weights = Features.defaultWeights.merging(weights) { _, new in new } self.weights = Features.defaultWeights.merging(weights) { _, new in new }
} }
+72 -13
View File
@@ -53,15 +53,21 @@ public final class CompositionEngine {
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
/// Index of a held space: an English word that is also a common one-syllable Chinese /// Index of a held space: the first word of a composition that also reads as Zhuyin is
/// reading ("up " = ㄧㄣ 因) is not committed on space; the next word decides /// not committed as English on space; the next word decides ("up jo4" = 因為,
/// ("up jo4" = 因為, "up the" = up the). /// "up the" = up the, "no su3" = no 你). Right after English text only common
/// one-syllable readings are held (up = 因, url = 教), so English typing isn't delayed.
private var held: Int? private var held: Int?
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol /// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
/// whatever the context; anything else makes it a plain backtick (so ``` still works). /// whatever the context; anything else makes it a plain backtick (so ``` still works).
public private(set) var pendingBacktick = false public private(set) var pendingBacktick = false
static let holdMinScore = -3.5 static let holdMinScore = -3.5
/// "AI 選字" (language-model reranking); nil = lexicon only.
public var reranker: Reranker?
/// The last characters committed, used as context for the reranker.
private var recentText = ""
private var previousMode: InputMode? private var previousMode: InputMode?
private var previousCommitTime = Date.distantPast private var previousCommitTime = Date.distantPast
/// How long the previous segment's mode keeps influencing the next one. /// How long the previous segment's mode keeps influencing the next one.
@@ -70,7 +76,7 @@ public final class CompositionEngine {
public init(classifier: Classifier) { public init(classifier: Classifier) {
self.classifier = classifier self.classifier = classifier
self.converter = Converter(lexicon: classifier.lexicon) self.converter = Converter(lexicon: classifier.lexicon, history: classifier.history)
} }
// MARK: - Derived state // MARK: - Derived state
@@ -94,19 +100,38 @@ public final class CompositionEngine {
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map { let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode) Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
} }
if let first = rest.first, first.mode == .zhuyin { if let first = rest.first, first.mode == .zhuyin,
heldReadsAsChinese(keys, held: h, end: first.range.upperBound, finishing: finishing) {
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst() return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
} }
return [Segment(range: 0..<base, mode: .literal)] + rest return [Segment(range: 0..<base, mode: .literal)] + rest
} }
/// Chinese all the way to `end` vs. the held English word followed by Chinese.
private func heldReadsAsChinese(_ keys: [Character], held h: Int, end: Int, finishing: Bool) -> Bool {
let c = classifier, w = classifier.weights, ctx = context
let isLast = end == keys.count
guard let zhWhole = c.segmentFeatures(Array(keys[..<end]), isLast: isLast, afterOther: false,
finishing: finishing, previous: ctx).0 else { return false }
guard let word = c.segmentFeatures(Array(keys[..<h]), isLast: false, afterOther: false,
finishing: true, previous: ctx).1,
let zhRest = c.segmentFeatures(Array(keys[(h + 1)..<end]), isLast: isLast, afterOther: true,
finishing: finishing, previous: nil).0 else { return true }
return Features.dot(w, zhWhole) >= Features.dot(w, word) + Features.dot(w, zhRest)
}
/// Whether keys + space also read as a common Chinese syllable (see `held`). /// Whether keys + space also read as a common Chinese syllable (see `held`).
private func isAmbiguousWord(_ keys: [Character]) -> Bool { private func isAmbiguousWord(_ keys: [Character]) -> Bool {
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false } guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
let p = ZhuyinParser.parse(keys + [" "]) let p = ZhuyinParser.parse(keys + [" "])
guard p.pending == nil, p.syllables.count == 1 else { return false } guard p.pending == nil, !p.syllables.isEmpty, p.strayTones == 0, p.foreignKeys == 0 else { return false }
guard let best = classifier.lexicon.entries(for: p.syllables.map(\.syllable)).first else { return false } var best: [Double] = []
return best.score >= CompositionEngine.holdMinScore for ps in p.syllables {
guard let e = classifier.lexicon.entries(for: [ps.syllable]).first else { return false }
best.append(e.score)
}
if context != .literal { return true } // first word: hold anything that reads as Zhuyin
return best.count == 1 && best[0] >= CompositionEngine.holdMinScore
} }
/// Current split of the buffer into Zhuyin / literal runs. /// Current split of the buffer into Zhuyin / literal runs.
@@ -166,9 +191,16 @@ public final class CompositionEngine {
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) { private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
let (syl, pending) = convertibleSyllables(seg, final: final) let (syl, pending) = convertibleSyllables(seg, final: final)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count } let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
if let reranker, let chosen = reranker.choose(converter, syl, fixed: fixedHere, context: rerankContext) {
return (chosen, pending)
}
return (converter.convert(syl, fixed: fixedHere), pending) return (converter.convert(syl, fixed: fixedHere), pending)
} }
private var rerankContext: String {
now().timeIntervalSince(previousCommitTime) < contextWindow ? recentText : ""
}
// MARK: - Cursor // MARK: - Cursor
/// Absolute key index after each syllable of a Zhuyin segment. /// Absolute key index after each syllable of a Zhuyin segment.
@@ -176,15 +208,19 @@ public final class CompositionEngine {
ZhuyinParser.parse(Array(keys[seg.range])).syllables.map { seg.range.lowerBound + $0.keyRange.upperBound } ZhuyinParser.parse(Array(keys[seg.range])).syllables.map { seg.range.lowerBound + $0.keyRange.upperBound }
} }
/// Key indices the cursor can stop at: the start, after every Zhuyin syllable, and at /// Key indices the cursor can stop at: after every Zhuyin syllable and at every segment
/// every segment boundary. /// boundary, but never before the first Chinese character (there would be nothing to
/// pick candidates for).
private var cursorPositions: [Int] { private var cursorPositions: [Int] {
var out: Set<Int> = [0, keys.count] var out: Set<Int> = [keys.count]
var ends: [Int] = []
for seg in segments { for seg in segments {
out.insert(seg.range.upperBound) out.insert(seg.range.upperBound)
if seg.mode == .zhuyin { out.formUnion(syllableEnds(seg)) } if seg.mode == .zhuyin { ends += syllableEnds(seg) }
} }
return out.sorted() out.formUnion(ends)
let first = ends.min() ?? keys.count
return out.filter { $0 >= first }.sorted()
} }
/// A place to open candidates: after the `local`-th syllable of Zhuyin segment `segIndex`. /// A place to open candidates: after the `local`-th syllable of Zhuyin segment `segIndex`.
@@ -381,6 +417,16 @@ public final class CompositionEngine {
} }
} }
/// Turns English mode on or off directly (e.g. the input mode was switched in the menu
/// bar). Commit any composition first.
public func setEnglishMode(_ on: Bool) {
guard on != isEnglishMode else { return }
reset()
isEnglishMode = on
previousMode = on ? .literal : nil
previousCommitTime = now()
}
/// Forces the whole buffer into the other mode (a correction that is learned). /// Forces the whole buffer into the other mode (a correction that is learned).
private func flip() { private func flip() {
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
@@ -517,6 +563,8 @@ public final class CompositionEngine {
private func commitCurrent(_ given: [Segment]? = nil) -> String { private func commitCurrent(_ given: [Segment]? = nil) -> String {
let segs = given ?? segments let segs = given ?? segments
let text = segs.map { render($0, final: true) }.joined() let text = segs.map { render($0, final: true) }.joined()
rememberPicks(segs)
recentText = String((rerankContext + text).suffix(20))
if let forced = forcedMode { if let forced = forcedMode {
classifier.learned.record(String(keys), as: forced) classifier.learned.record(String(keys), as: forced)
} }
@@ -526,6 +574,17 @@ public final class CompositionEngine {
return text return text
} }
/// Candidates the user picked are remembered and come first next time.
private func rememberPicks(_ segs: [Segment]) {
for seg in segs where seg.mode == .zhuyin {
guard let picks = fixed[seg.range.lowerBound], !picks.isEmpty else { continue }
let syl = convertibleSyllables(seg, final: true).syllables
for (range, text) in picks where range.upperBound <= syl.count {
classifier.history.record(Array(syl[range]), phrase: text)
}
}
}
/// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty. /// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty.
public func commitAll() -> String { public func commitAll() -> String {
keys.isEmpty ? "" : commitCurrent() keys.isEmpty ? "" : commitCurrent()
+39 -3
View File
@@ -13,12 +13,22 @@ public struct Candidate: Equatable {
/// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon). /// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon).
public struct Converter { public struct Converter {
public let lexicon: Lexicon public let lexicon: Lexicon
/// Picked candidates come first.
public let history: UserHistory?
/// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted. /// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted.
static let unknownScore = -30.0 static let unknownScore = -30.0
/// Penalty for matching an untoned syllable against all tones. /// Penalty for matching an untoned syllable against all tones.
static let tonelessPenalty = -1.5 static let tonelessPenalty = -1.5
public init(lexicon: Lexicon) { self.lexicon = lexicon } public init(lexicon: Lexicon, history: UserHistory? = nil) {
self.lexicon = lexicon
self.history = history
}
private func entries(_ syllables: [Syllable]) -> [LexiconEntry] {
let e = lexicon.entries(for: syllables)
return history?.adjust(syllables, e) ?? e
}
/// - Parameter fixed: user-chosen phrases by syllable range; they are always kept. /// - Parameter fixed: user-chosen phrases by syllable range; they are always kept.
public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] { public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] {
@@ -51,13 +61,39 @@ public struct Converter {
return segments.reversed() return segments.reversed()
} }
/// The `k` best paths (log10 score, segments), best first — for language-model reranking.
public func kBest(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:], k: Int) -> [(score: Double, segments: [ConvertedSegment])] {
let n = syllables.count
guard n > 0, k > 0 else { return [] }
let maxLen = max(1, lexicon.maxPhraseLength)
typealias Path = (score: Double, segments: [ConvertedSegment])
var beams: [[Path]] = Array(repeating: [], count: n + 1)
beams[0] = [(0, [])]
for j in 1...n {
var cand: [Path] = []
for len in 1...min(maxLen, j) {
let i = j - len
guard !beams[i].isEmpty else { continue }
let range = i..<j
for (text, score) in options(for: range, syllables: syllables, fixed: fixed) {
for path in beams[i] {
cand.append((path.score + score, path.segments + [ConvertedSegment(range: range, text: text)]))
}
}
}
cand.sort { $0.score > $1.score }
beams[j] = Array(cand.prefix(k))
}
return beams[n]
}
private func options(for range: Range<Int>, syllables: [Syllable], fixed: [Range<Int>: String]) -> [(String, Double)] { private func options(for range: Range<Int>, syllables: [Syllable], fixed: [Range<Int>: String]) -> [(String, Double)] {
if let text = fixed[range] { return [(text, 0)] } if let text = fixed[range] { return [(text, 0)] }
// A span may not partially overlap or swallow a fixed span. // A span may not partially overlap or swallow a fixed span.
if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] } if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] }
let slice = Array(syllables[range]) let slice = Array(syllables[range])
var result = lexicon.entries(for: slice).map { entry -> (String, Double) in var result = entries(slice).map { entry -> (String, Double) in
let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0 let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0
return (entry.phrase, entry.score + penalty) return (entry.phrase, entry.score + penalty)
} }
@@ -74,7 +110,7 @@ public struct Converter {
var seen = Set<String>() var seen = Set<String>()
for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) { for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) {
let range = (end - len)..<end let range = (end - len)..<end
for entry in lexicon.entries(for: Array(syllables[range])) where !seen.contains(entry.phrase) { for entry in entries(Array(syllables[range])) where !seen.contains(entry.phrase) {
seen.insert(entry.phrase) seen.insert(entry.phrase)
out.append(Candidate(range: range, text: entry.phrase)) out.append(Candidate(range: range, text: entry.phrase))
} }
+242
View File
@@ -0,0 +1,242 @@
import Accelerate
import Foundation
/// Scores Chinese text: natural-log probability of `text` following `context`.
public protocol LanguageModel: AnyObject {
func logProb(_ text: String, context: String) -> Double
}
/// GPT-2 forward pass (CPU, Accelerate) for the character-level CKIP gpt2-tiny-chinese
/// model (GPL-3.0), stored by training/lm/export_lm.py as float16 ("AZLM" format).
/// Weights are expanded to float32 in memory (~46 MB for gpt2-tiny).
public final class GPT2LanguageModel: LanguageModel {
public enum LoadError: Error { case badFormat(String) }
let nLayer: Int, nHead: Int, d: Int, vocabSize: Int
let eps: Float
private var weights: [Float] = []
private var offsets: [String: Int] = [:]
private var vocab: [Character: Int32] = [:]
static let cls: Int32 = 101, unk: Int32 = 100
/// Longest input scored (context + text); longer contexts are cut from the left.
public var maxTokens = 64
public init(modelURL: URL, vocabURL: URL) throws {
let data = try Data(contentsOf: modelURL, options: .mappedIfSafe)
guard data.count > 8, data.prefix(4) == Data("AZLM".utf8) else { throw LoadError.badFormat("magic") }
let headerLength = Int(data[4]) | Int(data[5]) << 8 | Int(data[6]) << 16 | Int(data[7]) << 24
guard let header = try JSONSerialization.jsonObject(with: data.subdata(in: 8..<(8 + headerLength))) as? [String: Any],
let nLayer = header["n_layer"] as? Int, let nHead = header["n_head"] as? Int,
let d = header["n_embd"] as? Int, let vocabSize = header["vocab"] as? Int,
let eps = header["eps"] as? Double, let tensors = header["tensors"] as? [[Any]]
else { throw LoadError.badFormat("header") }
self.nLayer = nLayer; self.nHead = nHead; self.d = d; self.vocabSize = vocabSize; self.eps = Float(eps)
for t in tensors {
guard let name = t[0] as? String, let offset = t[2] as? Int else { throw LoadError.badFormat("tensor") }
offsets[name] = offset
}
let start = 8 + headerLength
let count = (data.count - start) / 2
weights = [Float](repeating: 0, count: count)
try data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
guard let base = raw.baseAddress?.advanced(by: start) else { throw LoadError.badFormat("data") }
try weights.withUnsafeMutableBufferPointer { out in
var src = vImage_Buffer(data: UnsafeMutableRawPointer(mutating: base), height: 1,
width: vImagePixelCount(count), rowBytes: count * 2)
var dst = vImage_Buffer(data: out.baseAddress, height: 1,
width: vImagePixelCount(count), rowBytes: count * 4)
guard vImageConvert_Planar16FtoPlanarF(&src, &dst, 0) == kvImageNoError else {
throw LoadError.badFormat("float16")
}
}
}
var index: Int32 = 0
for line in try String(contentsOf: vocabURL, encoding: .utf8).split(separator: "\n", omittingEmptySubsequences: false) {
if line.count == 1, let c = line.first, vocab[c] == nil { vocab[c] = index }
index += 1
}
for name in ["wte.weight", "wpe.weight", "ln_f.weight"] where offsets[name] == nil {
throw LoadError.badFormat("missing \(name)")
}
}
func tokens(_ text: String) -> [Int32] { text.map { vocab[$0] ?? GPT2LanguageModel.unk } }
public func logProb(_ text: String, context: String) -> Double {
let target = tokens(text)
guard !target.isEmpty else { return 0 }
var ctx = tokens(context)
if ctx.count + target.count + 1 > maxTokens { ctx = Array(ctx.suffix(max(0, maxTokens - target.count - 1))) }
let input = [GPT2LanguageModel.cls] + ctx + target.dropLast()
let h = hidden(input) // [T, d]
return weights.withUnsafeBufferPointer { w in
logSoftmaxScore(h, rows: input.count, from: ctx.count, targets: target, w: w)
}
}
// MARK: - Forward pass
private func w(_ name: String) -> Int { offsets[name]! }
/// Final hidden states [T, d] (row-major).
func hidden(_ ids: [Int32]) -> [Float] {
let T = ids.count, d = self.d
var x = [Float](repeating: 0, count: T * d)
weights.withUnsafeBufferPointer { W in
let wte = w("wte.weight"), wpe = w("wpe.weight")
for t in 0..<T {
vDSP_vadd(W.baseAddress! + wte + Int(ids[t]) * d, 1, W.baseAddress! + wpe + t * d, 1,
&x[t * d], 1, vDSP_Length(d))
}
for l in 0..<nLayer {
let p = "h.\(l)."
var h = layerNorm(x, rows: T, p + "ln_1", W)
let qkv = linear(h, rows: T, inDim: d, outDim: 3 * d, p + "attn.c_attn", W)
let a = attention(qkv, T: T)
var proj = linear(a, rows: T, inDim: d, outDim: d, p + "attn.c_proj", W)
GPT2LanguageModel.add(&x, proj)
h = layerNorm(x, rows: T, p + "ln_2", W)
var f = linear(h, rows: T, inDim: d, outDim: 4 * d, p + "mlp.c_fc", W)
geluNew(&f)
proj = linear(f, rows: T, inDim: 4 * d, outDim: d, p + "mlp.c_proj", W)
GPT2LanguageModel.add(&x, proj)
}
x = layerNorm(x, rows: T, "ln_f", W)
}
return x
}
private func layerNorm(_ x: [Float], rows: Int, _ name: String, _ W: UnsafeBufferPointer<Float>) -> [Float] {
let d = self.d
var out = [Float](repeating: 0, count: x.count)
let g = W.baseAddress! + w(name + ".weight"), b = W.baseAddress! + w(name + ".bias")
for r in 0..<rows {
var mean: Float = 0, sd: Float = 0
x.withUnsafeBufferPointer { xp in
out.withUnsafeMutableBufferPointer { op in
let src = xp.baseAddress! + r * d, dst = op.baseAddress! + r * d
vDSP_normalize(src, 1, dst, 1, &mean, &sd, vDSP_Length(d))
// vDSP_normalize divides by the population std without eps; redo with eps.
let scale = sd / (sd * sd + eps).squareRoot()
var s = scale
vDSP_vsmul(dst, 1, &s, dst, 1, vDSP_Length(d))
vDSP_vma(dst, 1, g, 1, b, 1, dst, 1, vDSP_Length(d))
}
}
}
return out
}
/// x [rows, inDim] · W [inDim, outDim] + bias.
private func linear(_ x: [Float], rows: Int, inDim: Int, outDim: Int, _ name: String,
_ W: UnsafeBufferPointer<Float>) -> [Float] {
var out = [Float](repeating: 0, count: rows * outDim)
vDSP_mmul(x, 1, W.baseAddress! + w(name + ".weight"), 1, &out, 1,
vDSP_Length(rows), vDSP_Length(outDim), vDSP_Length(inDim))
let bias = W.baseAddress! + w(name + ".bias")
out.withUnsafeMutableBufferPointer { op in
for r in 0..<rows {
vDSP_vadd(op.baseAddress! + r * outDim, 1, bias, 1, op.baseAddress! + r * outDim, 1, vDSP_Length(outDim))
}
}
return out
}
/// Causal multi-head self-attention. qkv is [T, 3d] (q | k | v).
private func attention(_ qkv: [Float], T: Int) -> [Float] {
let d = self.d, hd = d / nHead
let scale = 1 / Float(hd).squareRoot()
var out = [Float](repeating: 0, count: T * d)
var scores = [Float](repeating: 0, count: T)
qkv.withUnsafeBufferPointer { q in
out.withUnsafeMutableBufferPointer { o in
for head in 0..<nHead {
for i in 0..<T {
let qi = q.baseAddress! + i * 3 * d + head * hd
var maxScore = -Float.infinity
for j in 0...i {
var s: Float = 0
vDSP_dotpr(qi, 1, q.baseAddress! + j * 3 * d + d + head * hd, 1, &s, vDSP_Length(hd))
scores[j] = s * scale
maxScore = max(maxScore, scores[j])
}
var sum: Float = 0
for j in 0...i { scores[j] = expf(scores[j] - maxScore); sum += scores[j] }
let dst = o.baseAddress! + i * d + head * hd
for j in 0...i {
var a = scores[j] / sum
vDSP_vsma(q.baseAddress! + j * 3 * d + 2 * d + head * hd, 1, &a, dst, 1, dst, 1, vDSP_Length(hd))
}
}
}
}
}
return out
}
/// x += y (same length).
private static func add(_ x: inout [Float], _ y: [Float]) {
x.withUnsafeMutableBufferPointer { xp in
vDSP_vadd(xp.baseAddress!, 1, y, 1, xp.baseAddress!, 1, vDSP_Length(xp.count))
}
}
/// gelu_new: 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))).
private func geluNew(_ x: inout [Float]) {
let n = x.count
var inner = [Float](repeating: 0, count: n)
x.withUnsafeMutableBufferPointer { xb in
inner.withUnsafeMutableBufferPointer { ib in
let xp = xb.baseAddress!, ip = ib.baseAddress!, len = vDSP_Length(n)
var c3: Float = 0.044715, one: Float = 1, k = (2 / Float.pi).squareRoot(), half: Float = 0.5
vDSP_vsq(xp, 1, ip, 1, len) // x²
vDSP_vsmsa(ip, 1, &c3, &one, ip, 1, len) // 1 + 0.044715·x²
vDSP_vmul(ip, 1, xp, 1, ip, 1, len) // x + 0.044715·x³
vDSP_vsmul(ip, 1, &k, ip, 1, len)
var count = Int32(n)
vvtanhf(ip, ip, &count)
vDSP_vsadd(ip, 1, &one, ip, 1, len) // 1 + tanh(…)
vDSP_vmul(ip, 1, xp, 1, xp, 1, len)
vDSP_vsmul(xp, 1, &half, xp, 1, len)
}
}
}
/// Sum over target positions of log softmax(h · wteᵀ)[target]. Row `from + i` predicts target i.
private func logSoftmaxScore(_ h: [Float], rows T: Int, from start: Int, targets: [Int32],
w W: UnsafeBufferPointer<Float>) -> Double {
let d = self.d, V = vocabSize
let rowsUsed = T - start
// hᵀ for the rows we need: [d, rowsUsed]
var hT = [Float](repeating: 0, count: d * rowsUsed)
h.withUnsafeBufferPointer { hp in
vDSP_mtrans(hp.baseAddress! + start * d, 1, &hT, 1, vDSP_Length(d), vDSP_Length(rowsUsed))
}
// logitsᵀ [V, rowsUsed] = wte [V, d] · hᵀ [d, rowsUsed]
var logitsT = [Float](repeating: 0, count: V * rowsUsed)
vDSP_mmul(W.baseAddress! + w("wte.weight"), 1, hT, 1, &logitsT, 1,
vDSP_Length(V), vDSP_Length(rowsUsed), vDSP_Length(d))
var logits = [Float](repeating: 0, count: V * rowsUsed) // [rowsUsed, V]
vDSP_mtrans(logitsT, 1, &logits, 1, vDSP_Length(rowsUsed), vDSP_Length(V))
var total = 0.0
var expBuf = [Float](repeating: 0, count: V)
logits.withUnsafeMutableBufferPointer { lp in
expBuf.withUnsafeMutableBufferPointer { eb in
let e = eb.baseAddress!
for i in 0..<rowsUsed {
let row = lp.baseAddress! + i * V
var m: Float = 0
vDSP_maxv(row, 1, &m, vDSP_Length(V))
var negM = -m
vDSP_vsadd(row, 1, &negM, e, 1, vDSP_Length(V))
var n = Int32(V)
vvexpf(e, e, &n)
var sum: Float = 0
vDSP_sve(e, 1, &sum, vDSP_Length(V))
total += Double(row[Int(targets[i])] - m) - Double(logf(sum))
}
}
}
return total
}
}
+90
View File
@@ -0,0 +1,90 @@
import Foundation
/// "AI 選字": reranks the converter's best paths with a language model.
///
/// score = lexicon score (log10 → ln) + λ · LM log-probability
///
/// It never blocks typing for long: each rerank has a time budget, and a rerank that runs
/// over it is dropped (the plain lexicon result is used). Three overruns in a row pause
/// AI for a minute. `systemState` lets the app pause or reduce it when the machine is
/// busy, hot or saving power.
public final class Reranker {
public enum Mode: String, CaseIterable {
case off, saver, standard, quality
/// How many paths are reranked, and the time budget for one rerank.
var k: Int { [.saver: 3, .standard: 10, .quality: 30][self] ?? 0 }
var budget: TimeInterval { [.saver: 0.015, .standard: 0.04, .quality: 0.12][self] ?? 0 }
}
/// What the system allows right now (decided by the app).
public enum SystemState { case normal, reduced, paused }
public enum Status: Equatable {
case off, loading, active, pausedBusy, pausedSlow
}
public var mode: Mode { didSet { cache.removeAll() } }
/// nil while loading (or if the model file is missing).
public var model: LanguageModel? { didSet { cache.removeAll() } }
public var lambda = 0.6
public var systemState: () -> SystemState = { .normal }
public var now: () -> Date = Date.init
public private(set) var status: Status = .off
private var overruns = 0
private var pausedUntil = Date.distantPast
private var cache: [String: [ConvertedSegment]] = [:]
static let cacheLimit = 512
static let pauseAfterOverruns = 3
static let slowPause: TimeInterval = 60
public init(mode: Mode = .standard, model: LanguageModel? = nil) {
self.mode = mode
self.model = model
}
/// The mode actually used now (after the system state), or nil when AI is not used.
var effectiveMode: Mode? {
guard mode != .off else { status = .off; return nil }
guard model != nil else { status = .loading; return nil }
if now() < pausedUntil { status = .pausedSlow; return nil }
switch systemState() {
case .paused: status = .pausedBusy; return nil
case .reduced: status = .active; return .saver
case .normal: status = .active; return mode
}
}
/// The reranked conversion, or nil to use the plain lexicon result.
func choose(_ converter: Converter, _ syllables: [Syllable], fixed: [Range<Int>: String],
context: String) -> [ConvertedSegment]? {
guard let mode = effectiveMode, let model, !syllables.isEmpty else { return nil }
let key = "\(mode.rawValue)|\(context)|\(syllables.map(\.reading).joined(separator: "-"))|"
+ fixed.sorted { $0.key.lowerBound < $1.key.lowerBound }.map { "\($0.key)=\($0.value)" }.joined(separator: ",")
if let hit = cache[key] { return hit }
let paths = converter.kBest(syllables, fixed: fixed, k: mode.k)
guard paths.count > 1 else { return paths.first?.segments }
let start = now()
var best: (score: Double, segments: [ConvertedSegment])?
for path in paths {
let text = path.segments.map(\.text).joined()
let score = path.score * log(10) + lambda * model.logProb(text, context: context)
if best == nil || score > best!.score { best = (score, path.segments) }
if now().timeIntervalSince(start) > mode.budget {
overruns += 1
if overruns >= Reranker.pauseAfterOverruns {
pausedUntil = now().addingTimeInterval(Reranker.slowPause)
overruns = 0
status = .pausedSlow
}
return nil
}
}
overruns = 0
if cache.count >= Reranker.cacheLimit { cache.removeAll() }
cache[key] = best?.segments
return best?.segments
}
}
+79
View File
@@ -0,0 +1,79 @@
import Foundation
/// Remembers the candidates the user picked, so they come first next time.
///
/// Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
/// where the lexicon has it for the reading being converted, so a toneless key is safe.
/// Stored as JSON: {reading: {phrase: [count, last use]}} (same format on every platform).
public final class UserHistory {
public typealias Data = [String: [String: [Int]]]
/// (reading, phrase) pairs kept; the least recently used go first.
public var maxEntries = 2000
static let maxCount = 10
public private(set) var data: Data = [:]
private var clock = 0
private let onChange: (Data) -> Void
public init(data: Data = [:], onChange: @escaping (Data) -> Void = { _ in }) {
self.onChange = onChange
for (reading, phrases) in data {
for (phrase, value) in phrases where value.count >= 2 {
self.data[reading, default: [:]][phrase] = [value[0], value[1]]
clock = max(clock, value[1])
}
}
}
/// Loads the JSON file written through `onChange`; empty if missing or unreadable.
public static func load(from url: URL) -> Data {
guard let raw = try? Foundation.Data(contentsOf: url),
let data = try? JSONDecoder().decode(Data.self, from: raw) else { return [:] }
return data
}
static func key(_ syllables: [Syllable]) -> String {
syllables.map(\.toneless).joined(separator: "-")
}
public func record(_ syllables: [Syllable], phrase: String) {
guard !syllables.isEmpty, !phrase.isEmpty else { return }
clock += 1
let key = UserHistory.key(syllables)
let count = data[key]?[phrase]?.first ?? 0
data[key, default: [:]][phrase] = [min(UserHistory.maxCount, count + 1), clock]
evict()
onChange(data)
}
private func evict() {
var pairs: [(used: Int, reading: String, phrase: String)] = []
for (r, phrases) in data { for (p, v) in phrases { pairs.append((v[1], r, p)) } }
guard pairs.count > maxEntries else { return }
pairs.sort { $0.used < $1.used }
for pair in pairs.prefix(pairs.count - maxEntries) {
data[pair.reading]?[pair.phrase] = nil
if data[pair.reading]?.isEmpty == true { data[pair.reading] = nil }
}
}
public func clear() {
data = [:]
onChange(data)
}
/// Lexicon entries with picked phrases moved to the top: just above the best entry,
/// more so the more often they were picked.
func adjust(_ syllables: [Syllable], _ entries: [LexiconEntry]) -> [LexiconEntry] {
guard let picked = data[UserHistory.key(syllables)], let top = entries.first?.score else { return entries }
var out = entries.map { e -> LexiconEntry in
guard let count = picked[e.phrase]?.first else { return e }
return LexiconEntry(phrase: e.phrase, score: max(e.score, top + 0.01 + 0.1 * Double(count)))
}
// Stable sort (keep lexicon order among equal scores), like Python's sort.
out = out.enumerated().sorted { $0.element.score != $1.element.score
? $0.element.score > $1.element.score : $0.offset < $1.offset }.map(\.element)
return out
}
}
+23
View File
@@ -48,3 +48,26 @@ https://github.com/EasyIME/PIME — GNU LGPL 2.1
The Windows version runs as a module inside PIME, which users install The Windows version runs as a module inside PIME, which users install
separately. No PIME code is included in this software. separately. No PIME code is included in this software.
4. CKIP GPT2 Tiny Chinese — AI 選字語言模型 (language model)
-----------------------------------------------------------
https://huggingface.co/ckiplab/gpt2-tiny-chinese
https://github.com/ckiplab/ckip-transformers
Author: Mu Yang at CKIP, Academia Sinica (中央研究院 詞庫小組)
License: GNU General Public License v3.0 (full text: GPL-3.0.txt)
lm.bin is the model's weights converted to float16 by training/lm/export_lm.py;
lm-vocab.txt is its vocabulary (from bert-base-chinese). They are used unmodified
otherwise, to rerank conversion candidates ("AI 選字"). The source code of this input
method, including the conversion script and the code that runs the model, is
available from the project repository.
5. NumPy — 數值運算 (Windows / Linux, for AI 選字)
------------------------------------------------
https://numpy.org — BSD 3-Clause License (and the licenses of the libraries it bundles,
such as OpenBLAS). The Windows package includes unmodified NumPy wheels for 32-bit
Python 3.8 and 3.15 under autozhuyin/vendor/; the Linux installer downloads the
matching wheel from PyPI when the system has no NumPy. Each copy keeps its license
files in its numpy-*.dist-info folder.
@@ -133,6 +133,16 @@ final class EngineTests: XCTestCase {
return out return out
} }
/// The menu-bar input mode can switch English mode directly.
func testSetEnglishMode() {
let e = makeEngine()
e.setEnglishMode(true)
XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(e.handle(.character("a")).handled, false) // goes to the app
e.setEnglishMode(false)
XCTAssertEqual(type("su3cl3 ", into: e), "你好")
}
func testMixedSentence() { func testMixedSentence() {
let e = makeEngine() let e = makeEngine()
// 我 + "hello world " + 你好 // 我 + "hello world " + 你好
@@ -384,7 +394,7 @@ final class CursorTests: XCTestCase {
_ = type("←", into: e) _ = type("←", into: e)
XCTAssertEqual(e.cursorOffset, 3) XCTAssertEqual(e.cursorOffset, 3)
_ = type("←←←←", into: e) _ = type("←←←←", into: e)
XCTAssertEqual(e.cursorOffset, 0) // stops at the start XCTAssertEqual(e.cursorOffset, 1) // stops after the first character
_ = type("→→→→", into: e) _ = type("→→→→", into: e)
XCTAssertEqual(e.cursorOffset, 4) XCTAssertEqual(e.cursorOffset, 4)
} }
@@ -419,8 +429,10 @@ final class CursorTests: XCTestCase {
XCTAssertEqual(type("\n", into: e), "今天氣") XCTAssertEqual(type("\n", into: e), "今天氣")
} }
func testInsertAtStart() { func testCursorStopsAfterFirstCharacter() {
XCTAssertEqual(type("su3cl3←←ji3\n", into: engine()), "我你好") let e = engine()
_ = type("su3cl3←←←", into: e)
XCTAssertEqual(e.cursorOffset, 1)
} }
func testBackspaceDeletesCharacterBeforeCursor() { func testBackspaceDeletesCharacterBeforeCursor() {
@@ -459,8 +471,10 @@ final class FirstToneAfterEnglishTests: XCTestCase {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = "" var out = ""
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } } for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out, "hello ", keys) out += e.handle(.enter).commit ?? ""
XCTAssertEqual(e.markedText.count, 1, keys) // still composing one Chinese character XCTAssertEqual(String(out.prefix(6)), "hello ", keys)
XCTAssertEqual(out.count, 7, keys) // one Chinese character after it
XCTAssertFalse(out.last!.isASCII, keys)
} }
} }
} }
@@ -537,6 +551,226 @@ final class BacktickSymbolTests: XCTestCase {
} }
} }
final class FirstWordHoldTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// The first word is not committed as English on space if it also reads as Zhuyin.
func testNextWordDecides() {
let cases = ["el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
"i love you ": "i love you ", "up the ": "up the "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testNotHeldAfterEnglish() {
let e = makeEngine()
XCTAssertEqual(type("hello world ", into: e), "hello world ")
XCTAssertEqual(type("so ", into: e), "so ") // right after English: commits at once
}
}
final class UserHistoryTests: XCTestCase {
var saved: [UserHistory.Data] = []
var classifier: Classifier!
override func setUp() {
saved = []
classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights,
history: UserHistory { [unowned self] in self.saved.append($0) })
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "↓" ? .down : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
@discardableResult
func pick(_ keys: String, _ phrase: String) -> String {
let e = CompositionEngine(classifier: classifier)
_ = type(keys + "↓", into: e)
e.selectCandidate(e.candidates.firstIndex { $0.text == phrase }!)
return type("\n", into: e)
}
func testPickedCandidateComesFirst() {
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "天")
XCTAssertEqual(pick("wu0 ", "添"), "添")
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
let e = CompositionEngine(classifier: classifier)
_ = type("wu0 ↓", into: e)
XCTAssertEqual(e.candidates.first?.text, "添")
XCTAssertEqual(saved.last, ["ㄊㄧㄢ": ["添": [1, 1]]])
}
func testPhrasesStillWin() {
pick("wu0 ", "添")
XCTAssertEqual(type("rup wu0 \n", into: CompositionEngine(classifier: classifier)), "今天")
}
func testMostPickedWins() {
pick("wu0 ", "添")
pick("wu0 ", "添")
pick("wu0 ", "天")
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
}
func testRoundTripAndClear() throws {
pick("wu0 ", "添")
let json = try JSONEncoder().encode(saved.last!)
let h = UserHistory(data: try JSONDecoder().decode(UserHistory.Data.self, from: json))
XCTAssertEqual(h.data, ["ㄊㄧㄢ": ["添": [1, 1]]])
h.record(ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable), phrase: "天")
XCTAssertEqual(h.data["ㄊㄧㄢ"]?["天"], [1, 2])
h.clear()
XCTAssertTrue(h.data.isEmpty)
}
func testEvictsLeastRecent() {
let h = UserHistory()
h.maxEntries = 2
let syl = ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable)
for p in ["天", "添", "田"] { h.record(syl, phrase: p) }
XCTAssertEqual(Set(h.data["ㄊㄧㄢ"]!.keys), ["添", "田"])
}
}
/// Needs build/lm/lm.bin (scripts/fetch-lm.sh); skipped otherwise.
final class GPT2LanguageModelTests: XCTestCase {
static let dir = resources.deletingLastPathComponent().appendingPathComponent("build/lm")
func model() throws -> GPT2LanguageModel {
let bin = GPT2LanguageModelTests.dir.appendingPathComponent("lm.bin")
try XCTSkipUnless(FileManager.default.fileExists(atPath: bin.path), "run scripts/fetch-lm.sh")
return try GPT2LanguageModel(modelURL: bin, vocabURL: GPT2LanguageModelTests.dir.appendingPathComponent("lm-vocab.txt"))
}
/// Same numbers as the numpy reference (training/lm/gpt2.py, float32 weights).
func testMatchesReference() throws {
let m = try model()
let cases: [(String, String, Double)] = [("他在家", "", -14.329), ("今天天氣很好", "", -30.08),
("今添天氣很好", "", -44.392), ("家", "他在", -5.514),
("天氣很好", "今天", -23.883)]
for (text, context, want) in cases {
XCTAssertEqual(m.logProb(text, context: context), want, accuracy: 0.05, text)
}
}
func testSpeed() throws {
let m = try model()
let start = Date()
for _ in 0..<20 { _ = m.logProb("我們明天早上八點在車站見面", context: "好啊") }
let ms = Date().timeIntervalSince(start) / 20 * 1000
print("GPT2 logProb: \(String(format: "%.2f", ms)) ms")
XCTAssertLessThan(ms, 20)
}
}
/// A language model that likes one string and can pretend to be slow.
private final class FakeLanguageModel: LanguageModel {
var liked: String
var onCall: () -> Void = {}
init(liked: String) { self.liked = liked }
func logProb(_ text: String, context: String) -> Double {
onCall()
return text.contains(liked) ? 0 : -50
}
}
final class RerankerTests: XCTestCase {
func engine(_ reranker: Reranker?) -> CompositionEngine {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
e.reranker = reranker
return e
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testRerankPicksWhatTheModelLikes() {
XCTAssertEqual(type("wu0 \n", into: engine(nil)), "天")
let r = Reranker(mode: .standard, model: FakeLanguageModel(liked: "添"))
XCTAssertEqual(type("wu0 \n", into: engine(r)), "添")
XCTAssertEqual(r.status, .active)
}
func testOffAndLoadingUseTheLexicon() {
let off = Reranker(mode: .off, model: FakeLanguageModel(liked: "添"))
XCTAssertEqual(type("wu0 \n", into: engine(off)), "天")
XCTAssertEqual(off.status, .off)
let loading = Reranker(mode: .standard, model: nil)
XCTAssertEqual(type("wu0 \n", into: engine(loading)), "天")
XCTAssertEqual(loading.status, .loading)
}
func testPausedWhenSystemBusy() {
let r = Reranker(mode: .standard, model: FakeLanguageModel(liked: "添"))
r.systemState = { .paused }
XCTAssertEqual(type("wu0 \n", into: engine(r)), "天")
XCTAssertEqual(r.status, .pausedBusy)
}
func testReducedUsesFewerPaths() {
let lm = FakeLanguageModel(liked: "添")
var calls = 0
lm.onCall = { calls += 1 }
let r = Reranker(mode: .quality, model: lm)
r.systemState = { .reduced }
_ = type("wu0 \n", into: engine(r))
XCTAssertLessThanOrEqual(calls, 3)
}
func testSlowModelIsDroppedThenPaused() {
var clock = Date(timeIntervalSince1970: 0)
let lm = FakeLanguageModel(liked: "添")
lm.onCall = { clock.addTimeInterval(1) } // every call takes a "second"
let r = Reranker(mode: .standard, model: lm)
r.now = { clock }
for _ in 0..<3 { XCTAssertEqual(type("wu0 \n", into: engine(r)), "天") }
XCTAssertEqual(r.status, .pausedSlow)
lm.onCall = {}
XCTAssertEqual(type("wu0 \n", into: engine(r)), "天") // still paused
clock.addTimeInterval(61)
XCTAssertEqual(type("wu0 \n", into: engine(r)), "添") // back after a minute
}
/// With the real model (build/lm, skipped if missing): sentences the lexicon alone gets wrong.
func testRealModelFixesHomophones() throws {
let dir = GPT2LanguageModelTests.dir
let bin = dir.appendingPathComponent("lm.bin")
try XCTSkipUnless(FileManager.default.fileExists(atPath: bin.path), "run scripts/fetch-lm.sh")
let lm = try GPT2LanguageModel(modelURL: bin, vocabURL: dir.appendingPathComponent("lm-vocab.txt"))
let cases = ["w8 ql32k71u3ji3dj94cp32ji ": "他跑得比我快很多",
"fu/3su3y94gji u h4cl3a87": "請你再說一次好嗎"]
for (keys, want) in cases {
XCTAssertNotEqual(type(keys + "\n", into: engine(nil)), want)
XCTAssertEqual(type(keys + "\n", into: engine(Reranker(mode: .standard, model: lm))), want)
}
}
}
final class StartupTimingTests: XCTestCase { final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws { func testStartupTiming() throws {
var t = Date() var t = Date()
+674
View File
@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.
+106 -13
View File
@@ -17,6 +17,10 @@ import time
HERE = os.path.dirname(os.path.abspath(__file__)) HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE) sys.path.insert(0, HERE)
# numpy for "AI 選字", downloaded by install.sh when the system has none.
_VENDOR = os.path.join(HERE, "vendor", "cp%d%d" % sys.version_info[:2])
if os.path.isdir(_VENDOR):
sys.path.insert(1, _VENDOR)
import gi # noqa: E402 import gi # noqa: E402
@@ -24,6 +28,9 @@ gi.require_version("IBus", "1.0")
from gi.repository import GLib, GObject, IBus # noqa: E402 from gi.repository import GLib, GObject, IBus # noqa: E402
import autozhuyin # noqa: E402 import autozhuyin # noqa: E402
from autozhuyin.core.ai import AI, MODE_NAMES # noqa: E402
from autozhuyin.core.history import UserHistory # noqa: E402
from autozhuyin.core.reranker import MODES # noqa: E402
from autozhuyin.core.classifier import LITERAL, Classifier, LearnedPreferences # noqa: E402 from autozhuyin.core.classifier import LITERAL, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import ( # noqa: E402 from autozhuyin.core.engine import ( # noqa: E402
BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine, BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine,
@@ -40,6 +47,7 @@ _DATA = os.path.join(os.path.dirname(os.path.abspath(autozhuyin.__file__)), "dat
_USER_DIR = os.path.join(os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config"), "autozhuyin") _USER_DIR = os.path.join(os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config"), "autozhuyin")
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json") _LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt") _USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
try: try:
with open(os.path.join(HERE, "version.txt"), encoding="utf-8") as _f: with open(os.path.join(HERE, "version.txt"), encoding="utf-8") as _f:
@@ -48,15 +56,40 @@ except OSError:
pass pass
_classifier = None _classifier = None
_ai = None
def shared_ai():
""""AI 選字": settings and the language model, loaded in the background once."""
global _ai
if _ai is None:
_ai = AI(_DATA, _USER_DIR)
return _ai
def _save_json(path, data):
try:
os.makedirs(_USER_DIR, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False)
except OSError:
pass
def _load_json(path):
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError):
return {}
def _save_learned(weights): def _save_learned(weights):
try: _save_json(_LEARNED_FILE, weights)
os.makedirs(_USER_DIR, exist_ok=True)
with open(_LEARNED_FILE, "w", encoding="utf-8") as f:
json.dump(weights, f, ensure_ascii=False) def _save_history(data):
except OSError: _save_json(_HISTORY_FILE, data)
pass
def shared_classifier(): def shared_classifier():
@@ -72,14 +105,9 @@ def shared_classifier():
os.path.join(_DATA, "english-dict.txt"), os.path.join(_DATA, "english-dict.txt"),
os.path.join(_DATA, "english-bigram.json"), os.path.join(_DATA, "english-bigram.json"),
) )
weights = {}
try:
with open(_LEARNED_FILE, encoding="utf-8") as f:
weights = json.load(f)
except (OSError, ValueError):
pass
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json")) trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained) _classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
return _classifier return _classifier
@@ -105,6 +133,7 @@ class AutoZhuyinEngine(IBus.Engine):
if getattr(self, "engine", None) is not None: if getattr(self, "engine", None) is not None:
return return
self.engine = CompositionEngine(shared_classifier()) self.engine = CompositionEngine(shared_classifier())
self.engine.reranker = shared_ai().reranker
self.shift_tap = ShiftTapDetector() self.shift_tap = ShiftTapDetector()
self.table = IBus.LookupTable.new(self.engine.page_size, 0, True, False) self.table = IBus.LookupTable.new(self.engine.page_size, 0, True, False)
self.table.set_orientation(IBus.Orientation.VERTICAL) self.table.set_orientation(IBus.Orientation.VERTICAL)
@@ -114,8 +143,43 @@ class AutoZhuyinEngine(IBus.Engine):
symbol=IBus.Text.new_from_string("自"), symbol=IBus.Text.new_from_string("自"),
tooltip=IBus.Text.new_from_string("單按 Shift 切換英文模式"), tooltip=IBus.Text.new_from_string("單按 Shift 切換英文模式"),
sensitive=True, visible=True, state=IBus.PropState.UNCHECKED) sensitive=True, visible=True, state=IBus.PropState.UNCHECKED)
# "AI 選字" menu: modes (radio), auto-pause (toggle), status, clear learned data.
ai = shared_ai()
self.ai_props = {}
sub = IBus.PropList()
for mode in MODES:
prop = IBus.Property(
key="ai." + mode, prop_type=IBus.PropType.RADIO,
label=IBus.Text.new_from_string("AI 選字:" + MODE_NAMES[mode]),
tooltip=IBus.Text.new_from_string(""), sensitive=True, visible=True,
state=IBus.PropState.CHECKED if ai.reranker.mode == mode else IBus.PropState.UNCHECKED)
self.ai_props[mode] = prop
sub.append(prop)
self.pause_prop = IBus.Property(
key="ai.autopause", prop_type=IBus.PropType.TOGGLE,
label=IBus.Text.new_from_string("系統忙碌時自動暫停 AI"),
tooltip=IBus.Text.new_from_string(""), sensitive=True, visible=True,
state=IBus.PropState.CHECKED if ai.auto_pause else IBus.PropState.UNCHECKED)
self.status_prop = IBus.Property(
key="ai.status", prop_type=IBus.PropType.NORMAL,
label=IBus.Text.new_from_string(ai.status_text()),
tooltip=IBus.Text.new_from_string(""), sensitive=False, visible=True,
state=IBus.PropState.UNCHECKED)
self.clear_prop = IBus.Property(
key="clear", prop_type=IBus.PropType.NORMAL,
label=IBus.Text.new_from_string("清除學習紀錄"),
tooltip=IBus.Text.new_from_string(""), sensitive=True, visible=True,
state=IBus.PropState.UNCHECKED)
for prop in (self.pause_prop, self.status_prop, self.clear_prop):
sub.append(prop)
self.ai_menu = IBus.Property(
key="ai", prop_type=IBus.PropType.MENU,
label=IBus.Text.new_from_string("設定"),
tooltip=IBus.Text.new_from_string("AI 選字與學習紀錄"),
sensitive=True, visible=True, state=IBus.PropState.UNCHECKED, sub_props=sub)
self.props = IBus.PropList() self.props = IBus.PropList()
self.props.append(self.mode_prop) self.props.append(self.mode_prop)
self.props.append(self.ai_menu)
# --- keys --- # --- keys ---
@@ -198,12 +262,38 @@ class AutoZhuyinEngine(IBus.Engine):
self._update(None) self._update(None)
return True return True
# --- settings menu ---
def do_property_activate(self, prop_name, state):
self._setup()
ai = shared_ai()
if prop_name.startswith("ai.") and prop_name[3:] in MODES:
if state == IBus.PropState.CHECKED:
ai.set_mode(prop_name[3:])
elif prop_name == "ai.autopause":
ai.set_auto_pause(state == IBus.PropState.CHECKED)
elif prop_name == "clear":
shared_classifier().learned.clear()
shared_classifier().history.clear()
self._refresh_ai_props()
def _refresh_ai_props(self):
ai = shared_ai()
for mode, prop in self.ai_props.items():
prop.set_state(IBus.PropState.CHECKED if ai.reranker.mode == mode else IBus.PropState.UNCHECKED)
self.update_property(prop)
self.pause_prop.set_state(IBus.PropState.CHECKED if ai.auto_pause else IBus.PropState.UNCHECKED)
self.update_property(self.pause_prop)
self.status_prop.set_label(IBus.Text.new_from_string(ai.status_text()))
self.update_property(self.status_prop)
# --- lifecycle --- # --- lifecycle ---
def do_focus_in(self): def do_focus_in(self):
self._setup() self._setup()
self.register_properties(self.props) self.register_properties(self.props)
self._update_mode() self._update_mode()
self._refresh_ai_props()
def do_focus_out(self): def do_focus_out(self):
self._setup() self._setup()
@@ -255,6 +345,9 @@ class AutoZhuyinEngine(IBus.Engine):
english = self.engine.english_mode english = self.engine.english_mode
self.mode_prop.set_label(IBus.Text.new_from_string("英文" if english else "中英自動")) self.mode_prop.set_label(IBus.Text.new_from_string("英文" if english else "中英自動"))
self.mode_prop.set_symbol(IBus.Text.new_from_string("英" if english else "自")) self.mode_prop.set_symbol(IBus.Text.new_from_string("英" if english else "自"))
icon = os.path.join(HERE, "icon-en.png" if english else "icon.png")
if os.path.exists(icon):
self.mode_prop.set_icon(icon) # panels that show property icons
self.update_property(self.mode_prop) self.update_property(self.mode_prop)
+14 -2
View File
@@ -30,15 +30,26 @@
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正 Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
← → 在組字區移動游標;游標在中間時直接打注音會插入新字, ← → 在組字區移動游標;游標在中間時直接打注音會插入新字,
Backspace 刪掉游標前的字 Backspace 刪掉游標前的字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字 ↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
選過的字會被記住,下次優先出現
Esc 取消組字 Esc 取消組字
單按 Shift 切換英文模式(按鍵直接送出),再單按一次回到中英自動判斷 單按 Shift 切換英文模式(按鍵直接送出,狀態圖示變成「英」),再單按一次回到中英自動判斷
Shift+字母 打大寫字母(也會強制英文) Shift+字母 打大寫字母(也會強制英文)
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ; ` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
AI 選字
-------
用中研院 CKIP 的繁體中文小型語言模型,看前後文挑同音字(他跑「得」很快、請你「再」說一次),
完全在本機運算。需要 numpy:系統沒有的話,install.sh 會從 PyPI 下載符合的版本(約 15 MB)。
在 IBus/系統匣的輸入法選單「設定」裡可以選:
AI 選字:關閉/省電/標準(預設)/高品質
系統忙碌時自動暫停 AI(預設開啟:CPU 很忙或過熱時暫停,省電設定或電量低時改用省電)
設定存在 ~/.config/autozhuyin/settings.json。
切換與移除 切換與移除
---------- ----------
暫時改回一般鍵盤: ibus engine xkb:us::eng 暫時改回一般鍵盤: ibus engine xkb:us::eng
@@ -47,6 +58,7 @@
個人資料(學習紀錄、自訂詞)在 ~/.config/autozhuyin/: 個人資料(學習紀錄、自訂詞)在 ~/.config/autozhuyin/:
learned.json 按 Tab 修正的學習紀錄 learned.json 按 Tab 修正的學習紀錄
user-history.json 選字紀錄(選過的候選字)
user-phrases.txt 自訂詞,一行一個,例如:ㄗˋ-ㄉㄨㄥˋ 自動 -1 user-phrases.txt 自訂詞,一行一個,例如:ㄗˋ-ㄉㄨㄥˋ 自動 -1
+21 -3
View File
@@ -23,13 +23,31 @@ fi
# 2. Copy the program. # 2. Copy the program.
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null || true pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null || true
VENDOR_BACKUP=""
if [ -d "$DEST/vendor" ]; then # keep numpy downloaded by an earlier install
VENDOR_BACKUP="$(mktemp -d)"
mv "$DEST/vendor" "$VENDOR_BACKUP/"
fi
rm -rf "$DEST" rm -rf "$DEST"
mkdir -p "$DEST" mkdir -p "$DEST"
cp -R autozhuyin autozhuyin_ibus.py version.txt "$DEST/" cp -R autozhuyin autozhuyin_ibus.py version.txt tools "$DEST/"
[ -f icon.png ] && cp icon.png "$DEST/" for f in icon.png icon-en.png THIRD_PARTY_LICENSES.txt GPL-3.0.txt; do [ -f "$f" ] && cp "$f" "$DEST/"; done
[ -f THIRD_PARTY_LICENSES.txt ] && cp THIRD_PARTY_LICENSES.txt "$DEST/" [ -n "$VENDOR_BACKUP" ] && mv "$VENDOR_BACKUP/vendor" "$DEST/" && rmdir "$VENDOR_BACKUP"
chmod +x "$DEST/autozhuyin_ibus.py" chmod +x "$DEST/autozhuyin_ibus.py"
# 2b. numpy for "AI 選字": use the system's, or download the matching version from PyPI.
TAG="$("$PY" -c 'import sys; print("cp%d%d" % sys.version_info[:2])')"
if "$PY" -c 'import numpy' 2>/dev/null; then
echo "AI 選字:使用系統的 numpy。"
elif [ -d "$DEST/vendor/$TAG/numpy" ]; then
echo "AI 選字:使用先前下載的 numpy。"
else
echo "AI 選字需要 numpy,正在從 PyPI 下載(約 15 MB)…"
if ! "$PY" -I "$DEST/tools/fetch_numpy.py" "$DEST/vendor" auto auto; then
echo "下載 numpy 失敗:輸入法照常可用,只是沒有 AI 選字。之後可以重新執行 install.sh。"
fi
fi
# 3. Start with the desktop session: registers the engine with IBus and selects it. # 3. Start with the desktop session: registers the engine with IBus and selects it.
mkdir -p "$AUTOSTART" mkdir -p "$AUTOSTART"
cat > "$AUTOSTART/autozhuyin.desktop" <<DESKTOP cat > "$AUTOSTART/autozhuyin.desktop" <<DESKTOP
+37 -4
View File
@@ -8,6 +8,9 @@ import types
import unittest import unittest
LINUX = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) LINUX = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
import tempfile # noqa: E402
os.environ["XDG_CONFIG_HOME"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
sys.path.insert(0, os.path.join(os.path.dirname(LINUX), "windows")) # autozhuyin package sys.path.insert(0, os.path.join(os.path.dirname(LINUX), "windows")) # autozhuyin package
sys.path.insert(0, LINUX) sys.path.insert(0, LINUX)
@@ -51,9 +54,11 @@ class LookupTable:
class Property: class Property:
def __init__(self, **kw): self.label = kw["label"].text def __init__(self, **kw): self.key, self.label, self.state = kw["key"], kw["label"].text, kw.get("state", 0)
def set_label(self, t): self.label = t.text def set_label(self, t): self.label = t.text
def set_symbol(self, t): pass def set_symbol(self, t): pass
def set_state(self, s): self.state = s
def set_icon(self, path): self.icon = path
class Engine: class Engine:
@@ -68,7 +73,9 @@ class Engine:
def update_lookup_table(self, table, visible): self.shown = list(table.items) def update_lookup_table(self, table, visible): self.shown = list(table.items)
def hide_lookup_table(self): self.shown = None def hide_lookup_table(self): self.shown = None
def register_properties(self, p): pass def register_properties(self, p): pass
def update_property(self, p): self.mode_label = p.label def update_property(self, p):
if p.key == "mode":
self.mode_label = p.label
IBus.Text, IBus.AttrList, IBus.LookupTable, IBus.Property, IBus.Engine = Text, AttrList, LookupTable, Property, Engine IBus.Text, IBus.AttrList, IBus.LookupTable, IBus.Property, IBus.Engine = Text, AttrList, LookupTable, Property, Engine
@@ -77,8 +84,8 @@ IBus.attr_underline_new = lambda kind, a, b: ("underline", a, b)
IBus.attr_foreground_new = lambda color, a, b: ("color", a, b) IBus.attr_foreground_new = lambda color, a, b: ("color", a, b)
IBus.AttrUnderline = types.SimpleNamespace(SINGLE=1) IBus.AttrUnderline = types.SimpleNamespace(SINGLE=1)
IBus.Orientation = types.SimpleNamespace(VERTICAL=1) IBus.Orientation = types.SimpleNamespace(VERTICAL=1)
IBus.PropType = types.SimpleNamespace(NORMAL=0) IBus.PropType = types.SimpleNamespace(NORMAL=0, TOGGLE=1, RADIO=2, MENU=3)
IBus.PropState = types.SimpleNamespace(UNCHECKED=0) IBus.PropState = types.SimpleNamespace(UNCHECKED=0, CHECKED=1)
IBus.PreeditFocusMode = types.SimpleNamespace(COMMIT=1) IBus.PreeditFocusMode = types.SimpleNamespace(COMMIT=1)
gi = types.ModuleType("gi") gi = types.ModuleType("gi")
@@ -107,6 +114,8 @@ def press(e, keys):
class IBusEngineTests(unittest.TestCase): class IBusEngineTests(unittest.TestCase):
def setUp(self): def setUp(self):
autozhuyin_ibus.shared_classifier().history.clear()
autozhuyin_ibus.shared_classifier().learned.clear()
self.e = autozhuyin_ibus.AutoZhuyinEngine() self.e = autozhuyin_ibus.AutoZhuyinEngine()
def test_mixed_sentence(self): def test_mixed_sentence(self):
@@ -157,6 +166,30 @@ class IBusEngineTests(unittest.TestCase):
press(self.e, "rup fu4←wu0 \n") press(self.e, "rup fu4←wu0 \n")
self.assertEqual(self.e.committed, "今天氣") self.assertEqual(self.e.committed, "今天氣")
def test_picked_candidate_saved(self):
press(self.e, "su3↓")
second = self.e.shown[1]
self.e.do_candidate_clicked(1, 1, 0)
press(self.e, "\n")
path = os.path.join(os.environ["XDG_CONFIG_HOME"], "autozhuyin", "user-history.json")
with open(path, encoding="utf-8") as f:
self.assertIn(second, f.read())
press(self.e, "su3 \n")
self.assertTrue(self.e.committed.endswith(second))
def test_ai_menu(self):
e = self.e
self.assertEqual(e.ai_props["standard"].state, 1)
e.do_property_activate("ai.saver", 1)
self.assertEqual(autozhuyin_ibus.shared_ai().reranker.mode, "saver")
self.assertEqual(e.ai_props["saver"].state, 1)
self.assertEqual(e.ai_props["standard"].state, 0)
e.do_property_activate("ai.autopause", 0)
self.assertFalse(autozhuyin_ibus.shared_ai().auto_pause)
self.assertTrue(e.status_prop.label)
e.do_property_activate("ai.standard", 1)
e.do_property_activate("ai.autopause", 1)
def test_focus_out_resets(self): def test_focus_out_resets(self):
press(self.e, "su3") press(self.e, "su3")
self.e.do_focus_out() self.e.do_focus_out()
+6
View File
@@ -20,6 +20,9 @@ cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/" cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
# "AI 選字" language model (CKIP gpt2-tiny-chinese, GPL-3.0), fetched and converted once.
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null
cp build/lm/lm.bin build/lm/lm-vocab.txt licenses/GPL-3.0.txt "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build). # Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/" cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
@@ -28,6 +31,9 @@ if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
fi fi
cp build/icon.tiff "$APP/Contents/Resources/icon.tiff" cp build/icon.tiff "$APP/Contents/Resources/icon.tiff"
# English-mode icon ("A"), shown in the menu bar while English mode is on.
[ -f build/icon-en.tiff ] || xcrun swift scripts/make-icon.swift build/icon-en.tiff A
cp build/icon-en.tiff "$APP/Contents/Resources/icon-en.tiff"
codesign --force --sign - "$APP" codesign --force --sign - "$APP"
echo "Built $APP" echo "Built $APP"
+18
View File
@@ -0,0 +1,18 @@
#!/bin/bash
# Downloads the CKIP gpt2-tiny-chinese language model (GPL-3.0,
# https://huggingface.co/ckiplab/gpt2-tiny-chinese) and converts it to build/lm/:
# lm.bin float16 weights (custom format, see training/lm/export_lm.py)
# lm-vocab.txt
# build.sh bundles them into the app when present ("AI 選字"). Safe to re-run.
set -euo pipefail
cd "$(dirname "$0")/.."
SRC=build/models/ckip-gpt2-tiny
BASE="https://huggingface.co/ckiplab/gpt2-tiny-chinese/resolve/refs%2Fpr%2F1"
mkdir -p "$SRC" build/lm
for f in model.safetensors vocab.txt config.json; do
[ -s "$SRC/$f" ] || curl -sfL -o "$SRC/$f" "$BASE/$f"
done
python3 -I training/lm/export_lm.py "$SRC" build/lm/lm.bin
cp "$SRC/vocab.txt" build/lm/lm-vocab.txt
ls -lh build/lm
+4 -3
View File
@@ -1,8 +1,9 @@
// Renders the menu-bar icon ("注" with a small "A") to a TIFF. // Renders a menu-bar icon (one character, "注" by default; "A" for English mode) to a TIFF.
// Usage: swift scripts/make-icon.swift <output.tiff> // Usage: swift scripts/make-icon.swift <output.tiff> [character]
import AppKit import AppKit
let out = CommandLine.arguments.count > 1 ? CommandLine.arguments[1] : "icon.tiff" let out = CommandLine.arguments.count > 1 ? CommandLine.arguments[1] : "icon.tiff"
let glyph = CommandLine.arguments.count > 2 ? CommandLine.arguments[2] : "注"
let reps: [NSBitmapImageRep] = [16, 32].map { size in let reps: [NSBitmapImageRep] = [16, 32].map { size in
let rep = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: size, pixelsHigh: size, let rep = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: size, pixelsHigh: size,
bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false, bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false,
@@ -11,7 +12,7 @@ let reps: [NSBitmapImageRep] = [16, 32].map { size in
NSGraphicsContext.saveGraphicsState() NSGraphicsContext.saveGraphicsState()
NSGraphicsContext.current = NSGraphicsContext(bitmapImageRep: rep) NSGraphicsContext.current = NSGraphicsContext(bitmapImageRep: rep)
let font = NSFont.systemFont(ofSize: 13, weight: .semibold) let font = NSFont.systemFont(ofSize: 13, weight: .semibold)
let text = NSAttributedString(string: "注", attributes: [.font: font, .foregroundColor: NSColor.black]) let text = NSAttributedString(string: glyph, attributes: [.font: font, .foregroundColor: NSColor.black])
let s = text.size() let s = text.size()
text.draw(at: NSPoint(x: (16 - s.width) / 2, y: (16 - s.height) / 2)) text.draw(at: NSPoint(x: (16 - s.width) / 2, y: (16 - s.height) / 2))
NSGraphicsContext.restoreGraphicsState() NSGraphicsContext.restoreGraphicsState()
+6 -1
View File
@@ -8,6 +8,7 @@ VERSION="$(python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["v
NAME="AutoZhuyin-Linux-$VERSION" NAME="AutoZhuyin-Linux-$VERSION"
OUT="build/$NAME" OUT="build/$NAME"
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null # "AI 選字" model
python3 -I windows/tools/build_data.py python3 -I windows/tools/build_data.py
(cd windows && python3 -m unittest discover -q tests) (cd windows && python3 -m unittest discover -q tests)
python3 -m unittest discover -q -s linux/tests python3 -m unittest discover -q -s linux/tests
@@ -21,9 +22,13 @@ mkdir -p "$OUT/autozhuyin"
rsync -a --exclude __pycache__ --exclude '*.pyc' \ rsync -a --exclude __pycache__ --exclude '*.pyc' \
windows/autozhuyin/__init__.py windows/autozhuyin/core windows/autozhuyin/data "$OUT/autozhuyin/" windows/autozhuyin/__init__.py windows/autozhuyin/core windows/autozhuyin/data "$OUT/autozhuyin/"
cp linux/autozhuyin_ibus.py linux/installer/install.sh linux/installer/uninstall.sh linux/installer/README.txt \ cp linux/autozhuyin_ibus.py linux/installer/install.sh linux/installer/uninstall.sh linux/installer/README.txt \
THIRD_PARTY_LICENSES.txt "$OUT/" THIRD_PARTY_LICENSES.txt licenses/GPL-3.0.txt "$OUT/"
mkdir -p "$OUT/tools"
cp windows/tools/fetch_numpy.py "$OUT/tools/" # install.sh downloads numpy if the system has none
echo "$VERSION" > "$OUT/version.txt" echo "$VERSION" > "$OUT/version.txt"
sips -s format png -z 64 64 build/icon.tiff --out "$OUT/icon.png" >/dev/null sips -s format png -z 64 64 build/icon.tiff --out "$OUT/icon.png" >/dev/null
[ -f build/icon-en.tiff ] || DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon-en.tiff A
sips -s format png -z 64 64 build/icon-en.tiff --out "$OUT/icon-en.png" >/dev/null
chmod +x "$OUT/install.sh" "$OUT/uninstall.sh" "$OUT/autozhuyin_ibus.py" chmod +x "$OUT/install.sh" "$OUT/uninstall.sh" "$OUT/autozhuyin_ibus.py"
# Plain ustar without macOS extended attributes, so GNU tar doesn't warn. # Plain ustar without macOS extended attributes, so GNU tar doesn't warn.
+13
View File
@@ -8,6 +8,7 @@ VERSION="$(python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["v
NAME="AutoZhuyin-Windows-$VERSION" NAME="AutoZhuyin-Windows-$VERSION"
OUT="build/$NAME" OUT="build/$NAME"
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null # "AI 選字" model
python3 -I windows/tools/build_data.py python3 -I windows/tools/build_data.py
(cd windows && python3 -m unittest discover -q tests) (cd windows && python3 -m unittest discover -q tests)
@@ -17,6 +18,10 @@ if [ ! -f build/icon.tiff ]; then
fi fi
sips -s format png -z 32 32 build/icon.tiff --out build/icon-32.png >/dev/null sips -s format png -z 32 32 build/icon.tiff --out build/icon-32.png >/dev/null
python3 -I windows/tools/png_to_ico.py build/icon-32.png windows/autozhuyin/icon.ico python3 -I windows/tools/png_to_ico.py build/icon-32.png windows/autozhuyin/icon.ico
# English-mode icon ("A") for the taskbar while English mode is on.
[ -f build/icon-en.tiff ] || DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon-en.tiff A
sips -s format png -z 32 32 build/icon-en.tiff --out build/icon-en-32.png >/dev/null
python3 -I windows/tools/png_to_ico.py build/icon-en-32.png windows/autozhuyin/icon-en.ico
rm -rf "$OUT" "$OUT.zip" rm -rf "$OUT" "$OUT.zip"
mkdir -p "$OUT" mkdir -p "$OUT"
@@ -29,6 +34,14 @@ done
# Third-party notices: next to the installer and inside the installed folder. # Third-party notices: next to the installer and inside the installed folder.
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt" sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/" cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
sed 's/$/\r/' licenses/GPL-3.0.txt > "$OUT/autozhuyin/GPL-3.0.txt"
# numpy for "AI 選字", for both Pythons PIME ships (1.3: 3.8, 2.x: 3.15; both 32-bit).
for tag in cp38 cp315; do
[ -d "build/numpy-vendor/$tag" ] || python3 -I windows/tools/fetch_numpy.py build/numpy-vendor "$tag" win32
done
mkdir -p "$OUT/autozhuyin/vendor"
cp -R build/numpy-vendor/cp38 build/numpy-vendor/cp315 "$OUT/autozhuyin/vendor/"
(cd build && zip -qr -X "$NAME.zip" "$NAME") (cd build && zip -qr -X "$NAME.zip" "$NAME")
echo "Built build/$NAME.zip" echo "Built build/$NAME.zip"
+147
View File
@@ -0,0 +1,147 @@
"""Measures how much language-model reranking improves Zhuyin → Chinese conversion.
For each test sentence: turn the text into Dachen keystrokes (via the lexicon's
readings), convert with the current Viterbi converter (baseline), take the K best
paths, and rerank them with lexicon score + λ · LM log-probability.
Usage (needs numpy and the model in build/models/ckip-gpt2-tiny):
python training/lm/eval_rerank.py build/lm-eval/chat.txt build/lm-eval/wiki.txt
"""
import heapq
import math
import os
import sys
import time
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.insert(0, os.path.join(ROOT, "windows"))
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from autozhuyin.core import bopomofo # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
from gpt2 import GPT2 # noqa: E402
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
LN10 = math.log(10)
def reading_keys(reading):
out = []
for syl in reading.split("-"):
tone = " "
for ch in syl:
if ch in TONE_KEY:
tone = TONE_KEY[ch]
else:
out.append(KEY_OF[ch])
out.append(tone)
return out
def build_reverse(lex):
best = {}
for reading, entries in lex.exact.items():
for phrase, score in entries:
if phrase not in best or score > best[phrase][1]:
best[phrase] = (reading, score)
return best
def text_to_keys(text, reverse, max_len):
"""Most likely segmentation of the text into lexicon phrases → keystrokes."""
n = len(text)
best = [(-math.inf, None)] * (n + 1)
best[0] = (0.0, None)
for j in range(1, n + 1):
for i in range(max(0, j - max_len), j):
if best[i][0] == -math.inf:
continue
hit = reverse.get(text[i:j])
if hit:
s = best[i][0] + hit[1]
if s > best[j][0]:
best[j] = (s, (i, hit[0]))
if best[n][1] is None:
return None
readings, j = [], n
while j > 0:
i, r = best[j][1]
readings.append(r)
j = i
keys = []
for r in reversed(readings):
keys += reading_keys(r)
return keys
def kbest(conv, syllables, k):
"""K best (score, text) paths through the converter's lattice (scores in log10)."""
n = len(syllables)
max_len = max(1, conv.lexicon.max_phrase_length)
beams = [[] for _ in range(n + 1)]
beams[0] = [(0.0, "")]
for j in range(1, n + 1):
cand = []
for length in range(1, min(max_len, j) + 1):
i = j - length
if not beams[i]:
continue
for text, score in conv._options(syllables, i, j, {}):
for s, t in beams[i]:
cand.append((s + score, t + text))
beams[j] = heapq.nlargest(k, cand, key=lambda e: e[0])
return beams[n]
def char_errors(a, b):
return sum(x != y for x, y in zip(a, b)) + abs(len(a) - len(b))
def main(paths, k=10):
lex = Lexicon.from_file(os.path.join(ROOT, "windows/autozhuyin/data/lexicon.txt"))
conv = Converter(lex)
lm = GPT2(os.path.join(ROOT, "build/models/ckip-gpt2-tiny"))
reverse = build_reverse(lex)
cases = []
for path in paths:
for line in open(path, encoding="utf-8"):
text = line.strip().replace(",", "").replace("、", "")
if not text:
continue
keys = text_to_keys(text, reverse, lex.max_phrase_length)
if keys is None:
continue
syl = [s.syllable for s in bopomofo.parse(keys).syllables]
paths_k = kbest(conv, syl, k)
t0 = time.time()
lm_scores = [lm.logprob(t) for _, t in paths_k]
cases.append((os.path.basename(path), text, paths_k, lm_scores, time.time() - t0))
def evaluate(lam, subset):
chars = errs = sent_ok = 0
for _, text, paths_k, lm_scores, _ in subset:
best = max(range(len(paths_k)), key=lambda i: paths_k[i][0] * LN10 + lam * lm_scores[i])
out = paths_k[best][1]
e = char_errors(out, text)
chars += len(text)
errs += e
sent_ok += e == 0
return 1 - errs / chars, sent_ok / len(subset)
lams = [0, 0.1, 0.2, 0.3, 0.5, 0.7, 1.0, 1.5, 2.0]
for name in sorted({c[0] for c in cases}):
sub = [c for c in cases if c[0] == name]
oracle = sum(any(t == c[1] for _, t in c[2]) for c in sub) / len(sub)
print(f"\n== {name}: {len(sub)} sentences, top-{k} contains the answer: {oracle:.1%}")
for lam in lams:
ca, sa = evaluate(lam, sub)
print(f" λ={lam:<4} char acc {ca:.2%} sentence acc {sa:.1%}" + (" (baseline)" if lam == 0 else ""))
ms = sum(c[4] for c in cases) / len(cases) * 1000
print(f"\nLM time per sentence (top-{k}): {ms:.1f} ms")
return cases
if __name__ == "__main__":
main(sys.argv[1:])
+48
View File
@@ -0,0 +1,48 @@
"""Converts the gpt2-tiny-chinese safetensors file to the input method's format.
Format (little-endian):
"AZLM" magic, uint32 header length, JSON header, then float16 tensors back to back.
Header: {"n_layer", "n_head", "n_embd", "vocab", "eps", "tensors": [[name, [shape], offset]]}
(offset in float16 elements from the start of the data). Linear weights keep the
GPT-2 Conv1D layout [in, out]. The attention mask buffers are dropped.
Usage: export_lm.py <model dir> <out.bin> (standard library only)
"""
import array
import json
import struct
import sys
def main(src, out):
cfg = json.load(open(f"{src}/config.json"))
with open(f"{src}/model.safetensors", "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(n))
base = 8 + n
names = sorted(k for k, v in header.items() if k != "__metadata__" and v["dtype"] == "F32" and v["shape"])
tensors, blobs, offset = [], [], 0
for name in names:
info = header[name]
a, b = info["data_offsets"]
f.seek(base + a)
floats = array.array("f")
floats.frombytes(f.read(b - a))
if sys.byteorder != "little":
floats.byteswap()
half = struct.pack(f"<{len(floats)}e", *floats)
tensors.append([name.replace("transformer.", ""), info["shape"], offset])
blobs.append(half)
offset += len(floats)
meta = json.dumps({"n_layer": cfg["n_layer"], "n_head": cfg["n_head"], "n_embd": cfg["n_embd"],
"vocab": cfg["vocab_size"], "eps": cfg["layer_norm_epsilon"],
"tensors": tensors}).encode()
with open(out, "wb") as f:
f.write(b"AZLM" + struct.pack("<I", len(meta)) + meta)
for blob in blobs:
f.write(blob)
print(f"wrote {out}: {offset} weights")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
+83
View File
@@ -0,0 +1,83 @@
"""Minimal GPT-2 forward pass in numpy, for the CKIP gpt2-tiny-chinese model
(https://huggingface.co/ckiplab/gpt2-tiny-chinese, GPL-3.0). Character-level:
each Chinese character is one token of the bert-base-chinese vocabulary.
Used to evaluate language-model reranking before porting it to the input methods.
"""
import json
import math
import struct
import numpy as np
CLS, UNK = 101, 100
def load_safetensors(path):
with open(path, "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(n))
base = 8 + n
out = {}
for name, info in header.items():
if name == "__metadata__" or info["dtype"] != "F32":
continue
a, b = info["data_offsets"]
f.seek(base + a)
out[name] = np.frombuffer(f.read(b - a), dtype=np.float32).reshape(info["shape"])
return out
class GPT2:
def __init__(self, model_dir):
self.w = load_safetensors(f"{model_dir}/model.safetensors")
cfg = json.load(open(f"{model_dir}/config.json"))
self.n_layer, self.n_head = cfg["n_layer"], cfg["n_head"]
self.eps = cfg["layer_norm_epsilon"]
self.vocab = {}
for i, line in enumerate(open(f"{model_dir}/vocab.txt", encoding="utf-8")):
self.vocab.setdefault(line.rstrip("\n"), i)
def ids(self, text):
return [self.vocab.get(c, UNK) for c in text]
def _ln(self, x, name):
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + self.eps) * self.w[name + ".weight"] + self.w[name + ".bias"]
def hidden(self, ids):
"""Final hidden states [T, d] for token ids."""
w, T = self.w, len(ids)
x = w["transformer.wte.weight"][ids] + w["transformer.wpe.weight"][:T]
mask = np.triu(np.full((T, T), -1e9, dtype=np.float32), 1)
for l in range(self.n_layer):
p = f"transformer.h.{l}."
h = self._ln(x, p + "ln_1")
qkv = h @ w[p + "attn.c_attn.weight"] + w[p + "attn.c_attn.bias"]
q, k, v = np.split(qkv, 3, axis=-1)
d = q.shape[-1] // self.n_head
q = q.reshape(T, self.n_head, d).transpose(1, 0, 2)
k = k.reshape(T, self.n_head, d).transpose(1, 0, 2)
v = v.reshape(T, self.n_head, d).transpose(1, 0, 2)
att = q @ k.transpose(0, 2, 1) / math.sqrt(d) + mask
att = np.exp(att - att.max(-1, keepdims=True))
att /= att.sum(-1, keepdims=True)
a = (att @ v).transpose(1, 0, 2).reshape(T, -1)
x = x + a @ w[p + "attn.c_proj.weight"] + w[p + "attn.c_proj.bias"]
h = self._ln(x, p + "ln_2")
h = h @ w[p + "mlp.c_fc.weight"] + w[p + "mlp.c_fc.bias"]
h = 0.5 * h * (1 + np.tanh(math.sqrt(2 / math.pi) * (h + 0.044715 * h ** 3))) # gelu_new
x = x + h @ w[p + "mlp.c_proj.weight"] + w[p + "mlp.c_proj.bias"]
return self._ln(x, "transformer.ln_f")
def logprob(self, text, context=""):
"""Natural-log probability of `text` given `context` (both strings)."""
ctx, tgt = self.ids(context), self.ids(text)
ids = [CLS] + ctx + tgt
h = self.hidden(ids[:-1])
logits = h @ self.w["transformer.wte.weight"].T
logits -= logits.max(-1, keepdims=True)
lse = np.log(np.exp(logits).sum(-1))
start = len(ctx)
return float(sum(logits[start + i, t] - lse[start + i] for i, t in enumerate(tgt)))
@@ -0,0 +1 @@
{"ㄋㄧ": {"妳": [1, 1]}}
+110 -15
View File
@@ -6,11 +6,21 @@ tests); this file only translates PIME key events and replies.
""" """
import json import json
import os import os
import sys
import time import time
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME) # numpy for "AI 選字" is bundled per Python version (PIME 1.3 has 3.8, PIME 2 has 3.15).
from textService import TextService _VENDOR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "vendor",
"cp%d%d" % sys.version_info[:2])
if os.path.isdir(_VENDOR) and _VENDOR not in sys.path:
sys.path.insert(0, _VENDOR)
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
from textService import COMMAND_LEFT_CLICK, COMMAND_MENU, TextService # noqa: F401
from .core.ai import AI, MODE_NAMES
from .core.history import UserHistory
from .core.reranker import MODES
from .core.classifier import Classifier, LearnedPreferences from .core.classifier import Classifier, LearnedPreferences
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
from .core.english import EnglishModel from .core.english import EnglishModel
@@ -22,17 +32,40 @@ _DATA = os.path.join(_HERE, "data")
_USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin") _USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin")
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json") _LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt") _USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
_classifier = None _classifier = None
_ai = None
ID_MODE_ICON, ID_AUTO_PAUSE, ID_STATUS, ID_CLEAR = 1, 2, 3, 4
ID_AI_MODE = {mode: 10 + i for i, mode in enumerate(MODES)} # 10..13
_ICON = os.path.join(_HERE, "icon.ico")
_ICON_EN = os.path.join(_HERE, "icon-en.ico") # shown while English mode is on
def _save_json(path, data):
try:
os.makedirs(_USER_DIR, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False)
except OSError:
pass
def _load_json(path):
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError):
return {}
def _save_learned(weights): def _save_learned(weights):
try: _save_json(_LEARNED_FILE, weights)
os.makedirs(_USER_DIR, exist_ok=True)
with open(_LEARNED_FILE, "w", encoding="utf-8") as f:
json.dump(weights, f, ensure_ascii=False) def _save_history(data):
except OSError: _save_json(_HISTORY_FILE, data)
pass
def _shared_classifier(): def _shared_classifier():
@@ -48,17 +81,20 @@ def _shared_classifier():
os.path.join(_DATA, "english-dict.txt"), os.path.join(_DATA, "english-dict.txt"),
os.path.join(_DATA, "english-bigram.json"), os.path.join(_DATA, "english-bigram.json"),
) )
weights = {}
try:
with open(_LEARNED_FILE, encoding="utf-8") as f:
weights = json.load(f)
except (OSError, ValueError):
pass
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json")) trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained) _classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
return _classifier return _classifier
def _shared_ai():
""""AI 選字": settings and the language model, loaded in the background once."""
global _ai
if _ai is None:
_ai = AI(_DATA, _USER_DIR)
return _ai
_SPECIAL = { _SPECIAL = {
VK_BACK: BACKSPACE, VK_RETURN: ENTER, VK_ESCAPE: ESCAPE, VK_TAB: TAB, # noqa: F405 VK_BACK: BACKSPACE, VK_RETURN: ENTER, VK_ESCAPE: ESCAPE, VK_TAB: TAB, # noqa: F405
VK_UP: UP, VK_DOWN: DOWN, VK_LEFT: LEFT, VK_RIGHT: RIGHT, # noqa: F405 VK_UP: UP, VK_DOWN: DOWN, VK_LEFT: LEFT, VK_RIGHT: RIGHT, # noqa: F405
@@ -70,6 +106,7 @@ class AutoZhuyinTextService(TextService):
TextService.__init__(self, client) TextService.__init__(self, client)
self.engine = CompositionEngine(_shared_classifier()) self.engine = CompositionEngine(_shared_classifier())
self.shift_tap = ShiftTapDetector() self.shift_tap = ShiftTapDetector()
self.engine.reranker = _shared_ai().reranker
# --- lifecycle --- # --- lifecycle ---
@@ -77,11 +114,68 @@ class AutoZhuyinTextService(TextService):
TextService.onActivate(self) TextService.onActivate(self)
self.setSelKeys("123456789") self.setSelKeys("123456789")
self.customizeUI(candFontSize=16, candPerRow=1, candUseCursor=True) self.customizeUI(candFontSize=16, candPerRow=1, candUseCursor=True)
# Taskbar IME icon (Windows 8+) and language bar button; both open the settings menu.
self._icon_english = self.engine.english_mode
self.addButton("windows-mode-icon", icon=self._mode_icon(), tooltip=self._mode_tooltip(),
commandId=ID_MODE_ICON)
self.addButton("autozhuyin-settings", icon=_ICON, tooltip="自動注音設定", type="menu")
def onDeactivate(self): def onDeactivate(self):
self.engine.reset() self.engine.reset()
self.removeButton("windows-mode-icon")
self.removeButton("autozhuyin-settings")
TextService.onDeactivate(self) TextService.onDeactivate(self)
# --- taskbar icon: "注" for 中英自動, "A" for English mode ---
def _mode_icon(self):
return _ICON_EN if self.engine.english_mode and os.path.exists(_ICON_EN) else _ICON
def _mode_tooltip(self):
return ("英文模式" if self.engine.english_mode else "中英自動") + "(左鍵切換,右鍵設定)"
def _update_mode_icon(self):
if getattr(self, "_icon_english", None) == self.engine.english_mode:
return
self._icon_english = self.engine.english_mode
self.changeButton("windows-mode-icon", icon=self._mode_icon(), tooltip=self._mode_tooltip())
# --- menu (AI 選字 settings) ---
def onMenu(self, buttonId):
if buttonId not in ("windows-mode-icon", "autozhuyin-settings"):
return None
ai = _shared_ai()
items = [{"text": "AI 選字:" + MODE_NAMES[m], "id": ID_AI_MODE[m], "checked": ai.reranker.mode == m}
for m in MODES]
items += [
{},
{"text": "系統忙碌時自動暫停 AI", "id": ID_AUTO_PAUSE, "checked": ai.auto_pause},
{"text": ai.status_text(), "id": ID_STATUS, "enabled": False},
{},
{"text": "清除學習紀錄", "id": ID_CLEAR},
]
return items
def onCommand(self, commandId, commandType):
ai = _shared_ai()
if commandId == ID_MODE_ICON and commandType == COMMAND_LEFT_CLICK and not self.engine.keys:
self.engine.handle(SHIFT_TAP) # left click: English mode on / off
self._update_mode_icon()
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
return
for mode, cid in ID_AI_MODE.items():
if commandId == cid:
ai.set_mode(mode)
self.showMessage("AI 選字:" + MODE_NAMES[mode], 1)
return
if commandId == ID_AUTO_PAUSE:
ai.set_auto_pause(not ai.auto_pause)
elif commandId == ID_CLEAR:
_shared_classifier().learned.clear()
_shared_classifier().history.clear()
self.showMessage("已清除學習紀錄", 1)
def onCompositionTerminated(self, forced): def onCompositionTerminated(self, forced):
# The app ended the composition (focus change, mouse click): the shown text stays. # The app ended the composition (focus change, mouse click): the shown text stays.
self.engine.reset() self.engine.reset()
@@ -142,6 +236,7 @@ class AutoZhuyinTextService(TextService):
return False return False
self.engine.handle(SHIFT_TAP) self.engine.handle(SHIFT_TAP)
self._update(None) self._update(None)
self._update_mode_icon()
if not self.engine.keys: if not self.engine.keys:
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1) self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
return True return True
+108
View File
@@ -0,0 +1,108 @@
""""AI 選字" for the Windows and Linux shells: settings, background model loading,
system-load detection and the status line shown in the menu."""
import json
import os
import threading
from . import lm
from .reranker import (MODES, NORMAL, OFF, ST_ACTIVE, ST_BUSY, ST_LOADING, ST_OFF, ST_SLOW,
STANDARD, Reranker)
from .system_monitor import REDUCED, SystemMonitor
MODE_NAMES = {OFF: "關閉", "saver": "省電", STANDARD: "標準", "quality": "高品質"}
class AI:
def __init__(self, data_dir, user_dir, monitor=None, background=True):
self.model_path = os.path.join(data_dir, "lm.bin")
self.vocab_path = os.path.join(data_dir, "lm-vocab.txt")
self.settings_path = os.path.join(user_dir, "settings.json")
self.monitor = monitor or SystemMonitor()
self.background = background
settings = self._load_settings()
mode = settings.get("ai_mode", STANDARD)
self.auto_pause = bool(settings.get("ai_auto_pause", True))
self.reranker = Reranker(mode if mode in MODES else STANDARD)
self.reranker.system_state = lambda: self.monitor.state() if self.auto_pause else NORMAL
self.error = None # why the model cannot be used, if it cannot
self._loading = False
self.load_model()
# --- settings ---
def _load_settings(self):
try:
with open(self.settings_path, encoding="utf-8") as f:
data = json.load(f)
return data if isinstance(data, dict) else {}
except (OSError, ValueError):
return {}
def _save_settings(self):
data = self._load_settings()
data.update(ai_mode=self.reranker.mode, ai_auto_pause=self.auto_pause)
try:
os.makedirs(os.path.dirname(self.settings_path), exist_ok=True)
with open(self.settings_path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=1)
except OSError:
pass
def set_mode(self, mode):
if mode not in MODES:
return
self.reranker.mode = mode
if mode == OFF:
self.reranker.model = None # free the memory
else:
self.load_model()
self._save_settings()
def set_auto_pause(self, on):
self.auto_pause = bool(on)
self._save_settings()
# --- model ---
def load_model(self):
if self.reranker.mode == OFF or self.reranker.model is not None or self._loading or self.error:
return
if not lm.available():
self.error = "需要 numpy"
return
if not (os.path.exists(self.model_path) and os.path.exists(self.vocab_path)):
self.error = "AI 模型未安裝"
return
self._loading = True
def work():
try:
model = lm.GPT2LanguageModel(self.model_path, self.vocab_path)
except Exception as e: # noqa: BLE001 — any failure just disables AI
self.error = f"AI 模型載入失敗:{e}"
model = None
self._loading = False
if self.reranker.mode != OFF:
self.reranker.model = model
if self.background:
threading.Thread(target=work, daemon=True).start()
else:
work()
def status_text(self):
if self.error:
return self.error
self.reranker.effective_mode() # refresh the status
st = self.reranker.status
if st == ST_OFF:
return "AI 選字已關閉"
if st == ST_LOADING:
return "AI 模型載入中…"
if st == ST_BUSY:
return "AI 已暫停:系統忙碌或過熱"
if st == ST_SLOW:
return "AI 已暫停:運算太慢,一分鐘後再試"
if st == ST_ACTIVE and self.auto_pause and self.monitor.state() == REDUCED:
return "AI 使用中(省電模式)"
return "AI 使用中"
+3 -1
View File
@@ -1,6 +1,7 @@
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift.""" """Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
import json import json
from .history import UserHistory
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
ZHUYIN = "zhuyin" ZHUYIN = "zhuyin"
@@ -31,10 +32,11 @@ class LearnedPreferences:
class Classifier: class Classifier:
def __init__(self, lexicon, english, learned=None, weights=None): def __init__(self, lexicon, english, learned=None, weights=None, history=None):
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned or LearnedPreferences() self.learned = learned or LearnedPreferences()
self.history = history or UserHistory() # picked candidates (shared by all engines)
self.weights = dict(DEFAULT_WEIGHTS) self.weights = dict(DEFAULT_WEIGHTS)
if weights: if weights:
self.weights.update(weights) self.weights.update(weights)
+60 -12
View File
@@ -8,6 +8,7 @@ import time
from . import bopomofo from . import bopomofo
from .classifier import LITERAL, ZHUYIN from .classifier import LITERAL, ZHUYIN
from .features import dot
from .lexicon import Converter from .lexicon import Converter
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>" BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
@@ -20,8 +21,10 @@ FULLWIDTH = {
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
} }
# An English word that is also a common one-syllable Chinese reading ("up " = ㄧㄣ 因) # The first word of a composition that also reads as Zhuyin is not committed as English
# is not committed on space; the next word decides ("up jo4" = 因為, "up the" = up the). # on space; the next word decides ("up jo4" = 因為, "up the" = up the, "no su3" = no 你).
# Right after English text only common one-syllable readings are held (up = 因, url = 教),
# so normal English typing is not delayed.
HOLD_MIN_SCORE = -3.5 HOLD_MIN_SCORE = -3.5
# "`" then one of these keys always gives the full-width symbol, whatever the context. # "`" then one of these keys always gives the full-width symbol, whatever the context.
@@ -43,7 +46,7 @@ class CompositionEngine:
def __init__(self, classifier, now=time.monotonic): def __init__(self, classifier, now=time.monotonic):
self.classifier = classifier self.classifier = classifier
self.converter = Converter(classifier.lexicon) self.converter = Converter(classifier.lexicon, classifier.history)
self.now = now self.now = now
self.keys = [] self.keys = []
self.forced_mode = None self.forced_mode = None
@@ -62,6 +65,8 @@ class CompositionEngine:
self.inserting = [] self.inserting = []
# Index of a held space (see HOLD_MIN_SCORE), or None. # Index of a held space (see HOLD_MIN_SCORE), or None.
self.held = None self.held = None
self.reranker = None # "AI 選字" (reranker.Reranker); None = lexicon only
self._recent_text = "" # last characters committed: context for the reranker
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None self._cache_key = None
@@ -94,19 +99,39 @@ class CompositionEngine:
rest = keys[held + 1:] rest = keys[held + 1:]
base = held + 1 base = held + 1
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)] rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
if rest_segs and rest_segs[0][2] == ZHUYIN: if rest_segs and rest_segs[0][2] == ZHUYIN and self._held_reads_as_chinese(keys, held, rest_segs[0][1], finishing):
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:] return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
return [(0, base, LITERAL)] + rest_segs return [(0, base, LITERAL)] + rest_segs
def _held_reads_as_chinese(self, keys, held, end, finishing):
"""Chinese all the way to `end` vs. the held English word followed by Chinese."""
c, w, ctx = self.classifier, self.classifier.weights, self.context
is_last = end == len(keys)
zh_whole = c.segment_features(keys[:end], is_last, False, finishing, ctx)[0]
word = c.segment_features(keys[:held], False, False, True, ctx)[1]
zh_rest = c.segment_features(keys[held + 1:end], is_last, True, finishing, None)[0]
if zh_whole is None:
return False
if word is None or zh_rest is None:
return True
return dot(w, zh_whole) >= dot(w, word) + dot(w, zh_rest)
def _is_ambiguous_word(self, keys): def _is_ambiguous_word(self, keys):
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE).""" """Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
if not all(k.isascii() and k.isalpha() for k in keys): if not all(k.isascii() and k.isalpha() for k in keys):
return False return False
p = bopomofo.parse(list(keys) + [" "]) p = bopomofo.parse(list(keys) + [" "])
if p.pending is not None or len(p.syllables) != 1: if p.pending is not None or not p.syllables or p.stray_tones or p.foreign_keys:
return False return False
entries = self.classifier.lexicon.entries([s.syllable for s in p.syllables]) best = []
return bool(entries) and entries[0][1] >= HOLD_MIN_SCORE for ps in p.syllables:
entries = self.classifier.lexicon.entries([ps.syllable])
if not entries:
return False
best.append(entries[0][1])
if self.context != LITERAL:
return True # first word: hold anything that reads as Zhuyin
return len(best) == 1 and best[0] >= HOLD_MIN_SCORE
@property @property
def segments(self): def segments(self):
@@ -136,8 +161,15 @@ class CompositionEngine:
def _conversion(self, seg, final): def _conversion(self, seg, final):
syl, pending = self._convertible(seg, final) syl, pending = self._convertible(seg, final)
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)} fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
if self.reranker is not None:
chosen = self.reranker.choose(self.converter, syl, fx, self._rerank_context())
if chosen is not None:
return chosen, pending
return self.converter.convert(syl, fx), pending return self.converter.convert(syl, fx), pending
def _rerank_context(self):
return self._recent_text if self.now() - self._previous_time < self.context_window else ""
def _render(self, seg, final): def _render(self, seg, final):
start, end, mode = seg start, end, mode = seg
if mode == LITERAL: if mode == LITERAL:
@@ -152,14 +184,17 @@ class CompositionEngine:
return [seg[0] + ps.end for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables] return [seg[0] + ps.end for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables]
def _cursor_positions(self): def _cursor_positions(self):
"""Key indices the cursor can stop at: the start, after every Zhuyin syllable, """Key indices the cursor can stop at: after every Zhuyin syllable and at every
and at every segment boundary.""" segment boundary, but never before the first Chinese character (there would be
out = {0, len(self.keys)} nothing to pick candidates for)."""
out, ends = {len(self.keys)}, []
for seg in self.segments: for seg in self.segments:
out.add(seg[1]) out.add(seg[1])
if seg[2] == ZHUYIN: if seg[2] == ZHUYIN:
out.update(self._syllable_ends(seg)) ends += self._syllable_ends(seg)
return sorted(out) out.update(ends)
first = min(ends) if ends else len(self.keys)
return sorted(p for p in out if p >= first)
def _stop_at(self, pos): def _stop_at(self, pos):
"""(segment index, syllables before pos) for the Zhuyin segment pos is in.""" """(segment index, syllables before pos) for the Zhuyin segment pos is in."""
@@ -529,6 +564,8 @@ class CompositionEngine:
def _commit(self, segs=None): def _commit(self, segs=None):
segs = segs if segs is not None else self.segments segs = segs if segs is not None else self.segments
text = "".join(self._render(s, True) for s in segs) text = "".join(self._render(s, True) for s in segs)
self._remember_picks(segs)
self._recent_text = (self._rerank_context() + text)[-20:]
if self.forced_mode: if self.forced_mode:
self.classifier.learned.record("".join(self.keys), self.forced_mode) self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = segs[-1][2] if segs else None self._previous_mode = segs[-1][2] if segs else None
@@ -536,6 +573,17 @@ class CompositionEngine:
self.reset() self.reset()
return text return text
def _remember_picks(self, segs):
"""Candidates the user picked are remembered and come first next time."""
for i, seg in enumerate(segs):
picks = self.fixed.get(seg[0])
if seg[2] != ZHUYIN or not picks:
continue
syl = self._convertible(seg, True)[0]
for (a, b), text in picks.items():
if b <= len(syl):
self.classifier.history.record(syl[a:b], text)
def commit_all(self): def commit_all(self):
return self._commit() if self.keys else "" return self._commit() if self.keys else ""
+65
View File
@@ -0,0 +1,65 @@
"""Remembers the candidates the user picked. Port of UserHistory.swift.
Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
where the lexicon has it for the reading being converted, so a toneless key is safe.
"""
class UserHistory:
max_entries = 2000 # (reading, phrase) pairs kept; the least recently used go first
max_count = 10
def __init__(self, data=None, on_change=None):
# reading -> {phrase: [count, last use]}
self.data = {}
self.on_change = on_change or (lambda data: None)
self._clock = 0
for reading, phrases in (data or {}).items():
if not isinstance(phrases, dict):
continue
for phrase, value in phrases.items():
try:
count, used = int(value[0]), int(value[1])
except (TypeError, ValueError, IndexError):
continue
self.data.setdefault(reading, {})[phrase] = [count, used]
self._clock = max(self._clock, used)
@staticmethod
def key(syllables):
return "-".join(s.toneless for s in syllables)
def record(self, syllables, phrase):
if not syllables or not phrase:
return
self._clock += 1
entry = self.data.setdefault(self.key(syllables), {}).setdefault(phrase, [0, 0])
entry[0] = min(self.max_count, entry[0] + 1)
entry[1] = self._clock
self._evict()
self.on_change(self.data)
def _evict(self):
pairs = [(v[1], r, p) for r, ps in self.data.items() for p, v in ps.items()]
if len(pairs) <= self.max_entries:
return
pairs.sort()
for _, r, p in pairs[:len(pairs) - self.max_entries]:
del self.data[r][p]
if not self.data[r]:
del self.data[r]
def clear(self):
self.data = {}
self.on_change(self.data)
def adjust(self, syllables, entries):
"""Lexicon entries [(phrase, score)] with picked phrases moved to the top: just above
the best entry, more so the more often they were picked. Returns a new list."""
picked = self.data.get(self.key(syllables))
if not picked or not entries:
return entries
top = entries[0][1]
out = [(p, max(s, top + 0.01 + 0.1 * picked[p][0]) if p in picked else s) for p, s in entries]
out.sort(key=lambda e: -e[1])
return out
+30 -3
View File
@@ -84,8 +84,13 @@ TONELESS_PENALTY = -1.5
class Converter: class Converter:
def __init__(self, lexicon): def __init__(self, lexicon, history=None):
self.lexicon = lexicon self.lexicon = lexicon
self.history = history # UserHistory: picked candidates come first
def _entries(self, syllables):
entries = self.lexicon.entries(syllables)
return self.history.adjust(syllables, entries) if self.history else entries
def _options(self, syllables, i, j, fixed): def _options(self, syllables, i, j, fixed):
if (i, j) in fixed: if (i, j) in fixed:
@@ -94,7 +99,7 @@ class Converter:
return [] return []
sl = syllables[i:j] sl = syllables[i:j]
penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0 penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0
out = [(p, s + penalty) for p, s in self.lexicon.entries(sl)] out = [(p, s + penalty) for p, s in self._entries(sl)]
if not out and j - i == 1: if not out and j - i == 1:
out = [(sl[0].reading, UNKNOWN_SCORE)] out = [(sl[0].reading, UNKNOWN_SCORE)]
return out return out
@@ -125,6 +130,28 @@ class Converter:
j = seg[0][0] j = seg[0][0]
return segs[::-1] return segs[::-1]
def kbest(self, syllables, k, fixed=None):
"""The k best paths [(log10 score, [((start, end), text)])], best first."""
fixed = fixed or {}
n = len(syllables)
if n == 0 or k <= 0:
return []
max_len = max(1, self.lexicon.max_phrase_length)
beams = [[] for _ in range(n + 1)]
beams[0] = [(0.0, [])]
for j in range(1, n + 1):
cand = []
for length in range(1, min(max_len, j) + 1):
i = j - length
if not beams[i]:
continue
for text, score in self._options(syllables, i, j, fixed):
for s, segs in beams[i]:
cand.append((s + score, segs + [((i, j), text)]))
cand.sort(key=lambda e: -e[0])
beams[j] = cand[:k]
return beams[n]
def candidates(self, syllables, end): def candidates(self, syllables, end):
"""[((start, end), text)] for phrases ending at syllable `end`, longest first.""" """[((start, end), text)] for phrases ending at syllable `end`, longest first."""
if end <= 0 or end > len(syllables): if end <= 0 or end > len(syllables):
@@ -132,7 +159,7 @@ class Converter:
out, seen = [], set() out, seen = [], set()
for length in range(min(self.lexicon.max_phrase_length, end), 0, -1): for length in range(min(self.lexicon.max_phrase_length, end), 0, -1):
i = end - length i = end - length
for phrase, _ in self.lexicon.entries(syllables[i:end]): for phrase, _ in self._entries(syllables[i:end]):
if phrase not in seen: if phrase not in seen:
seen.add(phrase) seen.add(phrase)
out.append(((i, end), phrase)) out.append(((i, end), phrase))
+109
View File
@@ -0,0 +1,109 @@
"""GPT-2 language model for "AI 選字" (numpy). Port of LanguageModel.swift.
Reads the "AZLM" file made by training/lm/export_lm.py (CKIP gpt2-tiny-chinese, GPL-3.0,
float16 weights, expanded to float32 in memory). numpy is optional: when it cannot be
imported, `available()` is False and the input method works without AI.
"""
import json
import math
import struct
try:
import numpy as np
except Exception: # ImportError, or a numpy build for another Python / architecture
np = None
CLS, UNK = 101, 100
def available():
return np is not None
class GPT2LanguageModel:
max_tokens = 64
def __init__(self, model_path, vocab_path):
if np is None:
raise RuntimeError("numpy is not available")
with open(model_path, "rb") as f:
raw = f.read()
if raw[:4] != b"AZLM":
raise ValueError("not an AZLM file")
n = struct.unpack("<I", raw[4:8])[0]
header = json.loads(raw[8:8 + n])
data = np.frombuffer(raw, dtype="<f2", offset=8 + n).astype(np.float32)
self.n_layer, self.n_head = header["n_layer"], header["n_head"]
self.d, self.eps = header["n_embd"], header["eps"]
self.w = {}
for name, shape, offset in header["tensors"]:
size = int(np.prod(shape))
self.w[name] = data[offset:offset + size].reshape(shape)
self.vocab = {}
with open(vocab_path, encoding="utf-8") as f:
for i, line in enumerate(f):
token = line.rstrip("\n")
if len(token) == 1:
self.vocab.setdefault(token, i)
def ids(self, text):
return [self.vocab.get(c, UNK) for c in text]
def _ln(self, x, name):
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + self.eps) * self.w[name + ".weight"] + self.w[name + ".bias"]
def _hidden(self, ids):
"""ids [B, T] → final hidden states [B, T, d]."""
w = self.w
B, T = ids.shape
x = w["wte.weight"][ids] + w["wpe.weight"][:T]
mask = np.triu(np.full((T, T), -1e9, dtype=np.float32), 1)
hd = self.d // self.n_head
for l in range(self.n_layer):
p = f"h.{l}."
h = self._ln(x, p + "ln_1")
qkv = h @ w[p + "attn.c_attn.weight"] + w[p + "attn.c_attn.bias"]
q, k, v = np.split(qkv, 3, axis=-1)
q = q.reshape(B, T, self.n_head, hd).transpose(0, 2, 1, 3)
k = k.reshape(B, T, self.n_head, hd).transpose(0, 2, 3, 1)
v = v.reshape(B, T, self.n_head, hd).transpose(0, 2, 1, 3)
att = q @ k / math.sqrt(hd) + mask
att = np.exp(att - att.max(-1, keepdims=True))
att /= att.sum(-1, keepdims=True)
a = (att @ v).transpose(0, 2, 1, 3).reshape(B, T, self.d)
x = x + a @ w[p + "attn.c_proj.weight"] + w[p + "attn.c_proj.bias"]
h = self._ln(x, p + "ln_2")
h = h @ w[p + "mlp.c_fc.weight"] + w[p + "mlp.c_fc.bias"]
h = 0.5 * h * (1 + np.tanh(math.sqrt(2 / math.pi) * (h + 0.044715 * h ** 3))) # gelu_new
x = x + h @ w[p + "mlp.c_proj.weight"] + w[p + "mlp.c_proj.bias"]
return self._ln(x, "ln_f")
def log_probs(self, texts, context=""):
"""Natural-log probabilities of several texts after the same context. Texts of the
same length are scored together in one batch."""
out = [0.0] * len(texts)
by_len = {}
for i, t in enumerate(texts):
if t:
by_len.setdefault(len(t), []).append(i)
for length, idx in by_len.items():
ctx = self.ids(context)
if len(ctx) + length + 1 > self.max_tokens:
ctx = ctx[max(0, len(ctx) - (self.max_tokens - length - 1)):]
targets = np.array([self.ids(texts[i]) for i in idx]) # [B, L]
prefix = np.array([CLS] + ctx)
inputs = np.concatenate([np.tile(prefix, (len(idx), 1)), targets[:, :-1]], axis=1)
h = self._hidden(inputs)[:, len(ctx):, :] # rows predicting targets
logits = h @ self.w["wte.weight"].T # [B, L, V]
m = logits.max(-1, keepdims=True)
lse = np.log(np.exp(logits - m).sum(-1)) + m[..., 0]
picked = np.take_along_axis(logits, targets[..., None], -1)[..., 0]
scores = (picked - lse).sum(-1)
for j, i in enumerate(idx):
out[i] = float(scores[j])
return out
def log_prob(self, text, context=""):
return self.log_probs([text], context)[0]
+102
View File
@@ -0,0 +1,102 @@
""""AI 選字": reranks the converter's best paths with a language model. Port of Reranker.swift.
score = lexicon score (log10 → ln) + λ · LM log-probability
Typing is never blocked for long: a rerank that runs over its time budget counts as an
overrun, and three in a row pause AI for a minute. `system_state` lets the shell pause
or reduce it when the machine is busy, hot or saving power.
"""
import math
import time
OFF, SAVER, STANDARD, QUALITY = "off", "saver", "standard", "quality"
MODES = (OFF, SAVER, STANDARD, QUALITY)
# Paths reranked, and the time budget (seconds) of one rerank. numpy is slower than
# the Mac's Accelerate code, so the budgets are larger than in Reranker.swift.
K = {SAVER: 3, STANDARD: 10, QUALITY: 30}
BUDGET = {SAVER: 0.04, STANDARD: 0.08, QUALITY: 0.2}
NORMAL, REDUCED, PAUSED = "normal", "reduced", "paused" # system states
ST_OFF, ST_LOADING, ST_ACTIVE, ST_BUSY, ST_SLOW = "off", "loading", "active", "paused-busy", "paused-slow"
PAUSE_AFTER_OVERRUNS = 3
SLOW_PAUSE = 60.0
CACHE_LIMIT = 512
class Reranker:
def __init__(self, mode=STANDARD, model=None, now=time.monotonic):
self._mode = mode
self._model = model
self.lam = 0.6
self.system_state = lambda: NORMAL
self.now = now
self.status = ST_OFF
self._overruns = 0
self._paused_until = float("-inf")
self._cache = {}
@property
def mode(self):
return self._mode
@mode.setter
def mode(self, value):
self._mode = value
self._cache.clear()
@property
def model(self):
return self._model
@model.setter
def model(self, value):
self._model = value
self._cache.clear()
def effective_mode(self):
"""The mode used now (after the system state), or None when AI is not used."""
if self._mode == OFF:
self.status = ST_OFF
return None
if self._model is None:
self.status = ST_LOADING
return None
if self.now() < self._paused_until:
self.status = ST_SLOW
return None
state = self.system_state()
if state == PAUSED:
self.status = ST_BUSY
return None
self.status = ST_ACTIVE
return SAVER if state == REDUCED else self._mode
def choose(self, converter, syllables, fixed, context):
"""The reranked conversion [((start, end), text)], or None for the plain result."""
mode = self.effective_mode()
if mode is None or not syllables:
return None
key = (mode, context, tuple(s.reading for s in syllables), tuple(sorted(fixed.items())))
if key in self._cache:
return self._cache[key]
paths = converter.kbest(syllables, K[mode], fixed)
if len(paths) <= 1:
return paths[0][1] if paths else None
start = self.now()
lm = self._model.log_probs(["".join(t for _, t in segs) for _, segs in paths], context)
best = max(range(len(paths)), key=lambda i: paths[i][0] * math.log(10) + self.lam * lm[i])
if self.now() - start > BUDGET[mode]:
# Too slow: use this result (it is computed) but don't cache it, and pause AI
# after a few overruns in a row.
self._overruns += 1
if self._overruns >= PAUSE_AFTER_OVERRUNS:
self._paused_until = self.now() + SLOW_PAUSE
self._overruns = 0
self.status = ST_SLOW
return paths[best][1]
self._overruns = 0
if len(self._cache) >= CACHE_LIMIT:
self._cache.clear()
self._cache[key] = paths[best][1]
return paths[best][1]
+95
View File
@@ -0,0 +1,95 @@
"""Decides when "AI 選字" should step back (Windows and Linux; the Mac app has its own).
paused — CPU busy (over 90%), or the machine is very hot (Linux thermal zones ≥ 95 °C)
reduced — power saving (Windows battery saver; Linux low-power profile or battery ≤ 20%)
normal — otherwise. Checked at most every 2 seconds.
"""
import glob
import os
import sys
import time
from .reranker import NORMAL, PAUSED, REDUCED
class SystemMonitor:
interval = 2.0
busy_load = 0.9
def __init__(self, now=time.monotonic):
self.now = now
self._cached = (float("-inf"), NORMAL)
self._cpu_prev = None
def state(self):
t, s = self._cached
if self.now() - t < self.interval:
return s
try:
s = self._compute()
except Exception:
s = NORMAL
self._cached = (self.now(), s)
return s
def _compute(self):
if sys.platform == "win32":
return self._windows()
return self._linux()
# --- Linux ---
def _linux(self):
if os.getloadavg()[0] / max(1, os.cpu_count() or 1) > self.busy_load:
return PAUSED
temps = []
for path in glob.glob("/sys/class/thermal/thermal_zone*/temp"):
try:
temps.append(int(open(path).read().strip()) / 1000)
except (OSError, ValueError):
pass
if temps and max(temps) >= 95:
return PAUSED
try:
if open("/sys/firmware/acpi/platform_profile").read().strip() in ("low-power", "quiet"):
return REDUCED
except OSError:
pass
for bat in glob.glob("/sys/class/power_supply/BAT*"):
try:
status = open(bat + "/status").read().strip()
capacity = int(open(bat + "/capacity").read().strip())
except (OSError, ValueError):
continue
if status == "Discharging" and capacity <= 20:
return REDUCED
return NORMAL
# --- Windows ---
def _windows(self):
import ctypes
from ctypes import wintypes
class FILETIME(ctypes.Structure):
_fields_ = [("lo", wintypes.DWORD), ("hi", wintypes.DWORD)]
idle, kernel, user = FILETIME(), FILETIME(), FILETIME()
ctypes.windll.kernel32.GetSystemTimes(ctypes.byref(idle), ctypes.byref(kernel), ctypes.byref(user))
values = [(f.hi << 32) | f.lo for f in (idle, kernel, user)]
prev, self._cpu_prev = self._cpu_prev, values
if prev:
d_idle, d_kernel, d_user = (a - b for a, b in zip(values, prev))
total = d_kernel + d_user # kernel time includes idle time
if total > 0 and 1 - d_idle / total > self.busy_load:
return PAUSED
class SYSTEM_POWER_STATUS(ctypes.Structure):
_fields_ = [("ACLineStatus", ctypes.c_ubyte), ("BatteryFlag", ctypes.c_ubyte),
("BatteryLifePercent", ctypes.c_ubyte), ("SystemStatusFlag", ctypes.c_ubyte),
("BatteryLifeTime", wintypes.DWORD), ("BatteryFullLifeTime", wintypes.DWORD)]
status = SYSTEM_POWER_STATUS()
if ctypes.windll.kernel32.GetSystemPowerStatus(ctypes.byref(status)) and status.SystemStatusFlag == 1:
return REDUCED # battery saver on
return NORMAL
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 KiB

+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.9.0", "version": "0.13.0",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+12 -2
View File
@@ -28,9 +28,10 @@
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正 Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
← → 在組字區移動游標,選要改哪個字;游標在中間時直接打注音 ← → 在組字區移動游標,選要改哪個字;游標在中間時直接打注音
會插入新字,Backspace 刪掉游標前的字 會插入新字,Backspace 刪掉游標前的字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字 ↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
選過的字會被記住,下次優先出現
Esc 取消組字 Esc 取消組字
單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線), 單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線,工作列圖示變成 A),
再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示; 再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
打到一半時按,同 Tab 整段翻轉 打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文) Shift+字母 打大寫字母(也會強制英文)
@@ -42,6 +43,15 @@
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
AI 選字
-------
用中研院 CKIP 的繁體中文小型語言模型,看前後文挑同音字(他跑「得」很快、請你「再」說一次),
完全在本機運算,不連網。在工作列的輸入法圖示上按右鍵(或語言列的設定按鈕)可以選:
AI 選字:關閉/省電/標準(預設)/高品質
系統忙碌時自動暫停 AI(預設開啟:CPU 很忙時暫停,省電模式時改用省電)
選單裡會顯示目前狀態。電腦比較慢時建議用「省電」,運算太慢時也會自動暫停一分鐘。
回報問題 回報問題
-------- --------
請告訴我: 請告訴我:
+180 -5
View File
@@ -15,6 +15,7 @@ from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BAC
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402 from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402 from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402 from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
from autozhuyin.core.history import UserHistory # noqa: E402
DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data") DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data")
_start = time.time() _start = time.time()
@@ -246,7 +247,7 @@ class CursorTests(unittest.TestCase):
type_keys("←", e) type_keys("←", e)
self.assertEqual(e.cursor_offset, 3) self.assertEqual(e.cursor_offset, 3)
type_keys("←←←←", e) type_keys("←←←←", e)
self.assertEqual(e.cursor_offset, 0) self.assertEqual(e.cursor_offset, 1) # stops after the first character
type_keys("→→→→", e) type_keys("→→→→", e)
self.assertEqual(e.cursor_offset, 4) self.assertEqual(e.cursor_offset, 4)
@@ -277,8 +278,18 @@ class InsertInMiddleTests(unittest.TestCase):
self.assertEqual(e.cursor_offset, 2) self.assertEqual(e.cursor_offset, 2)
self.assertEqual(type_keys("\n", e), "今天氣") self.assertEqual(type_keys("\n", e), "今天氣")
def test_insert_at_start(self): def test_cursor_stops_after_first_character(self):
self.assertEqual(type_keys("su3cl3←←ji3\n", make_engine()), "我你好") e = make_engine()
type_keys("su3cl3←←←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("ji3\n", e)
self.assertEqual(e.marked_text, "")
def test_no_stop_before_first_chinese(self):
e = make_engine()
type_keys("hello ji3←←", e) # "hello " held, then 我: nothing to pick before 我
type_keys("↓", e)
self.assertTrue(e.showing_candidates)
def test_backspace_deletes_character_before_cursor(self): def test_backspace_deletes_character_before_cursor(self):
e = make_engine() e = make_engine()
@@ -308,8 +319,10 @@ class FirstToneAfterEnglishTests(unittest.TestCase):
def test_first_tone_after_english(self): def test_first_tone_after_english(self):
for keys in ["ru", "t", "gj", "vup", "tj", "el"]: for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
e = make_engine() e = make_engine()
self.assertEqual(type_keys("hello " + keys + " ", e), "hello ", keys) out = type_keys("hello " + keys + " \n", e)
self.assertEqual(len(e.marked_text), 1, keys) self.assertEqual(out[:6], "hello ", keys)
self.assertEqual(len(out), 7, keys) # one Chinese character after it
self.assertFalse(out[6].isascii(), keys)
class HeldWordTests(unittest.TestCase): class HeldWordTests(unittest.TestCase):
@@ -348,5 +361,167 @@ class BacktickSymbolTests(unittest.TestCase):
self.assertEqual(e.marked_text, "你") self.assertEqual(e.marked_text, "你")
class FirstWordHoldTests(unittest.TestCase):
"""The first word is not committed as English on space if it also reads as Zhuyin."""
def test_next_word_decides(self):
cases = {"el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
"i love you ": "i love you ", "up the ": "up the "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_not_held_after_english(self):
e = make_engine()
self.assertEqual(type_keys("hello world ", e), "hello world ")
self.assertEqual(type_keys("so ", e), "so ") # right after English: commits at once
class UserHistoryTests(unittest.TestCase):
def setUp(self):
self.saved = []
self.c = Classifier(LEX, ENG, weights=WEIGHTS, history=UserHistory(on_change=self.saved.append))
def pick(self, keys, phrase):
e = CompositionEngine(self.c)
type_keys(keys + "↓", e)
e.select_candidate([t for _, t in e.candidates].index(phrase))
return type_keys("\n", e)
def test_picked_candidate_comes_first(self):
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "天")
self.assertEqual(self.pick("wu0 ", "添"), "添")
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
e = CompositionEngine(self.c)
type_keys("wu0 ↓", e)
self.assertEqual(e.candidates[0][1], "添")
self.assertEqual(self.saved[-1], {"ㄊㄧㄢ": {"添": [1, 1]}})
def test_phrases_still_win(self):
self.pick("wu0 ", "添")
self.assertEqual(type_keys("rup wu0 \n", CompositionEngine(self.c)), "今天")
def test_most_picked_wins(self):
self.pick("wu0 ", "添")
self.pick("wu0 ", "添")
self.pick("wu0 ", "天")
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
def test_round_trip_and_clear(self):
self.pick("wu0 ", "添")
h = UserHistory(self.saved[-1])
self.assertEqual(h.data, {"ㄊㄧㄢ": {"添": [1, 1]}})
h.record([bopomofo.parse(list("wu0 ")).syllables[0].syllable], "天")
self.assertEqual(h.data["ㄊㄧㄢ"]["天"], [1, 2])
h.clear()
self.assertEqual(h.data, {})
def test_evicts_least_recent(self):
h = UserHistory()
h.max_entries = 2
syl = [bopomofo.parse(list("wu0 ")).syllables[0].syllable]
for p in ["天", "添", "田"]:
h.record(syl, p)
self.assertEqual(set(h.data["ㄊㄧㄢ"]), {"添", "田"})
class FakeLM:
"""Likes one string; `on_call` can advance a fake clock."""
def __init__(self, liked):
self.liked, self.calls, self.on_call = liked, 0, lambda: None
def log_probs(self, texts, context=""):
self.calls += len(texts)
self.on_call()
return [0.0 if self.liked in t else -50.0 for t in texts]
class RerankerTests(unittest.TestCase):
def engine(self, reranker):
e = make_engine()
e.reranker = reranker
return e
def test_rerank_picks_what_the_model_likes(self):
from autozhuyin.core.reranker import Reranker, ST_ACTIVE
self.assertEqual(type_keys("wu0 \n", self.engine(None)), "天")
r = Reranker("standard", FakeLM("添"))
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "添")
self.assertEqual(r.status, ST_ACTIVE)
def test_off_and_loading_use_the_lexicon(self):
from autozhuyin.core.reranker import Reranker, ST_LOADING, ST_OFF
off = Reranker("off", FakeLM("添"))
self.assertEqual(type_keys("wu0 \n", self.engine(off)), "天")
self.assertEqual(off.status, ST_OFF)
loading = Reranker("standard", None)
self.assertEqual(type_keys("wu0 \n", self.engine(loading)), "天")
self.assertEqual(loading.status, ST_LOADING)
def test_paused_and_reduced(self):
from autozhuyin.core.reranker import PAUSED, REDUCED, Reranker, ST_BUSY
r = Reranker("standard", FakeLM("添"))
r.system_state = lambda: PAUSED
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "天")
self.assertEqual(r.status, ST_BUSY)
lm = FakeLM("添")
r = Reranker("quality", lm)
r.system_state = lambda: REDUCED
type_keys("wu0 \n", self.engine(r))
self.assertLessEqual(lm.calls, 3)
def test_slow_model_is_paused(self):
from autozhuyin.core.reranker import Reranker, ST_SLOW
clock = [0.0]
lm = FakeLM("添")
lm.on_call = lambda: clock.__setitem__(0, clock[0] + 1) # every batch takes a "second"
r = Reranker("standard", lm, now=lambda: clock[0])
for _ in range(3):
type_keys("wu0 \n", self.engine(r))
self.assertEqual(r.status, ST_SLOW)
lm.on_call = lambda: None
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "天") # paused
clock[0] += 61
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "添") # back after a minute
def test_kbest_first_is_viterbi(self):
syl = [s.syllable for s in bopomofo.parse(list("rup wu0 wu0 fu4")).syllables]
conv = Converter(LEX)
best = conv.kbest(syl, 5)
self.assertEqual("".join(t for _, t in best[0][1]), "".join(t for _, t in conv.convert(syl)))
self.assertEqual(len(best), 5)
class RealModelTests(unittest.TestCase):
"""Needs numpy and data/lm.bin (scripts/fetch-lm.sh, tools/build_data.py); skipped otherwise."""
def test_fixes_homophones(self):
from autozhuyin.core import lm
from autozhuyin.core.reranker import Reranker
path = os.path.join(DATA, "lm.bin")
if not lm.available() or not os.path.exists(path):
self.skipTest("needs numpy and the model")
model = lm.GPT2LanguageModel(path, os.path.join(DATA, "lm-vocab.txt"))
self.assertAlmostEqual(model.log_prob("他在家"), -14.329, delta=0.05)
cases = {"w8 ql32k71u3ji3dj94cp32ji ": "他跑得比我快很多", "fu/3su3y94gji u h4cl3a87": "請你再說一次好嗎"}
for keys, want in cases.items():
self.assertNotEqual(type_keys(keys + "\n", make_engine()), want)
e = make_engine()
e.reranker = Reranker("standard", model)
self.assertEqual(type_keys(keys + "\n", e), want)
class AISettingsTests(unittest.TestCase):
def test_settings_round_trip_and_status(self):
import tempfile
from autozhuyin.core.ai import AI
user = tempfile.mkdtemp()
ai = AI(tempfile.mkdtemp(), user, background=False) # no model files here
self.assertTrue(ai.error) # "需要 numpy" or "AI 模型未安裝"
self.assertEqual(ai.status_text(), ai.error)
ai.set_mode("saver")
ai.set_auto_pause(False)
again = AI(tempfile.mkdtemp(), user, background=False)
self.assertEqual(again.reranker.mode, "saver")
self.assertFalse(again.auto_pause)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+41 -1
View File
@@ -6,6 +6,9 @@ import types
import unittest import unittest
HERE = os.path.dirname(os.path.abspath(__file__)) HERE = os.path.dirname(os.path.abspath(__file__))
import tempfile # noqa: E402
os.environ["APPDATA"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
sys.path.insert(0, os.path.dirname(HERE)) sys.path.insert(0, os.path.dirname(HERE))
# --- stand-ins for PIME modules (only what the wrapper uses) --- # --- stand-ins for PIME modules (only what the wrapper uses) ---
@@ -37,6 +40,9 @@ class FakeTextService:
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
def showMessage(self, message, duration=3): self.reply["showMessage"] = message def showMessage(self, message, duration=3): self.reply["showMessage"] = message
def addButton(self, button_id, **kw): self.reply.setdefault("addButton", []).append(button_id)
def removeButton(self, button_id): self.reply.setdefault("removeButton", []).append(button_id)
def changeButton(self, button_id, **kw): self.reply.setdefault("changeButton", []).append((button_id, kw))
def setShowCandidates(self, show): def setShowCandidates(self, show):
self.showCandidates = show self.showCandidates = show
@@ -44,9 +50,10 @@ class FakeTextService:
textService.TextService = FakeTextService textService.TextService = FakeTextService
textService.COMMAND_LEFT_CLICK, textService.COMMAND_RIGHT_CLICK, textService.COMMAND_MENU = 0, 1, 2
sys.modules["textService"] = textService sys.modules["textService"] = textService
from autozhuyin.autozhuyin_ime import AutoZhuyinTextService # noqa: E402 from autozhuyin.autozhuyin_ime import AutoZhuyinTextService, _shared_classifier # noqa: E402
VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28} VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
@@ -99,6 +106,8 @@ def type_text(svc, text):
class WrapperTests(unittest.TestCase): class WrapperTests(unittest.TestCase):
def setUp(self): def setUp(self):
_shared_classifier().history.clear()
_shared_classifier().learned.clear()
self.svc = AutoZhuyinTextService(client=None) self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate() self.svc.onActivate()
@@ -109,6 +118,29 @@ class WrapperTests(unittest.TestCase):
self.assertEqual(type_text(self.svc, "`<"), ",") self.assertEqual(type_text(self.svc, "`<"), ",")
self.assertEqual(type_text(self.svc, "ji3`."), "我。") self.assertEqual(type_text(self.svc, "ji3`."), "我。")
def test_settings_menu(self):
from autozhuyin import autozhuyin_ime as ime
self.assertIn("autozhuyin-settings", self.svc.reply["addButton"])
menu = self.svc.onMenu("autozhuyin-settings")
texts = [m.get("text") for m in menu]
self.assertIn("AI 選字:標準", texts)
self.assertIn("清除學習紀錄", texts)
self.svc.onCommand(ime.ID_AI_MODE["saver"], 2)
checked = [m["text"] for m in self.svc.onMenu("autozhuyin-settings") if m.get("checked")]
self.assertIn("AI 選字:省電", checked)
self.svc.onCommand(ime.ID_AI_MODE["standard"], 2)
self.assertIsNone(self.svc.onMenu("other"))
def test_mode_icon_click_toggles_english(self):
self.svc.reply = {}
self.svc.onCommand(1, 0)
self.assertTrue(self.svc.engine.english_mode)
(button, kw), = self.svc.reply["changeButton"]
self.assertEqual(button, "windows-mode-icon")
self.assertTrue(kw["icon"].endswith("icon-en.ico") or kw["icon"].endswith("icon.ico"))
self.svc.onCommand(1, 0)
self.assertFalse(self.svc.engine.english_mode)
def test_composition_shown(self): def test_composition_shown(self):
type_text(self.svc, "su3c") type_text(self.svc, "su3c")
self.assertEqual(self.svc.reply["compositionString"], "你ㄏ") self.assertEqual(self.svc.reply["compositionString"], "你ㄏ")
@@ -142,9 +174,17 @@ class WrapperTests(unittest.TestCase):
class ShiftTapWrapperTests(unittest.TestCase): class ShiftTapWrapperTests(unittest.TestCase):
def setUp(self): def setUp(self):
_shared_classifier().history.clear()
_shared_classifier().learned.clear()
self.svc = AutoZhuyinTextService(client=None) self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate() self.svc.onActivate()
def test_shift_tap_changes_icon(self):
tap_shift(self.svc)
self.assertEqual(self.svc.reply["changeButton"][0][0], "windows-mode-icon")
tap_shift(self.svc)
self.assertIn("changeButton", self.svc.reply)
def test_tap_forces_english_lowercase(self): def test_tap_forces_english_lowercase(self):
reply = tap_shift(self.svc) reply = tap_shift(self.svc)
self.assertEqual(reply.get("showMessage"), "英文") self.assertEqual(reply.get("showMessage"), "英文")
+7
View File
@@ -5,6 +5,7 @@
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words, - english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
Webster's 2nd edition, public domain), since Windows has no such file Webster's 2nd edition, public domain), since Windows has no such file
- english-bigram.json: precomputed letter-bigram table - english-bigram.json: precomputed letter-bigram table
- lm.bin, lm-vocab.txt: the "AI 選字" language model, copied from build/lm/ if present
Usage: build_data.py [word-list] Usage: build_data.py [word-list]
""" """
@@ -34,4 +35,10 @@ common = {w.strip().lower() for w in (data / "english-common.txt").read_text(enc
bigram = train_bigram(sorted(common | set(words)), common) bigram = train_bigram(sorted(common | set(words)), common)
(data / "english-bigram.json").write_text(json.dumps(bigram), encoding="utf-8") (data / "english-bigram.json").write_text(json.dumps(bigram), encoding="utf-8")
# "AI 選字" language model (scripts/fetch-lm.sh makes build/lm/); optional.
lm_dir = ROOT / "build" / "lm"
for name in ("lm.bin", "lm-vocab.txt"):
if (lm_dir / name).exists():
shutil.copy(lm_dir / name, data / name)
print("data ready: %d dictionary words, %d bigrams" % (len(words), len(bigram["table"]))) print("data ready: %d dictionary words, %d bigrams" % (len(words), len(bigram["table"])))
+79
View File
@@ -0,0 +1,79 @@
"""Downloads numpy for one Python version / platform from PyPI and unpacks it into
<dest>/<python tag>/ (numpy, numpy.libs, dist-info; test folders are dropped).
fetch_numpy.py <dest> <python tag> <platform>
fetch_numpy.py autozhuyin/vendor cp38 win32
fetch_numpy.py ~/.local/share/autozhuyin/vendor auto auto (this Python, Linux)
The newest numpy release that has a wheel for that combination is used, and its
SHA-256 is checked against PyPI. Standard library only.
"""
import hashlib
import io
import json
import platform
import re
import shutil
import sys
import urllib.request
import zipfile
from pathlib import Path
def version_key(v):
return tuple(int(x) for x in re.findall(r"\d+", v))
def linux_platforms():
machine = platform.machine().lower()
arch = {"amd64": "x86_64", "arm64": "aarch64"}.get(machine, machine)
return [f"manylinux_2_17_{arch}", f"manylinux2014_{arch}", f"manylinux_2_27_{arch}", f"manylinux_2_28_{arch}"]
def find_wheel(tag, platforms):
with urllib.request.urlopen("https://pypi.org/pypi/numpy/json", timeout=60) as r:
releases = json.load(r)["releases"]
for version in sorted(releases, key=version_key, reverse=True):
if re.search(r"[a-z]", version): # skip pre-releases (rc, b, a, dev)
continue
for f in releases[version]:
name = f["filename"]
if not name.endswith(".whl") or f.get("yanked"):
continue
if f"-{tag}-{tag}-" not in name: # also skips free-threaded "cp315t" builds
continue
plat = name[:-4].split("-")[-1]
if any(p in plat.split(".") for p in platforms):
return version, f
return None, None
def main(dest, tag, plat):
if tag == "auto":
tag = "cp%d%d" % sys.version_info[:2]
platforms = linux_platforms() if plat == "auto" else [plat]
version, f = find_wheel(tag, platforms)
if not f:
sys.exit(f"no numpy wheel for {tag} {platforms}")
print(f"numpy {version} for {tag}: {f['filename']} ({f['size'] / 1e6:.1f} MB)")
with urllib.request.urlopen(f["url"], timeout=300) as r:
data = r.read()
if hashlib.sha256(data).hexdigest() != f["digests"]["sha256"]:
sys.exit("checksum mismatch")
out = Path(dest).expanduser() / tag
if out.exists():
shutil.rmtree(out)
out.mkdir(parents=True)
with zipfile.ZipFile(io.BytesIO(data)) as z:
for info in z.infolist():
parts = Path(info.filename).parts
if "tests" in parts or parts[0].endswith(".data"):
continue
z.extract(info, out)
print(f"unpacked to {out}")
if __name__ == "__main__":
if len(sys.argv) != 4:
sys.exit(__doc__)
main(*sys.argv[1:])