Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cfd095b3b0 | ||
|
|
f323a29552 | ||
|
|
aa00b793ce | ||
|
|
b3b7589379 | ||
|
|
c440cd7177 | ||
|
|
6443f771a0 | ||
|
|
0155ddd16e | ||
|
|
c8b3c4ae6f | ||
|
|
38e0758a13 | ||
|
|
fd87da8061 | ||
|
|
acafd8f96e | ||
|
|
fa5b266ae4 | ||
|
|
6448bf4ca2 | ||
|
|
2cebf0e700 | ||
|
|
815dcf979d | ||
|
|
8e82105af9 | ||
|
|
9cac3467a0 | ||
|
|
49e0b52fd5 | ||
|
|
fcc0f9d2db | ||
|
|
31d9991e16 |
@@ -1,6 +1,6 @@
|
|||||||
# 自動注音 AutoZhuyin
|
# 自動注音 AutoZhuyin
|
||||||
|
|
||||||
給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
|
給繁體中文使用者的輸入法(macOS、Windows、Linux):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
|
||||||
|
|
||||||
```
|
```
|
||||||
按鍵: ji3 hello world su3cl3␣
|
按鍵: ji3 hello world su3cl3␣
|
||||||
@@ -12,10 +12,13 @@
|
|||||||
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
|
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
|
||||||
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
|
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
|
||||||
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
|
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
|
||||||
- **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷
|
- **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷;英文模式時輸入法圖示會從「注」變成「A」
|
||||||
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
|
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
|
||||||
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
|
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
|
||||||
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
|
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
|
||||||
|
- **記住選字**:從候選字選過的字詞會被記住,下次打同樣的讀音會優先出現(選越多次越優先)
|
||||||
|
- **AI 選字**:用中研院 CKIP 的繁中小型語言模型看前後文挑同音字(他跑「得」很快、請你「再」說一次),完全在本機運算;可調整或關閉,系統忙碌時自動暫停
|
||||||
|
- **中間補字、刪字**:游標移到前面後直接打注音,會插在游標處;Backspace 刪掉游標前的那個字
|
||||||
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
|
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
|
||||||
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
|
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
|
||||||
|
|
||||||
@@ -31,6 +34,8 @@
|
|||||||
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
|
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
|
||||||
3. 清單裡沒出現的話,登出再登入一次。
|
3. 清單裡沒出現的話,登出再登入一次。
|
||||||
|
|
||||||
|
從 0.12 以前的版本更新:輸入法現在分成「中英自動」和「英文」兩個模式(各有圖示),安裝時會自動加入。如果輸入法清單裡還留著舊的「自動注音」項目,把它移除即可。
|
||||||
|
|
||||||
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
|
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
|
||||||
|
|
||||||
### Windows
|
### Windows
|
||||||
@@ -44,6 +49,16 @@ PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
|
|||||||
|
|
||||||
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
|
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
|
||||||
|
|
||||||
|
### Linux(Bazzite、SteamOS 桌面模式、Fedora…)
|
||||||
|
|
||||||
|
需求:系統內建 IBus 輸入法框架(Bazzite、Fedora 預設就有)。只在桌面模式可用,Steam 遊戲模式不支援系統輸入法。
|
||||||
|
|
||||||
|
1. 解壓縮 `AutoZhuyin-Linux-<版本>.tar.gz`,在終端機進到資料夾執行 `./install.sh`。不需要 sudo,只會裝在家目錄(`~/.local/share/autozhuyin`),所以唯讀系統也能用。
|
||||||
|
2. KDE(Bazzite / SteamOS 預設):系統設定 → 鍵盤 → 虛擬鍵盤,選 **IBus Wayland** → 套用,登出再登入。GNOME 不用設定。
|
||||||
|
3. 之後每次登入會自動切換到自動注音(`~/.config/autostart/autozhuyin.desktop`)。
|
||||||
|
|
||||||
|
暫時改回一般鍵盤:`ibus engine xkb:us::eng`;切回:`ibus engine autozhuyin`;移除:`./uninstall.sh`。
|
||||||
|
|
||||||
## 按鍵
|
## 按鍵
|
||||||
|
|
||||||
| 按鍵 | 作用 |
|
| 按鍵 | 作用 |
|
||||||
@@ -51,10 +66,10 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
|
|||||||
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
|
||||||
| Enter | 送出目前組字 |
|
| Enter | 送出目前組字 |
|
||||||
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
|
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
|
||||||
| ← → | 在組字區移動游標,選要改哪個字 |
|
| ← → | 在組字區移動游標(最左停在第一個中文字後面)。游標在中間時:直接打注音會**插入**新字,Backspace 刪掉游標前的字,↓ 選游標前的字 |
|
||||||
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
|
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
|
||||||
| Esc | 取消組字 |
|
| Esc | 取消組字 |
|
||||||
| **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
|
| **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷,圖示會變成「A」(Mac 的選單列、Windows 的工作列、Linux 的 IBus 狀態)。Mac 也可以直接在選單列選「英文」模式。打到一半時:同 Tab,整段翻轉 |
|
||||||
| Shift + 字母 | 打大寫字母(也會強制英文) |
|
| Shift + 字母 | 打大寫字母(也會強制英文) |
|
||||||
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
|
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
|
||||||
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
|
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
|
||||||
@@ -84,15 +99,59 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
|
|||||||
|
|
||||||
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
|
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
|
||||||
|
|
||||||
|
### AI 選字
|
||||||
|
|
||||||
|
詞庫只知道每個詞本身有多常見,不看前後文。AI 選字讓語言模型當裁判:
|
||||||
|
|
||||||
|
1. 詞庫先找出前 K 名的整句組合(例如「請你在說一次」「請你再說一次」)。
|
||||||
|
2. 模型 [CKIP GPT2-tiny](https://huggingface.co/ckiplab/gpt2-tiny-chinese)(中研院詞庫小組,繁體中文,約 1200 萬參數,22 MB)計算每個組合出現的機率,並參考剛送出的前文。
|
||||||
|
3. 分數 = 詞庫分數 + 0.6 × 模型分數,取最高的。選過的候選字仍然固定不變。
|
||||||
|
|
||||||
|
輸入法選單可以調整(Mac:選單列的輸入法選單;Windows:工作列輸入法圖示按右鍵或語言列的設定按鈕;Linux:IBus 選單的「設定」):
|
||||||
|
|
||||||
|
| 模式 | 比較幾個組合 | 每次運算上限 |
|
||||||
|
|---|---|---|
|
||||||
|
| 關閉 | — | 不載入模型、不佔記憶體 |
|
||||||
|
| 省電 | 3 | 15 毫秒 |
|
||||||
|
| 標準(預設) | 10 | 40 毫秒 |
|
||||||
|
| 高品質 | 30 | 120 毫秒 |
|
||||||
|
|
||||||
|
不會拖慢打字的機制:
|
||||||
|
- 每次運算超過上限就直接放棄,改用詞庫結果;連續 3 次超時就暫停一分鐘。
|
||||||
|
- 「系統忙碌時自動暫停 AI」(預設開啟),選單裡會顯示目前狀態:
|
||||||
|
|
||||||
|
| | 暫停 AI | 降到省電模式 |
|
||||||
|
|---|---|---|
|
||||||
|
| macOS | 過熱,或 CPU 負載(每核心 1 分鐘平均)超過 0.9 | 低耗電模式 |
|
||||||
|
| Windows | CPU 使用率超過 90% | 省電模式(電池保護) |
|
||||||
|
| Linux | CPU 負載超過 0.9,或溫度達 95 °C | 低耗電設定檔,或用電池且電量 ≤ 20% |
|
||||||
|
- 模型在背景載入,載入完成前照常用詞庫。
|
||||||
|
|
||||||
|
實測(`training/lm/eval_rerank.py`,標準模式):
|
||||||
|
|
||||||
|
| 測試資料 | 只用詞庫 | AI 選字 |
|
||||||
|
|---|---|---|
|
||||||
|
| 日常對話 60 句 | 字正確率 96.6%,整句全對 73% | 97.5%,80% |
|
||||||
|
| 維基百科 714 句 | 92.9%,31% | 94.7%,44% |
|
||||||
|
|
||||||
|
錯字大約少了 25–30%。
|
||||||
|
|
||||||
|
Windows、Linux 版用 numpy 執行同一個模型(`windows/autozhuyin/core/lm.py`),所以時間上限放寬為省電 40、標準 80、高品質 200 毫秒,超時的那次結果照用但不快取,連續 3 次超時暫停一分鐘。
|
||||||
|
- Windows:安裝包附上 PIME 兩種 Python(1.3 版的 3.8、2.x 版的 3.15,都是 32 位元)用的 numpy,安裝檔約 40 MB。
|
||||||
|
- Linux:系統沒有 numpy 時,`install.sh` 會從 PyPI 下載符合的版本到 `~/.local/share/autozhuyin/vendor/`。
|
||||||
|
- 設定存在 `%APPDATA%\PIME\autozhuyin\settings.json`、`~/.config/autozhuyin/settings.json`。
|
||||||
|
|
||||||
## 個人資料
|
## 個人資料
|
||||||
|
|
||||||
| | macOS | Windows |
|
| | macOS | Windows | Linux |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` |
|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | `~/.config/autozhuyin/learned.json` |
|
||||||
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` |
|
| 選字紀錄 | `~/Library/Application Support/AutoZhuyin/user-history.json` | `%APPDATA%\PIME\autozhuyin\user-history.json` | `~/.config/autozhuyin/user-history.json` |
|
||||||
|
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | `~/.config/autozhuyin/user-phrases.txt` |
|
||||||
|
|
||||||
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
|
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
|
||||||
Mac 版可以在輸入法選單選「清除學習紀錄」。
|
選字紀錄最多保留 2000 筆,太久沒用的會先淘汰。被選過的字只會排到同讀音第一順位,不會蓋過整個詞組(例如選過「添」之後,ㄊㄧㄢ 單打會出「添」,但「今天」還是「今天」)。
|
||||||
|
Mac 版可以在輸入法選單選「清除學習紀錄」(中英判斷和選字紀錄一起清除);Windows、Linux 刪除上面的檔案即可。
|
||||||
|
|
||||||
## 開發
|
## 開發
|
||||||
|
|
||||||
@@ -125,6 +184,18 @@ scripts/package-windows.sh # 產生 build/AutoZhuyin-Wi
|
|||||||
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
|
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
|
||||||
- 版本號在 `windows/autozhuyin/ime.json`。
|
- 版本號在 `windows/autozhuyin/ime.json`。
|
||||||
|
|
||||||
|
### Linux
|
||||||
|
|
||||||
|
Linux 版是 IBus 輸入法(`linux/autozhuyin_ibus.py`),直接使用 Windows 版的 Python 核心(`windows/autozhuyin/core/`)和資料。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m unittest discover -s linux/tests # 用假的 IBus 模組測試外殼
|
||||||
|
scripts/package-linux.sh # 產生 build/AutoZhuyin-Linux-<版本>.tar.gz(會先跑測試)
|
||||||
|
```
|
||||||
|
|
||||||
|
- 安裝在使用者目錄,不寫入 `/usr`:開機時由 autostart 啟動 `autozhuyin_ibus.py --activate`,它會把引擎動態註冊給 ibus-daemon 並切換過去。
|
||||||
|
- 也支援由 ibus-daemon 啟動(`--ibus`,配合 `/usr/share/ibus/component` 的 XML),之後要做 RPM 時可以用。
|
||||||
|
|
||||||
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
|
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
|
||||||
|
|
||||||
### 訓練判斷權重
|
### 訓練判斷權重
|
||||||
@@ -164,6 +235,22 @@ python3 training/make_parity.py # 權重或特徵改了之後都要重
|
|||||||
scripts/import-mcbopomofo.sh
|
scripts/import-mcbopomofo.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### AI 選字模型
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/fetch-lm.sh # 下載 CKIP gpt2-tiny-chinese,轉成 build/lm/lm.bin(float16)+ lm-vocab.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
`build.sh` 會自動執行並把模型打包進 app。模型推論寫在 `Sources/AutoZhuyinCore/LanguageModel.swift`(Accelerate,Apple Silicon 和 Intel 都可用),重排在 `Reranker.swift`。
|
||||||
|
|
||||||
|
評估效果(需要 numpy;`build/lm-eval/` 放測試句子,一行一句):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python training/lm/eval_rerank.py build/lm-eval/chat.txt build/lm-eval/wiki.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
`training/lm/gpt2.py` 是 numpy 版的參考實作,Swift 版的測試會比對它算出的分數。
|
||||||
|
|
||||||
### 發布新版本
|
### 發布新版本
|
||||||
|
|
||||||
先 commit 所有修改,再執行:
|
先 commit 所有修改,再執行:
|
||||||
@@ -178,12 +265,12 @@ scripts/release.sh minor --push # 同時 push commit 和 tag
|
|||||||
`release.sh` 會自動:
|
`release.sh` 會自動:
|
||||||
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
|
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
|
||||||
2. 跑 Swift 和 Python 全部測試
|
2. 跑 Swift 和 Python 全部測試
|
||||||
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip`
|
3. 產生 `build/AutoZhuyin-<版本>.pkg`、`build/AutoZhuyin-Windows-<版本>.zip`、`build/AutoZhuyin-Linux-<版本>.tar.gz`
|
||||||
4. commit「Release v<版本>」並打上 `v<版本>` tag
|
4. commit「Release v<版本>」並打上 `v<版本>` tag
|
||||||
|
|
||||||
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
|
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
|
||||||
|
|
||||||
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。
|
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg`、`.zip` 和 `.tar.gz`。
|
||||||
|
|
||||||
## 專案結構
|
## 專案結構
|
||||||
|
|
||||||
@@ -211,15 +298,20 @@ windows/
|
|||||||
installer/ install.bat、uninstall.bat、給使用者的 README.txt
|
installer/ install.bat、uninstall.bat、給使用者的 README.txt
|
||||||
tests/ Python 測試
|
tests/ Python 測試
|
||||||
tools/ 產生資料檔、圖示
|
tools/ 產生資料檔、圖示
|
||||||
|
linux/
|
||||||
|
autozhuyin_ibus.py IBus 外殼(共用 windows/autozhuyin/core)
|
||||||
|
installer/ install.sh、uninstall.sh、給使用者的 README.txt
|
||||||
|
tests/ 用假 IBus 模組的外殼測試
|
||||||
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
|
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
|
||||||
THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝檔)
|
THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝檔)
|
||||||
```
|
```
|
||||||
|
|
||||||
## 已知限制
|
## 已知限制
|
||||||
|
|
||||||
- Windows 版還沒在實機上測試過。
|
- Windows 版、Linux 版還沒在實機上測試過。
|
||||||
|
- Linux 版只支援 IBus。KDE 上有些 Flatpak 程式(例如 Flatpak 版 Chrome)在 IBus Wayland 下可能打不出字。
|
||||||
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
|
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
|
||||||
- 很短的字串本質上就有歧義。像 `up`(也是 ㄧㄣ 因)這種常用的,按空白後會先保留,看下一個詞再決定(`up jo4` = 因為、`up the` = up the);其他判斷錯的可以按 Tab 修正,修正後會記住。
|
- 很短的字串本質上就有歧義。開頭第一個詞如果也能讀成注音,按空白後會先保留(顯示成英文),看下一個詞再決定:`up jo4` = 因為、`el ji3` = 高我、`no su3cl3` = no 你好、`up the` = up the。接在英文後面時只保留 `up`(因)這種很常見的讀音,不會拖慢打英文。還是判斷錯的話按 Tab 修正,修正後會記住。
|
||||||
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
|
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
|
||||||
|
|
||||||
## 下一步(Roadmap)
|
## 下一步(Roadmap)
|
||||||
@@ -227,11 +319,13 @@ THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝
|
|||||||
- [x] 匯入完整詞庫
|
- [x] 匯入完整詞庫
|
||||||
- [x] 注音順序錯誤修正
|
- [x] 注音順序錯誤修正
|
||||||
- [x] Windows 版
|
- [x] Windows 版
|
||||||
|
- [x] Linux 版(IBus)
|
||||||
|
- [ ] Linux Fcitx5 版(KDE 上相容性較好)
|
||||||
- [x] 用資料訓練判斷權重(合成資料)
|
- [x] 用資料訓練判斷權重(合成資料)
|
||||||
- [ ] 加入真實打字資料一起訓練
|
- [ ] 加入真實打字資料一起訓練
|
||||||
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
|
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
|
||||||
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境
|
- [x] AI 選字:語言模型參考前後文挑同音字(macOS、Windows、Linux)
|
||||||
- [ ] 使用者詞頻學習(選過的候選字優先)
|
- [x] 使用者詞頻學習(選過的候選字優先)
|
||||||
- [ ] 偏好設定視窗
|
- [ ] 偏好設定視窗
|
||||||
|
|
||||||
## 致謝與授權
|
## 致謝與授權
|
||||||
@@ -239,5 +333,7 @@ THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝
|
|||||||
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
|
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
|
||||||
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
|
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
|
||||||
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
|
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
|
||||||
|
- AI 選字語言模型:[CKIP GPT2 Tiny Chinese](https://github.com/ckiplab/ckip-transformers)(中研院詞庫小組,GPL-3.0,全文在 `licenses/GPL-3.0.txt`,並一起打包進三個平台的安裝檔)
|
||||||
|
- 數值運算:[NumPy](https://numpy.org)(BSD 3-Clause,Windows 版附上、Linux 版安裝時下載)
|
||||||
|
|
||||||
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
|
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
/* Names shown in System Settings and the input menu. */
|
||||||
|
CFBundleName = "自動注音";
|
||||||
|
CFBundleDisplayName = "自動注音";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin" = "自動注音";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin.Auto" = "自動注音(中英自動)";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin.English" = "自動注音(英文)";
|
||||||
+63
-2
@@ -15,9 +15,70 @@
|
|||||||
<key>CFBundlePackageType</key>
|
<key>CFBundlePackageType</key>
|
||||||
<string>APPL</string>
|
<string>APPL</string>
|
||||||
<key>CFBundleShortVersionString</key>
|
<key>CFBundleShortVersionString</key>
|
||||||
<string>0.7.1</string>
|
<string>0.13.2</string>
|
||||||
<key>CFBundleVersion</key>
|
<key>CFBundleVersion</key>
|
||||||
<string>13</string>
|
<string>23</string>
|
||||||
|
<key>ComponentInputModeDict</key>
|
||||||
|
<dict>
|
||||||
|
<key>tsInputModeListKey</key>
|
||||||
|
<dict>
|
||||||
|
<key>com.autozhuyin.inputmethod.AutoZhuyin.Auto</key>
|
||||||
|
<dict>
|
||||||
|
<key>TISInputSourceID</key>
|
||||||
|
<string>com.autozhuyin.inputmethod.AutoZhuyin.Auto</string>
|
||||||
|
<key>TISIntendedLanguage</key>
|
||||||
|
<string>zh-Hant</string>
|
||||||
|
<key>tsInputModeAlternateMenuIconFileKey</key>
|
||||||
|
<string>icon.tiff</string>
|
||||||
|
<key>tsInputModeCharacterRepertoireKey</key>
|
||||||
|
<array>
|
||||||
|
<string>Hant</string>
|
||||||
|
</array>
|
||||||
|
<key>tsInputModeDefaultStateKey</key>
|
||||||
|
<true/>
|
||||||
|
<key>tsInputModeIsVisibleKey</key>
|
||||||
|
<true/>
|
||||||
|
<key>tsInputModeMenuIconFileKey</key>
|
||||||
|
<string>icon.tiff</string>
|
||||||
|
<key>tsInputModePaletteIconFileKey</key>
|
||||||
|
<string>icon.tiff</string>
|
||||||
|
<key>tsInputModePrimaryInScriptKey</key>
|
||||||
|
<true/>
|
||||||
|
<key>tsInputModeScriptKey</key>
|
||||||
|
<string>smTradChinese</string>
|
||||||
|
</dict>
|
||||||
|
<key>com.autozhuyin.inputmethod.AutoZhuyin.English</key>
|
||||||
|
<dict>
|
||||||
|
<key>TISInputSourceID</key>
|
||||||
|
<string>com.autozhuyin.inputmethod.AutoZhuyin.English</string>
|
||||||
|
<key>TISIntendedLanguage</key>
|
||||||
|
<string>zh-Hant</string>
|
||||||
|
<key>tsInputModeAlternateMenuIconFileKey</key>
|
||||||
|
<string>icon-en.tiff</string>
|
||||||
|
<key>tsInputModeCharacterRepertoireKey</key>
|
||||||
|
<array>
|
||||||
|
<string>Hant</string>
|
||||||
|
</array>
|
||||||
|
<key>tsInputModeDefaultStateKey</key>
|
||||||
|
<true/>
|
||||||
|
<key>tsInputModeIsVisibleKey</key>
|
||||||
|
<true/>
|
||||||
|
<key>tsInputModeMenuIconFileKey</key>
|
||||||
|
<string>icon-en.tiff</string>
|
||||||
|
<key>tsInputModePaletteIconFileKey</key>
|
||||||
|
<string>icon-en.tiff</string>
|
||||||
|
<key>tsInputModePrimaryInScriptKey</key>
|
||||||
|
<false/>
|
||||||
|
<key>tsInputModeScriptKey</key>
|
||||||
|
<string>smTradChinese</string>
|
||||||
|
</dict>
|
||||||
|
</dict>
|
||||||
|
<key>tsVisibleInputModeOrderedArrayKey</key>
|
||||||
|
<array>
|
||||||
|
<string>com.autozhuyin.inputmethod.AutoZhuyin.Auto</string>
|
||||||
|
<string>com.autozhuyin.inputmethod.AutoZhuyin.English</string>
|
||||||
|
</array>
|
||||||
|
</dict>
|
||||||
<key>InputMethodConnectionName</key>
|
<key>InputMethodConnectionName</key>
|
||||||
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
|
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
|
||||||
<key>InputMethodServerControllerClass</key>
|
<key>InputMethodServerControllerClass</key>
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
/* Names shown in System Settings and the input menu. */
|
||||||
|
CFBundleName = "AutoZhuyin";
|
||||||
|
CFBundleDisplayName = "AutoZhuyin";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin" = "AutoZhuyin";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin.Auto" = "AutoZhuyin (Auto)";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin.English" = "AutoZhuyin (English)";
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
/* Names shown in System Settings and the input menu. */
|
||||||
|
CFBundleName = "自動注音";
|
||||||
|
CFBundleDisplayName = "自動注音";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin" = "自動注音";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin.Auto" = "自動注音(中英自動)";
|
||||||
|
"com.autozhuyin.inputmethod.AutoZhuyin.English" = "自動注音(英文)";
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
import Foundation
|
||||||
|
import AutoZhuyinCore
|
||||||
|
|
||||||
|
/// "AI 選字" settings, model loading and system-load detection for the Mac app.
|
||||||
|
enum AI {
|
||||||
|
static let modeKey = "AIMode"
|
||||||
|
static let autoPauseKey = "AIAutoPause"
|
||||||
|
|
||||||
|
static let reranker: Reranker = {
|
||||||
|
let saved = UserDefaults.standard.string(forKey: modeKey).flatMap(Reranker.Mode.init(rawValue:))
|
||||||
|
let r = Reranker(mode: saved ?? .standard)
|
||||||
|
r.systemState = { SystemMonitor.shared.state() }
|
||||||
|
return r
|
||||||
|
}()
|
||||||
|
|
||||||
|
/// The model file is missing from the app bundle (built without scripts/fetch-lm.sh).
|
||||||
|
private(set) static var modelMissing = false
|
||||||
|
private static var loading = false
|
||||||
|
|
||||||
|
static var autoPause: Bool {
|
||||||
|
get { UserDefaults.standard.object(forKey: autoPauseKey) as? Bool ?? true }
|
||||||
|
set { UserDefaults.standard.set(newValue, forKey: autoPauseKey); SystemMonitor.shared.invalidate() }
|
||||||
|
}
|
||||||
|
|
||||||
|
static func setMode(_ mode: Reranker.Mode) {
|
||||||
|
reranker.mode = mode
|
||||||
|
UserDefaults.standard.set(mode.rawValue, forKey: modeKey)
|
||||||
|
if mode == .off {
|
||||||
|
reranker.model = nil // free the memory
|
||||||
|
} else {
|
||||||
|
loadModelIfNeeded()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Loads the model in the background (only when AI is on).
|
||||||
|
static func loadModelIfNeeded() {
|
||||||
|
guard reranker.mode != .off, reranker.model == nil, !loading, !modelMissing else { return }
|
||||||
|
guard let bin = Bundle.main.url(forResource: "lm", withExtension: "bin"),
|
||||||
|
let vocab = Bundle.main.url(forResource: "lm-vocab", withExtension: "txt") else {
|
||||||
|
modelMissing = true
|
||||||
|
return
|
||||||
|
}
|
||||||
|
loading = true
|
||||||
|
DispatchQueue.global(qos: .utility).async {
|
||||||
|
let start = Date()
|
||||||
|
let model = try? GPT2LanguageModel(modelURL: bin, vocabURL: vocab)
|
||||||
|
NSLog("AutoZhuyin: language model %@ in %.0f ms", model == nil ? "failed" : "loaded",
|
||||||
|
Date().timeIntervalSince(start) * 1000)
|
||||||
|
DispatchQueue.main.async {
|
||||||
|
loading = false
|
||||||
|
if model == nil { modelMissing = true }
|
||||||
|
if reranker.mode != .off { reranker.model = model }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static var statusText: String {
|
||||||
|
if modelMissing { return "AI 模型未安裝" }
|
||||||
|
switch reranker.status {
|
||||||
|
case .off: return "AI 選字已關閉"
|
||||||
|
case .loading: return "AI 模型載入中…"
|
||||||
|
case .active:
|
||||||
|
return SystemMonitor.shared.state() == .reduced ? "AI 使用中(省電:低耗電模式)" : "AI 使用中"
|
||||||
|
case .pausedBusy: return "AI 已暫停:系統忙碌或過熱"
|
||||||
|
case .pausedSlow: return "AI 已暫停:運算太慢,一分鐘後再試"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Decides when AI should step back: paused when the Mac is hot or the CPU is busy,
|
||||||
|
/// reduced (fewest candidates) in Low Power Mode. Checked at most every 2 seconds.
|
||||||
|
final class SystemMonitor {
|
||||||
|
static let shared = SystemMonitor()
|
||||||
|
private var cached: (time: Date, state: Reranker.SystemState) = (.distantPast, .normal)
|
||||||
|
/// CPU load (1-minute average per core) above which AI pauses.
|
||||||
|
var busyLoad = 0.9
|
||||||
|
|
||||||
|
func invalidate() { cached.time = .distantPast }
|
||||||
|
|
||||||
|
func state() -> Reranker.SystemState {
|
||||||
|
if Date().timeIntervalSince(cached.time) < 2 { return cached.state }
|
||||||
|
cached = (Date(), compute())
|
||||||
|
return cached.state
|
||||||
|
}
|
||||||
|
|
||||||
|
private func compute() -> Reranker.SystemState {
|
||||||
|
guard AI.autoPause else { return .normal }
|
||||||
|
let info = ProcessInfo.processInfo
|
||||||
|
if info.thermalState == .serious || info.thermalState == .critical { return .paused }
|
||||||
|
var load = [Double](repeating: 0, count: 3)
|
||||||
|
if getloadavg(&load, 3) > 0, load[0] / Double(max(1, info.activeProcessorCount)) > busyLoad {
|
||||||
|
return .paused
|
||||||
|
}
|
||||||
|
if info.isLowPowerModeEnabled { return .reduced }
|
||||||
|
return .normal
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import Carbon
|
||||||
import Cocoa
|
import Cocoa
|
||||||
import InputMethodKit
|
import InputMethodKit
|
||||||
import AutoZhuyinCore
|
import AutoZhuyinCore
|
||||||
@@ -9,6 +10,11 @@ enum Shared {
|
|||||||
|
|
||||||
static let learnedKey = "LearnedPreferences"
|
static let learnedKey = "LearnedPreferences"
|
||||||
|
|
||||||
|
/// ~/Library/Application Support/AutoZhuyin (user phrases, picked-candidate history).
|
||||||
|
static let supportDir: URL? = FileManager.default
|
||||||
|
.urls(for: .applicationSupportDirectory, in: .userDomainMask).first?
|
||||||
|
.appendingPathComponent("AutoZhuyin")
|
||||||
|
|
||||||
static let classifier: Classifier = {
|
static let classifier: Classifier = {
|
||||||
let bundle = Bundle.main
|
let bundle = Bundle.main
|
||||||
let start = Date()
|
let start = Date()
|
||||||
@@ -18,9 +24,9 @@ enum Shared {
|
|||||||
lexicon.load(data: data)
|
lexicon.load(data: data)
|
||||||
}
|
}
|
||||||
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
|
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
|
||||||
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
|
if let userFile = supportDir?.appendingPathComponent("user-phrases.txt"),
|
||||||
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
|
let text = try? String(contentsOf: userFile, encoding: .utf8) {
|
||||||
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
|
lexicon.load(text)
|
||||||
}
|
}
|
||||||
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
|
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
|
||||||
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
|
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
|
||||||
@@ -29,17 +35,27 @@ enum Shared {
|
|||||||
let learned = LearnedPreferences(weights: saved) { weights in
|
let learned = LearnedPreferences(weights: saved) { weights in
|
||||||
UserDefaults.standard.set(weights, forKey: learnedKey)
|
UserDefaults.standard.set(weights, forKey: learnedKey)
|
||||||
}
|
}
|
||||||
|
let historyFile = supportDir?.appendingPathComponent("user-history.json")
|
||||||
|
let history = UserHistory(data: historyFile.map(UserHistory.load(from:)) ?? [:]) { data in
|
||||||
|
guard let url = historyFile, let json = try? JSONEncoder().encode(data) else { return }
|
||||||
|
try? FileManager.default.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
|
||||||
|
try? json.write(to: url, options: .atomic)
|
||||||
|
}
|
||||||
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
|
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
|
||||||
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
|
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
|
||||||
// Only needed for untoned syllables; build it off the startup path.
|
// Only needed for untoned syllables; build it off the startup path.
|
||||||
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
|
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
|
||||||
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
|
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained, history: history)
|
||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
@objc(AutoZhuyinInputController)
|
@objc(AutoZhuyinInputController)
|
||||||
final class InputController: IMKInputController {
|
final class InputController: IMKInputController {
|
||||||
private let engine = CompositionEngine(classifier: Shared.classifier)
|
private lazy var engine: CompositionEngine = {
|
||||||
|
let e = CompositionEngine(classifier: Shared.classifier)
|
||||||
|
e.reranker = AI.reranker
|
||||||
|
return e
|
||||||
|
}()
|
||||||
private var shiftTap = ShiftTapDetector()
|
private var shiftTap = ShiftTapDetector()
|
||||||
|
|
||||||
// MARK: - Key events
|
// MARK: - Key events
|
||||||
@@ -59,8 +75,11 @@ final class InputController: IMKInputController {
|
|||||||
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
|
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
|
||||||
time: event.timestamp)
|
time: event.timestamp)
|
||||||
if tapped {
|
if tapped {
|
||||||
|
let idle = engine.keys.isEmpty
|
||||||
_ = engine.handle(.shiftTap)
|
_ = engine.handle(.shiftTap)
|
||||||
updateComposition(client)
|
updateComposition(client)
|
||||||
|
// English mode on/off: switch input mode so the menu-bar icon shows it.
|
||||||
|
if idle { InputModes.select(english: engine.isEnglishMode) }
|
||||||
}
|
}
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
@@ -137,6 +156,18 @@ final class InputController: IMKInputController {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The input mode changed (menu bar, keyboard shortcut, or our own Shift tap).
|
||||||
|
override func setValue(_ value: Any!, forTag tag: Int, client sender: Any!) {
|
||||||
|
if tag == kTextServiceInputModePropertyTag, let id = value as? String {
|
||||||
|
let english = InputModes.isEnglish(id)
|
||||||
|
if english != engine.isEnglishMode {
|
||||||
|
commitComposition(sender)
|
||||||
|
engine.setEnglishMode(english)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
super.setValue(value, forTag: tag, client: sender)
|
||||||
|
}
|
||||||
|
|
||||||
override func commitComposition(_ sender: Any!) {
|
override func commitComposition(_ sender: Any!) {
|
||||||
guard let client = sender as? IMKTextInput else { return }
|
guard let client = sender as? IMKTextInput else { return }
|
||||||
commit(engine.commitAll(), to: client)
|
commit(engine.commitAll(), to: client)
|
||||||
@@ -171,11 +202,33 @@ final class InputController: IMKInputController {
|
|||||||
|
|
||||||
override func menu() -> NSMenu! {
|
override func menu() -> NSMenu! {
|
||||||
let menu = NSMenu()
|
let menu = NSMenu()
|
||||||
|
let modes: [(Reranker.Mode, String, Selector)] = [
|
||||||
|
(.off, "AI 選字:關閉", #selector(aiOff(_:))),
|
||||||
|
(.saver, "AI 選字:省電", #selector(aiSaver(_:))),
|
||||||
|
(.standard, "AI 選字:標準", #selector(aiStandard(_:))),
|
||||||
|
(.quality, "AI 選字:高品質", #selector(aiQuality(_:))),
|
||||||
|
]
|
||||||
|
for (mode, title, action) in modes {
|
||||||
|
let item = menu.addItem(withTitle: title, action: action, keyEquivalent: "")
|
||||||
|
item.state = AI.reranker.mode == mode ? .on : .off
|
||||||
|
}
|
||||||
|
let pause = menu.addItem(withTitle: "系統忙碌時自動暫停 AI", action: #selector(toggleAutoPause(_:)), keyEquivalent: "")
|
||||||
|
pause.state = AI.autoPause ? .on : .off
|
||||||
|
let status = menu.addItem(withTitle: AI.statusText, action: nil, keyEquivalent: "")
|
||||||
|
status.isEnabled = false
|
||||||
|
menu.addItem(.separator())
|
||||||
menu.addItem(withTitle: "清除學習紀錄", action: #selector(clearLearned(_:)), keyEquivalent: "")
|
menu.addItem(withTitle: "清除學習紀錄", action: #selector(clearLearned(_:)), keyEquivalent: "")
|
||||||
return menu
|
return menu
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@objc private func aiOff(_ sender: Any?) { AI.setMode(.off) }
|
||||||
|
@objc private func aiSaver(_ sender: Any?) { AI.setMode(.saver) }
|
||||||
|
@objc private func aiStandard(_ sender: Any?) { AI.setMode(.standard) }
|
||||||
|
@objc private func aiQuality(_ sender: Any?) { AI.setMode(.quality) }
|
||||||
|
@objc private func toggleAutoPause(_ sender: Any?) { AI.autoPause.toggle() }
|
||||||
|
|
||||||
@objc private func clearLearned(_ sender: Any?) {
|
@objc private func clearLearned(_ sender: Any?) {
|
||||||
Shared.classifier.learned.clear()
|
Shared.classifier.learned.clear()
|
||||||
|
Shared.classifier.history.clear()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
import Carbon
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// The two input modes declared in Info.plist (ComponentInputModeDict). Each has its own
|
||||||
|
/// menu-bar icon, so switching modes is how the icon shows "中英自動" vs "英文".
|
||||||
|
enum InputModes {
|
||||||
|
static let auto = "com.autozhuyin.inputmethod.AutoZhuyin.Auto"
|
||||||
|
static let english = "com.autozhuyin.inputmethod.AutoZhuyin.English"
|
||||||
|
|
||||||
|
static func isEnglish(_ modeID: String) -> Bool { modeID == english }
|
||||||
|
|
||||||
|
private static func source(_ id: String) -> TISInputSource? {
|
||||||
|
let filter = [kTISPropertyInputSourceID as String: id] as CFDictionary
|
||||||
|
guard let list = TISCreateInputSourceList(filter, true)?.takeRetainedValue() as? [TISInputSource] else { return nil }
|
||||||
|
return list.first
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func isEnabled(_ src: TISInputSource) -> Bool {
|
||||||
|
guard let p = TISGetInputSourceProperty(src, kTISPropertyInputSourceIsEnabled) else { return false }
|
||||||
|
return CFBooleanGetValue(Unmanaged<CFBoolean>.fromOpaque(p).takeUnretainedValue())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Makes the given mode current. Returns false on failure.
|
||||||
|
/// Only enables the mode if it is not enabled yet: TISEnableInputSource opens System
|
||||||
|
/// Settings → Keyboard, even for a source that is already enabled.
|
||||||
|
@discardableResult
|
||||||
|
static func select(english: Bool) -> Bool {
|
||||||
|
guard let src = source(english ? InputModes.english : InputModes.auto) else { return false }
|
||||||
|
if !isEnabled(src) { TISEnableInputSource(src) }
|
||||||
|
return TISSelectInputSource(src) == noErr
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Called by `--register`: enable both modes (if not yet) so they are in the input menu.
|
||||||
|
static func enableAll() {
|
||||||
|
for id in [auto, english] {
|
||||||
|
if let src = source(id), !isEnabled(src) { TISEnableInputSource(src) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,6 +6,7 @@ import InputMethodKit
|
|||||||
// in System Settings without logging out (used by the installer's postinstall script).
|
// in System Settings without logging out (used by the installer's postinstall script).
|
||||||
if CommandLine.arguments.contains("--register") {
|
if CommandLine.arguments.contains("--register") {
|
||||||
let status = TISRegisterInputSource(Bundle.main.bundleURL as CFURL)
|
let status = TISRegisterInputSource(Bundle.main.bundleURL as CFURL)
|
||||||
|
if status == noErr { InputModes.enableAll() } // "中英自動" and "英文" (menu-bar icons)
|
||||||
print(status == noErr ? "Registered \(Bundle.main.bundlePath)" : "TISRegisterInputSource failed: \(status)")
|
print(status == noErr ? "Registered \(Bundle.main.bundlePath)" : "TISRegisterInputSource failed: \(status)")
|
||||||
exit(status == noErr ? 0 : 1)
|
exit(status == noErr ? 0 : 1)
|
||||||
}
|
}
|
||||||
@@ -22,5 +23,6 @@ else {
|
|||||||
Shared.server = server
|
Shared.server = server
|
||||||
Shared.candidates = IMKCandidates(server: server, panelType: kIMKSingleColumnScrollingCandidatePanel)
|
Shared.candidates = IMKCandidates(server: server, panelType: kIMKSingleColumnScrollingCandidatePanel)
|
||||||
_ = Shared.classifier // load the lexicon now rather than on the first keystroke
|
_ = Shared.classifier // load the lexicon now rather than on the first keystroke
|
||||||
|
AI.loadModelIfNeeded() // "AI 選字" language model, in the background
|
||||||
|
|
||||||
NSApplication.shared.run()
|
NSApplication.shared.run()
|
||||||
|
|||||||
@@ -55,14 +55,17 @@ public final class Classifier {
|
|||||||
public let lexicon: Lexicon
|
public let lexicon: Lexicon
|
||||||
public let english: EnglishModel
|
public let english: EnglishModel
|
||||||
public let learned: LearnedPreferences
|
public let learned: LearnedPreferences
|
||||||
|
/// Picked candidates (shared by every engine using this classifier).
|
||||||
|
public let history: UserHistory
|
||||||
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
|
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
|
||||||
public private(set) var weights: FeatureVector
|
public private(set) var weights: FeatureVector
|
||||||
|
|
||||||
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
|
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
|
||||||
weights: FeatureVector = [:]) {
|
weights: FeatureVector = [:], history: UserHistory = UserHistory()) {
|
||||||
self.lexicon = lexicon
|
self.lexicon = lexicon
|
||||||
self.english = english
|
self.english = english
|
||||||
self.learned = learned
|
self.learned = learned
|
||||||
|
self.history = history
|
||||||
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
|
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -45,21 +45,29 @@ public final class CompositionEngine {
|
|||||||
private var fixed: [Int: [Range<Int>: String]] = [:]
|
private var fixed: [Int: [Range<Int>: String]] = [:]
|
||||||
public private(set) var candidates: [Candidate] = []
|
public private(set) var candidates: [Candidate] = []
|
||||||
private var candidateSegmentStart = 0
|
private var candidateSegmentStart = 0
|
||||||
/// Composition cursor for choosing candidates: index into `cursorStops`, nil = at the end.
|
/// Composition cursor: a key index into `keys` (nil = at the end). ← → move it between
|
||||||
/// ← → move it; ↓ shows candidates for the phrase ending at the cursor.
|
/// syllables; ↓ shows candidates for the phrase before it; typing there inserts a
|
||||||
|
/// syllable (kept in `inserting` until its tone key, then spliced into `keys`).
|
||||||
private var cursor: Int?
|
private var cursor: Int?
|
||||||
|
private var inserting: [Character] = []
|
||||||
public var candidateIndex = 0
|
public var candidateIndex = 0
|
||||||
public var isShowingCandidates: Bool { !candidates.isEmpty }
|
public var isShowingCandidates: Bool { !candidates.isEmpty }
|
||||||
|
|
||||||
/// Index of a held space: an English word that is also a common one-syllable Chinese
|
/// Index of a held space: the first word of a composition that also reads as Zhuyin is
|
||||||
/// reading ("up " = ㄧㄣ 因) is not committed on space; the next word decides
|
/// not committed as English on space; the next word decides ("up jo4" = 因為,
|
||||||
/// ("up jo4" = 因為, "up the" = up the).
|
/// "up the" = up the, "no su3" = no 你). Right after English text only common
|
||||||
|
/// one-syllable readings are held (up = 因, url = 教), so English typing isn't delayed.
|
||||||
private var held: Int?
|
private var held: Int?
|
||||||
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
|
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
|
||||||
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
|
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
|
||||||
public private(set) var pendingBacktick = false
|
public private(set) var pendingBacktick = false
|
||||||
static let holdMinScore = -3.5
|
static let holdMinScore = -3.5
|
||||||
|
|
||||||
|
/// "AI 選字" (language-model reranking); nil = lexicon only.
|
||||||
|
public var reranker: Reranker?
|
||||||
|
/// The last characters committed, used as context for the reranker.
|
||||||
|
private var recentText = ""
|
||||||
|
|
||||||
private var previousMode: InputMode?
|
private var previousMode: InputMode?
|
||||||
private var previousCommitTime = Date.distantPast
|
private var previousCommitTime = Date.distantPast
|
||||||
/// How long the previous segment's mode keeps influencing the next one.
|
/// How long the previous segment's mode keeps influencing the next one.
|
||||||
@@ -68,7 +76,7 @@ public final class CompositionEngine {
|
|||||||
|
|
||||||
public init(classifier: Classifier) {
|
public init(classifier: Classifier) {
|
||||||
self.classifier = classifier
|
self.classifier = classifier
|
||||||
self.converter = Converter(lexicon: classifier.lexicon)
|
self.converter = Converter(lexicon: classifier.lexicon, history: classifier.history)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Derived state
|
// MARK: - Derived state
|
||||||
@@ -92,19 +100,38 @@ public final class CompositionEngine {
|
|||||||
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
|
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
|
||||||
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
|
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
|
||||||
}
|
}
|
||||||
if let first = rest.first, first.mode == .zhuyin {
|
if let first = rest.first, first.mode == .zhuyin,
|
||||||
|
heldReadsAsChinese(keys, held: h, end: first.range.upperBound, finishing: finishing) {
|
||||||
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
|
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
|
||||||
}
|
}
|
||||||
return [Segment(range: 0..<base, mode: .literal)] + rest
|
return [Segment(range: 0..<base, mode: .literal)] + rest
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Chinese all the way to `end` vs. the held English word followed by Chinese.
|
||||||
|
private func heldReadsAsChinese(_ keys: [Character], held h: Int, end: Int, finishing: Bool) -> Bool {
|
||||||
|
let c = classifier, w = classifier.weights, ctx = context
|
||||||
|
let isLast = end == keys.count
|
||||||
|
guard let zhWhole = c.segmentFeatures(Array(keys[..<end]), isLast: isLast, afterOther: false,
|
||||||
|
finishing: finishing, previous: ctx).0 else { return false }
|
||||||
|
guard let word = c.segmentFeatures(Array(keys[..<h]), isLast: false, afterOther: false,
|
||||||
|
finishing: true, previous: ctx).1,
|
||||||
|
let zhRest = c.segmentFeatures(Array(keys[(h + 1)..<end]), isLast: isLast, afterOther: true,
|
||||||
|
finishing: finishing, previous: nil).0 else { return true }
|
||||||
|
return Features.dot(w, zhWhole) >= Features.dot(w, word) + Features.dot(w, zhRest)
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether keys + space also read as a common Chinese syllable (see `held`).
|
/// Whether keys + space also read as a common Chinese syllable (see `held`).
|
||||||
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
|
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
|
||||||
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
|
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
|
||||||
let p = ZhuyinParser.parse(keys + [" "])
|
let p = ZhuyinParser.parse(keys + [" "])
|
||||||
guard p.pending == nil, p.syllables.count == 1 else { return false }
|
guard p.pending == nil, !p.syllables.isEmpty, p.strayTones == 0, p.foreignKeys == 0 else { return false }
|
||||||
guard let best = classifier.lexicon.entries(for: p.syllables.map(\.syllable)).first else { return false }
|
var best: [Double] = []
|
||||||
return best.score >= CompositionEngine.holdMinScore
|
for ps in p.syllables {
|
||||||
|
guard let e = classifier.lexicon.entries(for: [ps.syllable]).first else { return false }
|
||||||
|
best.append(e.score)
|
||||||
|
}
|
||||||
|
if context != .literal { return true } // first word: hold anything that reads as Zhuyin
|
||||||
|
return best.count == 1 && best[0] >= CompositionEngine.holdMinScore
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Current split of the buffer into Zhuyin / literal runs.
|
/// Current split of the buffer into Zhuyin / literal runs.
|
||||||
@@ -117,14 +144,35 @@ public final class CompositionEngine {
|
|||||||
public var markedText: String { markedPieces.map(\.text).joined() }
|
public var markedText: String { markedPieces.map(\.text).joined() }
|
||||||
|
|
||||||
public var markedPieces: [MarkedPiece] {
|
public var markedPieces: [MarkedPiece] {
|
||||||
let segs = segments
|
var pieces = segmentPieces
|
||||||
var pieces = segs.enumerated().map { i, seg in
|
if !inserting.isEmpty {
|
||||||
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
|
// Show the syllable being inserted at the cursor.
|
||||||
|
var at = baseCursorOffset, out: [MarkedPiece] = [], done = false
|
||||||
|
for piece in pieces {
|
||||||
|
if !done && at <= piece.text.count {
|
||||||
|
let i = piece.text.index(piece.text.startIndex, offsetBy: at)
|
||||||
|
out.append(MarkedPiece(text: String(piece.text[..<i]), mode: piece.mode))
|
||||||
|
out.append(MarkedPiece(text: insertingText, mode: .zhuyin))
|
||||||
|
out.append(MarkedPiece(text: String(piece.text[i...]), mode: piece.mode))
|
||||||
|
done = true
|
||||||
|
} else {
|
||||||
|
out.append(piece)
|
||||||
|
at -= piece.text.count
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pieces = out.filter { !$0.text.isEmpty }
|
||||||
}
|
}
|
||||||
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
|
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
|
||||||
return pieces
|
return pieces
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private var segmentPieces: [MarkedPiece] {
|
||||||
|
let segs = segments
|
||||||
|
return segs.enumerated().map { i, seg in
|
||||||
|
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
|
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
|
||||||
private func render(_ seg: Segment, final: Bool) -> String {
|
private func render(_ seg: Segment, final: Bool) -> String {
|
||||||
guard seg.mode == .zhuyin else { return String(keys[seg.range]) }
|
guard seg.mode == .zhuyin else { return String(keys[seg.range]) }
|
||||||
@@ -143,48 +191,167 @@ public final class CompositionEngine {
|
|||||||
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
|
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
|
||||||
let (syl, pending) = convertibleSyllables(seg, final: final)
|
let (syl, pending) = convertibleSyllables(seg, final: final)
|
||||||
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
|
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
|
||||||
|
if let reranker, let chosen = reranker.choose(converter, syl, fixed: fixedHere, context: rerankContext) {
|
||||||
|
return (chosen, pending)
|
||||||
|
}
|
||||||
return (converter.convert(syl, fixed: fixedHere), pending)
|
return (converter.convert(syl, fixed: fixedHere), pending)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private var rerankContext: String {
|
||||||
|
now().timeIntervalSince(previousCommitTime) < contextWindow ? recentText : ""
|
||||||
|
}
|
||||||
|
|
||||||
// MARK: - Cursor
|
// MARK: - Cursor
|
||||||
|
|
||||||
/// A place the cursor can stop: after the `local`-th syllable of Zhuyin segment `segIndex`.
|
/// Absolute key index after each syllable of a Zhuyin segment.
|
||||||
|
private func syllableEnds(_ seg: Segment) -> [Int] {
|
||||||
|
ZhuyinParser.parse(Array(keys[seg.range])).syllables.map { seg.range.lowerBound + $0.keyRange.upperBound }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Key indices the cursor can stop at: after every Zhuyin syllable and at every segment
|
||||||
|
/// boundary, but never before the first Chinese character (there would be nothing to
|
||||||
|
/// pick candidates for).
|
||||||
|
private var cursorPositions: [Int] {
|
||||||
|
var out: Set<Int> = [keys.count]
|
||||||
|
var ends: [Int] = []
|
||||||
|
for seg in segments {
|
||||||
|
out.insert(seg.range.upperBound)
|
||||||
|
if seg.mode == .zhuyin { ends += syllableEnds(seg) }
|
||||||
|
}
|
||||||
|
out.formUnion(ends)
|
||||||
|
let first = ends.min() ?? keys.count
|
||||||
|
return out.filter { $0 >= first }.sorted()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A place to open candidates: after the `local`-th syllable of Zhuyin segment `segIndex`.
|
||||||
private struct CursorStop { let segIndex: Int; let local: Int }
|
private struct CursorStop { let segIndex: Int; let local: Int }
|
||||||
|
|
||||||
private var cursorStops: [CursorStop] {
|
private func stop(at pos: Int) -> CursorStop? {
|
||||||
let segs = segments
|
let segs = segments
|
||||||
var out: [CursorStop] = []
|
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin
|
||||||
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin {
|
&& seg.range.lowerBound < pos && pos <= seg.range.upperBound {
|
||||||
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
|
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
|
||||||
for k in stride(from: 1, through: n, by: 1) { out.append(CursorStop(segIndex: i, local: k)) }
|
let local = syllableEnds(seg).filter { $0 <= pos }.count
|
||||||
|
return local > 0 ? CursorStop(segIndex: i, local: min(local, n)) : nil
|
||||||
}
|
}
|
||||||
return out
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Where ↓ opens candidates: before the cursor, or the last syllable.
|
||||||
|
private var candidateStop: CursorStop? {
|
||||||
|
if let c = cursor { return stop(at: c) }
|
||||||
|
let segs = segments
|
||||||
|
for i in segs.indices.reversed() where segs[i].mode == .zhuyin {
|
||||||
|
let n = convertibleSyllables(segs[i], final: i < segs.count - 1).syllables.count
|
||||||
|
if n > 0 { return CursorStop(segIndex: i, local: n) }
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cursor position in the rendered segments (without the syllable being inserted).
|
||||||
|
private var baseCursorOffset: Int {
|
||||||
|
let pieces = segmentPieces
|
||||||
|
guard let c = cursor else { return pieces.reduce(0) { $0 + $1.text.count } }
|
||||||
|
let segs = segments
|
||||||
|
for (i, seg) in segs.enumerated() where seg.range.lowerBound < c && c <= seg.range.upperBound {
|
||||||
|
var offset = pieces[..<i].reduce(0) { $0 + $1.text.count }
|
||||||
|
if seg.mode == .literal { return offset + c - seg.range.lowerBound }
|
||||||
|
let local = stop(at: c)?.local ?? 0
|
||||||
|
for conv in conversion(of: seg, final: i < segs.count - 1).segments {
|
||||||
|
if conv.range.upperBound <= local {
|
||||||
|
offset += conv.text.count
|
||||||
|
} else if conv.range.lowerBound < local {
|
||||||
|
offset += min(local - conv.range.lowerBound, conv.text.count)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return offset
|
||||||
|
}
|
||||||
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Cursor position in `markedText`, in Characters.
|
/// Cursor position in `markedText`, in Characters.
|
||||||
public var cursorOffset: Int {
|
public var cursorOffset: Int { baseCursorOffset + insertingText.count }
|
||||||
let pieces = markedPieces
|
|
||||||
let total = pieces.reduce(0) { $0 + $1.text.count }
|
private var insertingText: String {
|
||||||
let stops = cursorStops
|
inserting.isEmpty ? "" : ZhuyinParser.parse(inserting).syllables.map(\.syllable.reading).joined()
|
||||||
guard let c = cursor, stops.indices.contains(c) else { return total }
|
|
||||||
let stop = stops[c]
|
|
||||||
var offset = pieces[..<stop.segIndex].reduce(0) { $0 + $1.text.count }
|
|
||||||
let segs = segments
|
|
||||||
for conv in conversion(of: segs[stop.segIndex], final: stop.segIndex < segs.count - 1).segments {
|
|
||||||
if conv.range.upperBound <= stop.local {
|
|
||||||
offset += conv.text.count
|
|
||||||
} else if conv.range.lowerBound < stop.local {
|
|
||||||
offset += min(stop.local - conv.range.lowerBound, conv.text.count)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return offset
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func moveCursor(by delta: Int) {
|
private func moveCursor(by delta: Int) {
|
||||||
let stops = cursorStops
|
inserting = []
|
||||||
guard !stops.isEmpty else { return }
|
let stops = cursorPositions
|
||||||
let next = (cursor ?? stops.count - 1) + delta
|
let here = cursor ?? keys.count
|
||||||
cursor = next >= stops.count - 1 ? nil : max(0, next)
|
let next = delta < 0 ? (stops.last { $0 < here } ?? here) : (stops.first { $0 > here } ?? here)
|
||||||
|
cursor = next >= keys.count ? nil : next
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Editing in the middle
|
||||||
|
|
||||||
|
/// Keeps picked phrases that end at or before key index `pos` (they don't move).
|
||||||
|
private func trimFixed(at pos: Int) {
|
||||||
|
var kept: [Int: [Range<Int>: String]] = [:]
|
||||||
|
for seg in segments where seg.mode == .zhuyin {
|
||||||
|
guard let picks = fixed[seg.range.lowerBound] else { continue }
|
||||||
|
if seg.range.upperBound <= pos {
|
||||||
|
kept[seg.range.lowerBound] = picks
|
||||||
|
} else if seg.range.lowerBound < pos {
|
||||||
|
let ends = syllableEnds(seg)
|
||||||
|
let here = picks.filter { $0.key.upperBound <= ends.count && ends[$0.key.upperBound - 1] <= pos }
|
||||||
|
if !here.isEmpty { kept[seg.range.lowerBound] = here }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fixed = kept
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Inserts keys at the cursor and moves the cursor after them.
|
||||||
|
private func splice(_ newKeys: [Character]) {
|
||||||
|
guard let pos = cursor else { keys += newKeys; return }
|
||||||
|
trimFixed(at: pos)
|
||||||
|
keys.insert(contentsOf: newKeys, at: pos)
|
||||||
|
if let h = held, h >= pos { held = h + newKeys.count }
|
||||||
|
let next = pos + newKeys.count
|
||||||
|
cursor = next >= keys.count ? nil : next
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Backspace in the middle: removes the syllable (or literal key) before the cursor.
|
||||||
|
private func deleteBeforeCursor(_ pos: Int) {
|
||||||
|
var start = pos - 1
|
||||||
|
for seg in segments where seg.mode == .zhuyin && seg.range.lowerBound < pos && pos <= seg.range.upperBound {
|
||||||
|
for ps in ZhuyinParser.parse(Array(keys[seg.range])).syllables
|
||||||
|
where seg.range.lowerBound + ps.keyRange.upperBound == pos {
|
||||||
|
start = seg.range.lowerBound + ps.keyRange.lowerBound
|
||||||
|
}
|
||||||
|
}
|
||||||
|
trimFixed(at: start)
|
||||||
|
keys.removeSubrange(start..<pos)
|
||||||
|
if let h = held {
|
||||||
|
if h >= start && h < pos { held = nil } else if h >= pos { held = h - (pos - start) }
|
||||||
|
}
|
||||||
|
cursor = start
|
||||||
|
if keys.isEmpty { reset() }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Typing with the cursor in the middle: compose one syllable, splice it in on its tone.
|
||||||
|
private func insertKey(_ c: Character) -> KeyResult {
|
||||||
|
if c == " " {
|
||||||
|
if inserting.isEmpty { return KeyResult(handled: true, commit: commitCurrent()) }
|
||||||
|
splice(inserting + [" "])
|
||||||
|
inserting = []
|
||||||
|
return .consumed
|
||||||
|
}
|
||||||
|
inserting.append(c)
|
||||||
|
let p = ZhuyinParser.parse(inserting)
|
||||||
|
if p.strayTones > 0 || p.foreignKeys > 0 {
|
||||||
|
inserting.removeLast() // a tone with nothing to attach to
|
||||||
|
} else if p.syllables.count == 1, p.syllables[0].toned {
|
||||||
|
splice(inserting)
|
||||||
|
inserting = []
|
||||||
|
} else if p.syllables.count > 1 {
|
||||||
|
// A new syllable started without a tone: splice the first one as it is.
|
||||||
|
let first = p.syllables[0].keyRange.upperBound
|
||||||
|
splice(Array(inserting[..<first]))
|
||||||
|
inserting = Array(inserting[first...])
|
||||||
|
}
|
||||||
|
return .consumed
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Key handling
|
// MARK: - Key handling
|
||||||
@@ -192,13 +359,20 @@ public final class CompositionEngine {
|
|||||||
public func handle(_ key: KeyInput) -> KeyResult {
|
public func handle(_ key: KeyInput) -> KeyResult {
|
||||||
if isShowingCandidates { return handleCandidateKey(key) }
|
if isShowingCandidates { return handleCandidateKey(key) }
|
||||||
if pendingBacktick { return afterBacktick(key) }
|
if pendingBacktick { return afterBacktick(key) }
|
||||||
|
if !inserting.isEmpty && (key == .backspace || key == .escape) {
|
||||||
|
if key == .backspace { inserting.removeLast() } else { inserting = [] }
|
||||||
|
return .consumed
|
||||||
|
}
|
||||||
|
|
||||||
switch key {
|
switch key {
|
||||||
case .character(let c):
|
case .character(let c):
|
||||||
return handleCharacter(c)
|
return handleCharacter(c)
|
||||||
case .backspace:
|
case .backspace:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
cursor = nil
|
if let c = cursor {
|
||||||
|
if c > 0 { deleteBeforeCursor(c) }
|
||||||
|
return .consumed
|
||||||
|
}
|
||||||
keys.removeLast()
|
keys.removeLast()
|
||||||
if let h = held, keys.count <= h { held = nil }
|
if let h = held, keys.count <= h { held = nil }
|
||||||
fixed = fixed.filter { $0.key < keys.count }
|
fixed = fixed.filter { $0.key < keys.count }
|
||||||
@@ -206,6 +380,7 @@ public final class CompositionEngine {
|
|||||||
return .consumed
|
return .consumed
|
||||||
case .enter:
|
case .enter:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
|
if !inserting.isEmpty { splice(inserting); inserting = [] }
|
||||||
return KeyResult(handled: true, commit: commitCurrent())
|
return KeyResult(handled: true, commit: commitCurrent())
|
||||||
case .escape:
|
case .escape:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
@@ -227,7 +402,7 @@ public final class CompositionEngine {
|
|||||||
return .consumed
|
return .consumed
|
||||||
case .down:
|
case .down:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
openCandidates()
|
if inserting.isEmpty { openCandidates() }
|
||||||
return .consumed
|
return .consumed
|
||||||
case .left:
|
case .left:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
@@ -242,13 +417,27 @@ public final class CompositionEngine {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Turns English mode on or off directly (e.g. the input mode was switched in the menu
|
||||||
|
/// bar). Commit any composition first.
|
||||||
|
public func setEnglishMode(_ on: Bool) {
|
||||||
|
guard on != isEnglishMode else { return }
|
||||||
|
reset()
|
||||||
|
isEnglishMode = on
|
||||||
|
previousMode = on ? .literal : nil
|
||||||
|
previousCommitTime = now()
|
||||||
|
}
|
||||||
|
|
||||||
/// Forces the whole buffer into the other mode (a correction that is learned).
|
/// Forces the whole buffer into the other mode (a correction that is learned).
|
||||||
private func flip() {
|
private func flip() {
|
||||||
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
|
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
|
||||||
}
|
}
|
||||||
|
|
||||||
private func handleCharacter(_ c: Character) -> KeyResult {
|
private func handleCharacter(_ c: Character) -> KeyResult {
|
||||||
cursor = nil // typing always continues at the end
|
if cursor != nil, !keys.isEmpty, c == " " || DachenLayout.isZhuyinKey(c) {
|
||||||
|
return insertKey(c)
|
||||||
|
}
|
||||||
|
if !inserting.isEmpty { splice(inserting); inserting = [] }
|
||||||
|
cursor = nil // anything else continues at the end
|
||||||
// English mode: every key goes to the app as typed.
|
// English mode: every key goes to the app as typed.
|
||||||
if isEnglishMode && keys.isEmpty { return .passThrough }
|
if isEnglishMode && keys.isEmpty { return .passThrough }
|
||||||
if c == "`" {
|
if c == "`" {
|
||||||
@@ -329,9 +518,7 @@ public final class CompositionEngine {
|
|||||||
|
|
||||||
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
|
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
|
||||||
private func openCandidates() {
|
private func openCandidates() {
|
||||||
let stops = cursorStops
|
guard let stop = candidateStop else { return }
|
||||||
guard !stops.isEmpty else { return }
|
|
||||||
let stop = stops[min(cursor ?? stops.count - 1, stops.count - 1)]
|
|
||||||
let segs = segments
|
let segs = segments
|
||||||
let seg = segs[stop.segIndex]
|
let seg = segs[stop.segIndex]
|
||||||
let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
|
let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
|
||||||
@@ -376,6 +563,8 @@ public final class CompositionEngine {
|
|||||||
private func commitCurrent(_ given: [Segment]? = nil) -> String {
|
private func commitCurrent(_ given: [Segment]? = nil) -> String {
|
||||||
let segs = given ?? segments
|
let segs = given ?? segments
|
||||||
let text = segs.map { render($0, final: true) }.joined()
|
let text = segs.map { render($0, final: true) }.joined()
|
||||||
|
rememberPicks(segs)
|
||||||
|
recentText = String((rerankContext + text).suffix(20))
|
||||||
if let forced = forcedMode {
|
if let forced = forcedMode {
|
||||||
classifier.learned.record(String(keys), as: forced)
|
classifier.learned.record(String(keys), as: forced)
|
||||||
}
|
}
|
||||||
@@ -385,6 +574,17 @@ public final class CompositionEngine {
|
|||||||
return text
|
return text
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Candidates the user picked are remembered and come first next time.
|
||||||
|
private func rememberPicks(_ segs: [Segment]) {
|
||||||
|
for seg in segs where seg.mode == .zhuyin {
|
||||||
|
guard let picks = fixed[seg.range.lowerBound], !picks.isEmpty else { continue }
|
||||||
|
let syl = convertibleSyllables(seg, final: true).syllables
|
||||||
|
for (range, text) in picks where range.upperBound <= syl.count {
|
||||||
|
classifier.history.record(Array(syl[range]), phrase: text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty.
|
/// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty.
|
||||||
public func commitAll() -> String {
|
public func commitAll() -> String {
|
||||||
keys.isEmpty ? "" : commitCurrent()
|
keys.isEmpty ? "" : commitCurrent()
|
||||||
@@ -392,6 +592,7 @@ public final class CompositionEngine {
|
|||||||
|
|
||||||
public func reset() {
|
public func reset() {
|
||||||
keys = []
|
keys = []
|
||||||
|
inserting = []
|
||||||
held = nil
|
held = nil
|
||||||
pendingBacktick = false
|
pendingBacktick = false
|
||||||
cursor = nil
|
cursor = nil
|
||||||
|
|||||||
@@ -13,12 +13,22 @@ public struct Candidate: Equatable {
|
|||||||
/// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon).
|
/// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon).
|
||||||
public struct Converter {
|
public struct Converter {
|
||||||
public let lexicon: Lexicon
|
public let lexicon: Lexicon
|
||||||
|
/// Picked candidates come first.
|
||||||
|
public let history: UserHistory?
|
||||||
/// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted.
|
/// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted.
|
||||||
static let unknownScore = -30.0
|
static let unknownScore = -30.0
|
||||||
/// Penalty for matching an untoned syllable against all tones.
|
/// Penalty for matching an untoned syllable against all tones.
|
||||||
static let tonelessPenalty = -1.5
|
static let tonelessPenalty = -1.5
|
||||||
|
|
||||||
public init(lexicon: Lexicon) { self.lexicon = lexicon }
|
public init(lexicon: Lexicon, history: UserHistory? = nil) {
|
||||||
|
self.lexicon = lexicon
|
||||||
|
self.history = history
|
||||||
|
}
|
||||||
|
|
||||||
|
private func entries(_ syllables: [Syllable]) -> [LexiconEntry] {
|
||||||
|
let e = lexicon.entries(for: syllables)
|
||||||
|
return history?.adjust(syllables, e) ?? e
|
||||||
|
}
|
||||||
|
|
||||||
/// - Parameter fixed: user-chosen phrases by syllable range; they are always kept.
|
/// - Parameter fixed: user-chosen phrases by syllable range; they are always kept.
|
||||||
public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] {
|
public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] {
|
||||||
@@ -51,13 +61,39 @@ public struct Converter {
|
|||||||
return segments.reversed()
|
return segments.reversed()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The `k` best paths (log10 score, segments), best first — for language-model reranking.
|
||||||
|
public func kBest(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:], k: Int) -> [(score: Double, segments: [ConvertedSegment])] {
|
||||||
|
let n = syllables.count
|
||||||
|
guard n > 0, k > 0 else { return [] }
|
||||||
|
let maxLen = max(1, lexicon.maxPhraseLength)
|
||||||
|
typealias Path = (score: Double, segments: [ConvertedSegment])
|
||||||
|
var beams: [[Path]] = Array(repeating: [], count: n + 1)
|
||||||
|
beams[0] = [(0, [])]
|
||||||
|
for j in 1...n {
|
||||||
|
var cand: [Path] = []
|
||||||
|
for len in 1...min(maxLen, j) {
|
||||||
|
let i = j - len
|
||||||
|
guard !beams[i].isEmpty else { continue }
|
||||||
|
let range = i..<j
|
||||||
|
for (text, score) in options(for: range, syllables: syllables, fixed: fixed) {
|
||||||
|
for path in beams[i] {
|
||||||
|
cand.append((path.score + score, path.segments + [ConvertedSegment(range: range, text: text)]))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cand.sort { $0.score > $1.score }
|
||||||
|
beams[j] = Array(cand.prefix(k))
|
||||||
|
}
|
||||||
|
return beams[n]
|
||||||
|
}
|
||||||
|
|
||||||
private func options(for range: Range<Int>, syllables: [Syllable], fixed: [Range<Int>: String]) -> [(String, Double)] {
|
private func options(for range: Range<Int>, syllables: [Syllable], fixed: [Range<Int>: String]) -> [(String, Double)] {
|
||||||
if let text = fixed[range] { return [(text, 0)] }
|
if let text = fixed[range] { return [(text, 0)] }
|
||||||
// A span may not partially overlap or swallow a fixed span.
|
// A span may not partially overlap or swallow a fixed span.
|
||||||
if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] }
|
if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] }
|
||||||
|
|
||||||
let slice = Array(syllables[range])
|
let slice = Array(syllables[range])
|
||||||
var result = lexicon.entries(for: slice).map { entry -> (String, Double) in
|
var result = entries(slice).map { entry -> (String, Double) in
|
||||||
let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0
|
let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0
|
||||||
return (entry.phrase, entry.score + penalty)
|
return (entry.phrase, entry.score + penalty)
|
||||||
}
|
}
|
||||||
@@ -74,7 +110,7 @@ public struct Converter {
|
|||||||
var seen = Set<String>()
|
var seen = Set<String>()
|
||||||
for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) {
|
for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) {
|
||||||
let range = (end - len)..<end
|
let range = (end - len)..<end
|
||||||
for entry in lexicon.entries(for: Array(syllables[range])) where !seen.contains(entry.phrase) {
|
for entry in entries(Array(syllables[range])) where !seen.contains(entry.phrase) {
|
||||||
seen.insert(entry.phrase)
|
seen.insert(entry.phrase)
|
||||||
out.append(Candidate(range: range, text: entry.phrase))
|
out.append(Candidate(range: range, text: entry.phrase))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,242 @@
|
|||||||
|
import Accelerate
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// Scores Chinese text: natural-log probability of `text` following `context`.
|
||||||
|
public protocol LanguageModel: AnyObject {
|
||||||
|
func logProb(_ text: String, context: String) -> Double
|
||||||
|
}
|
||||||
|
|
||||||
|
/// GPT-2 forward pass (CPU, Accelerate) for the character-level CKIP gpt2-tiny-chinese
|
||||||
|
/// model (GPL-3.0), stored by training/lm/export_lm.py as float16 ("AZLM" format).
|
||||||
|
/// Weights are expanded to float32 in memory (~46 MB for gpt2-tiny).
|
||||||
|
public final class GPT2LanguageModel: LanguageModel {
|
||||||
|
public enum LoadError: Error { case badFormat(String) }
|
||||||
|
|
||||||
|
let nLayer: Int, nHead: Int, d: Int, vocabSize: Int
|
||||||
|
let eps: Float
|
||||||
|
private var weights: [Float] = []
|
||||||
|
private var offsets: [String: Int] = [:]
|
||||||
|
private var vocab: [Character: Int32] = [:]
|
||||||
|
static let cls: Int32 = 101, unk: Int32 = 100
|
||||||
|
/// Longest input scored (context + text); longer contexts are cut from the left.
|
||||||
|
public var maxTokens = 64
|
||||||
|
|
||||||
|
public init(modelURL: URL, vocabURL: URL) throws {
|
||||||
|
let data = try Data(contentsOf: modelURL, options: .mappedIfSafe)
|
||||||
|
guard data.count > 8, data.prefix(4) == Data("AZLM".utf8) else { throw LoadError.badFormat("magic") }
|
||||||
|
let headerLength = Int(data[4]) | Int(data[5]) << 8 | Int(data[6]) << 16 | Int(data[7]) << 24
|
||||||
|
guard let header = try JSONSerialization.jsonObject(with: data.subdata(in: 8..<(8 + headerLength))) as? [String: Any],
|
||||||
|
let nLayer = header["n_layer"] as? Int, let nHead = header["n_head"] as? Int,
|
||||||
|
let d = header["n_embd"] as? Int, let vocabSize = header["vocab"] as? Int,
|
||||||
|
let eps = header["eps"] as? Double, let tensors = header["tensors"] as? [[Any]]
|
||||||
|
else { throw LoadError.badFormat("header") }
|
||||||
|
self.nLayer = nLayer; self.nHead = nHead; self.d = d; self.vocabSize = vocabSize; self.eps = Float(eps)
|
||||||
|
for t in tensors {
|
||||||
|
guard let name = t[0] as? String, let offset = t[2] as? Int else { throw LoadError.badFormat("tensor") }
|
||||||
|
offsets[name] = offset
|
||||||
|
}
|
||||||
|
let start = 8 + headerLength
|
||||||
|
let count = (data.count - start) / 2
|
||||||
|
weights = [Float](repeating: 0, count: count)
|
||||||
|
try data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
|
||||||
|
guard let base = raw.baseAddress?.advanced(by: start) else { throw LoadError.badFormat("data") }
|
||||||
|
try weights.withUnsafeMutableBufferPointer { out in
|
||||||
|
var src = vImage_Buffer(data: UnsafeMutableRawPointer(mutating: base), height: 1,
|
||||||
|
width: vImagePixelCount(count), rowBytes: count * 2)
|
||||||
|
var dst = vImage_Buffer(data: out.baseAddress, height: 1,
|
||||||
|
width: vImagePixelCount(count), rowBytes: count * 4)
|
||||||
|
guard vImageConvert_Planar16FtoPlanarF(&src, &dst, 0) == kvImageNoError else {
|
||||||
|
throw LoadError.badFormat("float16")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var index: Int32 = 0
|
||||||
|
for line in try String(contentsOf: vocabURL, encoding: .utf8).split(separator: "\n", omittingEmptySubsequences: false) {
|
||||||
|
if line.count == 1, let c = line.first, vocab[c] == nil { vocab[c] = index }
|
||||||
|
index += 1
|
||||||
|
}
|
||||||
|
for name in ["wte.weight", "wpe.weight", "ln_f.weight"] where offsets[name] == nil {
|
||||||
|
throw LoadError.badFormat("missing \(name)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func tokens(_ text: String) -> [Int32] { text.map { vocab[$0] ?? GPT2LanguageModel.unk } }
|
||||||
|
|
||||||
|
public func logProb(_ text: String, context: String) -> Double {
|
||||||
|
let target = tokens(text)
|
||||||
|
guard !target.isEmpty else { return 0 }
|
||||||
|
var ctx = tokens(context)
|
||||||
|
if ctx.count + target.count + 1 > maxTokens { ctx = Array(ctx.suffix(max(0, maxTokens - target.count - 1))) }
|
||||||
|
let input = [GPT2LanguageModel.cls] + ctx + target.dropLast()
|
||||||
|
let h = hidden(input) // [T, d]
|
||||||
|
return weights.withUnsafeBufferPointer { w in
|
||||||
|
logSoftmaxScore(h, rows: input.count, from: ctx.count, targets: target, w: w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Forward pass
|
||||||
|
|
||||||
|
private func w(_ name: String) -> Int { offsets[name]! }
|
||||||
|
|
||||||
|
/// Final hidden states [T, d] (row-major).
|
||||||
|
func hidden(_ ids: [Int32]) -> [Float] {
|
||||||
|
let T = ids.count, d = self.d
|
||||||
|
var x = [Float](repeating: 0, count: T * d)
|
||||||
|
weights.withUnsafeBufferPointer { W in
|
||||||
|
let wte = w("wte.weight"), wpe = w("wpe.weight")
|
||||||
|
for t in 0..<T {
|
||||||
|
vDSP_vadd(W.baseAddress! + wte + Int(ids[t]) * d, 1, W.baseAddress! + wpe + t * d, 1,
|
||||||
|
&x[t * d], 1, vDSP_Length(d))
|
||||||
|
}
|
||||||
|
for l in 0..<nLayer {
|
||||||
|
let p = "h.\(l)."
|
||||||
|
var h = layerNorm(x, rows: T, p + "ln_1", W)
|
||||||
|
let qkv = linear(h, rows: T, inDim: d, outDim: 3 * d, p + "attn.c_attn", W)
|
||||||
|
let a = attention(qkv, T: T)
|
||||||
|
var proj = linear(a, rows: T, inDim: d, outDim: d, p + "attn.c_proj", W)
|
||||||
|
GPT2LanguageModel.add(&x, proj)
|
||||||
|
h = layerNorm(x, rows: T, p + "ln_2", W)
|
||||||
|
var f = linear(h, rows: T, inDim: d, outDim: 4 * d, p + "mlp.c_fc", W)
|
||||||
|
geluNew(&f)
|
||||||
|
proj = linear(f, rows: T, inDim: 4 * d, outDim: d, p + "mlp.c_proj", W)
|
||||||
|
GPT2LanguageModel.add(&x, proj)
|
||||||
|
}
|
||||||
|
x = layerNorm(x, rows: T, "ln_f", W)
|
||||||
|
}
|
||||||
|
return x
|
||||||
|
}
|
||||||
|
|
||||||
|
private func layerNorm(_ x: [Float], rows: Int, _ name: String, _ W: UnsafeBufferPointer<Float>) -> [Float] {
|
||||||
|
let d = self.d
|
||||||
|
var out = [Float](repeating: 0, count: x.count)
|
||||||
|
let g = W.baseAddress! + w(name + ".weight"), b = W.baseAddress! + w(name + ".bias")
|
||||||
|
for r in 0..<rows {
|
||||||
|
var mean: Float = 0, sd: Float = 0
|
||||||
|
x.withUnsafeBufferPointer { xp in
|
||||||
|
out.withUnsafeMutableBufferPointer { op in
|
||||||
|
let src = xp.baseAddress! + r * d, dst = op.baseAddress! + r * d
|
||||||
|
vDSP_normalize(src, 1, dst, 1, &mean, &sd, vDSP_Length(d))
|
||||||
|
// vDSP_normalize divides by the population std without eps; redo with eps.
|
||||||
|
let scale = sd / (sd * sd + eps).squareRoot()
|
||||||
|
var s = scale
|
||||||
|
vDSP_vsmul(dst, 1, &s, dst, 1, vDSP_Length(d))
|
||||||
|
vDSP_vma(dst, 1, g, 1, b, 1, dst, 1, vDSP_Length(d))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// x [rows, inDim] · W [inDim, outDim] + bias.
|
||||||
|
private func linear(_ x: [Float], rows: Int, inDim: Int, outDim: Int, _ name: String,
|
||||||
|
_ W: UnsafeBufferPointer<Float>) -> [Float] {
|
||||||
|
var out = [Float](repeating: 0, count: rows * outDim)
|
||||||
|
vDSP_mmul(x, 1, W.baseAddress! + w(name + ".weight"), 1, &out, 1,
|
||||||
|
vDSP_Length(rows), vDSP_Length(outDim), vDSP_Length(inDim))
|
||||||
|
let bias = W.baseAddress! + w(name + ".bias")
|
||||||
|
out.withUnsafeMutableBufferPointer { op in
|
||||||
|
for r in 0..<rows {
|
||||||
|
vDSP_vadd(op.baseAddress! + r * outDim, 1, bias, 1, op.baseAddress! + r * outDim, 1, vDSP_Length(outDim))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Causal multi-head self-attention. qkv is [T, 3d] (q | k | v).
|
||||||
|
private func attention(_ qkv: [Float], T: Int) -> [Float] {
|
||||||
|
let d = self.d, hd = d / nHead
|
||||||
|
let scale = 1 / Float(hd).squareRoot()
|
||||||
|
var out = [Float](repeating: 0, count: T * d)
|
||||||
|
var scores = [Float](repeating: 0, count: T)
|
||||||
|
qkv.withUnsafeBufferPointer { q in
|
||||||
|
out.withUnsafeMutableBufferPointer { o in
|
||||||
|
for head in 0..<nHead {
|
||||||
|
for i in 0..<T {
|
||||||
|
let qi = q.baseAddress! + i * 3 * d + head * hd
|
||||||
|
var maxScore = -Float.infinity
|
||||||
|
for j in 0...i {
|
||||||
|
var s: Float = 0
|
||||||
|
vDSP_dotpr(qi, 1, q.baseAddress! + j * 3 * d + d + head * hd, 1, &s, vDSP_Length(hd))
|
||||||
|
scores[j] = s * scale
|
||||||
|
maxScore = max(maxScore, scores[j])
|
||||||
|
}
|
||||||
|
var sum: Float = 0
|
||||||
|
for j in 0...i { scores[j] = expf(scores[j] - maxScore); sum += scores[j] }
|
||||||
|
let dst = o.baseAddress! + i * d + head * hd
|
||||||
|
for j in 0...i {
|
||||||
|
var a = scores[j] / sum
|
||||||
|
vDSP_vsma(q.baseAddress! + j * 3 * d + 2 * d + head * hd, 1, &a, dst, 1, dst, 1, vDSP_Length(hd))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// x += y (same length).
|
||||||
|
private static func add(_ x: inout [Float], _ y: [Float]) {
|
||||||
|
x.withUnsafeMutableBufferPointer { xp in
|
||||||
|
vDSP_vadd(xp.baseAddress!, 1, y, 1, xp.baseAddress!, 1, vDSP_Length(xp.count))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// gelu_new: 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))).
|
||||||
|
private func geluNew(_ x: inout [Float]) {
|
||||||
|
let n = x.count
|
||||||
|
var inner = [Float](repeating: 0, count: n)
|
||||||
|
x.withUnsafeMutableBufferPointer { xb in
|
||||||
|
inner.withUnsafeMutableBufferPointer { ib in
|
||||||
|
let xp = xb.baseAddress!, ip = ib.baseAddress!, len = vDSP_Length(n)
|
||||||
|
var c3: Float = 0.044715, one: Float = 1, k = (2 / Float.pi).squareRoot(), half: Float = 0.5
|
||||||
|
vDSP_vsq(xp, 1, ip, 1, len) // x²
|
||||||
|
vDSP_vsmsa(ip, 1, &c3, &one, ip, 1, len) // 1 + 0.044715·x²
|
||||||
|
vDSP_vmul(ip, 1, xp, 1, ip, 1, len) // x + 0.044715·x³
|
||||||
|
vDSP_vsmul(ip, 1, &k, ip, 1, len)
|
||||||
|
var count = Int32(n)
|
||||||
|
vvtanhf(ip, ip, &count)
|
||||||
|
vDSP_vsadd(ip, 1, &one, ip, 1, len) // 1 + tanh(…)
|
||||||
|
vDSP_vmul(ip, 1, xp, 1, xp, 1, len)
|
||||||
|
vDSP_vsmul(xp, 1, &half, xp, 1, len)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Sum over target positions of log softmax(h · wteᵀ)[target]. Row `from + i` predicts target i.
|
||||||
|
private func logSoftmaxScore(_ h: [Float], rows T: Int, from start: Int, targets: [Int32],
|
||||||
|
w W: UnsafeBufferPointer<Float>) -> Double {
|
||||||
|
let d = self.d, V = vocabSize
|
||||||
|
let rowsUsed = T - start
|
||||||
|
// hᵀ for the rows we need: [d, rowsUsed]
|
||||||
|
var hT = [Float](repeating: 0, count: d * rowsUsed)
|
||||||
|
h.withUnsafeBufferPointer { hp in
|
||||||
|
vDSP_mtrans(hp.baseAddress! + start * d, 1, &hT, 1, vDSP_Length(d), vDSP_Length(rowsUsed))
|
||||||
|
}
|
||||||
|
// logitsᵀ [V, rowsUsed] = wte [V, d] · hᵀ [d, rowsUsed]
|
||||||
|
var logitsT = [Float](repeating: 0, count: V * rowsUsed)
|
||||||
|
vDSP_mmul(W.baseAddress! + w("wte.weight"), 1, hT, 1, &logitsT, 1,
|
||||||
|
vDSP_Length(V), vDSP_Length(rowsUsed), vDSP_Length(d))
|
||||||
|
var logits = [Float](repeating: 0, count: V * rowsUsed) // [rowsUsed, V]
|
||||||
|
vDSP_mtrans(logitsT, 1, &logits, 1, vDSP_Length(rowsUsed), vDSP_Length(V))
|
||||||
|
var total = 0.0
|
||||||
|
var expBuf = [Float](repeating: 0, count: V)
|
||||||
|
logits.withUnsafeMutableBufferPointer { lp in
|
||||||
|
expBuf.withUnsafeMutableBufferPointer { eb in
|
||||||
|
let e = eb.baseAddress!
|
||||||
|
for i in 0..<rowsUsed {
|
||||||
|
let row = lp.baseAddress! + i * V
|
||||||
|
var m: Float = 0
|
||||||
|
vDSP_maxv(row, 1, &m, vDSP_Length(V))
|
||||||
|
var negM = -m
|
||||||
|
vDSP_vsadd(row, 1, &negM, e, 1, vDSP_Length(V))
|
||||||
|
var n = Int32(V)
|
||||||
|
vvexpf(e, e, &n)
|
||||||
|
var sum: Float = 0
|
||||||
|
vDSP_sve(e, 1, &sum, vDSP_Length(V))
|
||||||
|
total += Double(row[Int(targets[i])] - m) - Double(logf(sum))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return total
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// "AI 選字": reranks the converter's best paths with a language model.
|
||||||
|
///
|
||||||
|
/// score = lexicon score (log10 → ln) + λ · LM log-probability
|
||||||
|
///
|
||||||
|
/// It never blocks typing for long: each rerank has a time budget, and a rerank that runs
|
||||||
|
/// over it is dropped (the plain lexicon result is used). Three overruns in a row pause
|
||||||
|
/// AI for a minute. `systemState` lets the app pause or reduce it when the machine is
|
||||||
|
/// busy, hot or saving power.
|
||||||
|
public final class Reranker {
|
||||||
|
public enum Mode: String, CaseIterable {
|
||||||
|
case off, saver, standard, quality
|
||||||
|
|
||||||
|
/// How many paths are reranked, and the time budget for one rerank.
|
||||||
|
var k: Int { [.saver: 3, .standard: 10, .quality: 30][self] ?? 0 }
|
||||||
|
var budget: TimeInterval { [.saver: 0.015, .standard: 0.04, .quality: 0.12][self] ?? 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What the system allows right now (decided by the app).
|
||||||
|
public enum SystemState { case normal, reduced, paused }
|
||||||
|
|
||||||
|
public enum Status: Equatable {
|
||||||
|
case off, loading, active, pausedBusy, pausedSlow
|
||||||
|
}
|
||||||
|
|
||||||
|
public var mode: Mode { didSet { cache.removeAll() } }
|
||||||
|
/// nil while loading (or if the model file is missing).
|
||||||
|
public var model: LanguageModel? { didSet { cache.removeAll() } }
|
||||||
|
public var lambda = 0.6
|
||||||
|
public var systemState: () -> SystemState = { .normal }
|
||||||
|
public var now: () -> Date = Date.init
|
||||||
|
public private(set) var status: Status = .off
|
||||||
|
|
||||||
|
private var overruns = 0
|
||||||
|
private var pausedUntil = Date.distantPast
|
||||||
|
private var cache: [String: [ConvertedSegment]] = [:]
|
||||||
|
static let cacheLimit = 512
|
||||||
|
static let pauseAfterOverruns = 3
|
||||||
|
static let slowPause: TimeInterval = 60
|
||||||
|
|
||||||
|
public init(mode: Mode = .standard, model: LanguageModel? = nil) {
|
||||||
|
self.mode = mode
|
||||||
|
self.model = model
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The mode actually used now (after the system state), or nil when AI is not used.
|
||||||
|
var effectiveMode: Mode? {
|
||||||
|
guard mode != .off else { status = .off; return nil }
|
||||||
|
guard model != nil else { status = .loading; return nil }
|
||||||
|
if now() < pausedUntil { status = .pausedSlow; return nil }
|
||||||
|
switch systemState() {
|
||||||
|
case .paused: status = .pausedBusy; return nil
|
||||||
|
case .reduced: status = .active; return .saver
|
||||||
|
case .normal: status = .active; return mode
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reranked conversion, or nil to use the plain lexicon result.
|
||||||
|
func choose(_ converter: Converter, _ syllables: [Syllable], fixed: [Range<Int>: String],
|
||||||
|
context: String) -> [ConvertedSegment]? {
|
||||||
|
guard let mode = effectiveMode, let model, !syllables.isEmpty else { return nil }
|
||||||
|
let key = "\(mode.rawValue)|\(context)|\(syllables.map(\.reading).joined(separator: "-"))|"
|
||||||
|
+ fixed.sorted { $0.key.lowerBound < $1.key.lowerBound }.map { "\($0.key)=\($0.value)" }.joined(separator: ",")
|
||||||
|
if let hit = cache[key] { return hit }
|
||||||
|
|
||||||
|
let paths = converter.kBest(syllables, fixed: fixed, k: mode.k)
|
||||||
|
guard paths.count > 1 else { return paths.first?.segments }
|
||||||
|
let start = now()
|
||||||
|
var best: (score: Double, segments: [ConvertedSegment])?
|
||||||
|
for path in paths {
|
||||||
|
let text = path.segments.map(\.text).joined()
|
||||||
|
let score = path.score * log(10) + lambda * model.logProb(text, context: context)
|
||||||
|
if best == nil || score > best!.score { best = (score, path.segments) }
|
||||||
|
if now().timeIntervalSince(start) > mode.budget {
|
||||||
|
overruns += 1
|
||||||
|
if overruns >= Reranker.pauseAfterOverruns {
|
||||||
|
pausedUntil = now().addingTimeInterval(Reranker.slowPause)
|
||||||
|
overruns = 0
|
||||||
|
status = .pausedSlow
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
overruns = 0
|
||||||
|
if cache.count >= Reranker.cacheLimit { cache.removeAll() }
|
||||||
|
cache[key] = best?.segments
|
||||||
|
return best?.segments
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// Remembers the candidates the user picked, so they come first next time.
|
||||||
|
///
|
||||||
|
/// Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
|
||||||
|
/// where the lexicon has it for the reading being converted, so a toneless key is safe.
|
||||||
|
/// Stored as JSON: {reading: {phrase: [count, last use]}} (same format on every platform).
|
||||||
|
public final class UserHistory {
|
||||||
|
public typealias Data = [String: [String: [Int]]]
|
||||||
|
|
||||||
|
/// (reading, phrase) pairs kept; the least recently used go first.
|
||||||
|
public var maxEntries = 2000
|
||||||
|
static let maxCount = 10
|
||||||
|
|
||||||
|
public private(set) var data: Data = [:]
|
||||||
|
private var clock = 0
|
||||||
|
private let onChange: (Data) -> Void
|
||||||
|
|
||||||
|
public init(data: Data = [:], onChange: @escaping (Data) -> Void = { _ in }) {
|
||||||
|
self.onChange = onChange
|
||||||
|
for (reading, phrases) in data {
|
||||||
|
for (phrase, value) in phrases where value.count >= 2 {
|
||||||
|
self.data[reading, default: [:]][phrase] = [value[0], value[1]]
|
||||||
|
clock = max(clock, value[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Loads the JSON file written through `onChange`; empty if missing or unreadable.
|
||||||
|
public static func load(from url: URL) -> Data {
|
||||||
|
guard let raw = try? Foundation.Data(contentsOf: url),
|
||||||
|
let data = try? JSONDecoder().decode(Data.self, from: raw) else { return [:] }
|
||||||
|
return data
|
||||||
|
}
|
||||||
|
|
||||||
|
static func key(_ syllables: [Syllable]) -> String {
|
||||||
|
syllables.map(\.toneless).joined(separator: "-")
|
||||||
|
}
|
||||||
|
|
||||||
|
public func record(_ syllables: [Syllable], phrase: String) {
|
||||||
|
guard !syllables.isEmpty, !phrase.isEmpty else { return }
|
||||||
|
clock += 1
|
||||||
|
let key = UserHistory.key(syllables)
|
||||||
|
let count = data[key]?[phrase]?.first ?? 0
|
||||||
|
data[key, default: [:]][phrase] = [min(UserHistory.maxCount, count + 1), clock]
|
||||||
|
evict()
|
||||||
|
onChange(data)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func evict() {
|
||||||
|
var pairs: [(used: Int, reading: String, phrase: String)] = []
|
||||||
|
for (r, phrases) in data { for (p, v) in phrases { pairs.append((v[1], r, p)) } }
|
||||||
|
guard pairs.count > maxEntries else { return }
|
||||||
|
pairs.sort { $0.used < $1.used }
|
||||||
|
for pair in pairs.prefix(pairs.count - maxEntries) {
|
||||||
|
data[pair.reading]?[pair.phrase] = nil
|
||||||
|
if data[pair.reading]?.isEmpty == true { data[pair.reading] = nil }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public func clear() {
|
||||||
|
data = [:]
|
||||||
|
onChange(data)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Lexicon entries with picked phrases moved to the top: just above the best entry,
|
||||||
|
/// more so the more often they were picked.
|
||||||
|
func adjust(_ syllables: [Syllable], _ entries: [LexiconEntry]) -> [LexiconEntry] {
|
||||||
|
guard let picked = data[UserHistory.key(syllables)], let top = entries.first?.score else { return entries }
|
||||||
|
var out = entries.map { e -> LexiconEntry in
|
||||||
|
guard let count = picked[e.phrase]?.first else { return e }
|
||||||
|
return LexiconEntry(phrase: e.phrase, score: max(e.score, top + 0.01 + 0.1 * Double(count)))
|
||||||
|
}
|
||||||
|
// Stable sort (keep lexicon order among equal scores), like Python's sort.
|
||||||
|
out = out.enumerated().sorted { $0.element.score != $1.element.score
|
||||||
|
? $0.element.score > $1.element.score : $0.offset < $1.offset }.map(\.element)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -48,3 +48,26 @@ https://github.com/EasyIME/PIME — GNU LGPL 2.1
|
|||||||
|
|
||||||
The Windows version runs as a module inside PIME, which users install
|
The Windows version runs as a module inside PIME, which users install
|
||||||
separately. No PIME code is included in this software.
|
separately. No PIME code is included in this software.
|
||||||
|
|
||||||
|
|
||||||
|
4. CKIP GPT2 Tiny Chinese — AI 選字語言模型 (language model)
|
||||||
|
-----------------------------------------------------------
|
||||||
|
https://huggingface.co/ckiplab/gpt2-tiny-chinese
|
||||||
|
https://github.com/ckiplab/ckip-transformers
|
||||||
|
Author: Mu Yang at CKIP, Academia Sinica (中央研究院 詞庫小組)
|
||||||
|
License: GNU General Public License v3.0 (full text: GPL-3.0.txt)
|
||||||
|
|
||||||
|
lm.bin is the model's weights converted to float16 by training/lm/export_lm.py;
|
||||||
|
lm-vocab.txt is its vocabulary (from bert-base-chinese). They are used unmodified
|
||||||
|
otherwise, to rerank conversion candidates ("AI 選字"). The source code of this input
|
||||||
|
method, including the conversion script and the code that runs the model, is
|
||||||
|
available from the project repository.
|
||||||
|
|
||||||
|
|
||||||
|
5. NumPy — 數值運算 (Windows / Linux, for AI 選字)
|
||||||
|
------------------------------------------------
|
||||||
|
https://numpy.org — BSD 3-Clause License (and the licenses of the libraries it bundles,
|
||||||
|
such as OpenBLAS). The Windows package includes unmodified NumPy wheels for 32-bit
|
||||||
|
Python 3.8 and 3.15 under autozhuyin/vendor/; the Linux installer downloads the
|
||||||
|
matching wheel from PyPI when the system has no NumPy. Each copy keeps its license
|
||||||
|
files in its numpy-*.dist-info folder.
|
||||||
|
|||||||
@@ -133,6 +133,16 @@ final class EngineTests: XCTestCase {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The menu-bar input mode can switch English mode directly.
|
||||||
|
func testSetEnglishMode() {
|
||||||
|
let e = makeEngine()
|
||||||
|
e.setEnglishMode(true)
|
||||||
|
XCTAssertTrue(e.isEnglishMode)
|
||||||
|
XCTAssertEqual(e.handle(.character("a")).handled, false) // goes to the app
|
||||||
|
e.setEnglishMode(false)
|
||||||
|
XCTAssertEqual(type("su3cl3 ", into: e), "你好")
|
||||||
|
}
|
||||||
|
|
||||||
func testMixedSentence() {
|
func testMixedSentence() {
|
||||||
let e = makeEngine()
|
let e = makeEngine()
|
||||||
// 我 + "hello world " + 你好
|
// 我 + "hello world " + 你好
|
||||||
@@ -369,7 +379,8 @@ final class CursorTests: XCTestCase {
|
|||||||
func type(_ s: String, into e: CompositionEngine) -> String {
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
var out = ""
|
var out = ""
|
||||||
for c in s {
|
for c in s {
|
||||||
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter : .character(c)
|
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter
|
||||||
|
: c == "\u{8}" ? .backspace : .character(c)
|
||||||
let r = e.handle(key)
|
let r = e.handle(key)
|
||||||
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
}
|
}
|
||||||
@@ -399,12 +410,50 @@ final class CursorTests: XCTestCase {
|
|||||||
XCTAssertEqual(type("\n", into: e), "今添天氣")
|
XCTAssertEqual(type("\n", into: e), "今添天氣")
|
||||||
}
|
}
|
||||||
|
|
||||||
func testTypingReturnsCursorToEnd() {
|
func testPunctuationCommitsFromTheMiddle() {
|
||||||
let e = engine()
|
let e = engine()
|
||||||
_ = type("rup wu0 ←", into: e)
|
_ = type("rup wu0 ←", into: e)
|
||||||
XCTAssertEqual(e.cursorOffset, 1)
|
XCTAssertEqual(e.cursorOffset, 1)
|
||||||
_ = type("wu0 ", into: e)
|
_ = type("?", into: e)
|
||||||
XCTAssertEqual(e.cursorOffset, e.markedText.count)
|
XCTAssertEqual(e.markedText, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testInsertSyllable() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup fu4←wu", into: e)
|
||||||
|
XCTAssertEqual(e.markedText, "今ㄊㄧ氣")
|
||||||
|
XCTAssertEqual(e.cursorOffset, 3)
|
||||||
|
_ = type("0 ", into: e)
|
||||||
|
XCTAssertEqual(e.markedText, "今天氣")
|
||||||
|
XCTAssertEqual(e.cursorOffset, 2)
|
||||||
|
XCTAssertEqual(type("\n", into: e), "今天氣")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCursorStopsAfterFirstCharacter() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("su3cl3←←←", into: e)
|
||||||
|
XCTAssertEqual(e.cursorOffset, 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testBackspaceDeletesCharacterBeforeCursor() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup wu0 wu0 fu4←←\u{8}", into: e)
|
||||||
|
XCTAssertEqual(e.markedText, "今天氣")
|
||||||
|
XCTAssertEqual(e.cursorOffset, 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testBackspaceInsideInsertedSyllable() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup fu4←wu\u{8}", into: e)
|
||||||
|
XCTAssertEqual(e.markedText, "今ㄊ氣")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testKeepsPickedPhraseBeforeCursor() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup wu0 ↓", into: e)
|
||||||
|
e.selectCandidate(e.candidates.firstIndex { $0.text == "添" }!)
|
||||||
|
_ = type("fu4←wu0 ", into: e) // insert 天 between 添 and 氣
|
||||||
|
XCTAssertEqual(String(e.markedText.prefix(3)), "今添天")
|
||||||
}
|
}
|
||||||
|
|
||||||
func testCandidatesForChineseBeforeEnglish() {
|
func testCandidatesForChineseBeforeEnglish() {
|
||||||
@@ -422,8 +471,10 @@ final class FirstToneAfterEnglishTests: XCTestCase {
|
|||||||
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
var out = ""
|
var out = ""
|
||||||
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
|
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
|
||||||
XCTAssertEqual(out, "hello ", keys)
|
out += e.handle(.enter).commit ?? ""
|
||||||
XCTAssertEqual(e.markedText.count, 1, keys) // still composing one Chinese character
|
XCTAssertEqual(String(out.prefix(6)), "hello ", keys)
|
||||||
|
XCTAssertEqual(out.count, 7, keys) // one Chinese character after it
|
||||||
|
XCTAssertFalse(out.last!.isASCII, keys)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -500,6 +551,226 @@ final class BacktickSymbolTests: XCTestCase {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
final class FirstWordHoldTests: XCTestCase {
|
||||||
|
func makeEngine() -> CompositionEngine {
|
||||||
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let r = e.handle(c == "\n" ? .enter : .character(c))
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The first word is not committed as English on space if it also reads as Zhuyin.
|
||||||
|
func testNextWordDecides() {
|
||||||
|
let cases = ["el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
|
||||||
|
"i love you ": "i love you ", "up the ": "up the "]
|
||||||
|
for (keys, want) in cases {
|
||||||
|
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testNotHeldAfterEnglish() {
|
||||||
|
let e = makeEngine()
|
||||||
|
XCTAssertEqual(type("hello world ", into: e), "hello world ")
|
||||||
|
XCTAssertEqual(type("so ", into: e), "so ") // right after English: commits at once
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class UserHistoryTests: XCTestCase {
|
||||||
|
var saved: [UserHistory.Data] = []
|
||||||
|
var classifier: Classifier!
|
||||||
|
|
||||||
|
override func setUp() {
|
||||||
|
saved = []
|
||||||
|
classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights,
|
||||||
|
history: UserHistory { [unowned self] in self.saved.append($0) })
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let key: KeyInput = c == "↓" ? .down : c == "\n" ? .enter : .character(c)
|
||||||
|
let r = e.handle(key)
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
@discardableResult
|
||||||
|
func pick(_ keys: String, _ phrase: String) -> String {
|
||||||
|
let e = CompositionEngine(classifier: classifier)
|
||||||
|
_ = type(keys + "↓", into: e)
|
||||||
|
e.selectCandidate(e.candidates.firstIndex { $0.text == phrase }!)
|
||||||
|
return type("\n", into: e)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testPickedCandidateComesFirst() {
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "天")
|
||||||
|
XCTAssertEqual(pick("wu0 ", "添"), "添")
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
|
||||||
|
let e = CompositionEngine(classifier: classifier)
|
||||||
|
_ = type("wu0 ↓", into: e)
|
||||||
|
XCTAssertEqual(e.candidates.first?.text, "添")
|
||||||
|
XCTAssertEqual(saved.last, ["ㄊㄧㄢ": ["添": [1, 1]]])
|
||||||
|
}
|
||||||
|
|
||||||
|
func testPhrasesStillWin() {
|
||||||
|
pick("wu0 ", "添")
|
||||||
|
XCTAssertEqual(type("rup wu0 \n", into: CompositionEngine(classifier: classifier)), "今天")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testMostPickedWins() {
|
||||||
|
pick("wu0 ", "添")
|
||||||
|
pick("wu0 ", "添")
|
||||||
|
pick("wu0 ", "天")
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testRoundTripAndClear() throws {
|
||||||
|
pick("wu0 ", "添")
|
||||||
|
let json = try JSONEncoder().encode(saved.last!)
|
||||||
|
let h = UserHistory(data: try JSONDecoder().decode(UserHistory.Data.self, from: json))
|
||||||
|
XCTAssertEqual(h.data, ["ㄊㄧㄢ": ["添": [1, 1]]])
|
||||||
|
h.record(ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable), phrase: "天")
|
||||||
|
XCTAssertEqual(h.data["ㄊㄧㄢ"]?["天"], [1, 2])
|
||||||
|
h.clear()
|
||||||
|
XCTAssertTrue(h.data.isEmpty)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testEvictsLeastRecent() {
|
||||||
|
let h = UserHistory()
|
||||||
|
h.maxEntries = 2
|
||||||
|
let syl = ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable)
|
||||||
|
for p in ["天", "添", "田"] { h.record(syl, phrase: p) }
|
||||||
|
XCTAssertEqual(Set(h.data["ㄊㄧㄢ"]!.keys), ["添", "田"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Needs build/lm/lm.bin (scripts/fetch-lm.sh); skipped otherwise.
|
||||||
|
final class GPT2LanguageModelTests: XCTestCase {
|
||||||
|
static let dir = resources.deletingLastPathComponent().appendingPathComponent("build/lm")
|
||||||
|
|
||||||
|
func model() throws -> GPT2LanguageModel {
|
||||||
|
let bin = GPT2LanguageModelTests.dir.appendingPathComponent("lm.bin")
|
||||||
|
try XCTSkipUnless(FileManager.default.fileExists(atPath: bin.path), "run scripts/fetch-lm.sh")
|
||||||
|
return try GPT2LanguageModel(modelURL: bin, vocabURL: GPT2LanguageModelTests.dir.appendingPathComponent("lm-vocab.txt"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Same numbers as the numpy reference (training/lm/gpt2.py, float32 weights).
|
||||||
|
func testMatchesReference() throws {
|
||||||
|
let m = try model()
|
||||||
|
let cases: [(String, String, Double)] = [("他在家", "", -14.329), ("今天天氣很好", "", -30.08),
|
||||||
|
("今添天氣很好", "", -44.392), ("家", "他在", -5.514),
|
||||||
|
("天氣很好", "今天", -23.883)]
|
||||||
|
for (text, context, want) in cases {
|
||||||
|
XCTAssertEqual(m.logProb(text, context: context), want, accuracy: 0.05, text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testSpeed() throws {
|
||||||
|
let m = try model()
|
||||||
|
let start = Date()
|
||||||
|
for _ in 0..<20 { _ = m.logProb("我們明天早上八點在車站見面", context: "好啊") }
|
||||||
|
let ms = Date().timeIntervalSince(start) / 20 * 1000
|
||||||
|
print("GPT2 logProb: \(String(format: "%.2f", ms)) ms")
|
||||||
|
XCTAssertLessThan(ms, 20)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A language model that likes one string and can pretend to be slow.
|
||||||
|
private final class FakeLanguageModel: LanguageModel {
|
||||||
|
var liked: String
|
||||||
|
var onCall: () -> Void = {}
|
||||||
|
init(liked: String) { self.liked = liked }
|
||||||
|
func logProb(_ text: String, context: String) -> Double {
|
||||||
|
onCall()
|
||||||
|
return text.contains(liked) ? 0 : -50
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class RerankerTests: XCTestCase {
|
||||||
|
func engine(_ reranker: Reranker?) -> CompositionEngine {
|
||||||
|
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
e.reranker = reranker
|
||||||
|
return e
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let r = e.handle(c == "\n" ? .enter : .character(c))
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func testRerankPicksWhatTheModelLikes() {
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(nil)), "天")
|
||||||
|
let r = Reranker(mode: .standard, model: FakeLanguageModel(liked: "添"))
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(r)), "添")
|
||||||
|
XCTAssertEqual(r.status, .active)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testOffAndLoadingUseTheLexicon() {
|
||||||
|
let off = Reranker(mode: .off, model: FakeLanguageModel(liked: "添"))
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(off)), "天")
|
||||||
|
XCTAssertEqual(off.status, .off)
|
||||||
|
let loading = Reranker(mode: .standard, model: nil)
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(loading)), "天")
|
||||||
|
XCTAssertEqual(loading.status, .loading)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testPausedWhenSystemBusy() {
|
||||||
|
let r = Reranker(mode: .standard, model: FakeLanguageModel(liked: "添"))
|
||||||
|
r.systemState = { .paused }
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(r)), "天")
|
||||||
|
XCTAssertEqual(r.status, .pausedBusy)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testReducedUsesFewerPaths() {
|
||||||
|
let lm = FakeLanguageModel(liked: "添")
|
||||||
|
var calls = 0
|
||||||
|
lm.onCall = { calls += 1 }
|
||||||
|
let r = Reranker(mode: .quality, model: lm)
|
||||||
|
r.systemState = { .reduced }
|
||||||
|
_ = type("wu0 \n", into: engine(r))
|
||||||
|
XCTAssertLessThanOrEqual(calls, 3)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testSlowModelIsDroppedThenPaused() {
|
||||||
|
var clock = Date(timeIntervalSince1970: 0)
|
||||||
|
let lm = FakeLanguageModel(liked: "添")
|
||||||
|
lm.onCall = { clock.addTimeInterval(1) } // every call takes a "second"
|
||||||
|
let r = Reranker(mode: .standard, model: lm)
|
||||||
|
r.now = { clock }
|
||||||
|
for _ in 0..<3 { XCTAssertEqual(type("wu0 \n", into: engine(r)), "天") }
|
||||||
|
XCTAssertEqual(r.status, .pausedSlow)
|
||||||
|
lm.onCall = {}
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(r)), "天") // still paused
|
||||||
|
clock.addTimeInterval(61)
|
||||||
|
XCTAssertEqual(type("wu0 \n", into: engine(r)), "添") // back after a minute
|
||||||
|
}
|
||||||
|
|
||||||
|
/// With the real model (build/lm, skipped if missing): sentences the lexicon alone gets wrong.
|
||||||
|
func testRealModelFixesHomophones() throws {
|
||||||
|
let dir = GPT2LanguageModelTests.dir
|
||||||
|
let bin = dir.appendingPathComponent("lm.bin")
|
||||||
|
try XCTSkipUnless(FileManager.default.fileExists(atPath: bin.path), "run scripts/fetch-lm.sh")
|
||||||
|
let lm = try GPT2LanguageModel(modelURL: bin, vocabURL: dir.appendingPathComponent("lm-vocab.txt"))
|
||||||
|
let cases = ["w8 ql32k71u3ji3dj94cp32ji ": "他跑得比我快很多",
|
||||||
|
"fu/3su3y94gji u h4cl3a87": "請你再說一次好嗎"]
|
||||||
|
for (keys, want) in cases {
|
||||||
|
XCTAssertNotEqual(type(keys + "\n", into: engine(nil)), want)
|
||||||
|
XCTAssertEqual(type(keys + "\n", into: engine(Reranker(mode: .standard, model: lm))), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
final class StartupTimingTests: XCTestCase {
|
final class StartupTimingTests: XCTestCase {
|
||||||
func testStartupTiming() throws {
|
func testStartupTiming() throws {
|
||||||
var t = Date()
|
var t = Date()
|
||||||
|
|||||||
@@ -0,0 +1,674 @@
|
|||||||
|
GNU GENERAL PUBLIC LICENSE
|
||||||
|
Version 3, 29 June 2007
|
||||||
|
|
||||||
|
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||||
|
Everyone is permitted to copy and distribute verbatim copies
|
||||||
|
of this license document, but changing it is not allowed.
|
||||||
|
|
||||||
|
Preamble
|
||||||
|
|
||||||
|
The GNU General Public License is a free, copyleft license for
|
||||||
|
software and other kinds of works.
|
||||||
|
|
||||||
|
The licenses for most software and other practical works are designed
|
||||||
|
to take away your freedom to share and change the works. By contrast,
|
||||||
|
the GNU General Public License is intended to guarantee your freedom to
|
||||||
|
share and change all versions of a program--to make sure it remains free
|
||||||
|
software for all its users. We, the Free Software Foundation, use the
|
||||||
|
GNU General Public License for most of our software; it applies also to
|
||||||
|
any other work released this way by its authors. You can apply it to
|
||||||
|
your programs, too.
|
||||||
|
|
||||||
|
When we speak of free software, we are referring to freedom, not
|
||||||
|
price. Our General Public Licenses are designed to make sure that you
|
||||||
|
have the freedom to distribute copies of free software (and charge for
|
||||||
|
them if you wish), that you receive source code or can get it if you
|
||||||
|
want it, that you can change the software or use pieces of it in new
|
||||||
|
free programs, and that you know you can do these things.
|
||||||
|
|
||||||
|
To protect your rights, we need to prevent others from denying you
|
||||||
|
these rights or asking you to surrender the rights. Therefore, you have
|
||||||
|
certain responsibilities if you distribute copies of the software, or if
|
||||||
|
you modify it: responsibilities to respect the freedom of others.
|
||||||
|
|
||||||
|
For example, if you distribute copies of such a program, whether
|
||||||
|
gratis or for a fee, you must pass on to the recipients the same
|
||||||
|
freedoms that you received. You must make sure that they, too, receive
|
||||||
|
or can get the source code. And you must show them these terms so they
|
||||||
|
know their rights.
|
||||||
|
|
||||||
|
Developers that use the GNU GPL protect your rights with two steps:
|
||||||
|
(1) assert copyright on the software, and (2) offer you this License
|
||||||
|
giving you legal permission to copy, distribute and/or modify it.
|
||||||
|
|
||||||
|
For the developers' and authors' protection, the GPL clearly explains
|
||||||
|
that there is no warranty for this free software. For both users' and
|
||||||
|
authors' sake, the GPL requires that modified versions be marked as
|
||||||
|
changed, so that their problems will not be attributed erroneously to
|
||||||
|
authors of previous versions.
|
||||||
|
|
||||||
|
Some devices are designed to deny users access to install or run
|
||||||
|
modified versions of the software inside them, although the manufacturer
|
||||||
|
can do so. This is fundamentally incompatible with the aim of
|
||||||
|
protecting users' freedom to change the software. The systematic
|
||||||
|
pattern of such abuse occurs in the area of products for individuals to
|
||||||
|
use, which is precisely where it is most unacceptable. Therefore, we
|
||||||
|
have designed this version of the GPL to prohibit the practice for those
|
||||||
|
products. If such problems arise substantially in other domains, we
|
||||||
|
stand ready to extend this provision to those domains in future versions
|
||||||
|
of the GPL, as needed to protect the freedom of users.
|
||||||
|
|
||||||
|
Finally, every program is threatened constantly by software patents.
|
||||||
|
States should not allow patents to restrict development and use of
|
||||||
|
software on general-purpose computers, but in those that do, we wish to
|
||||||
|
avoid the special danger that patents applied to a free program could
|
||||||
|
make it effectively proprietary. To prevent this, the GPL assures that
|
||||||
|
patents cannot be used to render the program non-free.
|
||||||
|
|
||||||
|
The precise terms and conditions for copying, distribution and
|
||||||
|
modification follow.
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
0. Definitions.
|
||||||
|
|
||||||
|
"This License" refers to version 3 of the GNU General Public License.
|
||||||
|
|
||||||
|
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||||
|
works, such as semiconductor masks.
|
||||||
|
|
||||||
|
"The Program" refers to any copyrightable work licensed under this
|
||||||
|
License. Each licensee is addressed as "you". "Licensees" and
|
||||||
|
"recipients" may be individuals or organizations.
|
||||||
|
|
||||||
|
To "modify" a work means to copy from or adapt all or part of the work
|
||||||
|
in a fashion requiring copyright permission, other than the making of an
|
||||||
|
exact copy. The resulting work is called a "modified version" of the
|
||||||
|
earlier work or a work "based on" the earlier work.
|
||||||
|
|
||||||
|
A "covered work" means either the unmodified Program or a work based
|
||||||
|
on the Program.
|
||||||
|
|
||||||
|
To "propagate" a work means to do anything with it that, without
|
||||||
|
permission, would make you directly or secondarily liable for
|
||||||
|
infringement under applicable copyright law, except executing it on a
|
||||||
|
computer or modifying a private copy. Propagation includes copying,
|
||||||
|
distribution (with or without modification), making available to the
|
||||||
|
public, and in some countries other activities as well.
|
||||||
|
|
||||||
|
To "convey" a work means any kind of propagation that enables other
|
||||||
|
parties to make or receive copies. Mere interaction with a user through
|
||||||
|
a computer network, with no transfer of a copy, is not conveying.
|
||||||
|
|
||||||
|
An interactive user interface displays "Appropriate Legal Notices"
|
||||||
|
to the extent that it includes a convenient and prominently visible
|
||||||
|
feature that (1) displays an appropriate copyright notice, and (2)
|
||||||
|
tells the user that there is no warranty for the work (except to the
|
||||||
|
extent that warranties are provided), that licensees may convey the
|
||||||
|
work under this License, and how to view a copy of this License. If
|
||||||
|
the interface presents a list of user commands or options, such as a
|
||||||
|
menu, a prominent item in the list meets this criterion.
|
||||||
|
|
||||||
|
1. Source Code.
|
||||||
|
|
||||||
|
The "source code" for a work means the preferred form of the work
|
||||||
|
for making modifications to it. "Object code" means any non-source
|
||||||
|
form of a work.
|
||||||
|
|
||||||
|
A "Standard Interface" means an interface that either is an official
|
||||||
|
standard defined by a recognized standards body, or, in the case of
|
||||||
|
interfaces specified for a particular programming language, one that
|
||||||
|
is widely used among developers working in that language.
|
||||||
|
|
||||||
|
The "System Libraries" of an executable work include anything, other
|
||||||
|
than the work as a whole, that (a) is included in the normal form of
|
||||||
|
packaging a Major Component, but which is not part of that Major
|
||||||
|
Component, and (b) serves only to enable use of the work with that
|
||||||
|
Major Component, or to implement a Standard Interface for which an
|
||||||
|
implementation is available to the public in source code form. A
|
||||||
|
"Major Component", in this context, means a major essential component
|
||||||
|
(kernel, window system, and so on) of the specific operating system
|
||||||
|
(if any) on which the executable work runs, or a compiler used to
|
||||||
|
produce the work, or an object code interpreter used to run it.
|
||||||
|
|
||||||
|
The "Corresponding Source" for a work in object code form means all
|
||||||
|
the source code needed to generate, install, and (for an executable
|
||||||
|
work) run the object code and to modify the work, including scripts to
|
||||||
|
control those activities. However, it does not include the work's
|
||||||
|
System Libraries, or general-purpose tools or generally available free
|
||||||
|
programs which are used unmodified in performing those activities but
|
||||||
|
which are not part of the work. For example, Corresponding Source
|
||||||
|
includes interface definition files associated with source files for
|
||||||
|
the work, and the source code for shared libraries and dynamically
|
||||||
|
linked subprograms that the work is specifically designed to require,
|
||||||
|
such as by intimate data communication or control flow between those
|
||||||
|
subprograms and other parts of the work.
|
||||||
|
|
||||||
|
The Corresponding Source need not include anything that users
|
||||||
|
can regenerate automatically from other parts of the Corresponding
|
||||||
|
Source.
|
||||||
|
|
||||||
|
The Corresponding Source for a work in source code form is that
|
||||||
|
same work.
|
||||||
|
|
||||||
|
2. Basic Permissions.
|
||||||
|
|
||||||
|
All rights granted under this License are granted for the term of
|
||||||
|
copyright on the Program, and are irrevocable provided the stated
|
||||||
|
conditions are met. This License explicitly affirms your unlimited
|
||||||
|
permission to run the unmodified Program. The output from running a
|
||||||
|
covered work is covered by this License only if the output, given its
|
||||||
|
content, constitutes a covered work. This License acknowledges your
|
||||||
|
rights of fair use or other equivalent, as provided by copyright law.
|
||||||
|
|
||||||
|
You may make, run and propagate covered works that you do not
|
||||||
|
convey, without conditions so long as your license otherwise remains
|
||||||
|
in force. You may convey covered works to others for the sole purpose
|
||||||
|
of having them make modifications exclusively for you, or provide you
|
||||||
|
with facilities for running those works, provided that you comply with
|
||||||
|
the terms of this License in conveying all material for which you do
|
||||||
|
not control copyright. Those thus making or running the covered works
|
||||||
|
for you must do so exclusively on your behalf, under your direction
|
||||||
|
and control, on terms that prohibit them from making any copies of
|
||||||
|
your copyrighted material outside their relationship with you.
|
||||||
|
|
||||||
|
Conveying under any other circumstances is permitted solely under
|
||||||
|
the conditions stated below. Sublicensing is not allowed; section 10
|
||||||
|
makes it unnecessary.
|
||||||
|
|
||||||
|
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||||
|
|
||||||
|
No covered work shall be deemed part of an effective technological
|
||||||
|
measure under any applicable law fulfilling obligations under article
|
||||||
|
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||||
|
similar laws prohibiting or restricting circumvention of such
|
||||||
|
measures.
|
||||||
|
|
||||||
|
When you convey a covered work, you waive any legal power to forbid
|
||||||
|
circumvention of technological measures to the extent such circumvention
|
||||||
|
is effected by exercising rights under this License with respect to
|
||||||
|
the covered work, and you disclaim any intention to limit operation or
|
||||||
|
modification of the work as a means of enforcing, against the work's
|
||||||
|
users, your or third parties' legal rights to forbid circumvention of
|
||||||
|
technological measures.
|
||||||
|
|
||||||
|
4. Conveying Verbatim Copies.
|
||||||
|
|
||||||
|
You may convey verbatim copies of the Program's source code as you
|
||||||
|
receive it, in any medium, provided that you conspicuously and
|
||||||
|
appropriately publish on each copy an appropriate copyright notice;
|
||||||
|
keep intact all notices stating that this License and any
|
||||||
|
non-permissive terms added in accord with section 7 apply to the code;
|
||||||
|
keep intact all notices of the absence of any warranty; and give all
|
||||||
|
recipients a copy of this License along with the Program.
|
||||||
|
|
||||||
|
You may charge any price or no price for each copy that you convey,
|
||||||
|
and you may offer support or warranty protection for a fee.
|
||||||
|
|
||||||
|
5. Conveying Modified Source Versions.
|
||||||
|
|
||||||
|
You may convey a work based on the Program, or the modifications to
|
||||||
|
produce it from the Program, in the form of source code under the
|
||||||
|
terms of section 4, provided that you also meet all of these conditions:
|
||||||
|
|
||||||
|
a) The work must carry prominent notices stating that you modified
|
||||||
|
it, and giving a relevant date.
|
||||||
|
|
||||||
|
b) The work must carry prominent notices stating that it is
|
||||||
|
released under this License and any conditions added under section
|
||||||
|
7. This requirement modifies the requirement in section 4 to
|
||||||
|
"keep intact all notices".
|
||||||
|
|
||||||
|
c) You must license the entire work, as a whole, under this
|
||||||
|
License to anyone who comes into possession of a copy. This
|
||||||
|
License will therefore apply, along with any applicable section 7
|
||||||
|
additional terms, to the whole of the work, and all its parts,
|
||||||
|
regardless of how they are packaged. This License gives no
|
||||||
|
permission to license the work in any other way, but it does not
|
||||||
|
invalidate such permission if you have separately received it.
|
||||||
|
|
||||||
|
d) If the work has interactive user interfaces, each must display
|
||||||
|
Appropriate Legal Notices; however, if the Program has interactive
|
||||||
|
interfaces that do not display Appropriate Legal Notices, your
|
||||||
|
work need not make them do so.
|
||||||
|
|
||||||
|
A compilation of a covered work with other separate and independent
|
||||||
|
works, which are not by their nature extensions of the covered work,
|
||||||
|
and which are not combined with it such as to form a larger program,
|
||||||
|
in or on a volume of a storage or distribution medium, is called an
|
||||||
|
"aggregate" if the compilation and its resulting copyright are not
|
||||||
|
used to limit the access or legal rights of the compilation's users
|
||||||
|
beyond what the individual works permit. Inclusion of a covered work
|
||||||
|
in an aggregate does not cause this License to apply to the other
|
||||||
|
parts of the aggregate.
|
||||||
|
|
||||||
|
6. Conveying Non-Source Forms.
|
||||||
|
|
||||||
|
You may convey a covered work in object code form under the terms
|
||||||
|
of sections 4 and 5, provided that you also convey the
|
||||||
|
machine-readable Corresponding Source under the terms of this License,
|
||||||
|
in one of these ways:
|
||||||
|
|
||||||
|
a) Convey the object code in, or embodied in, a physical product
|
||||||
|
(including a physical distribution medium), accompanied by the
|
||||||
|
Corresponding Source fixed on a durable physical medium
|
||||||
|
customarily used for software interchange.
|
||||||
|
|
||||||
|
b) Convey the object code in, or embodied in, a physical product
|
||||||
|
(including a physical distribution medium), accompanied by a
|
||||||
|
written offer, valid for at least three years and valid for as
|
||||||
|
long as you offer spare parts or customer support for that product
|
||||||
|
model, to give anyone who possesses the object code either (1) a
|
||||||
|
copy of the Corresponding Source for all the software in the
|
||||||
|
product that is covered by this License, on a durable physical
|
||||||
|
medium customarily used for software interchange, for a price no
|
||||||
|
more than your reasonable cost of physically performing this
|
||||||
|
conveying of source, or (2) access to copy the
|
||||||
|
Corresponding Source from a network server at no charge.
|
||||||
|
|
||||||
|
c) Convey individual copies of the object code with a copy of the
|
||||||
|
written offer to provide the Corresponding Source. This
|
||||||
|
alternative is allowed only occasionally and noncommercially, and
|
||||||
|
only if you received the object code with such an offer, in accord
|
||||||
|
with subsection 6b.
|
||||||
|
|
||||||
|
d) Convey the object code by offering access from a designated
|
||||||
|
place (gratis or for a charge), and offer equivalent access to the
|
||||||
|
Corresponding Source in the same way through the same place at no
|
||||||
|
further charge. You need not require recipients to copy the
|
||||||
|
Corresponding Source along with the object code. If the place to
|
||||||
|
copy the object code is a network server, the Corresponding Source
|
||||||
|
may be on a different server (operated by you or a third party)
|
||||||
|
that supports equivalent copying facilities, provided you maintain
|
||||||
|
clear directions next to the object code saying where to find the
|
||||||
|
Corresponding Source. Regardless of what server hosts the
|
||||||
|
Corresponding Source, you remain obligated to ensure that it is
|
||||||
|
available for as long as needed to satisfy these requirements.
|
||||||
|
|
||||||
|
e) Convey the object code using peer-to-peer transmission, provided
|
||||||
|
you inform other peers where the object code and Corresponding
|
||||||
|
Source of the work are being offered to the general public at no
|
||||||
|
charge under subsection 6d.
|
||||||
|
|
||||||
|
A separable portion of the object code, whose source code is excluded
|
||||||
|
from the Corresponding Source as a System Library, need not be
|
||||||
|
included in conveying the object code work.
|
||||||
|
|
||||||
|
A "User Product" is either (1) a "consumer product", which means any
|
||||||
|
tangible personal property which is normally used for personal, family,
|
||||||
|
or household purposes, or (2) anything designed or sold for incorporation
|
||||||
|
into a dwelling. In determining whether a product is a consumer product,
|
||||||
|
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||||
|
product received by a particular user, "normally used" refers to a
|
||||||
|
typical or common use of that class of product, regardless of the status
|
||||||
|
of the particular user or of the way in which the particular user
|
||||||
|
actually uses, or expects or is expected to use, the product. A product
|
||||||
|
is a consumer product regardless of whether the product has substantial
|
||||||
|
commercial, industrial or non-consumer uses, unless such uses represent
|
||||||
|
the only significant mode of use of the product.
|
||||||
|
|
||||||
|
"Installation Information" for a User Product means any methods,
|
||||||
|
procedures, authorization keys, or other information required to install
|
||||||
|
and execute modified versions of a covered work in that User Product from
|
||||||
|
a modified version of its Corresponding Source. The information must
|
||||||
|
suffice to ensure that the continued functioning of the modified object
|
||||||
|
code is in no case prevented or interfered with solely because
|
||||||
|
modification has been made.
|
||||||
|
|
||||||
|
If you convey an object code work under this section in, or with, or
|
||||||
|
specifically for use in, a User Product, and the conveying occurs as
|
||||||
|
part of a transaction in which the right of possession and use of the
|
||||||
|
User Product is transferred to the recipient in perpetuity or for a
|
||||||
|
fixed term (regardless of how the transaction is characterized), the
|
||||||
|
Corresponding Source conveyed under this section must be accompanied
|
||||||
|
by the Installation Information. But this requirement does not apply
|
||||||
|
if neither you nor any third party retains the ability to install
|
||||||
|
modified object code on the User Product (for example, the work has
|
||||||
|
been installed in ROM).
|
||||||
|
|
||||||
|
The requirement to provide Installation Information does not include a
|
||||||
|
requirement to continue to provide support service, warranty, or updates
|
||||||
|
for a work that has been modified or installed by the recipient, or for
|
||||||
|
the User Product in which it has been modified or installed. Access to a
|
||||||
|
network may be denied when the modification itself materially and
|
||||||
|
adversely affects the operation of the network or violates the rules and
|
||||||
|
protocols for communication across the network.
|
||||||
|
|
||||||
|
Corresponding Source conveyed, and Installation Information provided,
|
||||||
|
in accord with this section must be in a format that is publicly
|
||||||
|
documented (and with an implementation available to the public in
|
||||||
|
source code form), and must require no special password or key for
|
||||||
|
unpacking, reading or copying.
|
||||||
|
|
||||||
|
7. Additional Terms.
|
||||||
|
|
||||||
|
"Additional permissions" are terms that supplement the terms of this
|
||||||
|
License by making exceptions from one or more of its conditions.
|
||||||
|
Additional permissions that are applicable to the entire Program shall
|
||||||
|
be treated as though they were included in this License, to the extent
|
||||||
|
that they are valid under applicable law. If additional permissions
|
||||||
|
apply only to part of the Program, that part may be used separately
|
||||||
|
under those permissions, but the entire Program remains governed by
|
||||||
|
this License without regard to the additional permissions.
|
||||||
|
|
||||||
|
When you convey a copy of a covered work, you may at your option
|
||||||
|
remove any additional permissions from that copy, or from any part of
|
||||||
|
it. (Additional permissions may be written to require their own
|
||||||
|
removal in certain cases when you modify the work.) You may place
|
||||||
|
additional permissions on material, added by you to a covered work,
|
||||||
|
for which you have or can give appropriate copyright permission.
|
||||||
|
|
||||||
|
Notwithstanding any other provision of this License, for material you
|
||||||
|
add to a covered work, you may (if authorized by the copyright holders of
|
||||||
|
that material) supplement the terms of this License with terms:
|
||||||
|
|
||||||
|
a) Disclaiming warranty or limiting liability differently from the
|
||||||
|
terms of sections 15 and 16 of this License; or
|
||||||
|
|
||||||
|
b) Requiring preservation of specified reasonable legal notices or
|
||||||
|
author attributions in that material or in the Appropriate Legal
|
||||||
|
Notices displayed by works containing it; or
|
||||||
|
|
||||||
|
c) Prohibiting misrepresentation of the origin of that material, or
|
||||||
|
requiring that modified versions of such material be marked in
|
||||||
|
reasonable ways as different from the original version; or
|
||||||
|
|
||||||
|
d) Limiting the use for publicity purposes of names of licensors or
|
||||||
|
authors of the material; or
|
||||||
|
|
||||||
|
e) Declining to grant rights under trademark law for use of some
|
||||||
|
trade names, trademarks, or service marks; or
|
||||||
|
|
||||||
|
f) Requiring indemnification of licensors and authors of that
|
||||||
|
material by anyone who conveys the material (or modified versions of
|
||||||
|
it) with contractual assumptions of liability to the recipient, for
|
||||||
|
any liability that these contractual assumptions directly impose on
|
||||||
|
those licensors and authors.
|
||||||
|
|
||||||
|
All other non-permissive additional terms are considered "further
|
||||||
|
restrictions" within the meaning of section 10. If the Program as you
|
||||||
|
received it, or any part of it, contains a notice stating that it is
|
||||||
|
governed by this License along with a term that is a further
|
||||||
|
restriction, you may remove that term. If a license document contains
|
||||||
|
a further restriction but permits relicensing or conveying under this
|
||||||
|
License, you may add to a covered work material governed by the terms
|
||||||
|
of that license document, provided that the further restriction does
|
||||||
|
not survive such relicensing or conveying.
|
||||||
|
|
||||||
|
If you add terms to a covered work in accord with this section, you
|
||||||
|
must place, in the relevant source files, a statement of the
|
||||||
|
additional terms that apply to those files, or a notice indicating
|
||||||
|
where to find the applicable terms.
|
||||||
|
|
||||||
|
Additional terms, permissive or non-permissive, may be stated in the
|
||||||
|
form of a separately written license, or stated as exceptions;
|
||||||
|
the above requirements apply either way.
|
||||||
|
|
||||||
|
8. Termination.
|
||||||
|
|
||||||
|
You may not propagate or modify a covered work except as expressly
|
||||||
|
provided under this License. Any attempt otherwise to propagate or
|
||||||
|
modify it is void, and will automatically terminate your rights under
|
||||||
|
this License (including any patent licenses granted under the third
|
||||||
|
paragraph of section 11).
|
||||||
|
|
||||||
|
However, if you cease all violation of this License, then your
|
||||||
|
license from a particular copyright holder is reinstated (a)
|
||||||
|
provisionally, unless and until the copyright holder explicitly and
|
||||||
|
finally terminates your license, and (b) permanently, if the copyright
|
||||||
|
holder fails to notify you of the violation by some reasonable means
|
||||||
|
prior to 60 days after the cessation.
|
||||||
|
|
||||||
|
Moreover, your license from a particular copyright holder is
|
||||||
|
reinstated permanently if the copyright holder notifies you of the
|
||||||
|
violation by some reasonable means, this is the first time you have
|
||||||
|
received notice of violation of this License (for any work) from that
|
||||||
|
copyright holder, and you cure the violation prior to 30 days after
|
||||||
|
your receipt of the notice.
|
||||||
|
|
||||||
|
Termination of your rights under this section does not terminate the
|
||||||
|
licenses of parties who have received copies or rights from you under
|
||||||
|
this License. If your rights have been terminated and not permanently
|
||||||
|
reinstated, you do not qualify to receive new licenses for the same
|
||||||
|
material under section 10.
|
||||||
|
|
||||||
|
9. Acceptance Not Required for Having Copies.
|
||||||
|
|
||||||
|
You are not required to accept this License in order to receive or
|
||||||
|
run a copy of the Program. Ancillary propagation of a covered work
|
||||||
|
occurring solely as a consequence of using peer-to-peer transmission
|
||||||
|
to receive a copy likewise does not require acceptance. However,
|
||||||
|
nothing other than this License grants you permission to propagate or
|
||||||
|
modify any covered work. These actions infringe copyright if you do
|
||||||
|
not accept this License. Therefore, by modifying or propagating a
|
||||||
|
covered work, you indicate your acceptance of this License to do so.
|
||||||
|
|
||||||
|
10. Automatic Licensing of Downstream Recipients.
|
||||||
|
|
||||||
|
Each time you convey a covered work, the recipient automatically
|
||||||
|
receives a license from the original licensors, to run, modify and
|
||||||
|
propagate that work, subject to this License. You are not responsible
|
||||||
|
for enforcing compliance by third parties with this License.
|
||||||
|
|
||||||
|
An "entity transaction" is a transaction transferring control of an
|
||||||
|
organization, or substantially all assets of one, or subdividing an
|
||||||
|
organization, or merging organizations. If propagation of a covered
|
||||||
|
work results from an entity transaction, each party to that
|
||||||
|
transaction who receives a copy of the work also receives whatever
|
||||||
|
licenses to the work the party's predecessor in interest had or could
|
||||||
|
give under the previous paragraph, plus a right to possession of the
|
||||||
|
Corresponding Source of the work from the predecessor in interest, if
|
||||||
|
the predecessor has it or can get it with reasonable efforts.
|
||||||
|
|
||||||
|
You may not impose any further restrictions on the exercise of the
|
||||||
|
rights granted or affirmed under this License. For example, you may
|
||||||
|
not impose a license fee, royalty, or other charge for exercise of
|
||||||
|
rights granted under this License, and you may not initiate litigation
|
||||||
|
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||||
|
any patent claim is infringed by making, using, selling, offering for
|
||||||
|
sale, or importing the Program or any portion of it.
|
||||||
|
|
||||||
|
11. Patents.
|
||||||
|
|
||||||
|
A "contributor" is a copyright holder who authorizes use under this
|
||||||
|
License of the Program or a work on which the Program is based. The
|
||||||
|
work thus licensed is called the contributor's "contributor version".
|
||||||
|
|
||||||
|
A contributor's "essential patent claims" are all patent claims
|
||||||
|
owned or controlled by the contributor, whether already acquired or
|
||||||
|
hereafter acquired, that would be infringed by some manner, permitted
|
||||||
|
by this License, of making, using, or selling its contributor version,
|
||||||
|
but do not include claims that would be infringed only as a
|
||||||
|
consequence of further modification of the contributor version. For
|
||||||
|
purposes of this definition, "control" includes the right to grant
|
||||||
|
patent sublicenses in a manner consistent with the requirements of
|
||||||
|
this License.
|
||||||
|
|
||||||
|
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||||
|
patent license under the contributor's essential patent claims, to
|
||||||
|
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||||
|
propagate the contents of its contributor version.
|
||||||
|
|
||||||
|
In the following three paragraphs, a "patent license" is any express
|
||||||
|
agreement or commitment, however denominated, not to enforce a patent
|
||||||
|
(such as an express permission to practice a patent or covenant not to
|
||||||
|
sue for patent infringement). To "grant" such a patent license to a
|
||||||
|
party means to make such an agreement or commitment not to enforce a
|
||||||
|
patent against the party.
|
||||||
|
|
||||||
|
If you convey a covered work, knowingly relying on a patent license,
|
||||||
|
and the Corresponding Source of the work is not available for anyone
|
||||||
|
to copy, free of charge and under the terms of this License, through a
|
||||||
|
publicly available network server or other readily accessible means,
|
||||||
|
then you must either (1) cause the Corresponding Source to be so
|
||||||
|
available, or (2) arrange to deprive yourself of the benefit of the
|
||||||
|
patent license for this particular work, or (3) arrange, in a manner
|
||||||
|
consistent with the requirements of this License, to extend the patent
|
||||||
|
license to downstream recipients. "Knowingly relying" means you have
|
||||||
|
actual knowledge that, but for the patent license, your conveying the
|
||||||
|
covered work in a country, or your recipient's use of the covered work
|
||||||
|
in a country, would infringe one or more identifiable patents in that
|
||||||
|
country that you have reason to believe are valid.
|
||||||
|
|
||||||
|
If, pursuant to or in connection with a single transaction or
|
||||||
|
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||||
|
covered work, and grant a patent license to some of the parties
|
||||||
|
receiving the covered work authorizing them to use, propagate, modify
|
||||||
|
or convey a specific copy of the covered work, then the patent license
|
||||||
|
you grant is automatically extended to all recipients of the covered
|
||||||
|
work and works based on it.
|
||||||
|
|
||||||
|
A patent license is "discriminatory" if it does not include within
|
||||||
|
the scope of its coverage, prohibits the exercise of, or is
|
||||||
|
conditioned on the non-exercise of one or more of the rights that are
|
||||||
|
specifically granted under this License. You may not convey a covered
|
||||||
|
work if you are a party to an arrangement with a third party that is
|
||||||
|
in the business of distributing software, under which you make payment
|
||||||
|
to the third party based on the extent of your activity of conveying
|
||||||
|
the work, and under which the third party grants, to any of the
|
||||||
|
parties who would receive the covered work from you, a discriminatory
|
||||||
|
patent license (a) in connection with copies of the covered work
|
||||||
|
conveyed by you (or copies made from those copies), or (b) primarily
|
||||||
|
for and in connection with specific products or compilations that
|
||||||
|
contain the covered work, unless you entered into that arrangement,
|
||||||
|
or that patent license was granted, prior to 28 March 2007.
|
||||||
|
|
||||||
|
Nothing in this License shall be construed as excluding or limiting
|
||||||
|
any implied license or other defenses to infringement that may
|
||||||
|
otherwise be available to you under applicable patent law.
|
||||||
|
|
||||||
|
12. No Surrender of Others' Freedom.
|
||||||
|
|
||||||
|
If conditions are imposed on you (whether by court order, agreement or
|
||||||
|
otherwise) that contradict the conditions of this License, they do not
|
||||||
|
excuse you from the conditions of this License. If you cannot convey a
|
||||||
|
covered work so as to satisfy simultaneously your obligations under this
|
||||||
|
License and any other pertinent obligations, then as a consequence you may
|
||||||
|
not convey it at all. For example, if you agree to terms that obligate you
|
||||||
|
to collect a royalty for further conveying from those to whom you convey
|
||||||
|
the Program, the only way you could satisfy both those terms and this
|
||||||
|
License would be to refrain entirely from conveying the Program.
|
||||||
|
|
||||||
|
13. Use with the GNU Affero General Public License.
|
||||||
|
|
||||||
|
Notwithstanding any other provision of this License, you have
|
||||||
|
permission to link or combine any covered work with a work licensed
|
||||||
|
under version 3 of the GNU Affero General Public License into a single
|
||||||
|
combined work, and to convey the resulting work. The terms of this
|
||||||
|
License will continue to apply to the part which is the covered work,
|
||||||
|
but the special requirements of the GNU Affero General Public License,
|
||||||
|
section 13, concerning interaction through a network will apply to the
|
||||||
|
combination as such.
|
||||||
|
|
||||||
|
14. Revised Versions of this License.
|
||||||
|
|
||||||
|
The Free Software Foundation may publish revised and/or new versions of
|
||||||
|
the GNU General Public License from time to time. Such new versions will
|
||||||
|
be similar in spirit to the present version, but may differ in detail to
|
||||||
|
address new problems or concerns.
|
||||||
|
|
||||||
|
Each version is given a distinguishing version number. If the
|
||||||
|
Program specifies that a certain numbered version of the GNU General
|
||||||
|
Public License "or any later version" applies to it, you have the
|
||||||
|
option of following the terms and conditions either of that numbered
|
||||||
|
version or of any later version published by the Free Software
|
||||||
|
Foundation. If the Program does not specify a version number of the
|
||||||
|
GNU General Public License, you may choose any version ever published
|
||||||
|
by the Free Software Foundation.
|
||||||
|
|
||||||
|
If the Program specifies that a proxy can decide which future
|
||||||
|
versions of the GNU General Public License can be used, that proxy's
|
||||||
|
public statement of acceptance of a version permanently authorizes you
|
||||||
|
to choose that version for the Program.
|
||||||
|
|
||||||
|
Later license versions may give you additional or different
|
||||||
|
permissions. However, no additional obligations are imposed on any
|
||||||
|
author or copyright holder as a result of your choosing to follow a
|
||||||
|
later version.
|
||||||
|
|
||||||
|
15. Disclaimer of Warranty.
|
||||||
|
|
||||||
|
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||||
|
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||||
|
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||||
|
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||||
|
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||||
|
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||||
|
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||||
|
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||||
|
|
||||||
|
16. Limitation of Liability.
|
||||||
|
|
||||||
|
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||||
|
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||||
|
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||||
|
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||||
|
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||||
|
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||||
|
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||||
|
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||||
|
SUCH DAMAGES.
|
||||||
|
|
||||||
|
17. Interpretation of Sections 15 and 16.
|
||||||
|
|
||||||
|
If the disclaimer of warranty and limitation of liability provided
|
||||||
|
above cannot be given local legal effect according to their terms,
|
||||||
|
reviewing courts shall apply local law that most closely approximates
|
||||||
|
an absolute waiver of all civil liability in connection with the
|
||||||
|
Program, unless a warranty or assumption of liability accompanies a
|
||||||
|
copy of the Program in return for a fee.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
How to Apply These Terms to Your New Programs
|
||||||
|
|
||||||
|
If you develop a new program, and you want it to be of the greatest
|
||||||
|
possible use to the public, the best way to achieve this is to make it
|
||||||
|
free software which everyone can redistribute and change under these terms.
|
||||||
|
|
||||||
|
To do so, attach the following notices to the program. It is safest
|
||||||
|
to attach them to the start of each source file to most effectively
|
||||||
|
state the exclusion of warranty; and each file should have at least
|
||||||
|
the "copyright" line and a pointer to where the full notice is found.
|
||||||
|
|
||||||
|
<one line to give the program's name and a brief idea of what it does.>
|
||||||
|
Copyright (C) <year> <name of author>
|
||||||
|
|
||||||
|
This program is free software: you can redistribute it and/or modify
|
||||||
|
it under the terms of the GNU General Public License as published by
|
||||||
|
the Free Software Foundation, either version 3 of the License, or
|
||||||
|
(at your option) any later version.
|
||||||
|
|
||||||
|
This program is distributed in the hope that it will be useful,
|
||||||
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
GNU General Public License for more details.
|
||||||
|
|
||||||
|
You should have received a copy of the GNU General Public License
|
||||||
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
Also add information on how to contact you by electronic and paper mail.
|
||||||
|
|
||||||
|
If the program does terminal interaction, make it output a short
|
||||||
|
notice like this when it starts in an interactive mode:
|
||||||
|
|
||||||
|
<program> Copyright (C) <year> <name of author>
|
||||||
|
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||||
|
This is free software, and you are welcome to redistribute it
|
||||||
|
under certain conditions; type `show c' for details.
|
||||||
|
|
||||||
|
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||||
|
parts of the General Public License. Of course, your program's commands
|
||||||
|
might be different; for a GUI interface, you would use an "about box".
|
||||||
|
|
||||||
|
You should also get your employer (if you work as a programmer) or school,
|
||||||
|
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||||
|
For more information on this, and how to apply and follow the GNU GPL, see
|
||||||
|
<https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
The GNU General Public License does not permit incorporating your program
|
||||||
|
into proprietary programs. If your program is a subroutine library, you
|
||||||
|
may consider it more useful to permit linking proprietary applications with
|
||||||
|
the library. If this is what you want to do, use the GNU Lesser General
|
||||||
|
Public License instead of this License. But first, please read
|
||||||
|
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
||||||
@@ -0,0 +1,408 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""IBus engine for AutoZhuyin (自動注音) on Linux.
|
||||||
|
|
||||||
|
All the logic lives in autozhuyin.core (shared with Windows and the tests); this file
|
||||||
|
only translates IBus key events and draws the preedit / candidate table.
|
||||||
|
|
||||||
|
Run modes:
|
||||||
|
autozhuyin_ibus.py --ibus started by ibus-daemon (from the component XML)
|
||||||
|
autozhuyin_ibus.py [--activate] started by the user/autostart: registers the engine
|
||||||
|
itself if ibus-daemon does not know it, and with
|
||||||
|
--activate makes it the current input method
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
sys.path.insert(0, HERE)
|
||||||
|
# numpy for "AI 選字", downloaded by install.sh when the system has none.
|
||||||
|
_VENDOR = os.path.join(HERE, "vendor", "cp%d%d" % sys.version_info[:2])
|
||||||
|
if os.path.isdir(_VENDOR):
|
||||||
|
sys.path.insert(1, _VENDOR)
|
||||||
|
|
||||||
|
import gi # noqa: E402
|
||||||
|
|
||||||
|
gi.require_version("IBus", "1.0")
|
||||||
|
from gi.repository import GLib, GObject, IBus # noqa: E402
|
||||||
|
|
||||||
|
import autozhuyin # noqa: E402
|
||||||
|
from autozhuyin.core.ai import AI, MODE_NAMES # noqa: E402
|
||||||
|
from autozhuyin.core.history import UserHistory # noqa: E402
|
||||||
|
from autozhuyin.core.reranker import MODES # noqa: E402
|
||||||
|
from autozhuyin.core.classifier import LITERAL, Classifier, LearnedPreferences # noqa: E402
|
||||||
|
from autozhuyin.core.engine import ( # noqa: E402
|
||||||
|
BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine,
|
||||||
|
)
|
||||||
|
from autozhuyin.core.english import EnglishModel # noqa: E402
|
||||||
|
from autozhuyin.core.lexicon import Lexicon # noqa: E402
|
||||||
|
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
|
||||||
|
|
||||||
|
ENGINE_NAME = "autozhuyin"
|
||||||
|
BUS_NAME = "org.freedesktop.IBus.AutoZhuyin"
|
||||||
|
VERSION = "0.0.0"
|
||||||
|
|
||||||
|
_DATA = os.path.join(os.path.dirname(os.path.abspath(autozhuyin.__file__)), "data")
|
||||||
|
_USER_DIR = os.path.join(os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config"), "autozhuyin")
|
||||||
|
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
|
||||||
|
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
|
||||||
|
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(os.path.join(HERE, "version.txt"), encoding="utf-8") as _f:
|
||||||
|
VERSION = _f.read().strip() or VERSION
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
_classifier = None
|
||||||
|
_ai = None
|
||||||
|
|
||||||
|
|
||||||
|
def shared_ai():
|
||||||
|
""""AI 選字": settings and the language model, loaded in the background once."""
|
||||||
|
global _ai
|
||||||
|
if _ai is None:
|
||||||
|
_ai = AI(_DATA, _USER_DIR)
|
||||||
|
return _ai
|
||||||
|
|
||||||
|
|
||||||
|
def _save_json(path, data):
|
||||||
|
try:
|
||||||
|
os.makedirs(_USER_DIR, exist_ok=True)
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, ensure_ascii=False)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json(path):
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _save_learned(weights):
|
||||||
|
_save_json(_LEARNED_FILE, weights)
|
||||||
|
|
||||||
|
|
||||||
|
def _save_history(data):
|
||||||
|
_save_json(_HISTORY_FILE, data)
|
||||||
|
|
||||||
|
|
||||||
|
def shared_classifier():
|
||||||
|
"""Loads the lexicon and English model once per process."""
|
||||||
|
global _classifier
|
||||||
|
if _classifier is None:
|
||||||
|
lexicon = Lexicon.from_file(os.path.join(_DATA, "lexicon.txt"))
|
||||||
|
if os.path.exists(_USER_PHRASES):
|
||||||
|
with open(_USER_PHRASES, encoding="utf-8") as f:
|
||||||
|
lexicon.load(f)
|
||||||
|
english = EnglishModel.from_files(
|
||||||
|
os.path.join(_DATA, "english-common.txt"),
|
||||||
|
os.path.join(_DATA, "english-dict.txt"),
|
||||||
|
os.path.join(_DATA, "english-bigram.json"),
|
||||||
|
)
|
||||||
|
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
|
||||||
|
_classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
|
||||||
|
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
|
||||||
|
return _classifier
|
||||||
|
|
||||||
|
|
||||||
|
_SPECIAL = {
|
||||||
|
IBus.KEY_BackSpace: BACKSPACE, IBus.KEY_Return: ENTER, IBus.KEY_KP_Enter: ENTER,
|
||||||
|
IBus.KEY_Escape: ESCAPE, IBus.KEY_Tab: TAB,
|
||||||
|
IBus.KEY_Up: UP, IBus.KEY_Down: DOWN, IBus.KEY_Left: LEFT, IBus.KEY_Right: RIGHT,
|
||||||
|
}
|
||||||
|
_SHIFT_KEYS = (IBus.KEY_Shift_L, IBus.KEY_Shift_R)
|
||||||
|
_SHORTCUT_MASK = (IBus.ModifierType.CONTROL_MASK | IBus.ModifierType.MOD1_MASK
|
||||||
|
| IBus.ModifierType.SUPER_MASK | IBus.ModifierType.MOD4_MASK)
|
||||||
|
_LITERAL_COLOR = 0x1E64DC # blue hint for English segments
|
||||||
|
|
||||||
|
|
||||||
|
class AutoZhuyinEngine(IBus.Engine):
|
||||||
|
__gtype_name__ = "AutoZhuyinEngine"
|
||||||
|
|
||||||
|
def __init__(self, *args, **kwargs):
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
self._setup()
|
||||||
|
|
||||||
|
def _setup(self):
|
||||||
|
if getattr(self, "engine", None) is not None:
|
||||||
|
return
|
||||||
|
self.engine = CompositionEngine(shared_classifier())
|
||||||
|
self.engine.reranker = shared_ai().reranker
|
||||||
|
self.shift_tap = ShiftTapDetector()
|
||||||
|
self.table = IBus.LookupTable.new(self.engine.page_size, 0, True, False)
|
||||||
|
self.table.set_orientation(IBus.Orientation.VERTICAL)
|
||||||
|
self.mode_prop = IBus.Property(
|
||||||
|
key="mode", prop_type=IBus.PropType.NORMAL,
|
||||||
|
label=IBus.Text.new_from_string("中英自動"),
|
||||||
|
symbol=IBus.Text.new_from_string("自"),
|
||||||
|
tooltip=IBus.Text.new_from_string("單按 Shift 切換英文模式"),
|
||||||
|
sensitive=True, visible=True, state=IBus.PropState.UNCHECKED)
|
||||||
|
# "AI 選字" menu: modes (radio), auto-pause (toggle), status, clear learned data.
|
||||||
|
ai = shared_ai()
|
||||||
|
self.ai_props = {}
|
||||||
|
sub = IBus.PropList()
|
||||||
|
for mode in MODES:
|
||||||
|
prop = IBus.Property(
|
||||||
|
key="ai." + mode, prop_type=IBus.PropType.RADIO,
|
||||||
|
label=IBus.Text.new_from_string("AI 選字:" + MODE_NAMES[mode]),
|
||||||
|
tooltip=IBus.Text.new_from_string(""), sensitive=True, visible=True,
|
||||||
|
state=IBus.PropState.CHECKED if ai.reranker.mode == mode else IBus.PropState.UNCHECKED)
|
||||||
|
self.ai_props[mode] = prop
|
||||||
|
sub.append(prop)
|
||||||
|
self.pause_prop = IBus.Property(
|
||||||
|
key="ai.autopause", prop_type=IBus.PropType.TOGGLE,
|
||||||
|
label=IBus.Text.new_from_string("系統忙碌時自動暫停 AI"),
|
||||||
|
tooltip=IBus.Text.new_from_string(""), sensitive=True, visible=True,
|
||||||
|
state=IBus.PropState.CHECKED if ai.auto_pause else IBus.PropState.UNCHECKED)
|
||||||
|
self.status_prop = IBus.Property(
|
||||||
|
key="ai.status", prop_type=IBus.PropType.NORMAL,
|
||||||
|
label=IBus.Text.new_from_string(ai.status_text()),
|
||||||
|
tooltip=IBus.Text.new_from_string(""), sensitive=False, visible=True,
|
||||||
|
state=IBus.PropState.UNCHECKED)
|
||||||
|
self.clear_prop = IBus.Property(
|
||||||
|
key="clear", prop_type=IBus.PropType.NORMAL,
|
||||||
|
label=IBus.Text.new_from_string("清除學習紀錄"),
|
||||||
|
tooltip=IBus.Text.new_from_string(""), sensitive=True, visible=True,
|
||||||
|
state=IBus.PropState.UNCHECKED)
|
||||||
|
for prop in (self.pause_prop, self.status_prop, self.clear_prop):
|
||||||
|
sub.append(prop)
|
||||||
|
self.ai_menu = IBus.Property(
|
||||||
|
key="ai", prop_type=IBus.PropType.MENU,
|
||||||
|
label=IBus.Text.new_from_string("設定"),
|
||||||
|
tooltip=IBus.Text.new_from_string("AI 選字與學習紀錄"),
|
||||||
|
sensitive=True, visible=True, state=IBus.PropState.UNCHECKED, sub_props=sub)
|
||||||
|
self.props = IBus.PropList()
|
||||||
|
self.props.append(self.mode_prop)
|
||||||
|
self.props.append(self.ai_menu)
|
||||||
|
|
||||||
|
# --- keys ---
|
||||||
|
|
||||||
|
def do_process_key_event(self, keyval, keycode, state):
|
||||||
|
self._setup()
|
||||||
|
released = bool(state & IBus.ModifierType.RELEASE_MASK)
|
||||||
|
|
||||||
|
# Shift pressed and released alone: English mode (or flip the current word).
|
||||||
|
if keyval in _SHIFT_KEYS:
|
||||||
|
now = time.monotonic()
|
||||||
|
if not released:
|
||||||
|
if state & _SHORTCUT_MASK:
|
||||||
|
self.shift_tap.key_pressed()
|
||||||
|
else:
|
||||||
|
self.shift_tap.shift_down(now)
|
||||||
|
return False
|
||||||
|
if self.shift_tap.shift_up(now):
|
||||||
|
self.engine.handle(SHIFT_TAP)
|
||||||
|
self._update(None)
|
||||||
|
self._update_mode()
|
||||||
|
return False
|
||||||
|
if released:
|
||||||
|
return False
|
||||||
|
self.shift_tap.key_pressed()
|
||||||
|
|
||||||
|
# Shortcuts (Ctrl/Alt/Super) go to the app; commit any composition first.
|
||||||
|
if state & _SHORTCUT_MASK:
|
||||||
|
if self.engine.keys:
|
||||||
|
self._update(self.engine.commit_all())
|
||||||
|
return False
|
||||||
|
|
||||||
|
key = self._key(keyval, state)
|
||||||
|
if key is None:
|
||||||
|
return self._busy() # swallow other keys (Delete, Home…) while composing
|
||||||
|
if not self.engine.wants(key):
|
||||||
|
return False
|
||||||
|
handled, commit = self.engine.handle(key)
|
||||||
|
self._update(commit)
|
||||||
|
return handled
|
||||||
|
|
||||||
|
def _key(self, keyval, state):
|
||||||
|
if keyval in _SPECIAL:
|
||||||
|
return _SPECIAL[keyval]
|
||||||
|
if keyval == IBus.KEY_space:
|
||||||
|
return " "
|
||||||
|
code = IBus.keyval_to_unicode(keyval)
|
||||||
|
if not code or not code.isprintable():
|
||||||
|
return None
|
||||||
|
# Caps Lock without Shift: keep Zhuyin working by using lowercase.
|
||||||
|
if state & IBus.ModifierType.LOCK_MASK and not state & IBus.ModifierType.SHIFT_MASK:
|
||||||
|
code = code.lower()
|
||||||
|
return code
|
||||||
|
|
||||||
|
def _busy(self):
|
||||||
|
return bool(self.engine.keys or self.engine.candidates)
|
||||||
|
|
||||||
|
# --- candidate panel (mouse) ---
|
||||||
|
|
||||||
|
def do_candidate_clicked(self, index, button, state):
|
||||||
|
self.engine.select_candidate(self.engine.page_start + index)
|
||||||
|
self._update(None)
|
||||||
|
|
||||||
|
def do_page_up(self):
|
||||||
|
return self._move_candidate(-self.engine.page_size)
|
||||||
|
|
||||||
|
def do_page_down(self):
|
||||||
|
return self._move_candidate(self.engine.page_size)
|
||||||
|
|
||||||
|
def do_cursor_up(self):
|
||||||
|
return self._move_candidate(-1)
|
||||||
|
|
||||||
|
def do_cursor_down(self):
|
||||||
|
return self._move_candidate(1)
|
||||||
|
|
||||||
|
def _move_candidate(self, delta):
|
||||||
|
if not self.engine.candidates:
|
||||||
|
return False
|
||||||
|
e = self.engine
|
||||||
|
e.candidate_index = max(0, min(len(e.candidates) - 1, e.candidate_index + delta))
|
||||||
|
self._update(None)
|
||||||
|
return True
|
||||||
|
|
||||||
|
# --- settings menu ---
|
||||||
|
|
||||||
|
def do_property_activate(self, prop_name, state):
|
||||||
|
self._setup()
|
||||||
|
ai = shared_ai()
|
||||||
|
if prop_name.startswith("ai.") and prop_name[3:] in MODES:
|
||||||
|
if state == IBus.PropState.CHECKED:
|
||||||
|
ai.set_mode(prop_name[3:])
|
||||||
|
elif prop_name == "ai.autopause":
|
||||||
|
ai.set_auto_pause(state == IBus.PropState.CHECKED)
|
||||||
|
elif prop_name == "clear":
|
||||||
|
shared_classifier().learned.clear()
|
||||||
|
shared_classifier().history.clear()
|
||||||
|
self._refresh_ai_props()
|
||||||
|
|
||||||
|
def _refresh_ai_props(self):
|
||||||
|
ai = shared_ai()
|
||||||
|
for mode, prop in self.ai_props.items():
|
||||||
|
prop.set_state(IBus.PropState.CHECKED if ai.reranker.mode == mode else IBus.PropState.UNCHECKED)
|
||||||
|
self.update_property(prop)
|
||||||
|
self.pause_prop.set_state(IBus.PropState.CHECKED if ai.auto_pause else IBus.PropState.UNCHECKED)
|
||||||
|
self.update_property(self.pause_prop)
|
||||||
|
self.status_prop.set_label(IBus.Text.new_from_string(ai.status_text()))
|
||||||
|
self.update_property(self.status_prop)
|
||||||
|
|
||||||
|
# --- lifecycle ---
|
||||||
|
|
||||||
|
def do_focus_in(self):
|
||||||
|
self._setup()
|
||||||
|
self.register_properties(self.props)
|
||||||
|
self._update_mode()
|
||||||
|
self._refresh_ai_props()
|
||||||
|
|
||||||
|
def do_focus_out(self):
|
||||||
|
self._setup()
|
||||||
|
# PreeditFocusMode.COMMIT: IBus commits the shown text itself on focus out.
|
||||||
|
self.engine.reset()
|
||||||
|
self.hide_lookup_table()
|
||||||
|
|
||||||
|
def do_reset(self):
|
||||||
|
self._setup()
|
||||||
|
self.engine.reset()
|
||||||
|
self.hide_preedit_text()
|
||||||
|
self.hide_lookup_table()
|
||||||
|
|
||||||
|
def do_disable(self):
|
||||||
|
self.do_reset()
|
||||||
|
|
||||||
|
# --- drawing ---
|
||||||
|
|
||||||
|
def _update(self, commit):
|
||||||
|
if commit:
|
||||||
|
self.commit_text(IBus.Text.new_from_string(commit))
|
||||||
|
pieces = self.engine.marked_pieces
|
||||||
|
text = "".join(t for t, _ in pieces)
|
||||||
|
if text:
|
||||||
|
ibus_text = IBus.Text.new_from_string(text)
|
||||||
|
attrs = IBus.AttrList()
|
||||||
|
attrs.append(IBus.attr_underline_new(IBus.AttrUnderline.SINGLE, 0, len(text)))
|
||||||
|
pos = 0
|
||||||
|
for piece, mode in pieces:
|
||||||
|
if mode == LITERAL and piece:
|
||||||
|
attrs.append(IBus.attr_foreground_new(_LITERAL_COLOR, pos, pos + len(piece)))
|
||||||
|
pos += len(piece)
|
||||||
|
ibus_text.set_attributes(attrs)
|
||||||
|
self.update_preedit_text_with_mode(ibus_text, self.engine.cursor_offset, True,
|
||||||
|
IBus.PreeditFocusMode.COMMIT)
|
||||||
|
else:
|
||||||
|
self.hide_preedit_text()
|
||||||
|
|
||||||
|
if self.engine.candidates:
|
||||||
|
self.table.clear()
|
||||||
|
for _, candidate in self.engine.candidates:
|
||||||
|
self.table.append_candidate(IBus.Text.new_from_string(candidate))
|
||||||
|
self.table.set_cursor_pos(self.engine.candidate_index)
|
||||||
|
self.update_lookup_table(self.table, True)
|
||||||
|
else:
|
||||||
|
self.hide_lookup_table()
|
||||||
|
|
||||||
|
def _update_mode(self):
|
||||||
|
english = self.engine.english_mode
|
||||||
|
self.mode_prop.set_label(IBus.Text.new_from_string("英文" if english else "中英自動"))
|
||||||
|
self.mode_prop.set_symbol(IBus.Text.new_from_string("英" if english else "自"))
|
||||||
|
icon = os.path.join(HERE, "icon-en.png" if english else "icon.png")
|
||||||
|
if os.path.exists(icon):
|
||||||
|
self.mode_prop.set_icon(icon) # panels that show property icons
|
||||||
|
self.update_property(self.mode_prop)
|
||||||
|
|
||||||
|
|
||||||
|
# --- process setup ---
|
||||||
|
|
||||||
|
def make_component(exec_path):
|
||||||
|
component = IBus.Component(
|
||||||
|
name=BUS_NAME, description="自動注音 AutoZhuyin", version=VERSION,
|
||||||
|
license="Private", author="AutoZhuyin", homepage="",
|
||||||
|
command_line=f"{sys.executable} {exec_path} --ibus", textdomain="")
|
||||||
|
icon = os.path.join(HERE, "icon.png")
|
||||||
|
component.add_engine(IBus.EngineDesc(
|
||||||
|
name=ENGINE_NAME, longname="自動注音", description="大千注音+英文,自動判斷中英",
|
||||||
|
language="zh_TW", license="Private", author="AutoZhuyin",
|
||||||
|
icon=icon if os.path.exists(icon) else "", layout="us", symbol="自", rank=0))
|
||||||
|
return component
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv):
|
||||||
|
IBus.init()
|
||||||
|
bus = IBus.Bus()
|
||||||
|
loop = GLib.MainLoop()
|
||||||
|
started_by_ibus = "--ibus" in argv
|
||||||
|
activate = "--activate" in argv
|
||||||
|
factory = None
|
||||||
|
|
||||||
|
def on_connected(*_):
|
||||||
|
nonlocal factory
|
||||||
|
factory = IBus.Factory.new(bus.get_connection())
|
||||||
|
factory.add_engine(ENGINE_NAME, GObject.type_from_name("AutoZhuyinEngine"))
|
||||||
|
if started_by_ibus:
|
||||||
|
bus.request_name(BUS_NAME, 0)
|
||||||
|
else:
|
||||||
|
known = any(e.get_name() == ENGINE_NAME for e in bus.list_engines())
|
||||||
|
if not known:
|
||||||
|
bus.register_component(make_component(os.path.abspath(__file__)))
|
||||||
|
if activate:
|
||||||
|
bus.set_global_engine(ENGINE_NAME)
|
||||||
|
|
||||||
|
def on_disconnected(*_):
|
||||||
|
if started_by_ibus:
|
||||||
|
loop.quit()
|
||||||
|
# Otherwise wait: IBus.Bus reconnects (and emits "connected") when ibus-daemon restarts.
|
||||||
|
|
||||||
|
bus.connect("connected", on_connected)
|
||||||
|
bus.connect("disconnected", on_disconnected)
|
||||||
|
shared_classifier() # load data before the first key
|
||||||
|
if bus.is_connected():
|
||||||
|
on_connected()
|
||||||
|
elif started_by_ibus:
|
||||||
|
print("autozhuyin: cannot connect to ibus-daemon", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
loop.run()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main(sys.argv[1:]))
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
自動注音 AutoZhuyin(Linux 測試版,IBus)
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
用大千注音鍵盤直接打字,不用切換中英文,輸入法會自動判斷你打的是注音還是英文。
|
||||||
|
|
||||||
|
按鍵: ji3 hello world su3cl3 (最後按空白)
|
||||||
|
輸出: 我hello world 你好
|
||||||
|
|
||||||
|
適用:Bazzite、SteamOS 桌面模式、Fedora 等有 IBus 的系統(KDE 或 GNOME)。
|
||||||
|
只在桌面模式可用;Steam 遊戲模式不支援系統輸入法。
|
||||||
|
|
||||||
|
|
||||||
|
安裝
|
||||||
|
----
|
||||||
|
1. 解壓縮,在終端機(Konsole)進到這個資料夾,執行:
|
||||||
|
./install.sh
|
||||||
|
不需要 sudo,檔案只會裝在你的家目錄(~/.local/share/autozhuyin)。
|
||||||
|
|
||||||
|
2. KDE(Bazzite / SteamOS 預設):
|
||||||
|
系統設定 → 鍵盤 → 虛擬鍵盤,選「IBus Wayland」→ 套用,然後登出再登入。
|
||||||
|
GNOME:IBus 是內建的,不用另外設定。
|
||||||
|
|
||||||
|
3. 之後每次登入都會自動切換到自動注音。
|
||||||
|
|
||||||
|
|
||||||
|
使用方式
|
||||||
|
--------
|
||||||
|
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
|
||||||
|
Enter 送出目前組字
|
||||||
|
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
|
||||||
|
← → 在組字區移動游標;游標在中間時直接打注音會插入新字,
|
||||||
|
Backspace 刪掉游標前的字
|
||||||
|
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
|
||||||
|
選過的字會被記住,下次優先出現
|
||||||
|
Esc 取消組字
|
||||||
|
單按 Shift 切換英文模式(按鍵直接送出,狀態圖示變成「英」),再單按一次回到中英自動判斷
|
||||||
|
Shift+字母 打大寫字母(也會強制英文)
|
||||||
|
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
|
||||||
|
|
||||||
|
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
|
||||||
|
|
||||||
|
|
||||||
|
AI 選字
|
||||||
|
-------
|
||||||
|
用中研院 CKIP 的繁體中文小型語言模型,看前後文挑同音字(他跑「得」很快、請你「再」說一次),
|
||||||
|
完全在本機運算。需要 numpy:系統沒有的話,install.sh 會從 PyPI 下載符合的版本(約 15 MB)。
|
||||||
|
在 IBus/系統匣的輸入法選單「設定」裡可以選:
|
||||||
|
AI 選字:關閉/省電/標準(預設)/高品質
|
||||||
|
系統忙碌時自動暫停 AI(預設開啟:CPU 很忙或過熱時暫停,省電設定或電量低時改用省電)
|
||||||
|
設定存在 ~/.config/autozhuyin/settings.json。
|
||||||
|
|
||||||
|
|
||||||
|
切換與移除
|
||||||
|
----------
|
||||||
|
暫時改回一般鍵盤: ibus engine xkb:us::eng
|
||||||
|
切回自動注音: ibus engine autozhuyin
|
||||||
|
移除: ./uninstall.sh
|
||||||
|
|
||||||
|
個人資料(學習紀錄、自訂詞)在 ~/.config/autozhuyin/:
|
||||||
|
learned.json 按 Tab 修正的學習紀錄
|
||||||
|
user-history.json 選字紀錄(選過的候選字)
|
||||||
|
user-phrases.txt 自訂詞,一行一個,例如:ㄗˋ-ㄉㄨㄥˋ 自動 -1
|
||||||
|
|
||||||
|
|
||||||
|
有問題時
|
||||||
|
--------
|
||||||
|
在終端機手動執行看看錯誤訊息:
|
||||||
|
/usr/bin/python3 ~/.local/share/autozhuyin/autozhuyin_ibus.py --activate
|
||||||
|
把畫面上的訊息回報給我。
|
||||||
Executable
+81
@@ -0,0 +1,81 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Installs AutoZhuyin (自動注音) for the current user as an IBus input method.
|
||||||
|
# Nothing outside your home folder is touched, so it works on read-only systems
|
||||||
|
# (Bazzite, SteamOS, Fedora Atomic). No sudo needed.
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")"
|
||||||
|
|
||||||
|
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/autozhuyin"
|
||||||
|
AUTOSTART="${XDG_CONFIG_HOME:-$HOME/.config}/autostart"
|
||||||
|
PY=/usr/bin/python3
|
||||||
|
|
||||||
|
echo "== 自動注音 AutoZhuyin 安裝 =="
|
||||||
|
|
||||||
|
# 1. IBus and its Python bindings must come with the system.
|
||||||
|
if ! command -v ibus >/dev/null 2>&1; then
|
||||||
|
echo "找不到 IBus(ibus 指令)。這個系統沒有內建 IBus,自動注音目前只支援 IBus。" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if ! "$PY" -c 'import gi; gi.require_version("IBus", "1.0"); from gi.repository import IBus' 2>/dev/null; then
|
||||||
|
echo "系統的 Python 缺少 IBus 模組(python3-gobject / IBus-1.0 typelib)。" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 2. Copy the program.
|
||||||
|
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null || true
|
||||||
|
VENDOR_BACKUP=""
|
||||||
|
if [ -d "$DEST/vendor" ]; then # keep numpy downloaded by an earlier install
|
||||||
|
VENDOR_BACKUP="$(mktemp -d)"
|
||||||
|
mv "$DEST/vendor" "$VENDOR_BACKUP/"
|
||||||
|
fi
|
||||||
|
rm -rf "$DEST"
|
||||||
|
mkdir -p "$DEST"
|
||||||
|
cp -R autozhuyin autozhuyin_ibus.py version.txt tools "$DEST/"
|
||||||
|
for f in icon.png icon-en.png THIRD_PARTY_LICENSES.txt GPL-3.0.txt; do [ -f "$f" ] && cp "$f" "$DEST/"; done
|
||||||
|
[ -n "$VENDOR_BACKUP" ] && mv "$VENDOR_BACKUP/vendor" "$DEST/" && rmdir "$VENDOR_BACKUP"
|
||||||
|
chmod +x "$DEST/autozhuyin_ibus.py"
|
||||||
|
|
||||||
|
# 2b. numpy for "AI 選字": use the system's, or download the matching version from PyPI.
|
||||||
|
TAG="$("$PY" -c 'import sys; print("cp%d%d" % sys.version_info[:2])')"
|
||||||
|
if "$PY" -c 'import numpy' 2>/dev/null; then
|
||||||
|
echo "AI 選字:使用系統的 numpy。"
|
||||||
|
elif [ -d "$DEST/vendor/$TAG/numpy" ]; then
|
||||||
|
echo "AI 選字:使用先前下載的 numpy。"
|
||||||
|
else
|
||||||
|
echo "AI 選字需要 numpy,正在從 PyPI 下載(約 15 MB)…"
|
||||||
|
if ! "$PY" -I "$DEST/tools/fetch_numpy.py" "$DEST/vendor" auto auto; then
|
||||||
|
echo "下載 numpy 失敗:輸入法照常可用,只是沒有 AI 選字。之後可以重新執行 install.sh。"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 3. Start with the desktop session: registers the engine with IBus and selects it.
|
||||||
|
mkdir -p "$AUTOSTART"
|
||||||
|
cat > "$AUTOSTART/autozhuyin.desktop" <<DESKTOP
|
||||||
|
[Desktop Entry]
|
||||||
|
Type=Application
|
||||||
|
Name=自動注音 AutoZhuyin
|
||||||
|
Comment=Registers the AutoZhuyin input method with IBus
|
||||||
|
Exec=$PY $DEST/autozhuyin_ibus.py --activate
|
||||||
|
Icon=$DEST/icon.png
|
||||||
|
X-GNOME-Autostart-enabled=true
|
||||||
|
X-KDE-autostart-phase=2
|
||||||
|
NoDisplay=true
|
||||||
|
DESKTOP
|
||||||
|
|
||||||
|
# 4. Start it now as well.
|
||||||
|
if pgrep -x ibus-daemon >/dev/null 2>&1; then
|
||||||
|
nohup "$PY" "$DEST/autozhuyin_ibus.py" --activate >/dev/null 2>&1 &
|
||||||
|
sleep 1
|
||||||
|
echo "已啟動並切換到自動注音。"
|
||||||
|
else
|
||||||
|
echo "IBus 目前沒有在執行,請照下面的步驟啟用 IBus 後登出再登入。"
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat <<'MSG'
|
||||||
|
|
||||||
|
安裝完成。
|
||||||
|
- KDE(Bazzite / SteamOS 桌面模式):系統設定 → 鍵盤 → 虛擬鍵盤,選「IBus Wayland」,
|
||||||
|
按套用後登出再登入。之後每次登入都會自動切換到自動注音。
|
||||||
|
- 想暫時改回一般鍵盤:ibus engine xkb:us::eng ;切回自動注音:ibus engine autozhuyin
|
||||||
|
- 移除:執行 uninstall.sh
|
||||||
|
MSG
|
||||||
Executable
+14
@@ -0,0 +1,14 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Removes AutoZhuyin (自動注音) for the current user. Your learned corrections and
|
||||||
|
# user phrases in ~/.config/autozhuyin are kept (delete that folder to remove them).
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/autozhuyin"
|
||||||
|
AUTOSTART="${XDG_CONFIG_HOME:-$HOME/.config}/autostart/autozhuyin.desktop"
|
||||||
|
|
||||||
|
if command -v ibus >/dev/null 2>&1 && [ "$(ibus engine 2>/dev/null)" = "autozhuyin" ]; then
|
||||||
|
ibus engine xkb:us::eng 2>/dev/null
|
||||||
|
fi
|
||||||
|
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null
|
||||||
|
rm -rf "$DEST" "$AUTOSTART"
|
||||||
|
echo "已移除自動注音。學習紀錄留在 ~/.config/autozhuyin,不需要可以自行刪除。"
|
||||||
@@ -0,0 +1,200 @@
|
|||||||
|
"""Drives autozhuyin_ibus.AutoZhuyinEngine through a minimal stand-in for
|
||||||
|
gi.repository.IBus, to catch wrapper bugs without Linux. Run from the repo root:
|
||||||
|
python3 -m unittest discover -s linux/tests
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
LINUX = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
import tempfile # noqa: E402
|
||||||
|
|
||||||
|
os.environ["XDG_CONFIG_HOME"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(LINUX), "windows")) # autozhuyin package
|
||||||
|
sys.path.insert(0, LINUX)
|
||||||
|
|
||||||
|
|
||||||
|
# --- stand-in for gi / IBus (only what the wrapper uses) ---
|
||||||
|
class Flag(int):
|
||||||
|
def __or__(self, o): return Flag(int(self) | int(o))
|
||||||
|
def __and__(self, o): return Flag(int(self) & int(o))
|
||||||
|
|
||||||
|
|
||||||
|
IBus = types.SimpleNamespace()
|
||||||
|
for i, name in enumerate(["BackSpace", "Return", "KP_Enter", "Escape", "Tab", "Up", "Down", "Left",
|
||||||
|
"Right", "space", "Shift_L", "Shift_R", "Delete"]):
|
||||||
|
setattr(IBus, "KEY_" + name, 0xff00 + i)
|
||||||
|
IBus.KEY_space = 0x20
|
||||||
|
IBus.ModifierType = types.SimpleNamespace(
|
||||||
|
SHIFT_MASK=Flag(1), LOCK_MASK=Flag(2), CONTROL_MASK=Flag(4), MOD1_MASK=Flag(8),
|
||||||
|
SUPER_MASK=Flag(1 << 26), MOD4_MASK=Flag(64), RELEASE_MASK=Flag(1 << 30))
|
||||||
|
IBus.keyval_to_unicode = lambda k: chr(k) if 0x20 <= k < 0x7f else "\x00"
|
||||||
|
|
||||||
|
|
||||||
|
class Text:
|
||||||
|
def __init__(self, s): self.text, self.attrs = s, None
|
||||||
|
@staticmethod
|
||||||
|
def new_from_string(s): return Text(s)
|
||||||
|
def set_attributes(self, a): self.attrs = a
|
||||||
|
|
||||||
|
|
||||||
|
class AttrList(list):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class LookupTable:
|
||||||
|
def __init__(self): self.items, self.cursor = [], 0
|
||||||
|
@staticmethod
|
||||||
|
def new(*a): return LookupTable()
|
||||||
|
def set_orientation(self, o): pass
|
||||||
|
def clear(self): self.items = []
|
||||||
|
def append_candidate(self, t): self.items.append(t.text)
|
||||||
|
def set_cursor_pos(self, p): self.cursor = p
|
||||||
|
|
||||||
|
|
||||||
|
class Property:
|
||||||
|
def __init__(self, **kw): self.key, self.label, self.state = kw["key"], kw["label"].text, kw.get("state", 0)
|
||||||
|
def set_label(self, t): self.label = t.text
|
||||||
|
def set_symbol(self, t): pass
|
||||||
|
def set_state(self, s): self.state = s
|
||||||
|
def set_icon(self, path): self.icon = path
|
||||||
|
|
||||||
|
|
||||||
|
class Engine:
|
||||||
|
def __init__(self, *a, **kw):
|
||||||
|
self.committed, self.preedit, self.cursor, self.shown = "", "", 0, None
|
||||||
|
self.preedit_attrs = None
|
||||||
|
|
||||||
|
def commit_text(self, t): self.committed += t.text
|
||||||
|
def update_preedit_text_with_mode(self, t, cursor, visible, mode):
|
||||||
|
self.preedit, self.cursor, self.preedit_attrs = t.text, cursor, t.attrs
|
||||||
|
def hide_preedit_text(self): self.preedit = ""
|
||||||
|
def update_lookup_table(self, table, visible): self.shown = list(table.items)
|
||||||
|
def hide_lookup_table(self): self.shown = None
|
||||||
|
def register_properties(self, p): pass
|
||||||
|
def update_property(self, p):
|
||||||
|
if p.key == "mode":
|
||||||
|
self.mode_label = p.label
|
||||||
|
|
||||||
|
|
||||||
|
IBus.Text, IBus.AttrList, IBus.LookupTable, IBus.Property, IBus.Engine = Text, AttrList, LookupTable, Property, Engine
|
||||||
|
IBus.PropList = lambda: types.SimpleNamespace(append=lambda p: None)
|
||||||
|
IBus.attr_underline_new = lambda kind, a, b: ("underline", a, b)
|
||||||
|
IBus.attr_foreground_new = lambda color, a, b: ("color", a, b)
|
||||||
|
IBus.AttrUnderline = types.SimpleNamespace(SINGLE=1)
|
||||||
|
IBus.Orientation = types.SimpleNamespace(VERTICAL=1)
|
||||||
|
IBus.PropType = types.SimpleNamespace(NORMAL=0, TOGGLE=1, RADIO=2, MENU=3)
|
||||||
|
IBus.PropState = types.SimpleNamespace(UNCHECKED=0, CHECKED=1)
|
||||||
|
IBus.PreeditFocusMode = types.SimpleNamespace(COMMIT=1)
|
||||||
|
|
||||||
|
gi = types.ModuleType("gi")
|
||||||
|
gi.require_version = lambda *a: None
|
||||||
|
repo = types.ModuleType("gi.repository")
|
||||||
|
repo.IBus, repo.GLib, repo.GObject = IBus, types.SimpleNamespace(), types.SimpleNamespace()
|
||||||
|
gi.repository = repo
|
||||||
|
sys.modules["gi"], sys.modules["gi.repository"] = gi, repo
|
||||||
|
|
||||||
|
import autozhuyin_ibus # noqa: E402
|
||||||
|
|
||||||
|
M = IBus.ModifierType
|
||||||
|
|
||||||
|
|
||||||
|
def press(e, keys):
|
||||||
|
"""Types keys; returns the list of 'handled' results."""
|
||||||
|
names = {"\n": IBus.KEY_Return, "\b": IBus.KEY_BackSpace, "\t": IBus.KEY_Tab, "←": IBus.KEY_Left,
|
||||||
|
"→": IBus.KEY_Right, "↓": IBus.KEY_Down, " ": IBus.KEY_space}
|
||||||
|
out = []
|
||||||
|
for c in keys:
|
||||||
|
kv = names.get(c, ord(c))
|
||||||
|
out.append(e.do_process_key_event(kv, 0, Flag(0)))
|
||||||
|
e.do_process_key_event(kv, 0, M.RELEASE_MASK)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class IBusEngineTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
autozhuyin_ibus.shared_classifier().history.clear()
|
||||||
|
autozhuyin_ibus.shared_classifier().learned.clear()
|
||||||
|
self.e = autozhuyin_ibus.AutoZhuyinEngine()
|
||||||
|
|
||||||
|
def test_mixed_sentence(self):
|
||||||
|
press(self.e, "ji3 hello world su3cl3 ")
|
||||||
|
self.assertEqual(self.e.committed, "我hello world 你好")
|
||||||
|
self.assertEqual(self.e.preedit, "")
|
||||||
|
|
||||||
|
def test_preedit_marks_english_blue(self):
|
||||||
|
press(self.e, "ji3hello")
|
||||||
|
self.assertEqual(self.e.preedit, "我hello")
|
||||||
|
self.assertIn(("color", 1, 6), self.e.preedit_attrs)
|
||||||
|
self.assertEqual(self.e.cursor, 6)
|
||||||
|
|
||||||
|
def test_space_passes_through_when_idle(self):
|
||||||
|
self.assertEqual(press(self.e, " "), [False])
|
||||||
|
|
||||||
|
def test_candidates_and_click(self):
|
||||||
|
press(self.e, "su3↓")
|
||||||
|
self.assertTrue(self.e.shown)
|
||||||
|
second = self.e.shown[1]
|
||||||
|
self.e.do_candidate_clicked(1, 1, 0)
|
||||||
|
self.assertIsNone(self.e.shown)
|
||||||
|
press(self.e, "\n")
|
||||||
|
self.assertEqual(self.e.committed, second)
|
||||||
|
|
||||||
|
def test_shift_tap_toggles_english(self):
|
||||||
|
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, Flag(0))
|
||||||
|
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, M.SHIFT_MASK | M.RELEASE_MASK)
|
||||||
|
self.assertEqual(self.e.mode_label, "英文")
|
||||||
|
self.assertEqual(press(self.e, "a"), [False]) # goes to the app as typed
|
||||||
|
|
||||||
|
def test_shift_as_modifier_is_not_a_tap(self):
|
||||||
|
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, Flag(0))
|
||||||
|
self.e.do_process_key_event(ord("A"), 0, M.SHIFT_MASK)
|
||||||
|
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, M.SHIFT_MASK | M.RELEASE_MASK)
|
||||||
|
self.assertFalse(self.e.engine.english_mode)
|
||||||
|
|
||||||
|
def test_ctrl_shortcut_commits_and_passes(self):
|
||||||
|
press(self.e, "su3")
|
||||||
|
self.assertFalse(self.e.do_process_key_event(ord("c"), 0, M.CONTROL_MASK))
|
||||||
|
self.assertEqual(self.e.committed, "你")
|
||||||
|
|
||||||
|
def test_backtick_symbol(self):
|
||||||
|
press(self.e, "`<")
|
||||||
|
self.assertEqual(self.e.committed, ",")
|
||||||
|
|
||||||
|
def test_insert_in_middle(self):
|
||||||
|
press(self.e, "rup fu4←wu0 \n")
|
||||||
|
self.assertEqual(self.e.committed, "今天氣")
|
||||||
|
|
||||||
|
def test_picked_candidate_saved(self):
|
||||||
|
press(self.e, "su3↓")
|
||||||
|
second = self.e.shown[1]
|
||||||
|
self.e.do_candidate_clicked(1, 1, 0)
|
||||||
|
press(self.e, "\n")
|
||||||
|
path = os.path.join(os.environ["XDG_CONFIG_HOME"], "autozhuyin", "user-history.json")
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
self.assertIn(second, f.read())
|
||||||
|
press(self.e, "su3 \n")
|
||||||
|
self.assertTrue(self.e.committed.endswith(second))
|
||||||
|
|
||||||
|
def test_ai_menu(self):
|
||||||
|
e = self.e
|
||||||
|
self.assertEqual(e.ai_props["standard"].state, 1)
|
||||||
|
e.do_property_activate("ai.saver", 1)
|
||||||
|
self.assertEqual(autozhuyin_ibus.shared_ai().reranker.mode, "saver")
|
||||||
|
self.assertEqual(e.ai_props["saver"].state, 1)
|
||||||
|
self.assertEqual(e.ai_props["standard"].state, 0)
|
||||||
|
e.do_property_activate("ai.autopause", 0)
|
||||||
|
self.assertFalse(autozhuyin_ibus.shared_ai().auto_pause)
|
||||||
|
self.assertTrue(e.status_prop.label)
|
||||||
|
e.do_property_activate("ai.standard", 1)
|
||||||
|
e.do_property_activate("ai.autopause", 1)
|
||||||
|
|
||||||
|
def test_focus_out_resets(self):
|
||||||
|
press(self.e, "su3")
|
||||||
|
self.e.do_focus_out()
|
||||||
|
self.assertEqual(self.e.engine.keys, [])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -18,8 +18,13 @@ rm -rf "$APP"
|
|||||||
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
||||||
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
|
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
|
||||||
cp Resources/Info.plist "$APP/Contents/Info.plist"
|
cp Resources/Info.plist "$APP/Contents/Info.plist"
|
||||||
|
# Localized names of the input method and its modes (System Settings, input menu).
|
||||||
|
cp -R Resources/*.lproj "$APP/Contents/Resources/"
|
||||||
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
|
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
|
||||||
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
|
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
|
||||||
|
# "AI 選字" language model (CKIP gpt2-tiny-chinese, GPL-3.0), fetched and converted once.
|
||||||
|
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null
|
||||||
|
cp build/lm/lm.bin build/lm/lm-vocab.txt licenses/GPL-3.0.txt "$APP/Contents/Resources/"
|
||||||
# Precomputed English dictionary and bigram table (shared with the Windows build).
|
# Precomputed English dictionary and bigram table (shared with the Windows build).
|
||||||
python3 -I windows/tools/build_data.py >/dev/null
|
python3 -I windows/tools/build_data.py >/dev/null
|
||||||
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
|
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
|
||||||
@@ -28,6 +33,9 @@ if [ ! -f build/icon.tiff ]; then
|
|||||||
xcrun swift scripts/make-icon.swift build/icon.tiff
|
xcrun swift scripts/make-icon.swift build/icon.tiff
|
||||||
fi
|
fi
|
||||||
cp build/icon.tiff "$APP/Contents/Resources/icon.tiff"
|
cp build/icon.tiff "$APP/Contents/Resources/icon.tiff"
|
||||||
|
# English-mode icon ("A"), shown in the menu bar while English mode is on.
|
||||||
|
[ -f build/icon-en.tiff ] || xcrun swift scripts/make-icon.swift build/icon-en.tiff A
|
||||||
|
cp build/icon-en.tiff "$APP/Contents/Resources/icon-en.tiff"
|
||||||
|
|
||||||
codesign --force --sign - "$APP"
|
codesign --force --sign - "$APP"
|
||||||
echo "Built $APP"
|
echo "Built $APP"
|
||||||
|
|||||||
Executable
+18
@@ -0,0 +1,18 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Downloads the CKIP gpt2-tiny-chinese language model (GPL-3.0,
|
||||||
|
# https://huggingface.co/ckiplab/gpt2-tiny-chinese) and converts it to build/lm/:
|
||||||
|
# lm.bin float16 weights (custom format, see training/lm/export_lm.py)
|
||||||
|
# lm-vocab.txt
|
||||||
|
# build.sh bundles them into the app when present ("AI 選字"). Safe to re-run.
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
SRC=build/models/ckip-gpt2-tiny
|
||||||
|
BASE="https://huggingface.co/ckiplab/gpt2-tiny-chinese/resolve/refs%2Fpr%2F1"
|
||||||
|
mkdir -p "$SRC" build/lm
|
||||||
|
for f in model.safetensors vocab.txt config.json; do
|
||||||
|
[ -s "$SRC/$f" ] || curl -sfL -o "$SRC/$f" "$BASE/$f"
|
||||||
|
done
|
||||||
|
python3 -I training/lm/export_lm.py "$SRC" build/lm/lm.bin
|
||||||
|
cp "$SRC/vocab.txt" build/lm/lm-vocab.txt
|
||||||
|
ls -lh build/lm
|
||||||
@@ -1,8 +1,9 @@
|
|||||||
// Renders the menu-bar icon ("注" with a small "A") to a TIFF.
|
// Renders a menu-bar icon (one character, "注" by default; "A" for English mode) to a TIFF.
|
||||||
// Usage: swift scripts/make-icon.swift <output.tiff>
|
// Usage: swift scripts/make-icon.swift <output.tiff> [character]
|
||||||
import AppKit
|
import AppKit
|
||||||
|
|
||||||
let out = CommandLine.arguments.count > 1 ? CommandLine.arguments[1] : "icon.tiff"
|
let out = CommandLine.arguments.count > 1 ? CommandLine.arguments[1] : "icon.tiff"
|
||||||
|
let glyph = CommandLine.arguments.count > 2 ? CommandLine.arguments[2] : "注"
|
||||||
let reps: [NSBitmapImageRep] = [16, 32].map { size in
|
let reps: [NSBitmapImageRep] = [16, 32].map { size in
|
||||||
let rep = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: size, pixelsHigh: size,
|
let rep = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: size, pixelsHigh: size,
|
||||||
bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false,
|
bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false,
|
||||||
@@ -11,7 +12,7 @@ let reps: [NSBitmapImageRep] = [16, 32].map { size in
|
|||||||
NSGraphicsContext.saveGraphicsState()
|
NSGraphicsContext.saveGraphicsState()
|
||||||
NSGraphicsContext.current = NSGraphicsContext(bitmapImageRep: rep)
|
NSGraphicsContext.current = NSGraphicsContext(bitmapImageRep: rep)
|
||||||
let font = NSFont.systemFont(ofSize: 13, weight: .semibold)
|
let font = NSFont.systemFont(ofSize: 13, weight: .semibold)
|
||||||
let text = NSAttributedString(string: "注", attributes: [.font: font, .foregroundColor: NSColor.black])
|
let text = NSAttributedString(string: glyph, attributes: [.font: font, .foregroundColor: NSColor.black])
|
||||||
let s = text.size()
|
let s = text.size()
|
||||||
text.draw(at: NSPoint(x: (16 - s.width) / 2, y: (16 - s.height) / 2))
|
text.draw(at: NSPoint(x: (16 - s.width) / 2, y: (16 - s.height) / 2))
|
||||||
NSGraphicsContext.restoreGraphicsState()
|
NSGraphicsContext.restoreGraphicsState()
|
||||||
|
|||||||
Executable
+36
@@ -0,0 +1,36 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Builds build/AutoZhuyin-Linux-<version>.tar.gz: the IBus engine, the shared Python
|
||||||
|
# core and data, and install/uninstall scripts (per-user install, no root needed).
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
VERSION="$(python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["version"])' windows/autozhuyin/ime.json)"
|
||||||
|
NAME="AutoZhuyin-Linux-$VERSION"
|
||||||
|
OUT="build/$NAME"
|
||||||
|
|
||||||
|
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null # "AI 選字" model
|
||||||
|
python3 -I windows/tools/build_data.py
|
||||||
|
(cd windows && python3 -m unittest discover -q tests)
|
||||||
|
python3 -m unittest discover -q -s linux/tests
|
||||||
|
|
||||||
|
if [ ! -f build/icon.tiff ]; then
|
||||||
|
DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon.tiff
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm -rf "$OUT" "$OUT.tar.gz"
|
||||||
|
mkdir -p "$OUT/autozhuyin"
|
||||||
|
rsync -a --exclude __pycache__ --exclude '*.pyc' \
|
||||||
|
windows/autozhuyin/__init__.py windows/autozhuyin/core windows/autozhuyin/data "$OUT/autozhuyin/"
|
||||||
|
cp linux/autozhuyin_ibus.py linux/installer/install.sh linux/installer/uninstall.sh linux/installer/README.txt \
|
||||||
|
THIRD_PARTY_LICENSES.txt licenses/GPL-3.0.txt "$OUT/"
|
||||||
|
mkdir -p "$OUT/tools"
|
||||||
|
cp windows/tools/fetch_numpy.py "$OUT/tools/" # install.sh downloads numpy if the system has none
|
||||||
|
echo "$VERSION" > "$OUT/version.txt"
|
||||||
|
sips -s format png -z 64 64 build/icon.tiff --out "$OUT/icon.png" >/dev/null
|
||||||
|
[ -f build/icon-en.tiff ] || DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon-en.tiff A
|
||||||
|
sips -s format png -z 64 64 build/icon-en.tiff --out "$OUT/icon-en.png" >/dev/null
|
||||||
|
chmod +x "$OUT/install.sh" "$OUT/uninstall.sh" "$OUT/autozhuyin_ibus.py"
|
||||||
|
|
||||||
|
# Plain ustar without macOS extended attributes, so GNU tar doesn't warn.
|
||||||
|
(cd build && COPYFILE_DISABLE=1 tar --no-xattrs --format ustar -czf "$NAME.tar.gz" "$NAME")
|
||||||
|
echo "Built build/$NAME.tar.gz"
|
||||||
@@ -8,6 +8,7 @@ VERSION="$(python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["v
|
|||||||
NAME="AutoZhuyin-Windows-$VERSION"
|
NAME="AutoZhuyin-Windows-$VERSION"
|
||||||
OUT="build/$NAME"
|
OUT="build/$NAME"
|
||||||
|
|
||||||
|
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null # "AI 選字" model
|
||||||
python3 -I windows/tools/build_data.py
|
python3 -I windows/tools/build_data.py
|
||||||
(cd windows && python3 -m unittest discover -q tests)
|
(cd windows && python3 -m unittest discover -q tests)
|
||||||
|
|
||||||
@@ -17,6 +18,10 @@ if [ ! -f build/icon.tiff ]; then
|
|||||||
fi
|
fi
|
||||||
sips -s format png -z 32 32 build/icon.tiff --out build/icon-32.png >/dev/null
|
sips -s format png -z 32 32 build/icon.tiff --out build/icon-32.png >/dev/null
|
||||||
python3 -I windows/tools/png_to_ico.py build/icon-32.png windows/autozhuyin/icon.ico
|
python3 -I windows/tools/png_to_ico.py build/icon-32.png windows/autozhuyin/icon.ico
|
||||||
|
# English-mode icon ("A") for the taskbar while English mode is on.
|
||||||
|
[ -f build/icon-en.tiff ] || DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon-en.tiff A
|
||||||
|
sips -s format png -z 32 32 build/icon-en.tiff --out build/icon-en-32.png >/dev/null
|
||||||
|
python3 -I windows/tools/png_to_ico.py build/icon-en-32.png windows/autozhuyin/icon-en.ico
|
||||||
|
|
||||||
rm -rf "$OUT" "$OUT.zip"
|
rm -rf "$OUT" "$OUT.zip"
|
||||||
mkdir -p "$OUT"
|
mkdir -p "$OUT"
|
||||||
@@ -29,6 +34,14 @@ done
|
|||||||
# Third-party notices: next to the installer and inside the installed folder.
|
# Third-party notices: next to the installer and inside the installed folder.
|
||||||
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
|
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
|
||||||
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
|
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
|
||||||
|
sed 's/$/\r/' licenses/GPL-3.0.txt > "$OUT/autozhuyin/GPL-3.0.txt"
|
||||||
|
|
||||||
|
# numpy for "AI 選字", for both Pythons PIME ships (1.3: 3.8, 2.x: 3.15; both 32-bit).
|
||||||
|
for tag in cp38 cp315; do
|
||||||
|
[ -d "build/numpy-vendor/$tag" ] || python3 -I windows/tools/fetch_numpy.py build/numpy-vendor "$tag" win32
|
||||||
|
done
|
||||||
|
mkdir -p "$OUT/autozhuyin/vendor"
|
||||||
|
cp -R build/numpy-vendor/cp38 build/numpy-vendor/cp315 "$OUT/autozhuyin/vendor/"
|
||||||
|
|
||||||
(cd build && zip -qr -X "$NAME.zip" "$NAME")
|
(cd build && zip -qr -X "$NAME.zip" "$NAME")
|
||||||
echo "Built build/$NAME.zip"
|
echo "Built build/$NAME.zip"
|
||||||
|
|||||||
+5
-3
@@ -1,5 +1,5 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# One-step release: bump the version, run all tests, build both installers, then
|
# One-step release: bump the version, run all tests, build all installers, then
|
||||||
# commit the version change and tag it.
|
# commit the version change and tag it.
|
||||||
#
|
#
|
||||||
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
|
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
|
||||||
@@ -38,14 +38,16 @@ xcrun swift test -q 2>&1 | tail -3
|
|||||||
echo "==> Packaging (runs the Python tests too)"
|
echo "==> Packaging (runs the Python tests too)"
|
||||||
scripts/package.sh >/dev/null
|
scripts/package.sh >/dev/null
|
||||||
scripts/package-windows.sh >/dev/null
|
scripts/package-windows.sh >/dev/null
|
||||||
|
scripts/package-linux.sh >/dev/null
|
||||||
|
|
||||||
trap - ERR
|
trap - ERR
|
||||||
git add Resources/Info.plist windows/autozhuyin/ime.json
|
git add Resources/Info.plist windows/autozhuyin/ime.json
|
||||||
git commit -q -m "Release v$VERSION"
|
git commit -q -m "Release v$VERSION"
|
||||||
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
|
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
|
||||||
echo "==> Tagged v$VERSION"
|
echo "==> Tagged v$VERSION"
|
||||||
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
|
FILES=("build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" "build/AutoZhuyin-Linux-$VERSION.tar.gz")
|
||||||
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
|
ls -lh "${FILES[@]}"
|
||||||
|
shasum -a 256 "${FILES[@]}"
|
||||||
|
|
||||||
if [ "$PUSH" = 1 ]; then
|
if [ "$PUSH" = 1 ]; then
|
||||||
git push origin HEAD "v$VERSION"
|
git push origin HEAD "v$VERSION"
|
||||||
|
|||||||
@@ -0,0 +1,147 @@
|
|||||||
|
"""Measures how much language-model reranking improves Zhuyin → Chinese conversion.
|
||||||
|
|
||||||
|
For each test sentence: turn the text into Dachen keystrokes (via the lexicon's
|
||||||
|
readings), convert with the current Viterbi converter (baseline), take the K best
|
||||||
|
paths, and rerank them with lexicon score + λ · LM log-probability.
|
||||||
|
|
||||||
|
Usage (needs numpy and the model in build/models/ckip-gpt2-tiny):
|
||||||
|
python training/lm/eval_rerank.py build/lm-eval/chat.txt build/lm-eval/wiki.txt
|
||||||
|
"""
|
||||||
|
import heapq
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
sys.path.insert(0, os.path.join(ROOT, "windows"))
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from autozhuyin.core import bopomofo # noqa: E402
|
||||||
|
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
|
||||||
|
from gpt2 import GPT2 # noqa: E402
|
||||||
|
|
||||||
|
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
|
||||||
|
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
|
||||||
|
LN10 = math.log(10)
|
||||||
|
|
||||||
|
|
||||||
|
def reading_keys(reading):
|
||||||
|
out = []
|
||||||
|
for syl in reading.split("-"):
|
||||||
|
tone = " "
|
||||||
|
for ch in syl:
|
||||||
|
if ch in TONE_KEY:
|
||||||
|
tone = TONE_KEY[ch]
|
||||||
|
else:
|
||||||
|
out.append(KEY_OF[ch])
|
||||||
|
out.append(tone)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def build_reverse(lex):
|
||||||
|
best = {}
|
||||||
|
for reading, entries in lex.exact.items():
|
||||||
|
for phrase, score in entries:
|
||||||
|
if phrase not in best or score > best[phrase][1]:
|
||||||
|
best[phrase] = (reading, score)
|
||||||
|
return best
|
||||||
|
|
||||||
|
|
||||||
|
def text_to_keys(text, reverse, max_len):
|
||||||
|
"""Most likely segmentation of the text into lexicon phrases → keystrokes."""
|
||||||
|
n = len(text)
|
||||||
|
best = [(-math.inf, None)] * (n + 1)
|
||||||
|
best[0] = (0.0, None)
|
||||||
|
for j in range(1, n + 1):
|
||||||
|
for i in range(max(0, j - max_len), j):
|
||||||
|
if best[i][0] == -math.inf:
|
||||||
|
continue
|
||||||
|
hit = reverse.get(text[i:j])
|
||||||
|
if hit:
|
||||||
|
s = best[i][0] + hit[1]
|
||||||
|
if s > best[j][0]:
|
||||||
|
best[j] = (s, (i, hit[0]))
|
||||||
|
if best[n][1] is None:
|
||||||
|
return None
|
||||||
|
readings, j = [], n
|
||||||
|
while j > 0:
|
||||||
|
i, r = best[j][1]
|
||||||
|
readings.append(r)
|
||||||
|
j = i
|
||||||
|
keys = []
|
||||||
|
for r in reversed(readings):
|
||||||
|
keys += reading_keys(r)
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def kbest(conv, syllables, k):
|
||||||
|
"""K best (score, text) paths through the converter's lattice (scores in log10)."""
|
||||||
|
n = len(syllables)
|
||||||
|
max_len = max(1, conv.lexicon.max_phrase_length)
|
||||||
|
beams = [[] for _ in range(n + 1)]
|
||||||
|
beams[0] = [(0.0, "")]
|
||||||
|
for j in range(1, n + 1):
|
||||||
|
cand = []
|
||||||
|
for length in range(1, min(max_len, j) + 1):
|
||||||
|
i = j - length
|
||||||
|
if not beams[i]:
|
||||||
|
continue
|
||||||
|
for text, score in conv._options(syllables, i, j, {}):
|
||||||
|
for s, t in beams[i]:
|
||||||
|
cand.append((s + score, t + text))
|
||||||
|
beams[j] = heapq.nlargest(k, cand, key=lambda e: e[0])
|
||||||
|
return beams[n]
|
||||||
|
|
||||||
|
|
||||||
|
def char_errors(a, b):
|
||||||
|
return sum(x != y for x, y in zip(a, b)) + abs(len(a) - len(b))
|
||||||
|
|
||||||
|
|
||||||
|
def main(paths, k=10):
|
||||||
|
lex = Lexicon.from_file(os.path.join(ROOT, "windows/autozhuyin/data/lexicon.txt"))
|
||||||
|
conv = Converter(lex)
|
||||||
|
lm = GPT2(os.path.join(ROOT, "build/models/ckip-gpt2-tiny"))
|
||||||
|
reverse = build_reverse(lex)
|
||||||
|
|
||||||
|
cases = []
|
||||||
|
for path in paths:
|
||||||
|
for line in open(path, encoding="utf-8"):
|
||||||
|
text = line.strip().replace(",", "").replace("、", "")
|
||||||
|
if not text:
|
||||||
|
continue
|
||||||
|
keys = text_to_keys(text, reverse, lex.max_phrase_length)
|
||||||
|
if keys is None:
|
||||||
|
continue
|
||||||
|
syl = [s.syllable for s in bopomofo.parse(keys).syllables]
|
||||||
|
paths_k = kbest(conv, syl, k)
|
||||||
|
t0 = time.time()
|
||||||
|
lm_scores = [lm.logprob(t) for _, t in paths_k]
|
||||||
|
cases.append((os.path.basename(path), text, paths_k, lm_scores, time.time() - t0))
|
||||||
|
|
||||||
|
def evaluate(lam, subset):
|
||||||
|
chars = errs = sent_ok = 0
|
||||||
|
for _, text, paths_k, lm_scores, _ in subset:
|
||||||
|
best = max(range(len(paths_k)), key=lambda i: paths_k[i][0] * LN10 + lam * lm_scores[i])
|
||||||
|
out = paths_k[best][1]
|
||||||
|
e = char_errors(out, text)
|
||||||
|
chars += len(text)
|
||||||
|
errs += e
|
||||||
|
sent_ok += e == 0
|
||||||
|
return 1 - errs / chars, sent_ok / len(subset)
|
||||||
|
|
||||||
|
lams = [0, 0.1, 0.2, 0.3, 0.5, 0.7, 1.0, 1.5, 2.0]
|
||||||
|
for name in sorted({c[0] for c in cases}):
|
||||||
|
sub = [c for c in cases if c[0] == name]
|
||||||
|
oracle = sum(any(t == c[1] for _, t in c[2]) for c in sub) / len(sub)
|
||||||
|
print(f"\n== {name}: {len(sub)} sentences, top-{k} contains the answer: {oracle:.1%}")
|
||||||
|
for lam in lams:
|
||||||
|
ca, sa = evaluate(lam, sub)
|
||||||
|
print(f" λ={lam:<4} char acc {ca:.2%} sentence acc {sa:.1%}" + (" (baseline)" if lam == 0 else ""))
|
||||||
|
ms = sum(c[4] for c in cases) / len(cases) * 1000
|
||||||
|
print(f"\nLM time per sentence (top-{k}): {ms:.1f} ms")
|
||||||
|
return cases
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main(sys.argv[1:])
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
"""Converts the gpt2-tiny-chinese safetensors file to the input method's format.
|
||||||
|
|
||||||
|
Format (little-endian):
|
||||||
|
"AZLM" magic, uint32 header length, JSON header, then float16 tensors back to back.
|
||||||
|
Header: {"n_layer", "n_head", "n_embd", "vocab", "eps", "tensors": [[name, [shape], offset]]}
|
||||||
|
(offset in float16 elements from the start of the data). Linear weights keep the
|
||||||
|
GPT-2 Conv1D layout [in, out]. The attention mask buffers are dropped.
|
||||||
|
|
||||||
|
Usage: export_lm.py <model dir> <out.bin> (standard library only)
|
||||||
|
"""
|
||||||
|
import array
|
||||||
|
import json
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
def main(src, out):
|
||||||
|
cfg = json.load(open(f"{src}/config.json"))
|
||||||
|
with open(f"{src}/model.safetensors", "rb") as f:
|
||||||
|
n = struct.unpack("<Q", f.read(8))[0]
|
||||||
|
header = json.loads(f.read(n))
|
||||||
|
base = 8 + n
|
||||||
|
names = sorted(k for k, v in header.items() if k != "__metadata__" and v["dtype"] == "F32" and v["shape"])
|
||||||
|
tensors, blobs, offset = [], [], 0
|
||||||
|
for name in names:
|
||||||
|
info = header[name]
|
||||||
|
a, b = info["data_offsets"]
|
||||||
|
f.seek(base + a)
|
||||||
|
floats = array.array("f")
|
||||||
|
floats.frombytes(f.read(b - a))
|
||||||
|
if sys.byteorder != "little":
|
||||||
|
floats.byteswap()
|
||||||
|
half = struct.pack(f"<{len(floats)}e", *floats)
|
||||||
|
tensors.append([name.replace("transformer.", ""), info["shape"], offset])
|
||||||
|
blobs.append(half)
|
||||||
|
offset += len(floats)
|
||||||
|
meta = json.dumps({"n_layer": cfg["n_layer"], "n_head": cfg["n_head"], "n_embd": cfg["n_embd"],
|
||||||
|
"vocab": cfg["vocab_size"], "eps": cfg["layer_norm_epsilon"],
|
||||||
|
"tensors": tensors}).encode()
|
||||||
|
with open(out, "wb") as f:
|
||||||
|
f.write(b"AZLM" + struct.pack("<I", len(meta)) + meta)
|
||||||
|
for blob in blobs:
|
||||||
|
f.write(blob)
|
||||||
|
print(f"wrote {out}: {offset} weights")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main(sys.argv[1], sys.argv[2])
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
"""Minimal GPT-2 forward pass in numpy, for the CKIP gpt2-tiny-chinese model
|
||||||
|
(https://huggingface.co/ckiplab/gpt2-tiny-chinese, GPL-3.0). Character-level:
|
||||||
|
each Chinese character is one token of the bert-base-chinese vocabulary.
|
||||||
|
|
||||||
|
Used to evaluate language-model reranking before porting it to the input methods.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import struct
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
CLS, UNK = 101, 100
|
||||||
|
|
||||||
|
|
||||||
|
def load_safetensors(path):
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
n = struct.unpack("<Q", f.read(8))[0]
|
||||||
|
header = json.loads(f.read(n))
|
||||||
|
base = 8 + n
|
||||||
|
out = {}
|
||||||
|
for name, info in header.items():
|
||||||
|
if name == "__metadata__" or info["dtype"] != "F32":
|
||||||
|
continue
|
||||||
|
a, b = info["data_offsets"]
|
||||||
|
f.seek(base + a)
|
||||||
|
out[name] = np.frombuffer(f.read(b - a), dtype=np.float32).reshape(info["shape"])
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class GPT2:
|
||||||
|
def __init__(self, model_dir):
|
||||||
|
self.w = load_safetensors(f"{model_dir}/model.safetensors")
|
||||||
|
cfg = json.load(open(f"{model_dir}/config.json"))
|
||||||
|
self.n_layer, self.n_head = cfg["n_layer"], cfg["n_head"]
|
||||||
|
self.eps = cfg["layer_norm_epsilon"]
|
||||||
|
self.vocab = {}
|
||||||
|
for i, line in enumerate(open(f"{model_dir}/vocab.txt", encoding="utf-8")):
|
||||||
|
self.vocab.setdefault(line.rstrip("\n"), i)
|
||||||
|
|
||||||
|
def ids(self, text):
|
||||||
|
return [self.vocab.get(c, UNK) for c in text]
|
||||||
|
|
||||||
|
def _ln(self, x, name):
|
||||||
|
mu = x.mean(-1, keepdims=True)
|
||||||
|
var = ((x - mu) ** 2).mean(-1, keepdims=True)
|
||||||
|
return (x - mu) / np.sqrt(var + self.eps) * self.w[name + ".weight"] + self.w[name + ".bias"]
|
||||||
|
|
||||||
|
def hidden(self, ids):
|
||||||
|
"""Final hidden states [T, d] for token ids."""
|
||||||
|
w, T = self.w, len(ids)
|
||||||
|
x = w["transformer.wte.weight"][ids] + w["transformer.wpe.weight"][:T]
|
||||||
|
mask = np.triu(np.full((T, T), -1e9, dtype=np.float32), 1)
|
||||||
|
for l in range(self.n_layer):
|
||||||
|
p = f"transformer.h.{l}."
|
||||||
|
h = self._ln(x, p + "ln_1")
|
||||||
|
qkv = h @ w[p + "attn.c_attn.weight"] + w[p + "attn.c_attn.bias"]
|
||||||
|
q, k, v = np.split(qkv, 3, axis=-1)
|
||||||
|
d = q.shape[-1] // self.n_head
|
||||||
|
q = q.reshape(T, self.n_head, d).transpose(1, 0, 2)
|
||||||
|
k = k.reshape(T, self.n_head, d).transpose(1, 0, 2)
|
||||||
|
v = v.reshape(T, self.n_head, d).transpose(1, 0, 2)
|
||||||
|
att = q @ k.transpose(0, 2, 1) / math.sqrt(d) + mask
|
||||||
|
att = np.exp(att - att.max(-1, keepdims=True))
|
||||||
|
att /= att.sum(-1, keepdims=True)
|
||||||
|
a = (att @ v).transpose(1, 0, 2).reshape(T, -1)
|
||||||
|
x = x + a @ w[p + "attn.c_proj.weight"] + w[p + "attn.c_proj.bias"]
|
||||||
|
h = self._ln(x, p + "ln_2")
|
||||||
|
h = h @ w[p + "mlp.c_fc.weight"] + w[p + "mlp.c_fc.bias"]
|
||||||
|
h = 0.5 * h * (1 + np.tanh(math.sqrt(2 / math.pi) * (h + 0.044715 * h ** 3))) # gelu_new
|
||||||
|
x = x + h @ w[p + "mlp.c_proj.weight"] + w[p + "mlp.c_proj.bias"]
|
||||||
|
return self._ln(x, "transformer.ln_f")
|
||||||
|
|
||||||
|
def logprob(self, text, context=""):
|
||||||
|
"""Natural-log probability of `text` given `context` (both strings)."""
|
||||||
|
ctx, tgt = self.ids(context), self.ids(text)
|
||||||
|
ids = [CLS] + ctx + tgt
|
||||||
|
h = self.hidden(ids[:-1])
|
||||||
|
logits = h @ self.w["transformer.wte.weight"].T
|
||||||
|
logits -= logits.max(-1, keepdims=True)
|
||||||
|
lse = np.log(np.exp(logits).sum(-1))
|
||||||
|
start = len(ctx)
|
||||||
|
return float(sum(logits[start + i, t] - lse[start + i] for i, t in enumerate(tgt)))
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"ㄋㄧ": {"妳": [1, 1]}}
|
||||||
@@ -6,11 +6,21 @@ tests); this file only translates PIME key events and replies.
|
|||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import sys
|
||||||
import time
|
import time
|
||||||
|
|
||||||
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
|
# numpy for "AI 選字" is bundled per Python version (PIME 1.3 has 3.8, PIME 2 has 3.15).
|
||||||
from textService import TextService
|
_VENDOR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "vendor",
|
||||||
|
"cp%d%d" % sys.version_info[:2])
|
||||||
|
if os.path.isdir(_VENDOR) and _VENDOR not in sys.path:
|
||||||
|
sys.path.insert(0, _VENDOR)
|
||||||
|
|
||||||
|
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
|
||||||
|
from textService import COMMAND_LEFT_CLICK, COMMAND_MENU, TextService # noqa: F401
|
||||||
|
|
||||||
|
from .core.ai import AI, MODE_NAMES
|
||||||
|
from .core.history import UserHistory
|
||||||
|
from .core.reranker import MODES
|
||||||
from .core.classifier import Classifier, LearnedPreferences
|
from .core.classifier import Classifier, LearnedPreferences
|
||||||
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
|
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
|
||||||
from .core.english import EnglishModel
|
from .core.english import EnglishModel
|
||||||
@@ -22,17 +32,40 @@ _DATA = os.path.join(_HERE, "data")
|
|||||||
_USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin")
|
_USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin")
|
||||||
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
|
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
|
||||||
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
|
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
|
||||||
|
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
|
||||||
|
|
||||||
_classifier = None
|
_classifier = None
|
||||||
|
_ai = None
|
||||||
|
|
||||||
|
ID_MODE_ICON, ID_AUTO_PAUSE, ID_STATUS, ID_CLEAR = 1, 2, 3, 4
|
||||||
|
ID_AI_MODE = {mode: 10 + i for i, mode in enumerate(MODES)} # 10..13
|
||||||
|
_ICON = os.path.join(_HERE, "icon.ico")
|
||||||
|
_ICON_EN = os.path.join(_HERE, "icon-en.ico") # shown while English mode is on
|
||||||
|
|
||||||
|
|
||||||
|
def _save_json(path, data):
|
||||||
|
try:
|
||||||
|
os.makedirs(_USER_DIR, exist_ok=True)
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, ensure_ascii=False)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json(path):
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
def _save_learned(weights):
|
def _save_learned(weights):
|
||||||
try:
|
_save_json(_LEARNED_FILE, weights)
|
||||||
os.makedirs(_USER_DIR, exist_ok=True)
|
|
||||||
with open(_LEARNED_FILE, "w", encoding="utf-8") as f:
|
|
||||||
json.dump(weights, f, ensure_ascii=False)
|
def _save_history(data):
|
||||||
except OSError:
|
_save_json(_HISTORY_FILE, data)
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
def _shared_classifier():
|
def _shared_classifier():
|
||||||
@@ -48,17 +81,20 @@ def _shared_classifier():
|
|||||||
os.path.join(_DATA, "english-dict.txt"),
|
os.path.join(_DATA, "english-dict.txt"),
|
||||||
os.path.join(_DATA, "english-bigram.json"),
|
os.path.join(_DATA, "english-bigram.json"),
|
||||||
)
|
)
|
||||||
weights = {}
|
|
||||||
try:
|
|
||||||
with open(_LEARNED_FILE, encoding="utf-8") as f:
|
|
||||||
weights = json.load(f)
|
|
||||||
except (OSError, ValueError):
|
|
||||||
pass
|
|
||||||
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
|
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
|
||||||
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained)
|
_classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
|
||||||
|
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
|
||||||
return _classifier
|
return _classifier
|
||||||
|
|
||||||
|
|
||||||
|
def _shared_ai():
|
||||||
|
""""AI 選字": settings and the language model, loaded in the background once."""
|
||||||
|
global _ai
|
||||||
|
if _ai is None:
|
||||||
|
_ai = AI(_DATA, _USER_DIR)
|
||||||
|
return _ai
|
||||||
|
|
||||||
|
|
||||||
_SPECIAL = {
|
_SPECIAL = {
|
||||||
VK_BACK: BACKSPACE, VK_RETURN: ENTER, VK_ESCAPE: ESCAPE, VK_TAB: TAB, # noqa: F405
|
VK_BACK: BACKSPACE, VK_RETURN: ENTER, VK_ESCAPE: ESCAPE, VK_TAB: TAB, # noqa: F405
|
||||||
VK_UP: UP, VK_DOWN: DOWN, VK_LEFT: LEFT, VK_RIGHT: RIGHT, # noqa: F405
|
VK_UP: UP, VK_DOWN: DOWN, VK_LEFT: LEFT, VK_RIGHT: RIGHT, # noqa: F405
|
||||||
@@ -70,6 +106,7 @@ class AutoZhuyinTextService(TextService):
|
|||||||
TextService.__init__(self, client)
|
TextService.__init__(self, client)
|
||||||
self.engine = CompositionEngine(_shared_classifier())
|
self.engine = CompositionEngine(_shared_classifier())
|
||||||
self.shift_tap = ShiftTapDetector()
|
self.shift_tap = ShiftTapDetector()
|
||||||
|
self.engine.reranker = _shared_ai().reranker
|
||||||
|
|
||||||
# --- lifecycle ---
|
# --- lifecycle ---
|
||||||
|
|
||||||
@@ -77,11 +114,68 @@ class AutoZhuyinTextService(TextService):
|
|||||||
TextService.onActivate(self)
|
TextService.onActivate(self)
|
||||||
self.setSelKeys("123456789")
|
self.setSelKeys("123456789")
|
||||||
self.customizeUI(candFontSize=16, candPerRow=1, candUseCursor=True)
|
self.customizeUI(candFontSize=16, candPerRow=1, candUseCursor=True)
|
||||||
|
# Taskbar IME icon (Windows 8+) and language bar button; both open the settings menu.
|
||||||
|
self._icon_english = self.engine.english_mode
|
||||||
|
self.addButton("windows-mode-icon", icon=self._mode_icon(), tooltip=self._mode_tooltip(),
|
||||||
|
commandId=ID_MODE_ICON)
|
||||||
|
self.addButton("autozhuyin-settings", icon=_ICON, tooltip="自動注音設定", type="menu")
|
||||||
|
|
||||||
def onDeactivate(self):
|
def onDeactivate(self):
|
||||||
self.engine.reset()
|
self.engine.reset()
|
||||||
|
self.removeButton("windows-mode-icon")
|
||||||
|
self.removeButton("autozhuyin-settings")
|
||||||
TextService.onDeactivate(self)
|
TextService.onDeactivate(self)
|
||||||
|
|
||||||
|
# --- taskbar icon: "注" for 中英自動, "A" for English mode ---
|
||||||
|
|
||||||
|
def _mode_icon(self):
|
||||||
|
return _ICON_EN if self.engine.english_mode and os.path.exists(_ICON_EN) else _ICON
|
||||||
|
|
||||||
|
def _mode_tooltip(self):
|
||||||
|
return ("英文模式" if self.engine.english_mode else "中英自動") + "(左鍵切換,右鍵設定)"
|
||||||
|
|
||||||
|
def _update_mode_icon(self):
|
||||||
|
if getattr(self, "_icon_english", None) == self.engine.english_mode:
|
||||||
|
return
|
||||||
|
self._icon_english = self.engine.english_mode
|
||||||
|
self.changeButton("windows-mode-icon", icon=self._mode_icon(), tooltip=self._mode_tooltip())
|
||||||
|
|
||||||
|
# --- menu (AI 選字 settings) ---
|
||||||
|
|
||||||
|
def onMenu(self, buttonId):
|
||||||
|
if buttonId not in ("windows-mode-icon", "autozhuyin-settings"):
|
||||||
|
return None
|
||||||
|
ai = _shared_ai()
|
||||||
|
items = [{"text": "AI 選字:" + MODE_NAMES[m], "id": ID_AI_MODE[m], "checked": ai.reranker.mode == m}
|
||||||
|
for m in MODES]
|
||||||
|
items += [
|
||||||
|
{},
|
||||||
|
{"text": "系統忙碌時自動暫停 AI", "id": ID_AUTO_PAUSE, "checked": ai.auto_pause},
|
||||||
|
{"text": ai.status_text(), "id": ID_STATUS, "enabled": False},
|
||||||
|
{},
|
||||||
|
{"text": "清除學習紀錄", "id": ID_CLEAR},
|
||||||
|
]
|
||||||
|
return items
|
||||||
|
|
||||||
|
def onCommand(self, commandId, commandType):
|
||||||
|
ai = _shared_ai()
|
||||||
|
if commandId == ID_MODE_ICON and commandType == COMMAND_LEFT_CLICK and not self.engine.keys:
|
||||||
|
self.engine.handle(SHIFT_TAP) # left click: English mode on / off
|
||||||
|
self._update_mode_icon()
|
||||||
|
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
|
||||||
|
return
|
||||||
|
for mode, cid in ID_AI_MODE.items():
|
||||||
|
if commandId == cid:
|
||||||
|
ai.set_mode(mode)
|
||||||
|
self.showMessage("AI 選字:" + MODE_NAMES[mode], 1)
|
||||||
|
return
|
||||||
|
if commandId == ID_AUTO_PAUSE:
|
||||||
|
ai.set_auto_pause(not ai.auto_pause)
|
||||||
|
elif commandId == ID_CLEAR:
|
||||||
|
_shared_classifier().learned.clear()
|
||||||
|
_shared_classifier().history.clear()
|
||||||
|
self.showMessage("已清除學習紀錄", 1)
|
||||||
|
|
||||||
def onCompositionTerminated(self, forced):
|
def onCompositionTerminated(self, forced):
|
||||||
# The app ended the composition (focus change, mouse click): the shown text stays.
|
# The app ended the composition (focus change, mouse click): the shown text stays.
|
||||||
self.engine.reset()
|
self.engine.reset()
|
||||||
@@ -142,6 +236,7 @@ class AutoZhuyinTextService(TextService):
|
|||||||
return False
|
return False
|
||||||
self.engine.handle(SHIFT_TAP)
|
self.engine.handle(SHIFT_TAP)
|
||||||
self._update(None)
|
self._update(None)
|
||||||
|
self._update_mode_icon()
|
||||||
if not self.engine.keys:
|
if not self.engine.keys:
|
||||||
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
|
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
|
||||||
return True
|
return True
|
||||||
|
|||||||
@@ -0,0 +1,108 @@
|
|||||||
|
""""AI 選字" for the Windows and Linux shells: settings, background model loading,
|
||||||
|
system-load detection and the status line shown in the menu."""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
|
||||||
|
from . import lm
|
||||||
|
from .reranker import (MODES, NORMAL, OFF, ST_ACTIVE, ST_BUSY, ST_LOADING, ST_OFF, ST_SLOW,
|
||||||
|
STANDARD, Reranker)
|
||||||
|
from .system_monitor import REDUCED, SystemMonitor
|
||||||
|
|
||||||
|
MODE_NAMES = {OFF: "關閉", "saver": "省電", STANDARD: "標準", "quality": "高品質"}
|
||||||
|
|
||||||
|
|
||||||
|
class AI:
|
||||||
|
def __init__(self, data_dir, user_dir, monitor=None, background=True):
|
||||||
|
self.model_path = os.path.join(data_dir, "lm.bin")
|
||||||
|
self.vocab_path = os.path.join(data_dir, "lm-vocab.txt")
|
||||||
|
self.settings_path = os.path.join(user_dir, "settings.json")
|
||||||
|
self.monitor = monitor or SystemMonitor()
|
||||||
|
self.background = background
|
||||||
|
settings = self._load_settings()
|
||||||
|
mode = settings.get("ai_mode", STANDARD)
|
||||||
|
self.auto_pause = bool(settings.get("ai_auto_pause", True))
|
||||||
|
self.reranker = Reranker(mode if mode in MODES else STANDARD)
|
||||||
|
self.reranker.system_state = lambda: self.monitor.state() if self.auto_pause else NORMAL
|
||||||
|
self.error = None # why the model cannot be used, if it cannot
|
||||||
|
self._loading = False
|
||||||
|
self.load_model()
|
||||||
|
|
||||||
|
# --- settings ---
|
||||||
|
|
||||||
|
def _load_settings(self):
|
||||||
|
try:
|
||||||
|
with open(self.settings_path, encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
return data if isinstance(data, dict) else {}
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def _save_settings(self):
|
||||||
|
data = self._load_settings()
|
||||||
|
data.update(ai_mode=self.reranker.mode, ai_auto_pause=self.auto_pause)
|
||||||
|
try:
|
||||||
|
os.makedirs(os.path.dirname(self.settings_path), exist_ok=True)
|
||||||
|
with open(self.settings_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, ensure_ascii=False, indent=1)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def set_mode(self, mode):
|
||||||
|
if mode not in MODES:
|
||||||
|
return
|
||||||
|
self.reranker.mode = mode
|
||||||
|
if mode == OFF:
|
||||||
|
self.reranker.model = None # free the memory
|
||||||
|
else:
|
||||||
|
self.load_model()
|
||||||
|
self._save_settings()
|
||||||
|
|
||||||
|
def set_auto_pause(self, on):
|
||||||
|
self.auto_pause = bool(on)
|
||||||
|
self._save_settings()
|
||||||
|
|
||||||
|
# --- model ---
|
||||||
|
|
||||||
|
def load_model(self):
|
||||||
|
if self.reranker.mode == OFF or self.reranker.model is not None or self._loading or self.error:
|
||||||
|
return
|
||||||
|
if not lm.available():
|
||||||
|
self.error = "需要 numpy"
|
||||||
|
return
|
||||||
|
if not (os.path.exists(self.model_path) and os.path.exists(self.vocab_path)):
|
||||||
|
self.error = "AI 模型未安裝"
|
||||||
|
return
|
||||||
|
self._loading = True
|
||||||
|
|
||||||
|
def work():
|
||||||
|
try:
|
||||||
|
model = lm.GPT2LanguageModel(self.model_path, self.vocab_path)
|
||||||
|
except Exception as e: # noqa: BLE001 — any failure just disables AI
|
||||||
|
self.error = f"AI 模型載入失敗:{e}"
|
||||||
|
model = None
|
||||||
|
self._loading = False
|
||||||
|
if self.reranker.mode != OFF:
|
||||||
|
self.reranker.model = model
|
||||||
|
|
||||||
|
if self.background:
|
||||||
|
threading.Thread(target=work, daemon=True).start()
|
||||||
|
else:
|
||||||
|
work()
|
||||||
|
|
||||||
|
def status_text(self):
|
||||||
|
if self.error:
|
||||||
|
return self.error
|
||||||
|
self.reranker.effective_mode() # refresh the status
|
||||||
|
st = self.reranker.status
|
||||||
|
if st == ST_OFF:
|
||||||
|
return "AI 選字已關閉"
|
||||||
|
if st == ST_LOADING:
|
||||||
|
return "AI 模型載入中…"
|
||||||
|
if st == ST_BUSY:
|
||||||
|
return "AI 已暫停:系統忙碌或過熱"
|
||||||
|
if st == ST_SLOW:
|
||||||
|
return "AI 已暫停:運算太慢,一分鐘後再試"
|
||||||
|
if st == ST_ACTIVE and self.auto_pause and self.monitor.state() == REDUCED:
|
||||||
|
return "AI 使用中(省電模式)"
|
||||||
|
return "AI 使用中"
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
|
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
|
||||||
import json
|
import json
|
||||||
|
|
||||||
|
from .history import UserHistory
|
||||||
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
|
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
|
||||||
|
|
||||||
ZHUYIN = "zhuyin"
|
ZHUYIN = "zhuyin"
|
||||||
@@ -31,10 +32,11 @@ class LearnedPreferences:
|
|||||||
|
|
||||||
class Classifier:
|
class Classifier:
|
||||||
|
|
||||||
def __init__(self, lexicon, english, learned=None, weights=None):
|
def __init__(self, lexicon, english, learned=None, weights=None, history=None):
|
||||||
self.lexicon = lexicon
|
self.lexicon = lexicon
|
||||||
self.english = english
|
self.english = english
|
||||||
self.learned = learned or LearnedPreferences()
|
self.learned = learned or LearnedPreferences()
|
||||||
|
self.history = history or UserHistory() # picked candidates (shared by all engines)
|
||||||
self.weights = dict(DEFAULT_WEIGHTS)
|
self.weights = dict(DEFAULT_WEIGHTS)
|
||||||
if weights:
|
if weights:
|
||||||
self.weights.update(weights)
|
self.weights.update(weights)
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import time
|
|||||||
|
|
||||||
from . import bopomofo
|
from . import bopomofo
|
||||||
from .classifier import LITERAL, ZHUYIN
|
from .classifier import LITERAL, ZHUYIN
|
||||||
|
from .features import dot
|
||||||
from .lexicon import Converter
|
from .lexicon import Converter
|
||||||
|
|
||||||
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
|
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
|
||||||
@@ -20,8 +21,10 @@ FULLWIDTH = {
|
|||||||
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
||||||
}
|
}
|
||||||
|
|
||||||
# An English word that is also a common one-syllable Chinese reading ("up " = ㄧㄣ 因)
|
# The first word of a composition that also reads as Zhuyin is not committed as English
|
||||||
# is not committed on space; the next word decides ("up jo4" = 因為, "up the" = up the).
|
# on space; the next word decides ("up jo4" = 因為, "up the" = up the, "no su3" = no 你).
|
||||||
|
# Right after English text only common one-syllable readings are held (up = 因, url = 教),
|
||||||
|
# so normal English typing is not delayed.
|
||||||
HOLD_MIN_SCORE = -3.5
|
HOLD_MIN_SCORE = -3.5
|
||||||
|
|
||||||
# "`" then one of these keys always gives the full-width symbol, whatever the context.
|
# "`" then one of these keys always gives the full-width symbol, whatever the context.
|
||||||
@@ -43,7 +46,7 @@ class CompositionEngine:
|
|||||||
|
|
||||||
def __init__(self, classifier, now=time.monotonic):
|
def __init__(self, classifier, now=time.monotonic):
|
||||||
self.classifier = classifier
|
self.classifier = classifier
|
||||||
self.converter = Converter(classifier.lexicon)
|
self.converter = Converter(classifier.lexicon, classifier.history)
|
||||||
self.now = now
|
self.now = now
|
||||||
self.keys = []
|
self.keys = []
|
||||||
self.forced_mode = None
|
self.forced_mode = None
|
||||||
@@ -55,10 +58,15 @@ class CompositionEngine:
|
|||||||
self.candidates = []
|
self.candidates = []
|
||||||
self.candidate_index = 0
|
self.candidate_index = 0
|
||||||
self._candidate_segment_start = 0
|
self._candidate_segment_start = 0
|
||||||
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
|
# Composition cursor: a key index into self.keys (None = at the end). ← → move it
|
||||||
|
# between syllables; ↓ opens candidates for the phrase before it; typing there
|
||||||
|
# inserts a syllable (kept in self.inserting until its tone key, then spliced in).
|
||||||
self.cursor = None
|
self.cursor = None
|
||||||
|
self.inserting = []
|
||||||
# Index of a held space (see HOLD_MIN_SCORE), or None.
|
# Index of a held space (see HOLD_MIN_SCORE), or None.
|
||||||
self.held = None
|
self.held = None
|
||||||
|
self.reranker = None # "AI 選字" (reranker.Reranker); None = lexicon only
|
||||||
|
self._recent_text = "" # last characters committed: context for the reranker
|
||||||
self._previous_mode = None
|
self._previous_mode = None
|
||||||
self._previous_time = float("-inf")
|
self._previous_time = float("-inf")
|
||||||
self._cache_key = None
|
self._cache_key = None
|
||||||
@@ -91,19 +99,39 @@ class CompositionEngine:
|
|||||||
rest = keys[held + 1:]
|
rest = keys[held + 1:]
|
||||||
base = held + 1
|
base = held + 1
|
||||||
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
|
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
|
||||||
if rest_segs and rest_segs[0][2] == ZHUYIN:
|
if rest_segs and rest_segs[0][2] == ZHUYIN and self._held_reads_as_chinese(keys, held, rest_segs[0][1], finishing):
|
||||||
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
|
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
|
||||||
return [(0, base, LITERAL)] + rest_segs
|
return [(0, base, LITERAL)] + rest_segs
|
||||||
|
|
||||||
|
def _held_reads_as_chinese(self, keys, held, end, finishing):
|
||||||
|
"""Chinese all the way to `end` vs. the held English word followed by Chinese."""
|
||||||
|
c, w, ctx = self.classifier, self.classifier.weights, self.context
|
||||||
|
is_last = end == len(keys)
|
||||||
|
zh_whole = c.segment_features(keys[:end], is_last, False, finishing, ctx)[0]
|
||||||
|
word = c.segment_features(keys[:held], False, False, True, ctx)[1]
|
||||||
|
zh_rest = c.segment_features(keys[held + 1:end], is_last, True, finishing, None)[0]
|
||||||
|
if zh_whole is None:
|
||||||
|
return False
|
||||||
|
if word is None or zh_rest is None:
|
||||||
|
return True
|
||||||
|
return dot(w, zh_whole) >= dot(w, word) + dot(w, zh_rest)
|
||||||
|
|
||||||
def _is_ambiguous_word(self, keys):
|
def _is_ambiguous_word(self, keys):
|
||||||
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
|
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
|
||||||
if not all(k.isascii() and k.isalpha() for k in keys):
|
if not all(k.isascii() and k.isalpha() for k in keys):
|
||||||
return False
|
return False
|
||||||
p = bopomofo.parse(list(keys) + [" "])
|
p = bopomofo.parse(list(keys) + [" "])
|
||||||
if p.pending is not None or len(p.syllables) != 1:
|
if p.pending is not None or not p.syllables or p.stray_tones or p.foreign_keys:
|
||||||
return False
|
return False
|
||||||
entries = self.classifier.lexicon.entries([s.syllable for s in p.syllables])
|
best = []
|
||||||
return bool(entries) and entries[0][1] >= HOLD_MIN_SCORE
|
for ps in p.syllables:
|
||||||
|
entries = self.classifier.lexicon.entries([ps.syllable])
|
||||||
|
if not entries:
|
||||||
|
return False
|
||||||
|
best.append(entries[0][1])
|
||||||
|
if self.context != LITERAL:
|
||||||
|
return True # first word: hold anything that reads as Zhuyin
|
||||||
|
return len(best) == 1 and best[0] >= HOLD_MIN_SCORE
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def segments(self):
|
def segments(self):
|
||||||
@@ -133,8 +161,15 @@ class CompositionEngine:
|
|||||||
def _conversion(self, seg, final):
|
def _conversion(self, seg, final):
|
||||||
syl, pending = self._convertible(seg, final)
|
syl, pending = self._convertible(seg, final)
|
||||||
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
|
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
|
||||||
|
if self.reranker is not None:
|
||||||
|
chosen = self.reranker.choose(self.converter, syl, fx, self._rerank_context())
|
||||||
|
if chosen is not None:
|
||||||
|
return chosen, pending
|
||||||
return self.converter.convert(syl, fx), pending
|
return self.converter.convert(syl, fx), pending
|
||||||
|
|
||||||
|
def _rerank_context(self):
|
||||||
|
return self._recent_text if self.now() - self._previous_time < self.context_window else ""
|
||||||
|
|
||||||
def _render(self, seg, final):
|
def _render(self, seg, final):
|
||||||
start, end, mode = seg
|
start, end, mode = seg
|
||||||
if mode == LITERAL:
|
if mode == LITERAL:
|
||||||
@@ -144,48 +179,183 @@ class CompositionEngine:
|
|||||||
|
|
||||||
# --- cursor ---
|
# --- cursor ---
|
||||||
|
|
||||||
def _cursor_stops(self):
|
def _syllable_ends(self, seg):
|
||||||
"""[(segment index, syllables before the stop)] for every Zhuyin syllable."""
|
"""Absolute key index after each syllable of a Zhuyin segment."""
|
||||||
segs = self.segments
|
return [seg[0] + ps.end for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables]
|
||||||
out = []
|
|
||||||
for i, seg in enumerate(segs):
|
def _cursor_positions(self):
|
||||||
|
"""Key indices the cursor can stop at: after every Zhuyin syllable and at every
|
||||||
|
segment boundary, but never before the first Chinese character (there would be
|
||||||
|
nothing to pick candidates for)."""
|
||||||
|
out, ends = {len(self.keys)}, []
|
||||||
|
for seg in self.segments:
|
||||||
|
out.add(seg[1])
|
||||||
if seg[2] == ZHUYIN:
|
if seg[2] == ZHUYIN:
|
||||||
|
ends += self._syllable_ends(seg)
|
||||||
|
out.update(ends)
|
||||||
|
first = min(ends) if ends else len(self.keys)
|
||||||
|
return sorted(p for p in out if p >= first)
|
||||||
|
|
||||||
|
def _stop_at(self, pos):
|
||||||
|
"""(segment index, syllables before pos) for the Zhuyin segment pos is in."""
|
||||||
|
segs = self.segments
|
||||||
|
for i, seg in enumerate(segs):
|
||||||
|
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
|
||||||
n = len(self._convertible(seg, i < len(segs) - 1)[0])
|
n = len(self._convertible(seg, i < len(segs) - 1)[0])
|
||||||
out.extend((i, k) for k in range(1, n + 1))
|
local = sum(1 for e in self._syllable_ends(seg) if e <= pos)
|
||||||
return out
|
return (i, min(local, n)) if local else None
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _candidate_stop(self):
|
||||||
|
"""Where ↓ opens candidates: before the cursor, or the last syllable."""
|
||||||
|
if self.cursor is not None:
|
||||||
|
return self._stop_at(self.cursor)
|
||||||
|
segs = self.segments
|
||||||
|
for i in range(len(segs) - 1, -1, -1):
|
||||||
|
if segs[i][2] == ZHUYIN:
|
||||||
|
n = len(self._convertible(segs[i], i < len(segs) - 1)[0])
|
||||||
|
if n:
|
||||||
|
return i, n
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _base_cursor_offset(self):
|
||||||
|
"""Cursor position in the rendered segments (without the syllable being inserted)."""
|
||||||
|
pieces = self._segment_pieces()
|
||||||
|
if self.cursor is None:
|
||||||
|
return sum(len(t) for t, _ in pieces)
|
||||||
|
segs = self.segments
|
||||||
|
for i, seg in enumerate(segs):
|
||||||
|
if not seg[0] < self.cursor <= seg[1]:
|
||||||
|
continue
|
||||||
|
offset = sum(len(t) for t, _ in pieces[:i])
|
||||||
|
if seg[2] == LITERAL:
|
||||||
|
return offset + self.cursor - seg[0]
|
||||||
|
stop = self._stop_at(self.cursor)
|
||||||
|
local = stop[1] if stop else 0
|
||||||
|
for (a, b), text in self._conversion(seg, i < len(segs) - 1)[0]:
|
||||||
|
if b <= local:
|
||||||
|
offset += len(text)
|
||||||
|
elif a < local:
|
||||||
|
offset += min(local - a, len(text))
|
||||||
|
return offset
|
||||||
|
return 0
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def cursor_offset(self):
|
def cursor_offset(self):
|
||||||
"""Cursor position in marked_text (characters)."""
|
"""Cursor position in marked_text (characters)."""
|
||||||
pieces = self.marked_pieces
|
return self._base_cursor_offset() + len(self._inserting_text())
|
||||||
stops = self._cursor_stops()
|
|
||||||
if self.cursor is None or not 0 <= self.cursor < len(stops):
|
|
||||||
return sum(len(t) for t, _ in pieces)
|
|
||||||
seg_index, local = stops[self.cursor]
|
|
||||||
offset = sum(len(t) for t, _ in pieces[:seg_index])
|
|
||||||
segs = self.segments
|
|
||||||
for (a, b), text in self._conversion(segs[seg_index], seg_index < len(segs) - 1)[0]:
|
|
||||||
if b <= local:
|
|
||||||
offset += len(text)
|
|
||||||
elif a < local:
|
|
||||||
offset += min(local - a, len(text))
|
|
||||||
return offset
|
|
||||||
|
|
||||||
def _move_cursor(self, delta):
|
def _move_cursor(self, delta):
|
||||||
stops = self._cursor_stops()
|
self.inserting = []
|
||||||
if not stops:
|
stops = self._cursor_positions()
|
||||||
return
|
here = len(self.keys) if self.cursor is None else self.cursor
|
||||||
nxt = (len(stops) - 1 if self.cursor is None else self.cursor) + delta
|
if delta < 0:
|
||||||
self.cursor = None if nxt >= len(stops) - 1 else max(0, nxt)
|
before = [p for p in stops if p < here]
|
||||||
|
nxt = before[-1] if before else here
|
||||||
|
else:
|
||||||
|
after = [p for p in stops if p > here]
|
||||||
|
nxt = after[0] if after else here
|
||||||
|
self.cursor = None if nxt >= len(self.keys) else nxt
|
||||||
|
|
||||||
|
def _inserting_text(self):
|
||||||
|
if not self.inserting:
|
||||||
|
return ""
|
||||||
|
return "".join(ps.syllable.reading for ps in bopomofo.parse(self.inserting).syllables)
|
||||||
|
|
||||||
|
# --- editing in the middle ---
|
||||||
|
|
||||||
|
def _trim_fixed(self, pos):
|
||||||
|
"""Keeps picked phrases that end at or before key index pos (they don't move)."""
|
||||||
|
kept = {}
|
||||||
|
for seg in self.segments:
|
||||||
|
picks = self.fixed.get(seg[0])
|
||||||
|
if not picks or seg[2] != ZHUYIN:
|
||||||
|
continue
|
||||||
|
if seg[1] <= pos:
|
||||||
|
kept[seg[0]] = picks
|
||||||
|
elif seg[0] < pos:
|
||||||
|
ends = self._syllable_ends(seg)
|
||||||
|
here = {r: t for r, t in picks.items() if r[1] <= len(ends) and ends[r[1] - 1] <= pos}
|
||||||
|
if here:
|
||||||
|
kept[seg[0]] = here
|
||||||
|
self.fixed = kept
|
||||||
|
|
||||||
|
def _splice(self, new_keys):
|
||||||
|
"""Inserts keys at the cursor and moves the cursor after them."""
|
||||||
|
pos = self.cursor
|
||||||
|
self._trim_fixed(pos)
|
||||||
|
self.keys[pos:pos] = new_keys
|
||||||
|
if self.held is not None and self.held >= pos:
|
||||||
|
self.held += len(new_keys)
|
||||||
|
self.cursor = pos + len(new_keys)
|
||||||
|
if self.cursor >= len(self.keys):
|
||||||
|
self.cursor = None
|
||||||
|
|
||||||
|
def _delete_before_cursor(self):
|
||||||
|
"""Backspace in the middle: removes the syllable (or literal key) before the cursor."""
|
||||||
|
pos = self.cursor
|
||||||
|
start = pos - 1
|
||||||
|
for seg in self.segments:
|
||||||
|
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
|
||||||
|
for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables:
|
||||||
|
if seg[0] + ps.end == pos:
|
||||||
|
start = seg[0] + ps.start
|
||||||
|
self._trim_fixed(start)
|
||||||
|
del self.keys[start:pos]
|
||||||
|
if self.held is not None:
|
||||||
|
if start <= self.held < pos:
|
||||||
|
self.held = None
|
||||||
|
elif self.held >= pos:
|
||||||
|
self.held -= pos - start
|
||||||
|
self.cursor = start
|
||||||
|
if not self.keys:
|
||||||
|
self.reset()
|
||||||
|
|
||||||
|
def _insert_key(self, c):
|
||||||
|
"""Typing with the cursor in the middle: compose one syllable, splice it in on its tone."""
|
||||||
|
if c == " ":
|
||||||
|
if not self.inserting:
|
||||||
|
return True, self._commit()
|
||||||
|
self._splice(self.inserting + [" "])
|
||||||
|
self.inserting = []
|
||||||
|
return CONSUMED
|
||||||
|
self.inserting.append(c)
|
||||||
|
p = bopomofo.parse(self.inserting)
|
||||||
|
if p.stray_tones or p.foreign_keys:
|
||||||
|
self.inserting.pop() # a tone with nothing to attach to
|
||||||
|
return CONSUMED
|
||||||
|
if p.syllables and p.syllables[-1].toned and len(p.syllables) == 1:
|
||||||
|
self._splice(self.inserting)
|
||||||
|
self.inserting = []
|
||||||
|
elif len(p.syllables) > 1:
|
||||||
|
# A new syllable started without a tone: splice the first one as it is.
|
||||||
|
first = p.syllables[0].end
|
||||||
|
self._splice(self.inserting[:first])
|
||||||
|
self.inserting = self.inserting[first:]
|
||||||
|
return CONSUMED
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def marked_pieces(self):
|
def marked_pieces(self):
|
||||||
segs = self.segments
|
pieces = self._segment_pieces()
|
||||||
pieces = [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
|
if self.inserting:
|
||||||
|
# Show the syllable being inserted at the cursor.
|
||||||
|
at, out, done = self._base_cursor_offset(), [], False
|
||||||
|
for text, mode in pieces:
|
||||||
|
if not done and at <= len(text):
|
||||||
|
out += [(text[:at], mode), (self._inserting_text(), ZHUYIN), (text[at:], mode)]
|
||||||
|
done = True
|
||||||
|
else:
|
||||||
|
out.append((text, mode))
|
||||||
|
at -= len(text)
|
||||||
|
pieces = [(t, m) for t, m in out if t]
|
||||||
if self.backtick:
|
if self.backtick:
|
||||||
pieces.append((BACKTICK, LITERAL))
|
pieces.append((BACKTICK, LITERAL))
|
||||||
return pieces
|
return pieces
|
||||||
|
|
||||||
|
def _segment_pieces(self):
|
||||||
|
segs = self.segments
|
||||||
|
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def marked_text(self):
|
def marked_text(self):
|
||||||
return "".join(t for t, _ in self.marked_pieces)
|
return "".join(t for t, _ in self.marked_pieces)
|
||||||
@@ -198,12 +368,14 @@ class CompositionEngine:
|
|||||||
|
|
||||||
def wants(self, key):
|
def wants(self, key):
|
||||||
"""Whether handle(key) would consume the key, without changing state."""
|
"""Whether handle(key) would consume the key, without changing state."""
|
||||||
if self.candidates or key == SHIFT_TAP or self.backtick:
|
if self.candidates or key == SHIFT_TAP or self.backtick or self.inserting:
|
||||||
return True
|
return True
|
||||||
if self.english_mode and not self.keys:
|
if self.english_mode and not self.keys:
|
||||||
return False # English mode: everything goes to the app as typed
|
return False # English mode: everything goes to the app as typed
|
||||||
if key in SPECIAL_KEYS:
|
if key in SPECIAL_KEYS:
|
||||||
return bool(self.keys)
|
return bool(self.keys)
|
||||||
|
if key == BACKTICK:
|
||||||
|
return True
|
||||||
if self.keys:
|
if self.keys:
|
||||||
return True
|
return True
|
||||||
if key == " ":
|
if key == " ":
|
||||||
@@ -217,10 +389,19 @@ class CompositionEngine:
|
|||||||
return self._handle_candidate_key(key)
|
return self._handle_candidate_key(key)
|
||||||
if self.backtick:
|
if self.backtick:
|
||||||
return self._after_backtick(key)
|
return self._after_backtick(key)
|
||||||
|
if self.inserting and key in (BACKSPACE, ESCAPE):
|
||||||
|
if key == BACKSPACE:
|
||||||
|
self.inserting.pop()
|
||||||
|
else:
|
||||||
|
self.inserting = []
|
||||||
|
return CONSUMED
|
||||||
if key == BACKSPACE:
|
if key == BACKSPACE:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
self.cursor = None
|
if self.cursor is not None:
|
||||||
|
if self.cursor > 0:
|
||||||
|
self._delete_before_cursor()
|
||||||
|
return CONSUMED
|
||||||
self.keys.pop()
|
self.keys.pop()
|
||||||
if self.held is not None and len(self.keys) <= self.held:
|
if self.held is not None and len(self.keys) <= self.held:
|
||||||
self.held = None
|
self.held = None
|
||||||
@@ -232,6 +413,9 @@ class CompositionEngine:
|
|||||||
if key == ENTER:
|
if key == ENTER:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
|
if self.inserting:
|
||||||
|
self._splice(self.inserting)
|
||||||
|
self.inserting = []
|
||||||
return True, self._commit()
|
return True, self._commit()
|
||||||
if key == ESCAPE:
|
if key == ESCAPE:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
@@ -255,7 +439,8 @@ class CompositionEngine:
|
|||||||
if key == DOWN:
|
if key == DOWN:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
self._open_candidates()
|
if not self.inserting:
|
||||||
|
self._open_candidates()
|
||||||
return CONSUMED
|
return CONSUMED
|
||||||
if key in (LEFT, RIGHT):
|
if key in (LEFT, RIGHT):
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
@@ -286,7 +471,12 @@ class CompositionEngine:
|
|||||||
return bopomofo.parse(self.keys[start:end])
|
return bopomofo.parse(self.keys[start:end])
|
||||||
|
|
||||||
def _handle_character(self, c):
|
def _handle_character(self, c):
|
||||||
self.cursor = None # typing always continues at the end
|
if self.cursor is not None and self.keys and (c == " " or bopomofo.is_zhuyin_key(c)):
|
||||||
|
return self._insert_key(c)
|
||||||
|
if self.inserting:
|
||||||
|
self._splice(self.inserting)
|
||||||
|
self.inserting = []
|
||||||
|
self.cursor = None # anything else continues at the end
|
||||||
if self.english_mode and not self.keys:
|
if self.english_mode and not self.keys:
|
||||||
return PASS # English mode: every key goes to the app as typed
|
return PASS # English mode: every key goes to the app as typed
|
||||||
if c == BACKTICK:
|
if c == BACKTICK:
|
||||||
@@ -330,10 +520,10 @@ class CompositionEngine:
|
|||||||
|
|
||||||
def _open_candidates(self):
|
def _open_candidates(self):
|
||||||
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
|
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
|
||||||
stops = self._cursor_stops()
|
stop = self._candidate_stop()
|
||||||
if not stops:
|
if stop is None:
|
||||||
return
|
return
|
||||||
seg_index, local = stops[min(len(stops) - 1 if self.cursor is None else self.cursor, len(stops) - 1)]
|
seg_index, local = stop
|
||||||
segs = self.segments
|
segs = self.segments
|
||||||
seg = segs[seg_index]
|
seg = segs[seg_index]
|
||||||
syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
|
syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
|
||||||
@@ -374,6 +564,8 @@ class CompositionEngine:
|
|||||||
def _commit(self, segs=None):
|
def _commit(self, segs=None):
|
||||||
segs = segs if segs is not None else self.segments
|
segs = segs if segs is not None else self.segments
|
||||||
text = "".join(self._render(s, True) for s in segs)
|
text = "".join(self._render(s, True) for s in segs)
|
||||||
|
self._remember_picks(segs)
|
||||||
|
self._recent_text = (self._rerank_context() + text)[-20:]
|
||||||
if self.forced_mode:
|
if self.forced_mode:
|
||||||
self.classifier.learned.record("".join(self.keys), self.forced_mode)
|
self.classifier.learned.record("".join(self.keys), self.forced_mode)
|
||||||
self._previous_mode = segs[-1][2] if segs else None
|
self._previous_mode = segs[-1][2] if segs else None
|
||||||
@@ -381,11 +573,23 @@ class CompositionEngine:
|
|||||||
self.reset()
|
self.reset()
|
||||||
return text
|
return text
|
||||||
|
|
||||||
|
def _remember_picks(self, segs):
|
||||||
|
"""Candidates the user picked are remembered and come first next time."""
|
||||||
|
for i, seg in enumerate(segs):
|
||||||
|
picks = self.fixed.get(seg[0])
|
||||||
|
if seg[2] != ZHUYIN or not picks:
|
||||||
|
continue
|
||||||
|
syl = self._convertible(seg, True)[0]
|
||||||
|
for (a, b), text in picks.items():
|
||||||
|
if b <= len(syl):
|
||||||
|
self.classifier.history.record(syl[a:b], text)
|
||||||
|
|
||||||
def commit_all(self):
|
def commit_all(self):
|
||||||
return self._commit() if self.keys else ""
|
return self._commit() if self.keys else ""
|
||||||
|
|
||||||
def reset(self):
|
def reset(self):
|
||||||
self.keys = []
|
self.keys = []
|
||||||
|
self.inserting = []
|
||||||
self.held = None
|
self.held = None
|
||||||
self.backtick = False
|
self.backtick = False
|
||||||
self.cursor = None
|
self.cursor = None
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
"""Remembers the candidates the user picked. Port of UserHistory.swift.
|
||||||
|
|
||||||
|
Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
|
||||||
|
where the lexicon has it for the reading being converted, so a toneless key is safe.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class UserHistory:
|
||||||
|
max_entries = 2000 # (reading, phrase) pairs kept; the least recently used go first
|
||||||
|
max_count = 10
|
||||||
|
|
||||||
|
def __init__(self, data=None, on_change=None):
|
||||||
|
# reading -> {phrase: [count, last use]}
|
||||||
|
self.data = {}
|
||||||
|
self.on_change = on_change or (lambda data: None)
|
||||||
|
self._clock = 0
|
||||||
|
for reading, phrases in (data or {}).items():
|
||||||
|
if not isinstance(phrases, dict):
|
||||||
|
continue
|
||||||
|
for phrase, value in phrases.items():
|
||||||
|
try:
|
||||||
|
count, used = int(value[0]), int(value[1])
|
||||||
|
except (TypeError, ValueError, IndexError):
|
||||||
|
continue
|
||||||
|
self.data.setdefault(reading, {})[phrase] = [count, used]
|
||||||
|
self._clock = max(self._clock, used)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def key(syllables):
|
||||||
|
return "-".join(s.toneless for s in syllables)
|
||||||
|
|
||||||
|
def record(self, syllables, phrase):
|
||||||
|
if not syllables or not phrase:
|
||||||
|
return
|
||||||
|
self._clock += 1
|
||||||
|
entry = self.data.setdefault(self.key(syllables), {}).setdefault(phrase, [0, 0])
|
||||||
|
entry[0] = min(self.max_count, entry[0] + 1)
|
||||||
|
entry[1] = self._clock
|
||||||
|
self._evict()
|
||||||
|
self.on_change(self.data)
|
||||||
|
|
||||||
|
def _evict(self):
|
||||||
|
pairs = [(v[1], r, p) for r, ps in self.data.items() for p, v in ps.items()]
|
||||||
|
if len(pairs) <= self.max_entries:
|
||||||
|
return
|
||||||
|
pairs.sort()
|
||||||
|
for _, r, p in pairs[:len(pairs) - self.max_entries]:
|
||||||
|
del self.data[r][p]
|
||||||
|
if not self.data[r]:
|
||||||
|
del self.data[r]
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
self.data = {}
|
||||||
|
self.on_change(self.data)
|
||||||
|
|
||||||
|
def adjust(self, syllables, entries):
|
||||||
|
"""Lexicon entries [(phrase, score)] with picked phrases moved to the top: just above
|
||||||
|
the best entry, more so the more often they were picked. Returns a new list."""
|
||||||
|
picked = self.data.get(self.key(syllables))
|
||||||
|
if not picked or not entries:
|
||||||
|
return entries
|
||||||
|
top = entries[0][1]
|
||||||
|
out = [(p, max(s, top + 0.01 + 0.1 * picked[p][0]) if p in picked else s) for p, s in entries]
|
||||||
|
out.sort(key=lambda e: -e[1])
|
||||||
|
return out
|
||||||
@@ -84,8 +84,13 @@ TONELESS_PENALTY = -1.5
|
|||||||
|
|
||||||
|
|
||||||
class Converter:
|
class Converter:
|
||||||
def __init__(self, lexicon):
|
def __init__(self, lexicon, history=None):
|
||||||
self.lexicon = lexicon
|
self.lexicon = lexicon
|
||||||
|
self.history = history # UserHistory: picked candidates come first
|
||||||
|
|
||||||
|
def _entries(self, syllables):
|
||||||
|
entries = self.lexicon.entries(syllables)
|
||||||
|
return self.history.adjust(syllables, entries) if self.history else entries
|
||||||
|
|
||||||
def _options(self, syllables, i, j, fixed):
|
def _options(self, syllables, i, j, fixed):
|
||||||
if (i, j) in fixed:
|
if (i, j) in fixed:
|
||||||
@@ -94,7 +99,7 @@ class Converter:
|
|||||||
return []
|
return []
|
||||||
sl = syllables[i:j]
|
sl = syllables[i:j]
|
||||||
penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0
|
penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0
|
||||||
out = [(p, s + penalty) for p, s in self.lexicon.entries(sl)]
|
out = [(p, s + penalty) for p, s in self._entries(sl)]
|
||||||
if not out and j - i == 1:
|
if not out and j - i == 1:
|
||||||
out = [(sl[0].reading, UNKNOWN_SCORE)]
|
out = [(sl[0].reading, UNKNOWN_SCORE)]
|
||||||
return out
|
return out
|
||||||
@@ -125,6 +130,28 @@ class Converter:
|
|||||||
j = seg[0][0]
|
j = seg[0][0]
|
||||||
return segs[::-1]
|
return segs[::-1]
|
||||||
|
|
||||||
|
def kbest(self, syllables, k, fixed=None):
|
||||||
|
"""The k best paths [(log10 score, [((start, end), text)])], best first."""
|
||||||
|
fixed = fixed or {}
|
||||||
|
n = len(syllables)
|
||||||
|
if n == 0 or k <= 0:
|
||||||
|
return []
|
||||||
|
max_len = max(1, self.lexicon.max_phrase_length)
|
||||||
|
beams = [[] for _ in range(n + 1)]
|
||||||
|
beams[0] = [(0.0, [])]
|
||||||
|
for j in range(1, n + 1):
|
||||||
|
cand = []
|
||||||
|
for length in range(1, min(max_len, j) + 1):
|
||||||
|
i = j - length
|
||||||
|
if not beams[i]:
|
||||||
|
continue
|
||||||
|
for text, score in self._options(syllables, i, j, fixed):
|
||||||
|
for s, segs in beams[i]:
|
||||||
|
cand.append((s + score, segs + [((i, j), text)]))
|
||||||
|
cand.sort(key=lambda e: -e[0])
|
||||||
|
beams[j] = cand[:k]
|
||||||
|
return beams[n]
|
||||||
|
|
||||||
def candidates(self, syllables, end):
|
def candidates(self, syllables, end):
|
||||||
"""[((start, end), text)] for phrases ending at syllable `end`, longest first."""
|
"""[((start, end), text)] for phrases ending at syllable `end`, longest first."""
|
||||||
if end <= 0 or end > len(syllables):
|
if end <= 0 or end > len(syllables):
|
||||||
@@ -132,7 +159,7 @@ class Converter:
|
|||||||
out, seen = [], set()
|
out, seen = [], set()
|
||||||
for length in range(min(self.lexicon.max_phrase_length, end), 0, -1):
|
for length in range(min(self.lexicon.max_phrase_length, end), 0, -1):
|
||||||
i = end - length
|
i = end - length
|
||||||
for phrase, _ in self.lexicon.entries(syllables[i:end]):
|
for phrase, _ in self._entries(syllables[i:end]):
|
||||||
if phrase not in seen:
|
if phrase not in seen:
|
||||||
seen.add(phrase)
|
seen.add(phrase)
|
||||||
out.append(((i, end), phrase))
|
out.append(((i, end), phrase))
|
||||||
|
|||||||
@@ -0,0 +1,109 @@
|
|||||||
|
"""GPT-2 language model for "AI 選字" (numpy). Port of LanguageModel.swift.
|
||||||
|
|
||||||
|
Reads the "AZLM" file made by training/lm/export_lm.py (CKIP gpt2-tiny-chinese, GPL-3.0,
|
||||||
|
float16 weights, expanded to float32 in memory). numpy is optional: when it cannot be
|
||||||
|
imported, `available()` is False and the input method works without AI.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import struct
|
||||||
|
|
||||||
|
try:
|
||||||
|
import numpy as np
|
||||||
|
except Exception: # ImportError, or a numpy build for another Python / architecture
|
||||||
|
np = None
|
||||||
|
|
||||||
|
CLS, UNK = 101, 100
|
||||||
|
|
||||||
|
|
||||||
|
def available():
|
||||||
|
return np is not None
|
||||||
|
|
||||||
|
|
||||||
|
class GPT2LanguageModel:
|
||||||
|
max_tokens = 64
|
||||||
|
|
||||||
|
def __init__(self, model_path, vocab_path):
|
||||||
|
if np is None:
|
||||||
|
raise RuntimeError("numpy is not available")
|
||||||
|
with open(model_path, "rb") as f:
|
||||||
|
raw = f.read()
|
||||||
|
if raw[:4] != b"AZLM":
|
||||||
|
raise ValueError("not an AZLM file")
|
||||||
|
n = struct.unpack("<I", raw[4:8])[0]
|
||||||
|
header = json.loads(raw[8:8 + n])
|
||||||
|
data = np.frombuffer(raw, dtype="<f2", offset=8 + n).astype(np.float32)
|
||||||
|
self.n_layer, self.n_head = header["n_layer"], header["n_head"]
|
||||||
|
self.d, self.eps = header["n_embd"], header["eps"]
|
||||||
|
self.w = {}
|
||||||
|
for name, shape, offset in header["tensors"]:
|
||||||
|
size = int(np.prod(shape))
|
||||||
|
self.w[name] = data[offset:offset + size].reshape(shape)
|
||||||
|
self.vocab = {}
|
||||||
|
with open(vocab_path, encoding="utf-8") as f:
|
||||||
|
for i, line in enumerate(f):
|
||||||
|
token = line.rstrip("\n")
|
||||||
|
if len(token) == 1:
|
||||||
|
self.vocab.setdefault(token, i)
|
||||||
|
|
||||||
|
def ids(self, text):
|
||||||
|
return [self.vocab.get(c, UNK) for c in text]
|
||||||
|
|
||||||
|
def _ln(self, x, name):
|
||||||
|
mu = x.mean(-1, keepdims=True)
|
||||||
|
var = ((x - mu) ** 2).mean(-1, keepdims=True)
|
||||||
|
return (x - mu) / np.sqrt(var + self.eps) * self.w[name + ".weight"] + self.w[name + ".bias"]
|
||||||
|
|
||||||
|
def _hidden(self, ids):
|
||||||
|
"""ids [B, T] → final hidden states [B, T, d]."""
|
||||||
|
w = self.w
|
||||||
|
B, T = ids.shape
|
||||||
|
x = w["wte.weight"][ids] + w["wpe.weight"][:T]
|
||||||
|
mask = np.triu(np.full((T, T), -1e9, dtype=np.float32), 1)
|
||||||
|
hd = self.d // self.n_head
|
||||||
|
for l in range(self.n_layer):
|
||||||
|
p = f"h.{l}."
|
||||||
|
h = self._ln(x, p + "ln_1")
|
||||||
|
qkv = h @ w[p + "attn.c_attn.weight"] + w[p + "attn.c_attn.bias"]
|
||||||
|
q, k, v = np.split(qkv, 3, axis=-1)
|
||||||
|
q = q.reshape(B, T, self.n_head, hd).transpose(0, 2, 1, 3)
|
||||||
|
k = k.reshape(B, T, self.n_head, hd).transpose(0, 2, 3, 1)
|
||||||
|
v = v.reshape(B, T, self.n_head, hd).transpose(0, 2, 1, 3)
|
||||||
|
att = q @ k / math.sqrt(hd) + mask
|
||||||
|
att = np.exp(att - att.max(-1, keepdims=True))
|
||||||
|
att /= att.sum(-1, keepdims=True)
|
||||||
|
a = (att @ v).transpose(0, 2, 1, 3).reshape(B, T, self.d)
|
||||||
|
x = x + a @ w[p + "attn.c_proj.weight"] + w[p + "attn.c_proj.bias"]
|
||||||
|
h = self._ln(x, p + "ln_2")
|
||||||
|
h = h @ w[p + "mlp.c_fc.weight"] + w[p + "mlp.c_fc.bias"]
|
||||||
|
h = 0.5 * h * (1 + np.tanh(math.sqrt(2 / math.pi) * (h + 0.044715 * h ** 3))) # gelu_new
|
||||||
|
x = x + h @ w[p + "mlp.c_proj.weight"] + w[p + "mlp.c_proj.bias"]
|
||||||
|
return self._ln(x, "ln_f")
|
||||||
|
|
||||||
|
def log_probs(self, texts, context=""):
|
||||||
|
"""Natural-log probabilities of several texts after the same context. Texts of the
|
||||||
|
same length are scored together in one batch."""
|
||||||
|
out = [0.0] * len(texts)
|
||||||
|
by_len = {}
|
||||||
|
for i, t in enumerate(texts):
|
||||||
|
if t:
|
||||||
|
by_len.setdefault(len(t), []).append(i)
|
||||||
|
for length, idx in by_len.items():
|
||||||
|
ctx = self.ids(context)
|
||||||
|
if len(ctx) + length + 1 > self.max_tokens:
|
||||||
|
ctx = ctx[max(0, len(ctx) - (self.max_tokens - length - 1)):]
|
||||||
|
targets = np.array([self.ids(texts[i]) for i in idx]) # [B, L]
|
||||||
|
prefix = np.array([CLS] + ctx)
|
||||||
|
inputs = np.concatenate([np.tile(prefix, (len(idx), 1)), targets[:, :-1]], axis=1)
|
||||||
|
h = self._hidden(inputs)[:, len(ctx):, :] # rows predicting targets
|
||||||
|
logits = h @ self.w["wte.weight"].T # [B, L, V]
|
||||||
|
m = logits.max(-1, keepdims=True)
|
||||||
|
lse = np.log(np.exp(logits - m).sum(-1)) + m[..., 0]
|
||||||
|
picked = np.take_along_axis(logits, targets[..., None], -1)[..., 0]
|
||||||
|
scores = (picked - lse).sum(-1)
|
||||||
|
for j, i in enumerate(idx):
|
||||||
|
out[i] = float(scores[j])
|
||||||
|
return out
|
||||||
|
|
||||||
|
def log_prob(self, text, context=""):
|
||||||
|
return self.log_probs([text], context)[0]
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
""""AI 選字": reranks the converter's best paths with a language model. Port of Reranker.swift.
|
||||||
|
|
||||||
|
score = lexicon score (log10 → ln) + λ · LM log-probability
|
||||||
|
|
||||||
|
Typing is never blocked for long: a rerank that runs over its time budget counts as an
|
||||||
|
overrun, and three in a row pause AI for a minute. `system_state` lets the shell pause
|
||||||
|
or reduce it when the machine is busy, hot or saving power.
|
||||||
|
"""
|
||||||
|
import math
|
||||||
|
import time
|
||||||
|
|
||||||
|
OFF, SAVER, STANDARD, QUALITY = "off", "saver", "standard", "quality"
|
||||||
|
MODES = (OFF, SAVER, STANDARD, QUALITY)
|
||||||
|
# Paths reranked, and the time budget (seconds) of one rerank. numpy is slower than
|
||||||
|
# the Mac's Accelerate code, so the budgets are larger than in Reranker.swift.
|
||||||
|
K = {SAVER: 3, STANDARD: 10, QUALITY: 30}
|
||||||
|
BUDGET = {SAVER: 0.04, STANDARD: 0.08, QUALITY: 0.2}
|
||||||
|
|
||||||
|
NORMAL, REDUCED, PAUSED = "normal", "reduced", "paused" # system states
|
||||||
|
ST_OFF, ST_LOADING, ST_ACTIVE, ST_BUSY, ST_SLOW = "off", "loading", "active", "paused-busy", "paused-slow"
|
||||||
|
|
||||||
|
PAUSE_AFTER_OVERRUNS = 3
|
||||||
|
SLOW_PAUSE = 60.0
|
||||||
|
CACHE_LIMIT = 512
|
||||||
|
|
||||||
|
|
||||||
|
class Reranker:
|
||||||
|
def __init__(self, mode=STANDARD, model=None, now=time.monotonic):
|
||||||
|
self._mode = mode
|
||||||
|
self._model = model
|
||||||
|
self.lam = 0.6
|
||||||
|
self.system_state = lambda: NORMAL
|
||||||
|
self.now = now
|
||||||
|
self.status = ST_OFF
|
||||||
|
self._overruns = 0
|
||||||
|
self._paused_until = float("-inf")
|
||||||
|
self._cache = {}
|
||||||
|
|
||||||
|
@property
|
||||||
|
def mode(self):
|
||||||
|
return self._mode
|
||||||
|
|
||||||
|
@mode.setter
|
||||||
|
def mode(self, value):
|
||||||
|
self._mode = value
|
||||||
|
self._cache.clear()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def model(self):
|
||||||
|
return self._model
|
||||||
|
|
||||||
|
@model.setter
|
||||||
|
def model(self, value):
|
||||||
|
self._model = value
|
||||||
|
self._cache.clear()
|
||||||
|
|
||||||
|
def effective_mode(self):
|
||||||
|
"""The mode used now (after the system state), or None when AI is not used."""
|
||||||
|
if self._mode == OFF:
|
||||||
|
self.status = ST_OFF
|
||||||
|
return None
|
||||||
|
if self._model is None:
|
||||||
|
self.status = ST_LOADING
|
||||||
|
return None
|
||||||
|
if self.now() < self._paused_until:
|
||||||
|
self.status = ST_SLOW
|
||||||
|
return None
|
||||||
|
state = self.system_state()
|
||||||
|
if state == PAUSED:
|
||||||
|
self.status = ST_BUSY
|
||||||
|
return None
|
||||||
|
self.status = ST_ACTIVE
|
||||||
|
return SAVER if state == REDUCED else self._mode
|
||||||
|
|
||||||
|
def choose(self, converter, syllables, fixed, context):
|
||||||
|
"""The reranked conversion [((start, end), text)], or None for the plain result."""
|
||||||
|
mode = self.effective_mode()
|
||||||
|
if mode is None or not syllables:
|
||||||
|
return None
|
||||||
|
key = (mode, context, tuple(s.reading for s in syllables), tuple(sorted(fixed.items())))
|
||||||
|
if key in self._cache:
|
||||||
|
return self._cache[key]
|
||||||
|
paths = converter.kbest(syllables, K[mode], fixed)
|
||||||
|
if len(paths) <= 1:
|
||||||
|
return paths[0][1] if paths else None
|
||||||
|
start = self.now()
|
||||||
|
lm = self._model.log_probs(["".join(t for _, t in segs) for _, segs in paths], context)
|
||||||
|
best = max(range(len(paths)), key=lambda i: paths[i][0] * math.log(10) + self.lam * lm[i])
|
||||||
|
if self.now() - start > BUDGET[mode]:
|
||||||
|
# Too slow: use this result (it is computed) but don't cache it, and pause AI
|
||||||
|
# after a few overruns in a row.
|
||||||
|
self._overruns += 1
|
||||||
|
if self._overruns >= PAUSE_AFTER_OVERRUNS:
|
||||||
|
self._paused_until = self.now() + SLOW_PAUSE
|
||||||
|
self._overruns = 0
|
||||||
|
self.status = ST_SLOW
|
||||||
|
return paths[best][1]
|
||||||
|
self._overruns = 0
|
||||||
|
if len(self._cache) >= CACHE_LIMIT:
|
||||||
|
self._cache.clear()
|
||||||
|
self._cache[key] = paths[best][1]
|
||||||
|
return paths[best][1]
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
"""Decides when "AI 選字" should step back (Windows and Linux; the Mac app has its own).
|
||||||
|
|
||||||
|
paused — CPU busy (over 90%), or the machine is very hot (Linux thermal zones ≥ 95 °C)
|
||||||
|
reduced — power saving (Windows battery saver; Linux low-power profile or battery ≤ 20%)
|
||||||
|
normal — otherwise. Checked at most every 2 seconds.
|
||||||
|
"""
|
||||||
|
import glob
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
from .reranker import NORMAL, PAUSED, REDUCED
|
||||||
|
|
||||||
|
|
||||||
|
class SystemMonitor:
|
||||||
|
interval = 2.0
|
||||||
|
busy_load = 0.9
|
||||||
|
|
||||||
|
def __init__(self, now=time.monotonic):
|
||||||
|
self.now = now
|
||||||
|
self._cached = (float("-inf"), NORMAL)
|
||||||
|
self._cpu_prev = None
|
||||||
|
|
||||||
|
def state(self):
|
||||||
|
t, s = self._cached
|
||||||
|
if self.now() - t < self.interval:
|
||||||
|
return s
|
||||||
|
try:
|
||||||
|
s = self._compute()
|
||||||
|
except Exception:
|
||||||
|
s = NORMAL
|
||||||
|
self._cached = (self.now(), s)
|
||||||
|
return s
|
||||||
|
|
||||||
|
def _compute(self):
|
||||||
|
if sys.platform == "win32":
|
||||||
|
return self._windows()
|
||||||
|
return self._linux()
|
||||||
|
|
||||||
|
# --- Linux ---
|
||||||
|
|
||||||
|
def _linux(self):
|
||||||
|
if os.getloadavg()[0] / max(1, os.cpu_count() or 1) > self.busy_load:
|
||||||
|
return PAUSED
|
||||||
|
temps = []
|
||||||
|
for path in glob.glob("/sys/class/thermal/thermal_zone*/temp"):
|
||||||
|
try:
|
||||||
|
temps.append(int(open(path).read().strip()) / 1000)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
pass
|
||||||
|
if temps and max(temps) >= 95:
|
||||||
|
return PAUSED
|
||||||
|
try:
|
||||||
|
if open("/sys/firmware/acpi/platform_profile").read().strip() in ("low-power", "quiet"):
|
||||||
|
return REDUCED
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
for bat in glob.glob("/sys/class/power_supply/BAT*"):
|
||||||
|
try:
|
||||||
|
status = open(bat + "/status").read().strip()
|
||||||
|
capacity = int(open(bat + "/capacity").read().strip())
|
||||||
|
except (OSError, ValueError):
|
||||||
|
continue
|
||||||
|
if status == "Discharging" and capacity <= 20:
|
||||||
|
return REDUCED
|
||||||
|
return NORMAL
|
||||||
|
|
||||||
|
# --- Windows ---
|
||||||
|
|
||||||
|
def _windows(self):
|
||||||
|
import ctypes
|
||||||
|
from ctypes import wintypes
|
||||||
|
|
||||||
|
class FILETIME(ctypes.Structure):
|
||||||
|
_fields_ = [("lo", wintypes.DWORD), ("hi", wintypes.DWORD)]
|
||||||
|
|
||||||
|
idle, kernel, user = FILETIME(), FILETIME(), FILETIME()
|
||||||
|
ctypes.windll.kernel32.GetSystemTimes(ctypes.byref(idle), ctypes.byref(kernel), ctypes.byref(user))
|
||||||
|
values = [(f.hi << 32) | f.lo for f in (idle, kernel, user)]
|
||||||
|
prev, self._cpu_prev = self._cpu_prev, values
|
||||||
|
if prev:
|
||||||
|
d_idle, d_kernel, d_user = (a - b for a, b in zip(values, prev))
|
||||||
|
total = d_kernel + d_user # kernel time includes idle time
|
||||||
|
if total > 0 and 1 - d_idle / total > self.busy_load:
|
||||||
|
return PAUSED
|
||||||
|
|
||||||
|
class SYSTEM_POWER_STATUS(ctypes.Structure):
|
||||||
|
_fields_ = [("ACLineStatus", ctypes.c_ubyte), ("BatteryFlag", ctypes.c_ubyte),
|
||||||
|
("BatteryLifePercent", ctypes.c_ubyte), ("SystemStatusFlag", ctypes.c_ubyte),
|
||||||
|
("BatteryLifeTime", wintypes.DWORD), ("BatteryFullLifeTime", wintypes.DWORD)]
|
||||||
|
|
||||||
|
status = SYSTEM_POWER_STATUS()
|
||||||
|
if ctypes.windll.kernel32.GetSystemPowerStatus(ctypes.byref(status)) and status.SystemStatusFlag == 1:
|
||||||
|
return REDUCED # battery saver on
|
||||||
|
return NORMAL
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 1.4 KiB |
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "自動注音 AutoZhuyin",
|
"name": "自動注音 AutoZhuyin",
|
||||||
"version": "0.7.1",
|
"version": "0.13.2",
|
||||||
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
|
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
|
||||||
"locale": "zh-Hant-TW",
|
"locale": "zh-Hant-TW",
|
||||||
"fallbackLocale": "zh-TW",
|
"fallbackLocale": "zh-TW",
|
||||||
|
|||||||
@@ -26,10 +26,12 @@
|
|||||||
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
|
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
|
||||||
Enter 送出目前組字
|
Enter 送出目前組字
|
||||||
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
|
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
|
||||||
← → 在組字區移動游標,選要改哪個字
|
← → 在組字區移動游標,選要改哪個字;游標在中間時直接打注音
|
||||||
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字
|
會插入新字,Backspace 刪掉游標前的字
|
||||||
|
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
|
||||||
|
選過的字會被記住,下次優先出現
|
||||||
Esc 取消組字
|
Esc 取消組字
|
||||||
單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
|
單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線,工作列圖示變成 A),
|
||||||
再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
|
再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
|
||||||
打到一半時按,同 Tab 整段翻轉
|
打到一半時按,同 Tab 整段翻轉
|
||||||
Shift+字母 打大寫字母(也會強制英文)
|
Shift+字母 打大寫字母(也會強制英文)
|
||||||
@@ -41,6 +43,15 @@
|
|||||||
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
|
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
|
||||||
|
|
||||||
|
|
||||||
|
AI 選字
|
||||||
|
-------
|
||||||
|
用中研院 CKIP 的繁體中文小型語言模型,看前後文挑同音字(他跑「得」很快、請你「再」說一次),
|
||||||
|
完全在本機運算,不連網。在工作列的輸入法圖示上按右鍵(或語言列的設定按鈕)可以選:
|
||||||
|
AI 選字:關閉/省電/標準(預設)/高品質
|
||||||
|
系統忙碌時自動暫停 AI(預設開啟:CPU 很忙時暫停,省電模式時改用省電)
|
||||||
|
選單裡會顯示目前狀態。電腦比較慢時建議用「省電」,運算太慢時也會自動暫停一分鐘。
|
||||||
|
|
||||||
|
|
||||||
回報問題
|
回報問題
|
||||||
--------
|
--------
|
||||||
請告訴我:
|
請告訴我:
|
||||||
|
|||||||
+215
-7
@@ -15,6 +15,7 @@ from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BAC
|
|||||||
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
|
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
|
||||||
from autozhuyin.core.english import EnglishModel # noqa: E402
|
from autozhuyin.core.english import EnglishModel # noqa: E402
|
||||||
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
|
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
|
||||||
|
from autozhuyin.core.history import UserHistory # noqa: E402
|
||||||
|
|
||||||
DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data")
|
DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data")
|
||||||
_start = time.time()
|
_start = time.time()
|
||||||
@@ -42,7 +43,7 @@ def make_engine():
|
|||||||
def type_keys(s, e):
|
def type_keys(s, e):
|
||||||
out = ""
|
out = ""
|
||||||
for c in s:
|
for c in s:
|
||||||
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN}.get(c, c)
|
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN, "\b": BACKSPACE}.get(c, c)
|
||||||
handled, commit = e.handle(key)
|
handled, commit = e.handle(key)
|
||||||
if commit is not None:
|
if commit is not None:
|
||||||
out += commit
|
out += commit
|
||||||
@@ -246,7 +247,7 @@ class CursorTests(unittest.TestCase):
|
|||||||
type_keys("←", e)
|
type_keys("←", e)
|
||||||
self.assertEqual(e.cursor_offset, 3)
|
self.assertEqual(e.cursor_offset, 3)
|
||||||
type_keys("←←←←", e)
|
type_keys("←←←←", e)
|
||||||
self.assertEqual(e.cursor_offset, 1)
|
self.assertEqual(e.cursor_offset, 1) # stops after the first character
|
||||||
type_keys("→→→→", e)
|
type_keys("→→→→", e)
|
||||||
self.assertEqual(e.cursor_offset, 4)
|
self.assertEqual(e.cursor_offset, 4)
|
||||||
|
|
||||||
@@ -258,12 +259,55 @@ class CursorTests(unittest.TestCase):
|
|||||||
e.select_candidate([t for _, t in e.candidates].index("添"))
|
e.select_candidate([t for _, t in e.candidates].index("添"))
|
||||||
self.assertEqual(type_keys("\n", e), "今添天氣")
|
self.assertEqual(type_keys("\n", e), "今添天氣")
|
||||||
|
|
||||||
def test_typing_returns_cursor_to_end(self):
|
def test_english_returns_cursor_to_end(self):
|
||||||
e = make_engine()
|
e = make_engine()
|
||||||
type_keys("rup wu0 ←", e)
|
type_keys("rup wu0 ←", e)
|
||||||
self.assertEqual(e.cursor_offset, 1)
|
self.assertEqual(e.cursor_offset, 1)
|
||||||
type_keys("wu0 ", e)
|
type_keys("?", e)
|
||||||
self.assertEqual(e.cursor_offset, len(e.marked_text))
|
self.assertEqual(e.marked_text, "")
|
||||||
|
|
||||||
|
|
||||||
|
class InsertInMiddleTests(unittest.TestCase):
|
||||||
|
def test_insert_syllable(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup fu4←wu", e)
|
||||||
|
self.assertEqual(e.marked_text, "今ㄊㄧ氣")
|
||||||
|
self.assertEqual(e.cursor_offset, 3)
|
||||||
|
type_keys("0 ", e)
|
||||||
|
self.assertEqual(e.marked_text, "今天氣")
|
||||||
|
self.assertEqual(e.cursor_offset, 2)
|
||||||
|
self.assertEqual(type_keys("\n", e), "今天氣")
|
||||||
|
|
||||||
|
def test_cursor_stops_after_first_character(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("su3cl3←←←", e)
|
||||||
|
self.assertEqual(e.cursor_offset, 1)
|
||||||
|
type_keys("ji3\n", e)
|
||||||
|
self.assertEqual(e.marked_text, "")
|
||||||
|
|
||||||
|
def test_no_stop_before_first_chinese(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("hello ji3←←", e) # "hello " held, then 我: nothing to pick before 我
|
||||||
|
type_keys("↓", e)
|
||||||
|
self.assertTrue(e.showing_candidates)
|
||||||
|
|
||||||
|
def test_backspace_deletes_character_before_cursor(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup wu0 wu0 fu4←←\b", e)
|
||||||
|
self.assertEqual(e.marked_text, "今天氣")
|
||||||
|
self.assertEqual(e.cursor_offset, 1)
|
||||||
|
|
||||||
|
def test_backspace_inside_inserted_syllable(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup fu4←wu\b", e)
|
||||||
|
self.assertEqual(e.marked_text, "今ㄊ氣")
|
||||||
|
|
||||||
|
def test_keeps_picked_phrase_before_cursor(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup wu0 ↓", e)
|
||||||
|
e.select_candidate([t for _, t in e.candidates].index("添"))
|
||||||
|
type_keys("fu4←wu0 ", e) # insert 天 between 添 and 氣
|
||||||
|
self.assertEqual(e.marked_text[:3], "今添天")
|
||||||
|
|
||||||
def test_candidates_for_chinese_before_english(self):
|
def test_candidates_for_chinese_before_english(self):
|
||||||
e = make_engine()
|
e = make_engine()
|
||||||
@@ -275,8 +319,10 @@ class FirstToneAfterEnglishTests(unittest.TestCase):
|
|||||||
def test_first_tone_after_english(self):
|
def test_first_tone_after_english(self):
|
||||||
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
|
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
|
||||||
e = make_engine()
|
e = make_engine()
|
||||||
self.assertEqual(type_keys("hello " + keys + " ", e), "hello ", keys)
|
out = type_keys("hello " + keys + " \n", e)
|
||||||
self.assertEqual(len(e.marked_text), 1, keys)
|
self.assertEqual(out[:6], "hello ", keys)
|
||||||
|
self.assertEqual(len(out), 7, keys) # one Chinese character after it
|
||||||
|
self.assertFalse(out[6].isascii(), keys)
|
||||||
|
|
||||||
|
|
||||||
class HeldWordTests(unittest.TestCase):
|
class HeldWordTests(unittest.TestCase):
|
||||||
@@ -315,5 +361,167 @@ class BacktickSymbolTests(unittest.TestCase):
|
|||||||
self.assertEqual(e.marked_text, "你")
|
self.assertEqual(e.marked_text, "你")
|
||||||
|
|
||||||
|
|
||||||
|
class FirstWordHoldTests(unittest.TestCase):
|
||||||
|
"""The first word is not committed as English on space if it also reads as Zhuyin."""
|
||||||
|
def test_next_word_decides(self):
|
||||||
|
cases = {"el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
|
||||||
|
"i love you ": "i love you ", "up the ": "up the "}
|
||||||
|
for keys, want in cases.items():
|
||||||
|
self.assertEqual(type_keys(keys, make_engine()), want, keys)
|
||||||
|
|
||||||
|
def test_not_held_after_english(self):
|
||||||
|
e = make_engine()
|
||||||
|
self.assertEqual(type_keys("hello world ", e), "hello world ")
|
||||||
|
self.assertEqual(type_keys("so ", e), "so ") # right after English: commits at once
|
||||||
|
|
||||||
|
|
||||||
|
class UserHistoryTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.saved = []
|
||||||
|
self.c = Classifier(LEX, ENG, weights=WEIGHTS, history=UserHistory(on_change=self.saved.append))
|
||||||
|
|
||||||
|
def pick(self, keys, phrase):
|
||||||
|
e = CompositionEngine(self.c)
|
||||||
|
type_keys(keys + "↓", e)
|
||||||
|
e.select_candidate([t for _, t in e.candidates].index(phrase))
|
||||||
|
return type_keys("\n", e)
|
||||||
|
|
||||||
|
def test_picked_candidate_comes_first(self):
|
||||||
|
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "天")
|
||||||
|
self.assertEqual(self.pick("wu0 ", "添"), "添")
|
||||||
|
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
|
||||||
|
e = CompositionEngine(self.c)
|
||||||
|
type_keys("wu0 ↓", e)
|
||||||
|
self.assertEqual(e.candidates[0][1], "添")
|
||||||
|
self.assertEqual(self.saved[-1], {"ㄊㄧㄢ": {"添": [1, 1]}})
|
||||||
|
|
||||||
|
def test_phrases_still_win(self):
|
||||||
|
self.pick("wu0 ", "添")
|
||||||
|
self.assertEqual(type_keys("rup wu0 \n", CompositionEngine(self.c)), "今天")
|
||||||
|
|
||||||
|
def test_most_picked_wins(self):
|
||||||
|
self.pick("wu0 ", "添")
|
||||||
|
self.pick("wu0 ", "添")
|
||||||
|
self.pick("wu0 ", "天")
|
||||||
|
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
|
||||||
|
|
||||||
|
def test_round_trip_and_clear(self):
|
||||||
|
self.pick("wu0 ", "添")
|
||||||
|
h = UserHistory(self.saved[-1])
|
||||||
|
self.assertEqual(h.data, {"ㄊㄧㄢ": {"添": [1, 1]}})
|
||||||
|
h.record([bopomofo.parse(list("wu0 ")).syllables[0].syllable], "天")
|
||||||
|
self.assertEqual(h.data["ㄊㄧㄢ"]["天"], [1, 2])
|
||||||
|
h.clear()
|
||||||
|
self.assertEqual(h.data, {})
|
||||||
|
|
||||||
|
def test_evicts_least_recent(self):
|
||||||
|
h = UserHistory()
|
||||||
|
h.max_entries = 2
|
||||||
|
syl = [bopomofo.parse(list("wu0 ")).syllables[0].syllable]
|
||||||
|
for p in ["天", "添", "田"]:
|
||||||
|
h.record(syl, p)
|
||||||
|
self.assertEqual(set(h.data["ㄊㄧㄢ"]), {"添", "田"})
|
||||||
|
|
||||||
|
|
||||||
|
class FakeLM:
|
||||||
|
"""Likes one string; `on_call` can advance a fake clock."""
|
||||||
|
def __init__(self, liked):
|
||||||
|
self.liked, self.calls, self.on_call = liked, 0, lambda: None
|
||||||
|
|
||||||
|
def log_probs(self, texts, context=""):
|
||||||
|
self.calls += len(texts)
|
||||||
|
self.on_call()
|
||||||
|
return [0.0 if self.liked in t else -50.0 for t in texts]
|
||||||
|
|
||||||
|
|
||||||
|
class RerankerTests(unittest.TestCase):
|
||||||
|
def engine(self, reranker):
|
||||||
|
e = make_engine()
|
||||||
|
e.reranker = reranker
|
||||||
|
return e
|
||||||
|
|
||||||
|
def test_rerank_picks_what_the_model_likes(self):
|
||||||
|
from autozhuyin.core.reranker import Reranker, ST_ACTIVE
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(None)), "天")
|
||||||
|
r = Reranker("standard", FakeLM("添"))
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "添")
|
||||||
|
self.assertEqual(r.status, ST_ACTIVE)
|
||||||
|
|
||||||
|
def test_off_and_loading_use_the_lexicon(self):
|
||||||
|
from autozhuyin.core.reranker import Reranker, ST_LOADING, ST_OFF
|
||||||
|
off = Reranker("off", FakeLM("添"))
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(off)), "天")
|
||||||
|
self.assertEqual(off.status, ST_OFF)
|
||||||
|
loading = Reranker("standard", None)
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(loading)), "天")
|
||||||
|
self.assertEqual(loading.status, ST_LOADING)
|
||||||
|
|
||||||
|
def test_paused_and_reduced(self):
|
||||||
|
from autozhuyin.core.reranker import PAUSED, REDUCED, Reranker, ST_BUSY
|
||||||
|
r = Reranker("standard", FakeLM("添"))
|
||||||
|
r.system_state = lambda: PAUSED
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "天")
|
||||||
|
self.assertEqual(r.status, ST_BUSY)
|
||||||
|
lm = FakeLM("添")
|
||||||
|
r = Reranker("quality", lm)
|
||||||
|
r.system_state = lambda: REDUCED
|
||||||
|
type_keys("wu0 \n", self.engine(r))
|
||||||
|
self.assertLessEqual(lm.calls, 3)
|
||||||
|
|
||||||
|
def test_slow_model_is_paused(self):
|
||||||
|
from autozhuyin.core.reranker import Reranker, ST_SLOW
|
||||||
|
clock = [0.0]
|
||||||
|
lm = FakeLM("添")
|
||||||
|
lm.on_call = lambda: clock.__setitem__(0, clock[0] + 1) # every batch takes a "second"
|
||||||
|
r = Reranker("standard", lm, now=lambda: clock[0])
|
||||||
|
for _ in range(3):
|
||||||
|
type_keys("wu0 \n", self.engine(r))
|
||||||
|
self.assertEqual(r.status, ST_SLOW)
|
||||||
|
lm.on_call = lambda: None
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "天") # paused
|
||||||
|
clock[0] += 61
|
||||||
|
self.assertEqual(type_keys("wu0 \n", self.engine(r)), "添") # back after a minute
|
||||||
|
|
||||||
|
def test_kbest_first_is_viterbi(self):
|
||||||
|
syl = [s.syllable for s in bopomofo.parse(list("rup wu0 wu0 fu4")).syllables]
|
||||||
|
conv = Converter(LEX)
|
||||||
|
best = conv.kbest(syl, 5)
|
||||||
|
self.assertEqual("".join(t for _, t in best[0][1]), "".join(t for _, t in conv.convert(syl)))
|
||||||
|
self.assertEqual(len(best), 5)
|
||||||
|
|
||||||
|
|
||||||
|
class RealModelTests(unittest.TestCase):
|
||||||
|
"""Needs numpy and data/lm.bin (scripts/fetch-lm.sh, tools/build_data.py); skipped otherwise."""
|
||||||
|
def test_fixes_homophones(self):
|
||||||
|
from autozhuyin.core import lm
|
||||||
|
from autozhuyin.core.reranker import Reranker
|
||||||
|
path = os.path.join(DATA, "lm.bin")
|
||||||
|
if not lm.available() or not os.path.exists(path):
|
||||||
|
self.skipTest("needs numpy and the model")
|
||||||
|
model = lm.GPT2LanguageModel(path, os.path.join(DATA, "lm-vocab.txt"))
|
||||||
|
self.assertAlmostEqual(model.log_prob("他在家"), -14.329, delta=0.05)
|
||||||
|
cases = {"w8 ql32k71u3ji3dj94cp32ji ": "他跑得比我快很多", "fu/3su3y94gji u h4cl3a87": "請你再說一次好嗎"}
|
||||||
|
for keys, want in cases.items():
|
||||||
|
self.assertNotEqual(type_keys(keys + "\n", make_engine()), want)
|
||||||
|
e = make_engine()
|
||||||
|
e.reranker = Reranker("standard", model)
|
||||||
|
self.assertEqual(type_keys(keys + "\n", e), want)
|
||||||
|
|
||||||
|
|
||||||
|
class AISettingsTests(unittest.TestCase):
|
||||||
|
def test_settings_round_trip_and_status(self):
|
||||||
|
import tempfile
|
||||||
|
from autozhuyin.core.ai import AI
|
||||||
|
user = tempfile.mkdtemp()
|
||||||
|
ai = AI(tempfile.mkdtemp(), user, background=False) # no model files here
|
||||||
|
self.assertTrue(ai.error) # "需要 numpy" or "AI 模型未安裝"
|
||||||
|
self.assertEqual(ai.status_text(), ai.error)
|
||||||
|
ai.set_mode("saver")
|
||||||
|
ai.set_auto_pause(False)
|
||||||
|
again = AI(tempfile.mkdtemp(), user, background=False)
|
||||||
|
self.assertEqual(again.reranker.mode, "saver")
|
||||||
|
self.assertFalse(again.auto_pause)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -6,6 +6,9 @@ import types
|
|||||||
import unittest
|
import unittest
|
||||||
|
|
||||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
import tempfile # noqa: E402
|
||||||
|
|
||||||
|
os.environ["APPDATA"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
|
||||||
sys.path.insert(0, os.path.dirname(HERE))
|
sys.path.insert(0, os.path.dirname(HERE))
|
||||||
|
|
||||||
# --- stand-ins for PIME modules (only what the wrapper uses) ---
|
# --- stand-ins for PIME modules (only what the wrapper uses) ---
|
||||||
@@ -37,6 +40,9 @@ class FakeTextService:
|
|||||||
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
|
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
|
||||||
|
|
||||||
def showMessage(self, message, duration=3): self.reply["showMessage"] = message
|
def showMessage(self, message, duration=3): self.reply["showMessage"] = message
|
||||||
|
def addButton(self, button_id, **kw): self.reply.setdefault("addButton", []).append(button_id)
|
||||||
|
def removeButton(self, button_id): self.reply.setdefault("removeButton", []).append(button_id)
|
||||||
|
def changeButton(self, button_id, **kw): self.reply.setdefault("changeButton", []).append((button_id, kw))
|
||||||
|
|
||||||
def setShowCandidates(self, show):
|
def setShowCandidates(self, show):
|
||||||
self.showCandidates = show
|
self.showCandidates = show
|
||||||
@@ -44,9 +50,10 @@ class FakeTextService:
|
|||||||
|
|
||||||
|
|
||||||
textService.TextService = FakeTextService
|
textService.TextService = FakeTextService
|
||||||
|
textService.COMMAND_LEFT_CLICK, textService.COMMAND_RIGHT_CLICK, textService.COMMAND_MENU = 0, 1, 2
|
||||||
sys.modules["textService"] = textService
|
sys.modules["textService"] = textService
|
||||||
|
|
||||||
from autozhuyin.autozhuyin_ime import AutoZhuyinTextService # noqa: E402
|
from autozhuyin.autozhuyin_ime import AutoZhuyinTextService, _shared_classifier # noqa: E402
|
||||||
|
|
||||||
VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
|
VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
|
||||||
|
|
||||||
@@ -99,12 +106,41 @@ def type_text(svc, text):
|
|||||||
|
|
||||||
class WrapperTests(unittest.TestCase):
|
class WrapperTests(unittest.TestCase):
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
|
_shared_classifier().history.clear()
|
||||||
|
_shared_classifier().learned.clear()
|
||||||
self.svc = AutoZhuyinTextService(client=None)
|
self.svc = AutoZhuyinTextService(client=None)
|
||||||
self.svc.onActivate()
|
self.svc.onActivate()
|
||||||
|
|
||||||
def test_mixed_sentence(self):
|
def test_mixed_sentence(self):
|
||||||
self.assertEqual(type_text(self.svc, "ji3 hello world su3cl3 "), "我hello world 你好")
|
self.assertEqual(type_text(self.svc, "ji3 hello world su3cl3 "), "我hello world 你好")
|
||||||
|
|
||||||
|
def test_backtick_symbol(self):
|
||||||
|
self.assertEqual(type_text(self.svc, "`<"), ",")
|
||||||
|
self.assertEqual(type_text(self.svc, "ji3`."), "我。")
|
||||||
|
|
||||||
|
def test_settings_menu(self):
|
||||||
|
from autozhuyin import autozhuyin_ime as ime
|
||||||
|
self.assertIn("autozhuyin-settings", self.svc.reply["addButton"])
|
||||||
|
menu = self.svc.onMenu("autozhuyin-settings")
|
||||||
|
texts = [m.get("text") for m in menu]
|
||||||
|
self.assertIn("AI 選字:標準", texts)
|
||||||
|
self.assertIn("清除學習紀錄", texts)
|
||||||
|
self.svc.onCommand(ime.ID_AI_MODE["saver"], 2)
|
||||||
|
checked = [m["text"] for m in self.svc.onMenu("autozhuyin-settings") if m.get("checked")]
|
||||||
|
self.assertIn("AI 選字:省電", checked)
|
||||||
|
self.svc.onCommand(ime.ID_AI_MODE["standard"], 2)
|
||||||
|
self.assertIsNone(self.svc.onMenu("other"))
|
||||||
|
|
||||||
|
def test_mode_icon_click_toggles_english(self):
|
||||||
|
self.svc.reply = {}
|
||||||
|
self.svc.onCommand(1, 0)
|
||||||
|
self.assertTrue(self.svc.engine.english_mode)
|
||||||
|
(button, kw), = self.svc.reply["changeButton"]
|
||||||
|
self.assertEqual(button, "windows-mode-icon")
|
||||||
|
self.assertTrue(kw["icon"].endswith("icon-en.ico") or kw["icon"].endswith("icon.ico"))
|
||||||
|
self.svc.onCommand(1, 0)
|
||||||
|
self.assertFalse(self.svc.engine.english_mode)
|
||||||
|
|
||||||
def test_composition_shown(self):
|
def test_composition_shown(self):
|
||||||
type_text(self.svc, "su3c")
|
type_text(self.svc, "su3c")
|
||||||
self.assertEqual(self.svc.reply["compositionString"], "你ㄏ")
|
self.assertEqual(self.svc.reply["compositionString"], "你ㄏ")
|
||||||
@@ -138,9 +174,17 @@ class WrapperTests(unittest.TestCase):
|
|||||||
|
|
||||||
class ShiftTapWrapperTests(unittest.TestCase):
|
class ShiftTapWrapperTests(unittest.TestCase):
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
|
_shared_classifier().history.clear()
|
||||||
|
_shared_classifier().learned.clear()
|
||||||
self.svc = AutoZhuyinTextService(client=None)
|
self.svc = AutoZhuyinTextService(client=None)
|
||||||
self.svc.onActivate()
|
self.svc.onActivate()
|
||||||
|
|
||||||
|
def test_shift_tap_changes_icon(self):
|
||||||
|
tap_shift(self.svc)
|
||||||
|
self.assertEqual(self.svc.reply["changeButton"][0][0], "windows-mode-icon")
|
||||||
|
tap_shift(self.svc)
|
||||||
|
self.assertIn("changeButton", self.svc.reply)
|
||||||
|
|
||||||
def test_tap_forces_english_lowercase(self):
|
def test_tap_forces_english_lowercase(self):
|
||||||
reply = tap_shift(self.svc)
|
reply = tap_shift(self.svc)
|
||||||
self.assertEqual(reply.get("showMessage"), "英文")
|
self.assertEqual(reply.get("showMessage"), "英文")
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
|
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
|
||||||
Webster's 2nd edition, public domain), since Windows has no such file
|
Webster's 2nd edition, public domain), since Windows has no such file
|
||||||
- english-bigram.json: precomputed letter-bigram table
|
- english-bigram.json: precomputed letter-bigram table
|
||||||
|
- lm.bin, lm-vocab.txt: the "AI 選字" language model, copied from build/lm/ if present
|
||||||
|
|
||||||
Usage: build_data.py [word-list]
|
Usage: build_data.py [word-list]
|
||||||
"""
|
"""
|
||||||
@@ -34,4 +35,10 @@ common = {w.strip().lower() for w in (data / "english-common.txt").read_text(enc
|
|||||||
bigram = train_bigram(sorted(common | set(words)), common)
|
bigram = train_bigram(sorted(common | set(words)), common)
|
||||||
(data / "english-bigram.json").write_text(json.dumps(bigram), encoding="utf-8")
|
(data / "english-bigram.json").write_text(json.dumps(bigram), encoding="utf-8")
|
||||||
|
|
||||||
|
# "AI 選字" language model (scripts/fetch-lm.sh makes build/lm/); optional.
|
||||||
|
lm_dir = ROOT / "build" / "lm"
|
||||||
|
for name in ("lm.bin", "lm-vocab.txt"):
|
||||||
|
if (lm_dir / name).exists():
|
||||||
|
shutil.copy(lm_dir / name, data / name)
|
||||||
|
|
||||||
print("data ready: %d dictionary words, %d bigrams" % (len(words), len(bigram["table"])))
|
print("data ready: %d dictionary words, %d bigrams" % (len(words), len(bigram["table"])))
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
"""Downloads numpy for one Python version / platform from PyPI and unpacks it into
|
||||||
|
<dest>/<python tag>/ (numpy, numpy.libs, dist-info; test folders are dropped).
|
||||||
|
|
||||||
|
fetch_numpy.py <dest> <python tag> <platform>
|
||||||
|
fetch_numpy.py autozhuyin/vendor cp38 win32
|
||||||
|
fetch_numpy.py ~/.local/share/autozhuyin/vendor auto auto (this Python, Linux)
|
||||||
|
|
||||||
|
The newest numpy release that has a wheel for that combination is used, and its
|
||||||
|
SHA-256 is checked against PyPI. Standard library only.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import platform
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import urllib.request
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def version_key(v):
|
||||||
|
return tuple(int(x) for x in re.findall(r"\d+", v))
|
||||||
|
|
||||||
|
|
||||||
|
def linux_platforms():
|
||||||
|
machine = platform.machine().lower()
|
||||||
|
arch = {"amd64": "x86_64", "arm64": "aarch64"}.get(machine, machine)
|
||||||
|
return [f"manylinux_2_17_{arch}", f"manylinux2014_{arch}", f"manylinux_2_27_{arch}", f"manylinux_2_28_{arch}"]
|
||||||
|
|
||||||
|
|
||||||
|
def find_wheel(tag, platforms):
|
||||||
|
with urllib.request.urlopen("https://pypi.org/pypi/numpy/json", timeout=60) as r:
|
||||||
|
releases = json.load(r)["releases"]
|
||||||
|
for version in sorted(releases, key=version_key, reverse=True):
|
||||||
|
if re.search(r"[a-z]", version): # skip pre-releases (rc, b, a, dev)
|
||||||
|
continue
|
||||||
|
for f in releases[version]:
|
||||||
|
name = f["filename"]
|
||||||
|
if not name.endswith(".whl") or f.get("yanked"):
|
||||||
|
continue
|
||||||
|
if f"-{tag}-{tag}-" not in name: # also skips free-threaded "cp315t" builds
|
||||||
|
continue
|
||||||
|
plat = name[:-4].split("-")[-1]
|
||||||
|
if any(p in plat.split(".") for p in platforms):
|
||||||
|
return version, f
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
def main(dest, tag, plat):
|
||||||
|
if tag == "auto":
|
||||||
|
tag = "cp%d%d" % sys.version_info[:2]
|
||||||
|
platforms = linux_platforms() if plat == "auto" else [plat]
|
||||||
|
version, f = find_wheel(tag, platforms)
|
||||||
|
if not f:
|
||||||
|
sys.exit(f"no numpy wheel for {tag} {platforms}")
|
||||||
|
print(f"numpy {version} for {tag}: {f['filename']} ({f['size'] / 1e6:.1f} MB)")
|
||||||
|
with urllib.request.urlopen(f["url"], timeout=300) as r:
|
||||||
|
data = r.read()
|
||||||
|
if hashlib.sha256(data).hexdigest() != f["digests"]["sha256"]:
|
||||||
|
sys.exit("checksum mismatch")
|
||||||
|
out = Path(dest).expanduser() / tag
|
||||||
|
if out.exists():
|
||||||
|
shutil.rmtree(out)
|
||||||
|
out.mkdir(parents=True)
|
||||||
|
with zipfile.ZipFile(io.BytesIO(data)) as z:
|
||||||
|
for info in z.infolist():
|
||||||
|
parts = Path(info.filename).parts
|
||||||
|
if "tests" in parts or parts[0].endswith(".data"):
|
||||||
|
continue
|
||||||
|
z.extract(info, out)
|
||||||
|
print(f"unpacked to {out}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if len(sys.argv) != 4:
|
||||||
|
sys.exit(__doc__)
|
||||||
|
main(*sys.argv[1:])
|
||||||
Reference in New Issue
Block a user