Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
251a16fa08 | ||
|
|
7cb1f62dfa | ||
|
|
3670305f0f | ||
|
|
ef9f91a6b7 | ||
|
|
1dacb8f164 | ||
|
|
bfeed22916 | ||
|
|
bb9cb0c4e0 | ||
|
|
2c5c70c066 | ||
|
|
17f7d6f50d | ||
|
|
7e173a1da8 | ||
|
|
fa0b78e5a4 | ||
|
|
e928d08fdc | ||
|
|
82f20a6551 | ||
|
|
e6e5004630 | ||
|
|
988ed471be | ||
|
|
6c36c01f0e | ||
|
|
0fc42a4b8c | ||
|
|
42a2d3a2a3 | ||
|
|
20291bf515 | ||
|
|
21142c2d98 | ||
|
|
476858d841 | ||
|
|
0a7242ddb3 | ||
|
|
d1f1112c65 | ||
|
|
dbe08e00a7 |
@@ -1,12 +1,65 @@
|
|||||||
# 自動注音 AutoZhuyin
|
# 自動注音 AutoZhuyin
|
||||||
|
|
||||||
macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
|
給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
|
||||||
|
|
||||||
```
|
```
|
||||||
按鍵: ji3 hello world su3cl3␣
|
按鍵: ji3 hello world su3cl3␣
|
||||||
輸出: 我hello world 你好
|
輸出: 我hello world 你好
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 功能
|
||||||
|
|
||||||
|
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
|
||||||
|
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
|
||||||
|
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
|
||||||
|
- **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷
|
||||||
|
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
|
||||||
|
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
|
||||||
|
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
|
||||||
|
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
|
||||||
|
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
|
||||||
|
|
||||||
|
## 下載與安裝
|
||||||
|
|
||||||
|
到 Gitea 的 [Releases](https://git.ays.ddnsfree.com/ays953/AutoZhuyin/releases) 頁面下載最新版本。
|
||||||
|
|
||||||
|
### macOS
|
||||||
|
|
||||||
|
需求:macOS 13 以上,Apple Silicon 和 Intel 都可以(universal binary)。
|
||||||
|
|
||||||
|
1. 雙擊 `AutoZhuyin-<版本>.pkg` 安裝。安裝檔沒有簽章,如果被擋下來,按右鍵 →「打開」,或到「系統設定 → 隱私權與安全性」按「仍要打開」。
|
||||||
|
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
|
||||||
|
3. 清單裡沒出現的話,登出再登入一次。
|
||||||
|
|
||||||
|
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
|
||||||
|
|
||||||
|
### Windows
|
||||||
|
|
||||||
|
需求:Windows 10 / 11,x64、x86、ARM64 都可以。
|
||||||
|
|
||||||
|
1. 先安裝輸入法框架 [PIME](https://github.com/EasyIME/PIME/releases)(免費、開源)。
|
||||||
|
2. 解壓縮 `AutoZhuyin-Windows-<版本>.zip`,在 `install.bat` 上按右鍵 →「以系統管理員身分執行」。
|
||||||
|
3. 設定 → 時間與語言 → 語言與地區 → 中文(台灣) → 語言選項 → 新增鍵盤 → **自動注音 AutoZhuyin**。
|
||||||
|
4. 用 Win + 空白鍵 切換過去。清單裡沒出現的話,重新開機一次。
|
||||||
|
|
||||||
|
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
|
||||||
|
|
||||||
|
## 按鍵
|
||||||
|
|
||||||
|
| 按鍵 | 作用 |
|
||||||
|
|---|---|
|
||||||
|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
|
||||||
|
| Enter | 送出目前組字 |
|
||||||
|
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
|
||||||
|
| ← → | 在組字區移動游標,選要改哪個字 |
|
||||||
|
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
|
||||||
|
| Esc | 取消組字 |
|
||||||
|
| **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
|
||||||
|
| Shift + 字母 | 打大寫字母(也會強制英文) |
|
||||||
|
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
|
||||||
|
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
|
||||||
|
| `` ` `` + 標點鍵 | **強制全形標點**,不管前後是中文還是英文:`` `, `` `` `< `` = `,`、`` `. `` `` `> `` = `。`、`` `/ `` = `?`、`` `; `` = `;`、`` `: `` = `:`、`` `' `` `` `\ `` = `、`、`` `[ `` `` `] `` = `「」`、`` `{ `` `` `} `` = `『』`、`` `( `` `` `) `` = `()`、`` `! `` = `!`、`` `~ `` = `~`、`` `- `` = `——`、`` `= `` = `=`。`` ` `` 後面接其他鍵就是普通的反引號(` ``` ` 照常可用) |
|
||||||
|
|
||||||
## 運作方式
|
## 運作方式
|
||||||
|
|
||||||
每按一個鍵,`Classifier` 會同時用兩種解讀幫目前的按鍵序列打分:
|
每按一個鍵,`Classifier` 會同時用兩種解讀幫目前的按鍵序列打分:
|
||||||
@@ -14,84 +67,174 @@ macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵
|
|||||||
| 注音分數 | 英文分數 |
|
| 注音分數 | 英文分數 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 是否能拆成合法注音音節(`SyllableTable`) | 是否為常用英文字(`english-common.txt`) |
|
| 是否能拆成合法注音音節(`SyllableTable`) | 是否為常用英文字(`english-common.txt`) |
|
||||||
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在系統字典 `/usr/share/dict/words` |
|
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在英文字典裡 |
|
||||||
| 讀音是否存在於詞庫 | 字母 bigram 模型判斷「像不像英文」 |
|
| 讀音是否存在於詞庫 | 字母 bigram 模型判斷「像不像英文」 |
|
||||||
| 前一段是中文 → 加分(語境) | 前一段是英文 → 加分(語境) |
|
| 前一段是中文 → 加分(語境) | 前一段是英文 → 加分(語境) |
|
||||||
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
|
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
|
||||||
|
| 一開始稍微偏向注音 | 純數字 → 數字 |
|
||||||
|
|
||||||
分數高的一方勝出。組字區即時顯示結果(英文以藍字顯示)。
|
每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。
|
||||||
|
|
||||||
## 按鍵
|
中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎:
|
||||||
|
- 每多切一刀會扣分(`seg.switch` 特徵)
|
||||||
|
- 後面還接著其他段的注音段,最後一個音節必須打了聲調
|
||||||
|
- 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
|
||||||
|
|
||||||
| 按鍵 | 作用 |
|
音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。
|
||||||
|---|---|
|
|
||||||
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
|
|
||||||
| Enter | 送出目前組字 |
|
|
||||||
| **Tab** | 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會**記住**這次修正 |
|
|
||||||
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字 |
|
|
||||||
| Esc | 取消組字 |
|
|
||||||
| Shift + 字母 | 強制英文 |
|
|
||||||
| `< > ? ! :` | 中文語境下自動轉成全形 `,。?!:` |
|
|
||||||
|
|
||||||
## 建置與安裝
|
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
|
||||||
|
|
||||||
需要 Xcode(使用 `/Applications/Xcode.app` 的工具鏈)。
|
## 個人資料
|
||||||
|
|
||||||
|
| | macOS | Windows |
|
||||||
|
|---|---|---|
|
||||||
|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` |
|
||||||
|
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` |
|
||||||
|
|
||||||
|
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
|
||||||
|
Mac 版可以在輸入法選單選「清除學習紀錄」。
|
||||||
|
|
||||||
|
## 開發
|
||||||
|
|
||||||
|
### macOS
|
||||||
|
|
||||||
|
需要 Xcode。腳本會自動使用 `/Applications/Xcode.app` 的工具鏈。
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
swift test # 執行單元測試(需 DEVELOPER_DIR 指向 Xcode)
|
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
|
||||||
scripts/build.sh # 產生 build/AutoZhuyin.app
|
xcrun swift test # 單元測試
|
||||||
scripts/install.sh # 安裝到 ~/Library/Input Methods 並重啟輸入法
|
scripts/build.sh # 產生 build/AutoZhuyin.app(universal)
|
||||||
|
scripts/install.sh # 裝到 ~/Library/Input Methods、重啟並向系統註冊
|
||||||
|
scripts/package.sh # 產生 build/AutoZhuyin-<版本>.pkg
|
||||||
```
|
```
|
||||||
|
|
||||||
第一次安裝後:系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
|
改完程式後執行 `scripts/install.sh`,再切換到其他輸入法後切回來,就會用到新版。
|
||||||
如果清單裡沒出現,登出再登入一次。
|
版本號在 `Resources/Info.plist` 的 `CFBundleShortVersionString`。
|
||||||
|
|
||||||
## 詞庫
|
### Windows
|
||||||
|
|
||||||
`Resources/lexicon.txt` 由開源的小麥注音原始資料轉換而來(MIT 授權,約 17 萬筆),
|
Windows 版建立在 [PIME](https://github.com/EasyIME/PIME) 上,核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣。不需要 Windows 也能開發和測試:
|
||||||
後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換只讀取資料檔,不執行小麥注音的程式碼。
|
|
||||||
要更新到小麥注音最新資料:
|
```bash
|
||||||
|
python3 windows/tools/build_data.py # 產生 windows/autozhuyin/data/
|
||||||
|
(cd windows && python3 -m unittest discover tests) # 核心測試 + 模擬 PIME 的外殼測試
|
||||||
|
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
|
||||||
|
```
|
||||||
|
|
||||||
|
- Windows 沒有內建英文字典,所以 `build_data.py` 會把 macOS 的 `/usr/share/dict/words`(韋氏字典,公有領域)轉成 `english-dict.txt` 一起打包,並預先算好 bigram 表。
|
||||||
|
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
|
||||||
|
- 版本號在 `windows/autozhuyin/ime.json`。
|
||||||
|
|
||||||
|
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
|
||||||
|
|
||||||
|
### 訓練判斷權重
|
||||||
|
|
||||||
|
中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法:
|
||||||
|
|
||||||
|
1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。
|
||||||
|
2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。
|
||||||
|
3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用)
|
||||||
|
python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘)
|
||||||
|
python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘
|
||||||
|
python3 training/train.py --eval-only # 只評估目前的權重
|
||||||
|
python3 training/make_parity.py # 權重或特徵改了之後都要重跑
|
||||||
|
```
|
||||||
|
|
||||||
|
目前的結果(10,000 筆測試資料):
|
||||||
|
|
||||||
|
| | 送出時整段判斷正確 | 打字過程中顯示正確 | 跟前一個字語言不同時 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 手調權重 | 97.84% | 88.19% | 96.2% |
|
||||||
|
| 訓練後 | **99.52%** | **91.49%** | **99.3%** |
|
||||||
|
|
||||||
|
訓練資料也模擬了「前一個字是中文或英文」和「按下空白(一聲)的那一刻」,所以「前一個字」的影響(`ctx.zh` / `ctx.lit`)也是訓練出來的,不是手調的。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
|
||||||
|
|
||||||
|
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
|
||||||
|
|
||||||
|
### 詞庫
|
||||||
|
|
||||||
|
`Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。
|
||||||
|
|
||||||
|
更新到小麥注音最新資料:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scripts/import-mcbopomofo.sh
|
scripts/import-mcbopomofo.sh
|
||||||
scripts/install.sh
|
|
||||||
```
|
```
|
||||||
|
|
||||||
自訂詞:`~/Library/Application Support/AutoZhuyin/user-phrases.txt`,格式同 `lexicon.txt`:
|
### 發布新版本
|
||||||
`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`
|
|
||||||
|
先 commit 所有修改,再執行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/release.sh # 修正版:0.1.0 → 0.1.1
|
||||||
|
scripts/release.sh minor # 新功能:0.1.1 → 0.2.0
|
||||||
|
scripts/release.sh major # 大改版:0.2.0 → 1.0.0
|
||||||
|
scripts/release.sh minor --push # 同時 push commit 和 tag
|
||||||
|
```
|
||||||
|
|
||||||
|
`release.sh` 會自動:
|
||||||
|
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
|
||||||
|
2. 跑 Swift 和 Python 全部測試
|
||||||
|
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip`
|
||||||
|
4. commit「Release v<版本>」並打上 `v<版本>` tag
|
||||||
|
|
||||||
|
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
|
||||||
|
|
||||||
|
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。
|
||||||
|
|
||||||
## 專案結構
|
## 專案結構
|
||||||
|
|
||||||
```
|
```
|
||||||
Sources/AutoZhuyinCore/ 純邏輯,可單元測試
|
Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試)
|
||||||
Bopomofo.swift 大千鍵盤對應、音節結構
|
Bopomofo.swift 大千鍵盤對應、音節結構
|
||||||
SyllableTable.swift 合法注音音節表
|
SyllableTable.swift 合法注音音節表
|
||||||
ZhuyinParser.swift 按鍵 → 音節
|
ZhuyinParser.swift 按鍵 → 音節(含順序錯誤修正)
|
||||||
Lexicon.swift 詞庫
|
Lexicon.swift 詞庫
|
||||||
Converter.swift 音節 → 中文(Viterbi)
|
Converter.swift 音節 → 中文(Viterbi)
|
||||||
EnglishModel.swift 英文判斷
|
EnglishModel.swift 英文判斷
|
||||||
Classifier.swift 中英判斷 + 學習
|
Features.swift 判斷用的特徵與預設權重
|
||||||
CompositionEngine.swift 輸入法狀態機
|
Classifier.swift 中英判斷(特徵 × 權重)、切分、學習
|
||||||
Sources/AutoZhuyin/ InputMethodKit 外殼
|
CompositionEngine.swift 輸入法狀態機
|
||||||
Tests/ 單元測試
|
Sources/AutoZhuyin/ macOS InputMethodKit 外殼
|
||||||
|
Tests/ Swift 單元測試
|
||||||
|
Resources/ Info.plist、詞庫、英文常用字、訓練好的權重
|
||||||
|
scripts/ 建置、安裝、打包、詞庫轉換
|
||||||
|
pkg/postinstall PKG 安裝後腳本
|
||||||
|
windows/
|
||||||
|
autozhuyin/ PIME 輸入法(整個資料夾會被複製到 PIME)
|
||||||
|
core/ 核心邏輯(Python 版)
|
||||||
|
autozhuyin_ime.py PIME 外殼
|
||||||
|
ime.json PIME 輸入法設定
|
||||||
|
installer/ install.bat、uninstall.bat、給使用者的 README.txt
|
||||||
|
tests/ Python 測試
|
||||||
|
tools/ 產生資料檔、圖示
|
||||||
|
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 已知限制
|
||||||
|
|
||||||
|
- Windows 版還沒在實機上測試過。
|
||||||
|
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
|
||||||
|
- 很短的字串本質上就有歧義。像 `up`(也是 ㄧㄣ 因)這種常用的,按空白後會先保留,看下一個詞再決定(`up jo4` = 因為、`up the` = up the);其他判斷錯的可以按 Tab 修正,修正後會記住。
|
||||||
|
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
|
||||||
|
|
||||||
## 下一步(Roadmap)
|
## 下一步(Roadmap)
|
||||||
|
|
||||||
- [ ] 匯入完整詞庫,並用真實打字資料調整判斷權重
|
- [x] 匯入完整詞庫
|
||||||
|
- [x] 注音順序錯誤修正
|
||||||
|
- [x] Windows 版
|
||||||
|
- [x] 用資料訓練判斷權重(合成資料)
|
||||||
|
- [ ] 加入真實打字資料一起訓練
|
||||||
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
|
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
|
||||||
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境
|
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境
|
||||||
- [ ] 使用者詞頻學習(選過的候選字優先)
|
- [ ] 使用者詞頻學習(選過的候選字優先)
|
||||||
- [ ] 偏好設定視窗
|
- [ ] 偏好設定視窗
|
||||||
|
|
||||||
## Windows 版
|
## 致謝與授權
|
||||||
|
|
||||||
`windows/` 是給 Windows 用的版本,建立在開源輸入法框架 [PIME](https://github.com/EasyIME/PIME) 上。
|
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
|
||||||
核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣;詞庫共用 `Resources/`。
|
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
|
||||||
|
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
|
||||||
```bash
|
|
||||||
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
|
|
||||||
```
|
|
||||||
|
|
||||||
zip 裡附有 `install.bat`、`uninstall.bat` 和給測試者看的 `README.txt`。使用者需要先安裝 PIME。
|
|
||||||
修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改。
|
|
||||||
|
|||||||
+14
-15
@@ -4,42 +4,41 @@
|
|||||||
<dict>
|
<dict>
|
||||||
<key>CFBundleDevelopmentRegion</key>
|
<key>CFBundleDevelopmentRegion</key>
|
||||||
<string>zh_TW</string>
|
<string>zh_TW</string>
|
||||||
|
<key>CFBundleDisplayName</key>
|
||||||
|
<string>自動注音</string>
|
||||||
<key>CFBundleExecutable</key>
|
<key>CFBundleExecutable</key>
|
||||||
<string>AutoZhuyin</string>
|
<string>AutoZhuyin</string>
|
||||||
<key>CFBundleIdentifier</key>
|
<key>CFBundleIdentifier</key>
|
||||||
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
|
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
|
||||||
<key>CFBundleName</key>
|
<key>CFBundleName</key>
|
||||||
<string>AutoZhuyin</string>
|
<string>AutoZhuyin</string>
|
||||||
<key>CFBundleDisplayName</key>
|
|
||||||
<string>自動注音</string>
|
|
||||||
<key>CFBundlePackageType</key>
|
<key>CFBundlePackageType</key>
|
||||||
<string>APPL</string>
|
<string>APPL</string>
|
||||||
<key>CFBundleShortVersionString</key>
|
<key>CFBundleShortVersionString</key>
|
||||||
<string>0.1.0</string>
|
<string>0.7.0</string>
|
||||||
<key>CFBundleVersion</key>
|
<key>CFBundleVersion</key>
|
||||||
<string>1</string>
|
<string>12</string>
|
||||||
<key>LSMinimumSystemVersion</key>
|
|
||||||
<string>13.0</string>
|
|
||||||
<key>LSBackgroundOnly</key>
|
|
||||||
<true/>
|
|
||||||
<key>LSUIElement</key>
|
|
||||||
<true/>
|
|
||||||
<key>NSPrincipalClass</key>
|
|
||||||
<string>NSApplication</string>
|
|
||||||
<key>InputMethodConnectionName</key>
|
<key>InputMethodConnectionName</key>
|
||||||
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
|
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
|
||||||
<key>InputMethodServerControllerClass</key>
|
<key>InputMethodServerControllerClass</key>
|
||||||
<string>AutoZhuyinInputController</string>
|
<string>AutoZhuyinInputController</string>
|
||||||
|
<key>LSBackgroundOnly</key>
|
||||||
|
<true/>
|
||||||
|
<key>LSMinimumSystemVersion</key>
|
||||||
|
<string>13.0</string>
|
||||||
|
<key>LSUIElement</key>
|
||||||
|
<true/>
|
||||||
|
<key>NSPrincipalClass</key>
|
||||||
|
<string>NSApplication</string>
|
||||||
<key>TISInputSourceID</key>
|
<key>TISInputSourceID</key>
|
||||||
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
|
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
|
||||||
<key>TISIntendedLanguage</key>
|
<key>TISIntendedLanguage</key>
|
||||||
<string>zh-Hant</string>
|
<string>zh-Hant</string>
|
||||||
<key>tsInputMethodIconFileKey</key>
|
|
||||||
<string>icon.tiff</string>
|
|
||||||
<key>tsInputMethodCharacterRepertoireKey</key>
|
<key>tsInputMethodCharacterRepertoireKey</key>
|
||||||
<array>
|
<array>
|
||||||
<string>Hant</string>
|
<string>Hant</string>
|
||||||
<string>Latn</string>
|
|
||||||
</array>
|
</array>
|
||||||
|
<key>tsInputMethodIconFileKey</key>
|
||||||
|
<string>icon.tiff</string>
|
||||||
</dict>
|
</dict>
|
||||||
</plist>
|
</plist>
|
||||||
|
|||||||
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"about": "AutoZhuyin classifier weights, learned by training/train.py",
|
||||||
|
"trained": "2026-11-11",
|
||||||
|
"examples": 40000,
|
||||||
|
"test_accuracy": {
|
||||||
|
"commit": 0.9952,
|
||||||
|
"typing": 0.9149
|
||||||
|
},
|
||||||
|
"weights": {
|
||||||
|
"ctx.lit": 0.0916,
|
||||||
|
"ctx.zh": 0.1349,
|
||||||
|
"lit.bias": 0.3982,
|
||||||
|
"lit.common": -0.2906,
|
||||||
|
"lit.common_len": 0.8046,
|
||||||
|
"lit.dict": -1.6662,
|
||||||
|
"lit.dict_len": 0.6441,
|
||||||
|
"lit.digits": -2.9689,
|
||||||
|
"lit.num_digits": -0.018,
|
||||||
|
"lit.num_others": -0.5554,
|
||||||
|
"lit.others": -2.0002,
|
||||||
|
"lit.prefix_common_len": 0.2831,
|
||||||
|
"lit.prefix_dict_len": 0.0299,
|
||||||
|
"lit.short": -0.2482,
|
||||||
|
"lit.unknown_score": 0.2479,
|
||||||
|
"lit.upper": 20.0,
|
||||||
|
"seg.switch": -1.7596,
|
||||||
|
"zh.bias": 1.3423,
|
||||||
|
"zh.lexscore": 0.2917,
|
||||||
|
"zh.pending_invalid": -5.0,
|
||||||
|
"zh.pending_prefix": -0.9162,
|
||||||
|
"zh.phrase": 0.4499,
|
||||||
|
"zh.reordered": -1.3079,
|
||||||
|
"zh.stray": -5.0,
|
||||||
|
"zh.toned_invalid": -5.0,
|
||||||
|
"zh.toned_valid_lex": -0.9018,
|
||||||
|
"zh.toned_valid_nolex": -1.6341,
|
||||||
|
"zh.unconfirmed": -0.7952,
|
||||||
|
"zh.untoned_mid_invalid": -5.0,
|
||||||
|
"zh.untoned_mid_valid": -2.7479
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,37 +11,61 @@ enum Shared {
|
|||||||
|
|
||||||
static let classifier: Classifier = {
|
static let classifier: Classifier = {
|
||||||
let bundle = Bundle.main
|
let bundle = Bundle.main
|
||||||
|
let start = Date()
|
||||||
let lexicon = Lexicon()
|
let lexicon = Lexicon()
|
||||||
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
|
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
|
||||||
let text = try? String(contentsOf: url, encoding: .utf8) {
|
let data = try? Data(contentsOf: url) {
|
||||||
lexicon.load(text)
|
lexicon.load(data: data)
|
||||||
}
|
}
|
||||||
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
|
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
|
||||||
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
|
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
|
||||||
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
|
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
|
||||||
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
|
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
|
||||||
}
|
}
|
||||||
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"))
|
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
|
||||||
|
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
|
||||||
|
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
|
||||||
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
|
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
|
||||||
let learned = LearnedPreferences(weights: saved) { weights in
|
let learned = LearnedPreferences(weights: saved) { weights in
|
||||||
UserDefaults.standard.set(weights, forKey: learnedKey)
|
UserDefaults.standard.set(weights, forKey: learnedKey)
|
||||||
}
|
}
|
||||||
return Classifier(lexicon: lexicon, english: english, learned: learned)
|
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
|
||||||
|
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
|
||||||
|
// Only needed for untoned syllables; build it off the startup path.
|
||||||
|
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
|
||||||
|
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
|
||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
@objc(AutoZhuyinInputController)
|
@objc(AutoZhuyinInputController)
|
||||||
final class InputController: IMKInputController {
|
final class InputController: IMKInputController {
|
||||||
private let engine = CompositionEngine(classifier: Shared.classifier)
|
private let engine = CompositionEngine(classifier: Shared.classifier)
|
||||||
|
private var shiftTap = ShiftTapDetector()
|
||||||
|
|
||||||
// MARK: - Key events
|
// MARK: - Key events
|
||||||
|
|
||||||
override func recognizedEvents(_ sender: Any!) -> Int {
|
override func recognizedEvents(_ sender: Any!) -> Int {
|
||||||
Int(NSEvent.EventTypeMask.keyDown.rawValue)
|
Int(NSEvent.EventTypeMask.keyDown.rawValue | NSEvent.EventTypeMask.flagsChanged.rawValue)
|
||||||
}
|
}
|
||||||
|
|
||||||
override func handle(_ event: NSEvent!, client sender: Any!) -> Bool {
|
override func handle(_ event: NSEvent!, client sender: Any!) -> Bool {
|
||||||
guard let event, event.type == .keyDown, let client = sender as? IMKTextInput else { return false }
|
guard let event, let client = sender as? IMKTextInput else { return false }
|
||||||
|
|
||||||
|
// Shift pressed and released alone: next word is English (or flip the current one).
|
||||||
|
if event.type == .flagsChanged {
|
||||||
|
let flags = event.modifierFlags
|
||||||
|
let tapped = shiftTap.modifiersChanged(
|
||||||
|
shift: flags.contains(.shift),
|
||||||
|
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
|
||||||
|
time: event.timestamp)
|
||||||
|
if tapped {
|
||||||
|
_ = engine.handle(.shiftTap)
|
||||||
|
updateComposition(client)
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
guard event.type == .keyDown else { return false }
|
||||||
|
shiftTap.keyPressed()
|
||||||
|
|
||||||
// Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first.
|
// Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first.
|
||||||
if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty {
|
if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty {
|
||||||
@@ -90,15 +114,19 @@ final class InputController: IMKInputController {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private func updateComposition(_ client: IMKTextInput) {
|
private func updateComposition(_ client: IMKTextInput) {
|
||||||
let text = engine.markedText
|
// Each segment is underlined; literal (English) segments are blue as a hint.
|
||||||
let attrs: [NSAttributedString.Key: Any] = [
|
let marked = NSMutableAttributedString()
|
||||||
.underlineStyle: NSUnderlineStyle.single.rawValue,
|
for piece in engine.markedPieces {
|
||||||
// Literal (English) composition is shown in a different colour as a hint.
|
marked.append(NSAttributedString(string: piece.text, attributes: [
|
||||||
.foregroundColor: engine.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
|
.underlineStyle: NSUnderlineStyle.single.rawValue,
|
||||||
]
|
.foregroundColor: piece.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
|
||||||
let marked = NSAttributedString(string: text, attributes: text.isEmpty ? [:] : attrs)
|
]))
|
||||||
|
}
|
||||||
|
let text = marked.string
|
||||||
|
// Caret at the candidate cursor (UTF-16 offset of the first `cursorOffset` characters).
|
||||||
|
let caret = String(text.prefix(engine.cursorOffset)).utf16.count
|
||||||
client.setMarkedText(marked,
|
client.setMarkedText(marked,
|
||||||
selectionRange: NSRange(location: (text as NSString).length, length: 0),
|
selectionRange: NSRange(location: caret, length: 0),
|
||||||
replacementRange: NSRange(location: NSNotFound, length: 0))
|
replacementRange: NSRange(location: NSNotFound, length: 0))
|
||||||
|
|
||||||
if engine.isShowingCandidates {
|
if engine.isShowingCandidates {
|
||||||
|
|||||||
@@ -7,6 +7,17 @@ public enum InputMode: String, Codable {
|
|||||||
case literal
|
case literal
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A run of keys (indices into the buffer) interpreted in one mode.
|
||||||
|
public struct Segment: Equatable {
|
||||||
|
public var range: Range<Int>
|
||||||
|
public var mode: InputMode
|
||||||
|
|
||||||
|
public init(range: Range<Int>, mode: InputMode) {
|
||||||
|
self.range = range
|
||||||
|
self.mode = mode
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public struct Classification {
|
public struct Classification {
|
||||||
public var mode: InputMode
|
public var mode: InputMode
|
||||||
public var zhuyinScore: Double
|
public var zhuyinScore: Double
|
||||||
@@ -39,82 +50,96 @@ public final class LearnedPreferences {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Decides whether a key sequence is Zhuyin or literal (English / numbers).
|
/// Decides whether a key sequence is Zhuyin or literal (English / numbers).
|
||||||
|
/// Every segment is scored as features · weights (see Features.swift).
|
||||||
public final class Classifier {
|
public final class Classifier {
|
||||||
public let lexicon: Lexicon
|
public let lexicon: Lexicon
|
||||||
public let english: EnglishModel
|
public let english: EnglishModel
|
||||||
public let learned: LearnedPreferences
|
public let learned: LearnedPreferences
|
||||||
|
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
|
||||||
|
public private(set) var weights: FeatureVector
|
||||||
|
|
||||||
/// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it.
|
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
|
||||||
public var zhuyinPrior = 0.5
|
weights: FeatureVector = [:]) {
|
||||||
/// How much the previous segment's mode pulls the current one.
|
|
||||||
public var contextWeight = 1.5
|
|
||||||
|
|
||||||
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) {
|
|
||||||
self.lexicon = lexicon
|
self.lexicon = lexicon
|
||||||
self.english = english
|
self.english = english
|
||||||
self.learned = learned
|
self.learned = learned
|
||||||
|
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// (Zhuyin features or nil, literal features or nil) for one segment. `previous` (mode of
|
||||||
|
/// the text committed just before) only affects the first segment.
|
||||||
|
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
|
||||||
|
previous: InputMode? = nil) -> (FeatureVector?, FeatureVector?) {
|
||||||
|
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther,
|
||||||
|
finishing: finishing, previous: previous),
|
||||||
|
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther,
|
||||||
|
finishing: finishing, previous: previous))
|
||||||
|
}
|
||||||
|
|
||||||
|
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
|
||||||
|
previous: InputMode?, whole: Bool) -> [Double] {
|
||||||
|
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther,
|
||||||
|
finishing: finishing, previous: previous)
|
||||||
|
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
|
||||||
|
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
|
||||||
|
if whole {
|
||||||
|
let b = learned.bias(for: String(seg))
|
||||||
|
if b > 0 { zh += b } else { lit -= b }
|
||||||
|
}
|
||||||
|
return [zh, lit]
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The whole buffer as one segment.
|
||||||
/// - Parameters:
|
/// - Parameters:
|
||||||
/// - finishing: the user pressed space. Zhuyin is scored as if the space were a
|
/// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as
|
||||||
/// first-tone key; literal is scored as a completed word.
|
/// first tone; literal is scored as a completed word.
|
||||||
/// - previous: mode of the previously committed segment, if recent.
|
/// - previous: mode of the previously committed segment, if recent.
|
||||||
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
|
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
|
||||||
var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior
|
let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true)
|
||||||
var lit = literalScore(keys, complete: finishing)
|
return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1])
|
||||||
switch previous {
|
|
||||||
case .zhuyin?: zh += contextWeight
|
|
||||||
case .literal?: lit += contextWeight
|
|
||||||
case nil: break
|
|
||||||
}
|
|
||||||
let learnedBias = learned.bias(for: String(keys))
|
|
||||||
if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias }
|
|
||||||
return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public func zhuyinScore(_ keys: [Character]) -> Double {
|
// MARK: - Segmentation
|
||||||
let parse = ZhuyinParser.parse(keys)
|
|
||||||
if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity }
|
/// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" →
|
||||||
var s = -5.0 * Double(parse.strayTones)
|
/// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the
|
||||||
for (i, p) in parse.syllables.enumerated() {
|
/// cost of each switch is the "seg.switch" feature.
|
||||||
let isLast = i == parse.syllables.count - 1
|
public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] {
|
||||||
// Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin.
|
let n = keys.count
|
||||||
if p.reordered { s -= 0.5 }
|
guard n > 0 else { return [] }
|
||||||
let valid = SyllableTable.isValid(p.syllable)
|
let modes: [InputMode] = [.zhuyin, .literal]
|
||||||
if p.toned {
|
// best[i][m]: best score for keys[0..<i] whose last segment has mode m.
|
||||||
if valid {
|
var best = Array(repeating: [Double](repeating: -.infinity, count: 2), count: n + 1)
|
||||||
s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5)
|
var back = Array(repeating: [Int](repeating: -1, count: 2), count: n + 1) // segment start j
|
||||||
} else {
|
|
||||||
s -= 5
|
for i in 1...n {
|
||||||
|
for j in 0..<i {
|
||||||
|
let s = scores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, finishing: finishing,
|
||||||
|
previous: j == 0 ? previous : nil, whole: j == 0 && i == n)
|
||||||
|
for m in 0..<2 where s[m] > -.infinity {
|
||||||
|
// Adjacent segments must alternate modes; same-mode neighbours are one segment.
|
||||||
|
if j > 0 && best[j][1 - m] == -.infinity { continue }
|
||||||
|
let total = j == 0 ? s[m] : best[j][1 - m] + s[m]
|
||||||
|
if total > best[i][m] {
|
||||||
|
best[i][m] = total
|
||||||
|
back[i][m] = j
|
||||||
|
}
|
||||||
}
|
}
|
||||||
} else if isLast {
|
|
||||||
s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5
|
|
||||||
} else {
|
|
||||||
// Untoned syllable in the middle: allowed (tone omission) but unusual.
|
|
||||||
s += valid ? -2.5 : -5
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
public func literalScore(_ keys: [Character], complete: Bool) -> Double {
|
guard best[n][0] > -.infinity || best[n][1] > -.infinity else {
|
||||||
if keys.isEmpty || keys.contains(" ") { return -.infinity }
|
return [Segment(range: 0..<n, mode: classify(keys, finishing: finishing, previous: previous).mode)]
|
||||||
var score = 0.0
|
|
||||||
if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English
|
|
||||||
|
|
||||||
let letters = String(keys.filter { $0.isLetter })
|
|
||||||
let digits = keys.filter { $0.isNumber }.count
|
|
||||||
let others = keys.count - letters.count - digits
|
|
||||||
|
|
||||||
if letters.isEmpty {
|
|
||||||
// Pure number like "2024" (with possible "." or ",").
|
|
||||||
return score + 0.8 * Double(digits) - 1.0 * Double(others)
|
|
||||||
}
|
}
|
||||||
score += english.score(letters, complete: complete || digits > 0)
|
var m = best[n][0] >= best[n][1] ? 0 : 1
|
||||||
score -= 1.5 * Double(digits)
|
var i = n
|
||||||
// `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not.
|
var out: [Segment] = []
|
||||||
let apostrophes = keys.filter { $0 == "'" }.count
|
while i > 0 {
|
||||||
score -= 1.5 * Double(others - apostrophes)
|
let j = back[i][m]
|
||||||
return score
|
out.append(Segment(range: j..<i, mode: modes[m]))
|
||||||
|
i = j
|
||||||
|
m = 1 - m
|
||||||
|
}
|
||||||
|
return out.reversed()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,8 @@ public enum KeyInput: Equatable {
|
|||||||
case character(Character)
|
case character(Character)
|
||||||
case backspace, enter, escape, tab
|
case backspace, enter, escape, tab
|
||||||
case up, down, left, right
|
case up, down, left, right
|
||||||
|
/// Shift pressed and released on its own.
|
||||||
|
case shiftTap
|
||||||
}
|
}
|
||||||
|
|
||||||
public struct KeyResult: Equatable {
|
public struct KeyResult: Equatable {
|
||||||
@@ -16,25 +18,48 @@ public struct KeyResult: Equatable {
|
|||||||
public static let consumed = KeyResult(handled: true, commit: nil)
|
public static let consumed = KeyResult(handled: true, commit: nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A piece of the composition as shown to the user.
|
||||||
|
public struct MarkedPiece: Equatable {
|
||||||
|
public var text: String
|
||||||
|
public var mode: InputMode
|
||||||
|
}
|
||||||
|
|
||||||
/// The whole input-method state machine, independent of InputMethodKit.
|
/// The whole input-method state machine, independent of InputMethodKit.
|
||||||
///
|
///
|
||||||
/// Keystrokes accumulate in a buffer. On every key the classifier decides whether the
|
/// Keystrokes accumulate in a buffer. On every key the classifier splits the buffer
|
||||||
/// buffer is Zhuyin or literal text; the composition (marked text) shows the result
|
/// into Zhuyin and literal segments (e.g. "ji3hello" → 我 + hello); the composition
|
||||||
/// live. Space / Enter / punctuation commit it. Tab flips the decision and the
|
/// (marked text) shows the result live. Space / Enter / punctuation commit it. Tab
|
||||||
/// correction is learned.
|
/// forces the whole buffer into the other mode and the correction is learned.
|
||||||
public final class CompositionEngine {
|
public final class CompositionEngine {
|
||||||
public let classifier: Classifier
|
public let classifier: Classifier
|
||||||
public let converter: Converter
|
public let converter: Converter
|
||||||
|
|
||||||
public private(set) var keys: [Character] = []
|
public private(set) var keys: [Character] = []
|
||||||
/// Set by Tab: overrides the classifier for the current buffer.
|
/// Set by Tab / a Shift tap while typing: overrides the classifier for the whole
|
||||||
|
/// buffer. It is a correction, learned on commit.
|
||||||
public private(set) var forcedMode: InputMode?
|
public private(set) var forcedMode: InputMode?
|
||||||
/// User-picked phrases, by syllable range.
|
/// English mode, toggled by a Shift tap when nothing is being typed. Keys go straight
|
||||||
public private(set) var fixed: [Range<Int>: String] = [:]
|
/// to the app (no composition, no underline) until Shift is tapped again.
|
||||||
|
public private(set) var isEnglishMode = false
|
||||||
|
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
|
||||||
|
private var fixed: [Int: [Range<Int>: String]] = [:]
|
||||||
public private(set) var candidates: [Candidate] = []
|
public private(set) var candidates: [Candidate] = []
|
||||||
|
private var candidateSegmentStart = 0
|
||||||
|
/// Composition cursor for choosing candidates: index into `cursorStops`, nil = at the end.
|
||||||
|
/// ← → move it; ↓ shows candidates for the phrase ending at the cursor.
|
||||||
|
private var cursor: Int?
|
||||||
public var candidateIndex = 0
|
public var candidateIndex = 0
|
||||||
public var isShowingCandidates: Bool { !candidates.isEmpty }
|
public var isShowingCandidates: Bool { !candidates.isEmpty }
|
||||||
|
|
||||||
|
/// Index of a held space: an English word that is also a common one-syllable Chinese
|
||||||
|
/// reading ("up " = ㄧㄣ 因) is not committed on space; the next word decides
|
||||||
|
/// ("up jo4" = 因為, "up the" = up the).
|
||||||
|
private var held: Int?
|
||||||
|
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
|
||||||
|
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
|
||||||
|
public private(set) var pendingBacktick = false
|
||||||
|
static let holdMinScore = -3.5
|
||||||
|
|
||||||
private var previousMode: InputMode?
|
private var previousMode: InputMode?
|
||||||
private var previousCommitTime = Date.distantPast
|
private var previousCommitTime = Date.distantPast
|
||||||
/// How long the previous segment's mode keeps influencing the next one.
|
/// How long the previous segment's mode keeps influencing the next one.
|
||||||
@@ -52,47 +77,131 @@ public final class CompositionEngine {
|
|||||||
now().timeIntervalSince(previousCommitTime) < contextWindow ? previousMode : nil
|
now().timeIntervalSince(previousCommitTime) < contextWindow ? previousMode : nil
|
||||||
}
|
}
|
||||||
|
|
||||||
public var mode: InputMode {
|
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
|
||||||
forcedMode ?? classifier.classify(keys, previous: context).mode
|
guard !keys.isEmpty else { return [] }
|
||||||
|
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
|
||||||
|
if let h = held, h < keys.count, keys[h] == " " {
|
||||||
|
return heldSegments(keys, held: h, finishing: finishing)
|
||||||
|
}
|
||||||
|
return classifier.segment(keys, finishing: finishing, previous: context)
|
||||||
}
|
}
|
||||||
|
|
||||||
public var parse: ZhuyinParse { ZhuyinParser.parse(keys) }
|
/// The held word is Chinese if the words typed after it start with Chinese.
|
||||||
|
private func heldSegments(_ keys: [Character], held h: Int, finishing: Bool) -> [Segment] {
|
||||||
|
let base = h + 1
|
||||||
|
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
|
||||||
|
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
|
||||||
|
}
|
||||||
|
if let first = rest.first, first.mode == .zhuyin {
|
||||||
|
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
|
||||||
|
}
|
||||||
|
return [Segment(range: 0..<base, mode: .literal)] + rest
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether keys + space also read as a common Chinese syllable (see `held`).
|
||||||
|
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
|
||||||
|
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
|
||||||
|
let p = ZhuyinParser.parse(keys + [" "])
|
||||||
|
guard p.pending == nil, p.syllables.count == 1 else { return false }
|
||||||
|
guard let best = classifier.lexicon.entries(for: p.syllables.map(\.syllable)).first else { return false }
|
||||||
|
return best.score >= CompositionEngine.holdMinScore
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Current split of the buffer into Zhuyin / literal runs.
|
||||||
|
public var segments: [Segment] { segments(of: keys) }
|
||||||
|
|
||||||
|
/// Mode of the last segment (what the user is typing right now).
|
||||||
|
public var mode: InputMode { segments.last?.mode ?? .zhuyin }
|
||||||
|
|
||||||
/// What to show underlined in the app while composing.
|
/// What to show underlined in the app while composing.
|
||||||
public var markedText: String {
|
public var markedText: String { markedPieces.map(\.text).joined() }
|
||||||
guard !keys.isEmpty else { return "" }
|
|
||||||
switch mode {
|
public var markedPieces: [MarkedPiece] {
|
||||||
case .literal:
|
let segs = segments
|
||||||
return String(keys)
|
var pieces = segs.enumerated().map { i, seg in
|
||||||
case .zhuyin:
|
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
|
||||||
let p = parse
|
|
||||||
let done = converter.convert(p.completed.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
|
|
||||||
return done + (p.pending?.syllable.reading ?? "")
|
|
||||||
}
|
}
|
||||||
|
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
|
||||||
|
return pieces
|
||||||
}
|
}
|
||||||
|
|
||||||
private func validFixed(_ p: ZhuyinParse) -> [Range<Int>: String] {
|
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
|
||||||
fixed.filter { $0.key.upperBound <= p.completed.count }
|
private func render(_ seg: Segment, final: Bool) -> String {
|
||||||
|
guard seg.mode == .zhuyin else { return String(keys[seg.range]) }
|
||||||
|
let (converted, pending) = conversion(of: seg, final: final)
|
||||||
|
return converted.map(\.text).joined() + pending
|
||||||
}
|
}
|
||||||
|
|
||||||
private func zhuyinCommitText() -> String {
|
/// Syllables of a Zhuyin segment that are converted to Chinese (all of them when
|
||||||
let p = parse
|
/// final, otherwise the finished ones).
|
||||||
// An unfinished trailing syllable is converted as untoned.
|
private func convertibleSyllables(_ seg: Segment, final: Bool) -> (syllables: [Syllable], pending: String) {
|
||||||
return converter.convert(p.syllables.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
|
let p = ZhuyinParser.parse(Array(keys[seg.range]))
|
||||||
|
if final { return (p.syllables.map(\.syllable), "") }
|
||||||
|
return (p.completed.map(\.syllable), p.pending?.syllable.reading ?? "")
|
||||||
|
}
|
||||||
|
|
||||||
|
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
|
||||||
|
let (syl, pending) = convertibleSyllables(seg, final: final)
|
||||||
|
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
|
||||||
|
return (converter.convert(syl, fixed: fixedHere), pending)
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Cursor
|
||||||
|
|
||||||
|
/// A place the cursor can stop: after the `local`-th syllable of Zhuyin segment `segIndex`.
|
||||||
|
private struct CursorStop { let segIndex: Int; let local: Int }
|
||||||
|
|
||||||
|
private var cursorStops: [CursorStop] {
|
||||||
|
let segs = segments
|
||||||
|
var out: [CursorStop] = []
|
||||||
|
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin {
|
||||||
|
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
|
||||||
|
for k in stride(from: 1, through: n, by: 1) { out.append(CursorStop(segIndex: i, local: k)) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cursor position in `markedText`, in Characters.
|
||||||
|
public var cursorOffset: Int {
|
||||||
|
let pieces = markedPieces
|
||||||
|
let total = pieces.reduce(0) { $0 + $1.text.count }
|
||||||
|
let stops = cursorStops
|
||||||
|
guard let c = cursor, stops.indices.contains(c) else { return total }
|
||||||
|
let stop = stops[c]
|
||||||
|
var offset = pieces[..<stop.segIndex].reduce(0) { $0 + $1.text.count }
|
||||||
|
let segs = segments
|
||||||
|
for conv in conversion(of: segs[stop.segIndex], final: stop.segIndex < segs.count - 1).segments {
|
||||||
|
if conv.range.upperBound <= stop.local {
|
||||||
|
offset += conv.text.count
|
||||||
|
} else if conv.range.lowerBound < stop.local {
|
||||||
|
offset += min(stop.local - conv.range.lowerBound, conv.text.count)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return offset
|
||||||
|
}
|
||||||
|
|
||||||
|
private func moveCursor(by delta: Int) {
|
||||||
|
let stops = cursorStops
|
||||||
|
guard !stops.isEmpty else { return }
|
||||||
|
let next = (cursor ?? stops.count - 1) + delta
|
||||||
|
cursor = next >= stops.count - 1 ? nil : max(0, next)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Key handling
|
// MARK: - Key handling
|
||||||
|
|
||||||
public func handle(_ key: KeyInput) -> KeyResult {
|
public func handle(_ key: KeyInput) -> KeyResult {
|
||||||
if isShowingCandidates { return handleCandidateKey(key) }
|
if isShowingCandidates { return handleCandidateKey(key) }
|
||||||
|
if pendingBacktick { return afterBacktick(key) }
|
||||||
|
|
||||||
switch key {
|
switch key {
|
||||||
case .character(let c):
|
case .character(let c):
|
||||||
return handleCharacter(c)
|
return handleCharacter(c)
|
||||||
case .backspace:
|
case .backspace:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
|
cursor = nil
|
||||||
keys.removeLast()
|
keys.removeLast()
|
||||||
fixed = validFixed(parse)
|
if let h = held, keys.count <= h { held = nil }
|
||||||
|
fixed = fixed.filter { $0.key < keys.count }
|
||||||
if keys.isEmpty { forcedMode = nil; fixed = [:] }
|
if keys.isEmpty { forcedMode = nil; fixed = [:] }
|
||||||
return .consumed
|
return .consumed
|
||||||
case .enter:
|
case .enter:
|
||||||
@@ -104,29 +213,64 @@ public final class CompositionEngine {
|
|||||||
return .consumed
|
return .consumed
|
||||||
case .tab:
|
case .tab:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
|
flip()
|
||||||
|
return .consumed
|
||||||
|
case .shiftTap:
|
||||||
|
if keys.isEmpty {
|
||||||
|
// Toggle English mode; it stays on until the next Shift tap.
|
||||||
|
isEnglishMode.toggle()
|
||||||
|
previousMode = isEnglishMode ? .literal : nil
|
||||||
|
previousCommitTime = now()
|
||||||
|
} else {
|
||||||
|
flip()
|
||||||
|
}
|
||||||
return .consumed
|
return .consumed
|
||||||
case .down:
|
case .down:
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
openCandidates()
|
openCandidates()
|
||||||
return .consumed
|
return .consumed
|
||||||
case .up, .left, .right:
|
case .left:
|
||||||
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
|
moveCursor(by: -1)
|
||||||
|
return .consumed
|
||||||
|
case .right:
|
||||||
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
|
moveCursor(by: 1)
|
||||||
|
return .consumed
|
||||||
|
case .up:
|
||||||
return keys.isEmpty ? .passThrough : .consumed
|
return keys.isEmpty ? .passThrough : .consumed
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Forces the whole buffer into the other mode (a correction that is learned).
|
||||||
|
private func flip() {
|
||||||
|
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
|
||||||
|
}
|
||||||
|
|
||||||
private func handleCharacter(_ c: Character) -> KeyResult {
|
private func handleCharacter(_ c: Character) -> KeyResult {
|
||||||
|
cursor = nil // typing always continues at the end
|
||||||
|
// English mode: every key goes to the app as typed.
|
||||||
|
if isEnglishMode && keys.isEmpty { return .passThrough }
|
||||||
|
if c == "`" {
|
||||||
|
pendingBacktick = true
|
||||||
|
return .consumed
|
||||||
|
}
|
||||||
if c == " " {
|
if c == " " {
|
||||||
guard !keys.isEmpty else { return .passThrough }
|
guard !keys.isEmpty else { return .passThrough }
|
||||||
let decided = forcedMode ?? classifier.classify(keys, finishing: true, previous: context).mode
|
let segs = segments(of: keys, finishing: true)
|
||||||
if decided == .literal {
|
if segs.last?.mode == .literal {
|
||||||
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + " ")
|
if held == nil, segs.count == 1, forcedMode == nil, isAmbiguousWord(keys) {
|
||||||
|
held = keys.count
|
||||||
|
keys.append(" ")
|
||||||
|
return .consumed
|
||||||
|
}
|
||||||
|
return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
|
||||||
}
|
}
|
||||||
if parse.pending != nil {
|
if parse(ofLast: segs)?.pending != nil {
|
||||||
keys.append(" ") // first tone
|
keys.append(" ") // first tone
|
||||||
return .consumed
|
return .consumed
|
||||||
}
|
}
|
||||||
return KeyResult(handled: true, commit: commitCurrent(as: .zhuyin))
|
return KeyResult(handled: true, commit: commitCurrent(segs))
|
||||||
}
|
}
|
||||||
|
|
||||||
let composable = DachenLayout.isZhuyinKey(c) || (c.isASCII && c.isLetter) || (c == "'" && !keys.isEmpty)
|
let composable = DachenLayout.isZhuyinKey(c) || (c.isASCII && c.isLetter) || (c == "'" && !keys.isEmpty)
|
||||||
@@ -134,9 +278,9 @@ public final class CompositionEngine {
|
|||||||
// ", . / ; -" are Zhuyin symbols (ㄝ ㄡ ㄥ ㄤ ㄦ) but act as punctuation after an
|
// ", . / ; -" are Zhuyin symbols (ㄝ ㄡ ㄥ ㄤ ㄦ) but act as punctuation after an
|
||||||
// English word. Decide with the key included: "2;" is ㄉㄤ (檔…), "hello," is English.
|
// English word. Decide with the key included: "2;" is ㄉㄤ (檔…), "hello," is English.
|
||||||
if !keys.isEmpty, DachenLayout.symbols[c] != nil, !c.isLetter, !c.isNumber, forcedMode != .zhuyin {
|
if !keys.isEmpty, DachenLayout.symbols[c] != nil, !c.isLetter, !c.isNumber, forcedMode != .zhuyin {
|
||||||
let withKey = forcedMode ?? classifier.classify(keys + [c], previous: context).mode
|
let withKey = segments(of: keys + [c])
|
||||||
if withKey == .literal && !keys.allSatisfy(\.isNumber) {
|
if withKey.last?.mode == .literal && !keys.allSatisfy(\.isNumber) {
|
||||||
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + String(c))
|
return KeyResult(handled: true, commit: commitCurrent() + String(c))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
keys.append(c)
|
keys.append(c)
|
||||||
@@ -151,25 +295,57 @@ public final class CompositionEngine {
|
|||||||
return KeyResult(handled: true, commit: committed + punct)
|
return KeyResult(handled: true, commit: committed + punct)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private func parse(ofLast segs: [Segment]) -> ZhuyinParse? {
|
||||||
|
guard let last = segs.last, last.mode == .zhuyin else { return nil }
|
||||||
|
return ZhuyinParser.parse(Array(keys[last.range]))
|
||||||
|
}
|
||||||
|
|
||||||
|
private func afterBacktick(_ key: KeyInput) -> KeyResult {
|
||||||
|
pendingBacktick = false
|
||||||
|
if key == .escape || key == .backspace { return .consumed }
|
||||||
|
let committed = keys.isEmpty ? "" : commitCurrent()
|
||||||
|
if case .character(let c) = key, let symbol = CompositionEngine.symbols[c] {
|
||||||
|
return KeyResult(handled: true, commit: committed + symbol)
|
||||||
|
}
|
||||||
|
// Not a symbol: the backtick was meant literally; then handle the key as usual.
|
||||||
|
var r = handle(key)
|
||||||
|
if !r.handled, case .character(let c) = key { r = KeyResult(handled: true, commit: String(c)) }
|
||||||
|
return KeyResult(handled: r.handled, commit: committed + "`" + (r.commit ?? ""))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// "`" + key → full-width symbol.
|
||||||
|
static let symbols: [Character: String] = [
|
||||||
|
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
|
||||||
|
"'": "、", "\\": "、", "\"": "「", "[": "「", "]": "」", "{": "『", "}": "』",
|
||||||
|
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
|
||||||
|
]
|
||||||
|
|
||||||
static let fullwidth: [Character: String] = [
|
static let fullwidth: [Character: String] = [
|
||||||
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
|
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
|
||||||
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
||||||
]
|
]
|
||||||
|
|
||||||
// MARK: - Candidates
|
// MARK: - Candidates (for the last Zhuyin segment)
|
||||||
|
|
||||||
|
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
|
||||||
private func openCandidates() {
|
private func openCandidates() {
|
||||||
let p = parse
|
let stops = cursorStops
|
||||||
guard mode == .zhuyin, !p.completed.isEmpty else { return }
|
guard !stops.isEmpty else { return }
|
||||||
candidates = converter.candidates(p.completed.map(\.syllable), endingAt: p.completed.count)
|
let stop = stops[min(cursor ?? stops.count - 1, stops.count - 1)]
|
||||||
|
let segs = segments
|
||||||
|
let seg = segs[stop.segIndex]
|
||||||
|
let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
|
||||||
|
candidates = converter.candidates(syl, endingAt: stop.local)
|
||||||
|
candidateSegmentStart = seg.range.lowerBound
|
||||||
candidateIndex = 0
|
candidateIndex = 0
|
||||||
}
|
}
|
||||||
|
|
||||||
public func selectCandidate(_ index: Int) {
|
public func selectCandidate(_ index: Int) {
|
||||||
guard candidates.indices.contains(index) else { return }
|
guard candidates.indices.contains(index) else { return }
|
||||||
let c = candidates[index]
|
let c = candidates[index]
|
||||||
fixed = fixed.filter { !$0.key.overlaps(c.range) }
|
var here = (fixed[candidateSegmentStart] ?? [:]).filter { !$0.key.overlaps(c.range) }
|
||||||
fixed[c.range] = c.text
|
here[c.range] = c.text
|
||||||
|
fixed[candidateSegmentStart] = here
|
||||||
candidates = []
|
candidates = []
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -197,13 +373,13 @@ public final class CompositionEngine {
|
|||||||
// MARK: - Commit
|
// MARK: - Commit
|
||||||
|
|
||||||
@discardableResult
|
@discardableResult
|
||||||
private func commitCurrent(as decided: InputMode? = nil) -> String {
|
private func commitCurrent(_ given: [Segment]? = nil) -> String {
|
||||||
let m = decided ?? mode
|
let segs = given ?? segments
|
||||||
let text = m == .literal ? String(keys) : zhuyinCommitText()
|
let text = segs.map { render($0, final: true) }.joined()
|
||||||
if let forced = forcedMode {
|
if let forced = forcedMode {
|
||||||
classifier.learned.record(String(keys), as: forced)
|
classifier.learned.record(String(keys), as: forced)
|
||||||
}
|
}
|
||||||
previousMode = m
|
previousMode = segs.last?.mode
|
||||||
previousCommitTime = now()
|
previousCommitTime = now()
|
||||||
reset()
|
reset()
|
||||||
return text
|
return text
|
||||||
@@ -216,6 +392,9 @@ public final class CompositionEngine {
|
|||||||
|
|
||||||
public func reset() {
|
public func reset() {
|
||||||
keys = []
|
keys = []
|
||||||
|
held = nil
|
||||||
|
pendingBacktick = false
|
||||||
|
cursor = nil
|
||||||
forcedMode = nil
|
forcedMode = nil
|
||||||
fixed = [:]
|
fixed = [:]
|
||||||
candidates = []
|
candidates = []
|
||||||
|
|||||||
@@ -8,33 +8,63 @@ import Foundation
|
|||||||
/// - a letter-bigram model trained on those words, for words not in either list.
|
/// - a letter-bigram model trained on those words, for words not in either list.
|
||||||
public final class EnglishModel {
|
public final class EnglishModel {
|
||||||
private var common = Set<String>()
|
private var common = Set<String>()
|
||||||
private var dictionary = Set<String>()
|
|
||||||
private var sortedWords: [String] = []
|
|
||||||
private var sortedCommon: [String] = []
|
private var sortedCommon: [String] = []
|
||||||
|
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
|
||||||
|
/// binary searches, so no 230k-entry Set has to be built at startup.
|
||||||
|
private var sortedDict: [String] = []
|
||||||
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
|
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
|
||||||
private var unseenBigram = -12.0
|
private var unseenBigram = -12.0
|
||||||
|
|
||||||
public init(commonWords: [String], dictionaryWords: [String] = []) {
|
/// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
|
||||||
|
public init(commonWords: [String], dictionaryWords: [String] = [],
|
||||||
|
bigram: (table: [String: Double], unseen: Double)? = nil) {
|
||||||
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
|
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
|
||||||
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
|
|
||||||
sortedCommon = common.sorted()
|
sortedCommon = common.sorted()
|
||||||
sortedWords = common.union(dictionary).sorted()
|
var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
|
||||||
trainBigram()
|
if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
|
||||||
|
dict = Array(Set(dict)).sorted()
|
||||||
|
}
|
||||||
|
sortedDict = dict
|
||||||
|
if let bigram {
|
||||||
|
self.bigram = bigram.table
|
||||||
|
unseenBigram = bigram.unseen
|
||||||
|
} else {
|
||||||
|
trainBigram()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Loads common words from a bundle resource file and the system dictionary if present.
|
/// Loads the bundled data files. Falls back to the system word list and to training
|
||||||
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) {
|
/// the bigram table when the precomputed files are missing.
|
||||||
|
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
|
||||||
func lines(_ url: URL?) -> [String] {
|
func lines(_ url: URL?) -> [String] {
|
||||||
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
|
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
|
||||||
return text.split(whereSeparator: \.isNewline)
|
return text.split(whereSeparator: \.isNewline)
|
||||||
.map { $0.trimmingCharacters(in: .whitespaces) }
|
.map { $0.trimmingCharacters(in: .whitespaces) }
|
||||||
.filter { !$0.hasPrefix("#") }
|
.filter { !$0.isEmpty && !$0.hasPrefix("#") }
|
||||||
}
|
}
|
||||||
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary))
|
var bigram: (table: [String: Double], unseen: Double)?
|
||||||
|
if let url = bigramURL, let data = try? Data(contentsOf: url),
|
||||||
|
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||||
|
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
|
||||||
|
bigram = (table, unseen)
|
||||||
|
}
|
||||||
|
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
|
||||||
|
?? URL(fileURLWithPath: "/usr/share/dict/words")
|
||||||
|
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
|
||||||
}
|
}
|
||||||
|
|
||||||
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
|
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
|
||||||
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) }
|
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
|
||||||
|
|
||||||
|
private func inDictionary(_ w: String) -> Bool {
|
||||||
|
let i = lowerBound(w, in: sortedDict)
|
||||||
|
return i < sortedDict.count && sortedDict[i] == w
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Some word strictly longer than `p` starts with it (common words or dictionary).
|
||||||
|
private func isPrefixOfWord(_ p: String) -> Bool {
|
||||||
|
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
|
||||||
|
}
|
||||||
|
|
||||||
/// - Parameter complete: true when the user ended the word (space / punctuation),
|
/// - Parameter complete: true when the user ended the word (space / punctuation),
|
||||||
/// false while still typing (prefixes of words get partial credit).
|
/// false while still typing (prefixes of words get partial credit).
|
||||||
@@ -43,16 +73,40 @@ public final class EnglishModel {
|
|||||||
let n = Double(w.count)
|
let n = Double(w.count)
|
||||||
guard !w.isEmpty else { return 0 }
|
guard !w.isEmpty else { return 0 }
|
||||||
if common.contains(w) { return 3 + n }
|
if common.contains(w) { return 3 + n }
|
||||||
if dictionary.contains(w) { return 1 + 0.6 * n }
|
if inDictionary(w) { return 1 + 0.6 * n }
|
||||||
if !complete {
|
if !complete {
|
||||||
if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
|
if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
|
||||||
if hasPrefix(w, in: sortedWords) { return 0.2 * n }
|
if isPrefixOfWord(w) { return 0.2 * n }
|
||||||
}
|
}
|
||||||
// Unknown word: judge by how English-like its letter sequence is.
|
// Unknown word: judge by how English-like its letter sequence is.
|
||||||
let plaus = plausibility(w)
|
let plaus = plausibility(w)
|
||||||
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Feature dict for the classifier (see Features.swift); `score` is their dot
|
||||||
|
/// product with the default weights.
|
||||||
|
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
|
||||||
|
let w = word.lowercased()
|
||||||
|
let n = Double(w.count)
|
||||||
|
var f: FeatureVector = [:]
|
||||||
|
if n <= 2 { f["lit.short"] = 1 }
|
||||||
|
if common.contains(w) {
|
||||||
|
f["lit.common"] = 1
|
||||||
|
f["lit.common_len"] = n
|
||||||
|
} else if inDictionary(w) {
|
||||||
|
f["lit.dict"] = 1
|
||||||
|
f["lit.dict_len"] = n
|
||||||
|
} else if !complete && hasPrefix(w, in: sortedCommon) {
|
||||||
|
f["lit.prefix_common_len"] = n
|
||||||
|
} else if !complete && isPrefixOfWord(w) {
|
||||||
|
f["lit.prefix_dict_len"] = n
|
||||||
|
} else {
|
||||||
|
let plaus = plausibility(w)
|
||||||
|
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
|
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
|
||||||
public func plausibility(_ w: String) -> Double {
|
public func plausibility(_ w: String) -> Double {
|
||||||
let chars = ["^"] + w.map(String.init) + ["$"]
|
let chars = ["^"] + w.map(String.init) + ["$"]
|
||||||
@@ -61,12 +115,17 @@ public final class EnglishModel {
|
|||||||
return total / Double(chars.count - 1)
|
return total / Double(chars.count - 1)
|
||||||
}
|
}
|
||||||
|
|
||||||
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
|
private func lowerBound(_ p: String, in words: [String]) -> Int {
|
||||||
var lo = 0, hi = words.count
|
var lo = 0, hi = words.count
|
||||||
while lo < hi {
|
while lo < hi {
|
||||||
let mid = (lo + hi) / 2
|
let mid = (lo + hi) / 2
|
||||||
if words[mid] < p { lo = mid + 1 } else { hi = mid }
|
if words[mid] < p { lo = mid + 1 } else { hi = mid }
|
||||||
}
|
}
|
||||||
|
return lo
|
||||||
|
}
|
||||||
|
|
||||||
|
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
|
||||||
|
var lo = lowerBound(p, in: words)
|
||||||
// words[lo] is the first word >= p; skip an exact match, then test the next one.
|
// words[lo] is the first word >= p; skip an exact match, then test the next one.
|
||||||
if lo < words.count && words[lo] == p { lo += 1 }
|
if lo < words.count && words[lo] == p { lo += 1 }
|
||||||
return lo < words.count && words[lo].hasPrefix(p)
|
return lo < words.count && words[lo].hasPrefix(p)
|
||||||
@@ -75,7 +134,7 @@ public final class EnglishModel {
|
|||||||
private func trainBigram() {
|
private func trainBigram() {
|
||||||
var pair: [String: Double] = [:]
|
var pair: [String: Double] = [:]
|
||||||
var first: [String: Double] = [:]
|
var first: [String: Double] = [:]
|
||||||
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
|
for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
|
||||||
let weight = common.contains(w) ? 5.0 : 1.0
|
let weight = common.contains(w) ? 5.0 : 1.0
|
||||||
let chars = ["^"] + w.map(String.init) + ["$"]
|
let chars = ["^"] + w.map(String.init) + ["$"]
|
||||||
for i in 1..<chars.count {
|
for i in 1..<chars.count {
|
||||||
|
|||||||
@@ -0,0 +1,151 @@
|
|||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode
|
||||||
|
/// is the dot product of its features with the weights. `defaultWeights` reproduces the
|
||||||
|
/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json
|
||||||
|
/// (training/train.py).
|
||||||
|
///
|
||||||
|
/// Python twin: windows/autozhuyin/core/features.py — both must produce identical
|
||||||
|
/// features (checked by ParityTests against training/parity-fixtures.json).
|
||||||
|
public typealias FeatureVector = [String: Double]
|
||||||
|
|
||||||
|
public enum Features {
|
||||||
|
public static let defaultWeights: FeatureVector = [
|
||||||
|
// Zhuyin
|
||||||
|
"zh.bias": 0.5,
|
||||||
|
"zh.stray": -5.0,
|
||||||
|
"zh.reordered": -0.5,
|
||||||
|
"zh.toned_valid_lex": 2.8,
|
||||||
|
"zh.toned_valid_nolex": 0.5,
|
||||||
|
"zh.toned_invalid": -5.0,
|
||||||
|
"zh.pending_prefix": 0.0,
|
||||||
|
"zh.pending_invalid": -5.0,
|
||||||
|
"zh.untoned_mid_valid": -2.5,
|
||||||
|
"zh.untoned_mid_invalid": -5.0,
|
||||||
|
"zh.unconfirmed": -2.0,
|
||||||
|
"zh.lexscore": 0.0,
|
||||||
|
"zh.phrase": 0.0,
|
||||||
|
// Literal
|
||||||
|
"lit.bias": 0.0,
|
||||||
|
"lit.upper": 20.0,
|
||||||
|
"lit.num_digits": 0.8,
|
||||||
|
"lit.num_others": -1.0,
|
||||||
|
"lit.common": 3.0,
|
||||||
|
"lit.common_len": 1.0,
|
||||||
|
"lit.dict": 1.0,
|
||||||
|
"lit.dict_len": 0.6,
|
||||||
|
"lit.prefix_common_len": 0.4,
|
||||||
|
"lit.prefix_dict_len": 0.2,
|
||||||
|
"lit.unknown_score": 1.0,
|
||||||
|
"lit.short": 0.0,
|
||||||
|
"lit.digits": -1.5,
|
||||||
|
"lit.others": -1.5,
|
||||||
|
// Segmentation
|
||||||
|
"seg.switch": -3.0,
|
||||||
|
// Context: the first segment has the same mode as the previously committed text
|
||||||
|
"ctx.zh": 1.5,
|
||||||
|
"ctx.lit": 1.5,
|
||||||
|
]
|
||||||
|
|
||||||
|
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
|
||||||
|
public static func loadWeights(from url: URL?) -> FeatureVector {
|
||||||
|
guard let url, let data = try? Data(contentsOf: url),
|
||||||
|
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||||
|
let w = obj["weights"] as? [String: Double] else { return [:] }
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double {
|
||||||
|
f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Features of `keys` read as Zhuyin, or nil if impossible.
|
||||||
|
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
|
||||||
|
afterOther: Bool = false, finishing: Bool = false,
|
||||||
|
previous: InputMode? = nil) -> FeatureVector? {
|
||||||
|
let parse = ZhuyinParser.parse(keys)
|
||||||
|
// A Zhuyin segment followed by another must end with a tone.
|
||||||
|
if !isLast && parse.pending != nil { return nil }
|
||||||
|
// Space finishes an untoned last syllable as first tone; after a toned syllable it
|
||||||
|
// just commits, so it must not count as an extra (stray) tone.
|
||||||
|
let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse
|
||||||
|
if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil }
|
||||||
|
|
||||||
|
var f: FeatureVector = ["zh.bias": 1]
|
||||||
|
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
|
||||||
|
if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) }
|
||||||
|
let last = scored.syllables.count - 1
|
||||||
|
var tonedRun: [Syllable] = []
|
||||||
|
for (i, p) in scored.syllables.enumerated() {
|
||||||
|
if p.reordered { add("zh.reordered") }
|
||||||
|
let valid = SyllableTable.isValid(p.syllable)
|
||||||
|
if p.toned {
|
||||||
|
if valid {
|
||||||
|
if let best = lexicon.bestScore(for: p.syllable) {
|
||||||
|
add("zh.toned_valid_lex")
|
||||||
|
add("zh.lexscore", best + 4)
|
||||||
|
} else {
|
||||||
|
add("zh.toned_valid_nolex")
|
||||||
|
}
|
||||||
|
tonedRun.append(p.syllable)
|
||||||
|
} else {
|
||||||
|
add("zh.toned_invalid")
|
||||||
|
}
|
||||||
|
} else if i == last {
|
||||||
|
add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid")
|
||||||
|
} else {
|
||||||
|
add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
|
||||||
|
var covered = [Bool](repeating: false, count: tonedRun.count)
|
||||||
|
for i in tonedRun.indices {
|
||||||
|
for n in 2...4 where i + n <= tonedRun.count {
|
||||||
|
if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty {
|
||||||
|
for k in i..<(i + n) { covered[k] = true }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let coveredCount = covered.filter { $0 }.count
|
||||||
|
if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) }
|
||||||
|
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
|
||||||
|
add("zh.unconfirmed")
|
||||||
|
}
|
||||||
|
if afterOther {
|
||||||
|
add("seg.switch")
|
||||||
|
} else if previous == .zhuyin {
|
||||||
|
add("ctx.zh")
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Features of `keys` output as typed, or nil if impossible.
|
||||||
|
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
|
||||||
|
afterOther: Bool = false, finishing: Bool = false,
|
||||||
|
previous: InputMode? = nil) -> FeatureVector? {
|
||||||
|
if keys.isEmpty || keys.contains(" ") { return nil }
|
||||||
|
let complete = !isLast || finishing
|
||||||
|
var f: FeatureVector = ["lit.bias": 1]
|
||||||
|
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
|
||||||
|
if afterOther {
|
||||||
|
add("seg.switch")
|
||||||
|
} else if previous == .literal {
|
||||||
|
add("ctx.lit")
|
||||||
|
}
|
||||||
|
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
|
||||||
|
|
||||||
|
let letters = String(keys.filter { $0.isLetter })
|
||||||
|
let digits = keys.filter { $0.isNumber }.count
|
||||||
|
let others = keys.count - letters.count - digits
|
||||||
|
if letters.isEmpty {
|
||||||
|
if digits > 0 { add("lit.num_digits", Double(digits)) }
|
||||||
|
if others > 0 { add("lit.num_others", Double(others)) }
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) }
|
||||||
|
if digits > 0 { add("lit.digits", Double(digits)) }
|
||||||
|
let nonApostrophe = others - keys.filter { $0 == "'" }.count
|
||||||
|
if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) }
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
|
|||||||
/// File format, one entry per line (`#` starts a comment):
|
/// File format, one entry per line (`#` starts a comment):
|
||||||
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
|
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
|
||||||
/// The phrase/reading columns may be in either order; the score is optional.
|
/// The phrase/reading columns may be in either order; the score is optional.
|
||||||
|
///
|
||||||
|
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
|
||||||
|
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
|
||||||
|
/// call `prepareTonelessIndex()` from a background queue to have it ready.
|
||||||
public final class Lexicon {
|
public final class Lexicon {
|
||||||
private var exact: [String: [LexiconEntry]] = [:]
|
private var exact: [String: [LexiconEntry]] = [:]
|
||||||
private var toneless: [String: [LexiconEntry]] = [:]
|
private var tonelessIndex: [String: [LexiconEntry]]?
|
||||||
|
private let lock = NSLock()
|
||||||
public private(set) var maxPhraseLength = 1
|
public private(set) var maxPhraseLength = 1
|
||||||
|
|
||||||
public init() {}
|
public init() {}
|
||||||
|
|
||||||
public convenience init(contentsOf url: URL) throws {
|
public convenience init(contentsOf url: URL) throws {
|
||||||
self.init()
|
self.init()
|
||||||
load(try String(contentsOf: url, encoding: .utf8))
|
load(data: try Data(contentsOf: url))
|
||||||
}
|
}
|
||||||
|
|
||||||
public func load(_ text: String) {
|
public func load(_ text: String) {
|
||||||
for line in text.split(whereSeparator: \.isNewline) {
|
load(data: Data(text.utf8))
|
||||||
let trimmed = line.trimmingCharacters(in: .whitespaces)
|
|
||||||
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
|
|
||||||
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
|
|
||||||
guard cols.count >= 2 else { continue }
|
|
||||||
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
|
|
||||||
guard Lexicon.isBopomofo(reading) else { continue }
|
|
||||||
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
|
|
||||||
add(reading: reading, phrase: phrase, score: score)
|
|
||||||
}
|
|
||||||
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
|
|
||||||
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public func add(reading rawReading: String, phrase: String, score: Double) {
|
public func load(data: Data) {
|
||||||
let entry = LexiconEntry(phrase: phrase, score: score)
|
lock.lock()
|
||||||
|
defer { lock.unlock() }
|
||||||
|
exact.reserveCapacity(exact.count + data.count / 30)
|
||||||
|
data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
|
||||||
|
let bytes = raw.bindMemory(to: UInt8.self)
|
||||||
|
let n = bytes.count
|
||||||
|
var i = 0
|
||||||
|
while i < n {
|
||||||
|
var end = i
|
||||||
|
while end < n && bytes[end] != 0x0A { end += 1 }
|
||||||
|
parseLine(bytes, i, end)
|
||||||
|
i = end + 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (key, entries) in exact where entries.count > 1 {
|
||||||
|
exact[key] = entries.sorted { $0.score > $1.score }
|
||||||
|
}
|
||||||
|
tonelessIndex = nil // rebuilt on demand
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Splits one line into up to three space/tab separated fields.
|
||||||
|
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
|
||||||
|
var fields: [Range<Int>] = []
|
||||||
|
var i = start
|
||||||
|
while i < end && fields.count < 3 {
|
||||||
|
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
|
||||||
|
if i >= end { break }
|
||||||
|
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
|
||||||
|
let s = i
|
||||||
|
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
|
||||||
|
fields.append(s..<i)
|
||||||
|
}
|
||||||
|
guard fields.count >= 2 else { return }
|
||||||
|
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
|
||||||
|
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
|
||||||
|
guard Lexicon.isBopomofo(b, r) else { return }
|
||||||
|
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
|
||||||
|
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
|
||||||
|
var prefixNeutral = b[r.lowerBound] == 0xCB
|
||||||
|
var k = r.lowerBound
|
||||||
|
while !prefixNeutral && k + 2 < r.upperBound {
|
||||||
|
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
|
||||||
|
k += 1
|
||||||
|
}
|
||||||
|
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parses "-3.9151"-style decimals without creating a String.
|
||||||
|
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
|
||||||
|
var i = r.lowerBound
|
||||||
|
var sign = 1.0
|
||||||
|
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
|
||||||
|
var value = 0.0, scale = 1.0, digits = 0, fraction = false
|
||||||
|
while i < r.upperBound {
|
||||||
|
let c = b[i]
|
||||||
|
if c >= 0x30 && c <= 0x39 {
|
||||||
|
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
|
||||||
|
digits += 1
|
||||||
|
} else if c == 0x2E && !fraction {
|
||||||
|
fraction = true
|
||||||
|
} else {
|
||||||
|
return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
|
||||||
|
}
|
||||||
|
i += 1
|
||||||
|
}
|
||||||
|
return digits > 0 ? sign * value : nil
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Adds one entry (call `load` for bulk data; this does not re-sort).
|
||||||
|
public func add(reading: String, phrase: String, score: Double) {
|
||||||
|
lock.lock()
|
||||||
|
defer { lock.unlock() }
|
||||||
|
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
|
||||||
|
exact[key]?.sort { $0.score > $1.score }
|
||||||
|
tonelessIndex = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
@discardableResult
|
||||||
|
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
|
||||||
|
var reading = rawReading
|
||||||
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
|
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
|
||||||
let syllables = rawReading.split(separator: "-").map { s -> String in
|
if normalise {
|
||||||
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
|
reading = reading.split(separator: "-").map { s -> String in
|
||||||
|
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
|
||||||
|
}.joined(separator: "-")
|
||||||
}
|
}
|
||||||
let reading = syllables.joined(separator: "-")
|
maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
|
||||||
maxPhraseLength = max(maxPhraseLength, syllables.count)
|
if exact[reading] == nil {
|
||||||
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) {
|
exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
|
||||||
exact[reading, default: []].append(entry)
|
} else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
|
||||||
|
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
|
||||||
}
|
}
|
||||||
let key = Lexicon.stripTones(reading)
|
return reading
|
||||||
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) {
|
}
|
||||||
toneless[key]![i].score = max(toneless[key]![i].score, score)
|
|
||||||
} else {
|
/// Builds the index that ignores tones (for untoned syllables). Safe to call from a
|
||||||
toneless[key, default: []].append(entry)
|
/// background queue; lookups build it themselves if it is not ready yet.
|
||||||
|
public func prepareTonelessIndex() {
|
||||||
|
lock.lock()
|
||||||
|
defer { lock.unlock() }
|
||||||
|
_ = buildTonelessIfNeeded()
|
||||||
|
}
|
||||||
|
|
||||||
|
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
|
||||||
|
if let index = tonelessIndex { return index }
|
||||||
|
var best: [String: [String: Double]] = [:]
|
||||||
|
for (reading, entries) in exact {
|
||||||
|
let key = Lexicon.stripTones(reading)
|
||||||
|
for e in entries {
|
||||||
|
if let s = best[key]?[e.phrase], s >= e.score { continue }
|
||||||
|
best[key, default: [:]][e.phrase] = e.score
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
var index: [String: [LexiconEntry]] = [:]
|
||||||
|
index.reserveCapacity(best.count)
|
||||||
|
for (key, phrases) in best {
|
||||||
|
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
|
||||||
|
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
|
||||||
|
}
|
||||||
|
tonelessIndex = index
|
||||||
|
return index
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Entries for a sequence of readings. Untoned syllables match any tone.
|
/// Entries for a sequence of readings. Untoned syllables match any tone.
|
||||||
public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
|
public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
|
||||||
|
lock.lock()
|
||||||
|
defer { lock.unlock() }
|
||||||
if syllables.allSatisfy({ $0.tone != nil }) {
|
if syllables.allSatisfy({ $0.tone != nil }) {
|
||||||
return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
|
return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
|
||||||
}
|
}
|
||||||
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? []
|
return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
|
||||||
}
|
}
|
||||||
|
|
||||||
public func bestScore(for syllable: Syllable) -> Double? {
|
public func bestScore(for syllable: Syllable) -> Double? {
|
||||||
@@ -81,4 +182,13 @@ public final class Lexicon {
|
|||||||
guard let first = s.unicodeScalars.first else { return false }
|
guard let first = s.unicodeScalars.first else { return false }
|
||||||
return (0x3105...0x312F).contains(first.value) || first == "˙"
|
return (0x3105...0x312F).contains(first.value) || first == "˙"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
|
||||||
|
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
|
||||||
|
guard r.count >= 2 else { return false }
|
||||||
|
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
|
||||||
|
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
|
||||||
|
let c = b[r.lowerBound + 2]
|
||||||
|
return c >= 0x85 && c <= 0xAF
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// Detects Shift pressed and released on its own (no other key in between, released
|
||||||
|
/// within `maxDuration`), as opposed to Shift used for capitals or shortcuts.
|
||||||
|
///
|
||||||
|
/// Python twin: windows/autozhuyin/core/shift_tap.py
|
||||||
|
public struct ShiftTapDetector {
|
||||||
|
public var maxDuration: TimeInterval = 0.5
|
||||||
|
private var pressedAt: TimeInterval?
|
||||||
|
private var used = false
|
||||||
|
|
||||||
|
public init() {}
|
||||||
|
|
||||||
|
/// Call when the modifier state changes. Returns true when a Shift tap completed.
|
||||||
|
public mutating func modifiersChanged(shift: Bool, otherModifiers: Bool, time: TimeInterval) -> Bool {
|
||||||
|
if shift && !otherModifiers {
|
||||||
|
if pressedAt == nil {
|
||||||
|
pressedAt = time
|
||||||
|
used = false
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
defer { pressedAt = nil }
|
||||||
|
guard let start = pressedAt, !shift, !otherModifiers, !used else { return false }
|
||||||
|
return time - start <= maxDuration
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Call for every other key press: Shift was used as a modifier, not tapped.
|
||||||
|
public mutating func keyPressed() {
|
||||||
|
if pressedAt != nil { used = true }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,7 +6,11 @@ private let resources = URL(fileURLWithPath: #filePath)
|
|||||||
.appendingPathComponent("Resources")
|
.appendingPathComponent("Resources")
|
||||||
|
|
||||||
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
|
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
|
||||||
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"))
|
private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
|
||||||
|
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
|
||||||
|
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
|
||||||
|
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
|
||||||
|
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
|
||||||
|
|
||||||
final class ParserTests: XCTestCase {
|
final class ParserTests: XCTestCase {
|
||||||
func testParsesNiHao() {
|
func testParsesNiHao() {
|
||||||
@@ -84,7 +88,7 @@ final class ConverterTests: XCTestCase {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final class ClassifierTests: XCTestCase {
|
final class ClassifierTests: XCTestCase {
|
||||||
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish)
|
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
|
||||||
|
|
||||||
func mode(_ s: String, finishing: Bool = false) -> InputMode {
|
func mode(_ s: String, finishing: Bool = false) -> InputMode {
|
||||||
classifier.classify(Array(s), finishing: finishing).mode
|
classifier.classify(Array(s), finishing: finishing).mode
|
||||||
@@ -106,7 +110,7 @@ final class ClassifierTests: XCTestCase {
|
|||||||
|
|
||||||
func testLearningFlipsDecision() {
|
func testLearningFlipsDecision() {
|
||||||
let learned = LearnedPreferences()
|
let learned = LearnedPreferences()
|
||||||
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned)
|
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights)
|
||||||
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
|
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
|
||||||
learned.record("so", as: .zhuyin)
|
learned.record("so", as: .zhuyin)
|
||||||
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
|
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
|
||||||
@@ -115,7 +119,7 @@ final class ClassifierTests: XCTestCase {
|
|||||||
|
|
||||||
final class EngineTests: XCTestCase {
|
final class EngineTests: XCTestCase {
|
||||||
func makeEngine() -> CompositionEngine {
|
func makeEngine() -> CompositionEngine {
|
||||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Types a string and returns everything committed.
|
/// Types a string and returns everything committed.
|
||||||
@@ -188,7 +192,7 @@ final class EngineTests: XCTestCase {
|
|||||||
|
|
||||||
final class DigitInitialTests: XCTestCase {
|
final class DigitInitialTests: XCTestCase {
|
||||||
func engine() -> CompositionEngine {
|
func engine() -> CompositionEngine {
|
||||||
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
}
|
}
|
||||||
|
|
||||||
func run(_ s: String) -> (out: String, marked: String) {
|
func run(_ s: String) -> (out: String, marked: String) {
|
||||||
@@ -219,3 +223,291 @@ final class DigitInitialTests: XCTestCase {
|
|||||||
XCTAssertEqual(run("hello, ").out, "hello, ")
|
XCTAssertEqual(run("hello, ").out, "hello, ")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
final class MixedSegmentTests: XCTestCase {
|
||||||
|
func run(_ s: String) -> (out: String, marked: String) {
|
||||||
|
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let r = e.handle(c == "\n" ? .enter : .character(c))
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return (out, e.markedText)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testChineseThenEnglish() {
|
||||||
|
XCTAssertEqual(run("ji3hello ").out, "我hello ")
|
||||||
|
XCTAssertEqual(run("su3cl3github\n").out, "你好github")
|
||||||
|
XCTAssertEqual(run("ji3hel").marked, "我hel")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testEnglishThenChinese() {
|
||||||
|
XCTAssertEqual(run("hellosu3cl3 ").out, "hello你好")
|
||||||
|
XCTAssertEqual(run("oksu3cl3\n").out, "ok你好")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testThreeSegments() {
|
||||||
|
XCTAssertEqual(run("ji3emailsu3cl3\n").out, "我email你好")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testSingleLanguageUnchanged() {
|
||||||
|
XCTAssertEqual(run("rup wu0 wu0 fu4cp3cl3\n").out, "今天天氣很好")
|
||||||
|
XCTAssertEqual(run("helloworld\n").out, "helloworld")
|
||||||
|
XCTAssertEqual(run("meeting ").out, "meeting ")
|
||||||
|
XCTAssertEqual(run("183\n").out.count, 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py).
|
||||||
|
final class ParityTests: XCTestCase {
|
||||||
|
func testMatchesPythonSegmentation() throws {
|
||||||
|
let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json")
|
||||||
|
let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]]
|
||||||
|
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
|
||||||
|
var mismatches: [String] = []
|
||||||
|
for fx in fixtures {
|
||||||
|
let keys = Array(fx["keys"] as! String)
|
||||||
|
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
|
||||||
|
let previous = (fx["previous"] as? String).flatMap(InputMode.init(rawValue:))
|
||||||
|
let got = c.segment(keys, finishing: fx["finishing"] as! Bool, previous: previous).map {
|
||||||
|
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
|
||||||
|
}
|
||||||
|
if got != expected {
|
||||||
|
mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!) previous=\(String(describing: previous)): \(got) vs \(expected)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
XCTAssertGreaterThan(fixtures.count, 300)
|
||||||
|
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func testToneThenSpaceStaysChinese() {
|
||||||
|
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
var out = ""
|
||||||
|
for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } }
|
||||||
|
XCTAssertEqual(out.count, 1, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class ShiftTapTests: XCTestCase {
|
||||||
|
func engine() -> CompositionEngine {
|
||||||
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let key: KeyInput = c == "⇧" ? .shiftTap : c == "\n" ? .enter : .character(c)
|
||||||
|
let r = e.handle(key)
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func testShiftTapTogglesEnglishMode() {
|
||||||
|
let e = engine()
|
||||||
|
// "ru" alone would be ㄐㄧ (機); in English mode keys pass straight through, lowercase.
|
||||||
|
XCTAssertEqual(type("⇧ru su ", into: e), "ru su ")
|
||||||
|
XCTAssertTrue(e.isEnglishMode) // still English after spaces
|
||||||
|
XCTAssertEqual(e.markedText, "") // never composed: no underline
|
||||||
|
XCTAssertEqual(type("\n", into: e), "\n") // Enter goes to the app, mode stays
|
||||||
|
XCTAssertTrue(e.isEnglishMode)
|
||||||
|
XCTAssertEqual(type("⇧su3cl3\n", into: e), "你好") // tap again: automatic again
|
||||||
|
XCTAssertFalse(e.isEnglishMode)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testShiftTapTwiceCancels() {
|
||||||
|
XCTAssertEqual(type("⇧⇧su3cl3\n", into: engine()), "你好")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testShiftTapMidWordFlips() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("so", into: e)
|
||||||
|
XCTAssertEqual(e.mode, .literal)
|
||||||
|
_ = e.handle(.shiftTap)
|
||||||
|
XCTAssertEqual(e.mode, .zhuyin)
|
||||||
|
XCTAssertFalse(e.isEnglishMode) // a flip, not English mode
|
||||||
|
_ = type("\n", into: e)
|
||||||
|
XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned
|
||||||
|
}
|
||||||
|
|
||||||
|
func testEnglishModeIsNotLearned() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("⇧hello ", into: e)
|
||||||
|
XCTAssertTrue(e.classifier.learned.weights.isEmpty)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testBackspaceInEnglishModeGoesToApp() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("⇧", into: e)
|
||||||
|
XCTAssertFalse(e.handle(.backspace).handled)
|
||||||
|
XCTAssertTrue(e.isEnglishMode)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testDetector() {
|
||||||
|
var d = ShiftTapDetector()
|
||||||
|
XCTAssertFalse(d.modifiersChanged(shift: true, otherModifiers: false, time: 0))
|
||||||
|
XCTAssertTrue(d.modifiersChanged(shift: false, otherModifiers: false, time: 0.1))
|
||||||
|
// Shift used for a capital letter: not a tap.
|
||||||
|
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 1)
|
||||||
|
d.keyPressed()
|
||||||
|
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 1.1))
|
||||||
|
// Held too long: not a tap.
|
||||||
|
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 2)
|
||||||
|
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 3))
|
||||||
|
// Shift + Command: not a tap.
|
||||||
|
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 4)
|
||||||
|
_ = d.modifiersChanged(shift: true, otherModifiers: true, time: 4.1)
|
||||||
|
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 4.2))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class CursorTests: XCTestCase {
|
||||||
|
func engine() -> CompositionEngine {
|
||||||
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter : .character(c)
|
||||||
|
let r = e.handle(key)
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCursorMovesAndClamps() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup wu0 wu0 fu4", into: e) // 今天天氣
|
||||||
|
XCTAssertEqual(e.cursorOffset, 4)
|
||||||
|
_ = type("←", into: e)
|
||||||
|
XCTAssertEqual(e.cursorOffset, 3)
|
||||||
|
_ = type("←←←←", into: e)
|
||||||
|
XCTAssertEqual(e.cursorOffset, 1) // stops after the first character
|
||||||
|
_ = type("→→→→", into: e)
|
||||||
|
XCTAssertEqual(e.cursorOffset, 4)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testChooseCandidateInTheMiddle() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup wu0 wu0 fu4", into: e)
|
||||||
|
_ = type("←←↓", into: e) // cursor after the 2nd syllable (天)
|
||||||
|
XCTAssertTrue(e.isShowingCandidates)
|
||||||
|
XCTAssertTrue(e.candidates.allSatisfy { $0.range.upperBound == 2 })
|
||||||
|
let i = e.candidates.firstIndex { $0.text == "添" }!
|
||||||
|
e.selectCandidate(i)
|
||||||
|
XCTAssertEqual(type("\n", into: e), "今添天氣")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testTypingReturnsCursorToEnd() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("rup wu0 ←", into: e)
|
||||||
|
XCTAssertEqual(e.cursorOffset, 1)
|
||||||
|
_ = type("wu0 ", into: e)
|
||||||
|
XCTAssertEqual(e.cursorOffset, e.markedText.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCandidatesForChineseBeforeEnglish() {
|
||||||
|
let e = engine()
|
||||||
|
_ = type("su3cl3hello", into: e) // 你好hello
|
||||||
|
_ = type("↓", into: e) // last Chinese syllable, not the English word
|
||||||
|
XCTAssertTrue(e.isShowingCandidates)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class FirstToneAfterEnglishTests: XCTestCase {
|
||||||
|
/// After an English word, a first-tone syllable + space must stay Chinese (機, 吃, 書…).
|
||||||
|
func testFirstToneAfterEnglish() {
|
||||||
|
for keys in ["ru", "t", "gj", "vup", "tj", "el"] {
|
||||||
|
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
var out = ""
|
||||||
|
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
|
||||||
|
XCTAssertEqual(out, "hello ", keys)
|
||||||
|
XCTAssertEqual(e.markedText.count, 1, keys) // still composing one Chinese character
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class HeldWordTests: XCTestCase {
|
||||||
|
func makeEngine() -> CompositionEngine {
|
||||||
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let key: KeyInput = c == "\n" ? .enter : .character(c)
|
||||||
|
let r = e.handle(key)
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// "up " is also ㄧㄣ (因): the next word decides.
|
||||||
|
func testNextWordDecides() {
|
||||||
|
let cases = ["up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
|
||||||
|
"up\n": "up", "i love ": "i love "]
|
||||||
|
for (keys, want) in cases {
|
||||||
|
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testHeldWordStaysComposing() {
|
||||||
|
let e = makeEngine()
|
||||||
|
XCTAssertEqual(type("up ", into: e), "")
|
||||||
|
XCTAssertEqual(e.markedText, "up ")
|
||||||
|
_ = e.handle(.backspace)
|
||||||
|
XCTAssertEqual(e.markedText, "up")
|
||||||
|
XCTAssertEqual(type(" the ", into: e), "up the ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class BacktickSymbolTests: XCTestCase {
|
||||||
|
func makeEngine() -> CompositionEngine {
|
||||||
|
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
|
||||||
|
}
|
||||||
|
|
||||||
|
func type(_ s: String, into e: CompositionEngine) -> String {
|
||||||
|
var out = ""
|
||||||
|
for c in s {
|
||||||
|
let key: KeyInput = c == "\n" ? .enter : .character(c)
|
||||||
|
let r = e.handle(key)
|
||||||
|
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func testForcedSymbols() {
|
||||||
|
let cases = ["`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
|
||||||
|
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"]
|
||||||
|
for (keys, want) in cases {
|
||||||
|
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testPlainBacktick() {
|
||||||
|
let e = makeEngine()
|
||||||
|
XCTAssertEqual(type("`", into: e), "")
|
||||||
|
XCTAssertEqual(e.markedText, "`")
|
||||||
|
XCTAssertEqual(type("ls` ", into: e), "`ls` ")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testBackspaceCancels() {
|
||||||
|
let e = makeEngine()
|
||||||
|
_ = type("su3`", into: e)
|
||||||
|
_ = e.handle(.backspace)
|
||||||
|
XCTAssertEqual(e.markedText, "你")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
final class StartupTimingTests: XCTestCase {
|
||||||
|
func testStartupTiming() throws {
|
||||||
|
var t = Date()
|
||||||
|
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
|
||||||
|
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
|
||||||
|
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
|
||||||
|
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
|
||||||
|
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
|
||||||
|
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+4
-1
@@ -18,7 +18,10 @@ rm -rf "$APP"
|
|||||||
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
||||||
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
|
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
|
||||||
cp Resources/Info.plist "$APP/Contents/Info.plist"
|
cp Resources/Info.plist "$APP/Contents/Info.plist"
|
||||||
cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/"
|
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
|
||||||
|
# Precomputed English dictionary and bigram table (shared with the Windows build).
|
||||||
|
python3 -I windows/tools/build_data.py >/dev/null
|
||||||
|
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
|
||||||
|
|
||||||
if [ ! -f build/icon.tiff ]; then
|
if [ ! -f build/icon.tiff ]; then
|
||||||
xcrun swift scripts/make-icon.swift build/icon.tiff
|
xcrun swift scripts/make-icon.swift build/icon.tiff
|
||||||
|
|||||||
Executable
+32
@@ -0,0 +1,32 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Increments the version in every place it lives:
|
||||||
|
# Resources/Info.plist CFBundleShortVersionString (x.y.z) and CFBundleVersion (build number)
|
||||||
|
# windows/autozhuyin/ime.json "version"
|
||||||
|
#
|
||||||
|
# Usage: scripts/bump-version.sh [patch|minor|major] (default: patch)
|
||||||
|
# Prints the new version.
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
LEVEL="${1:-patch}"
|
||||||
|
PLIST=Resources/Info.plist
|
||||||
|
IME=windows/autozhuyin/ime.json
|
||||||
|
|
||||||
|
CURRENT="$(plutil -extract CFBundleShortVersionString raw "$PLIST")"
|
||||||
|
IFS=. read -r MAJOR MINOR PATCH <<< "$CURRENT"
|
||||||
|
case "$LEVEL" in
|
||||||
|
major) MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0 ;;
|
||||||
|
minor) MINOR=$((MINOR + 1)); PATCH=0 ;;
|
||||||
|
patch) PATCH=$((PATCH + 1)) ;;
|
||||||
|
*) echo "usage: $0 [patch|minor|major]" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
NEW="$MAJOR.$MINOR.$PATCH"
|
||||||
|
BUILD=$(( $(plutil -extract CFBundleVersion raw "$PLIST") + 1 ))
|
||||||
|
|
||||||
|
plutil -replace CFBundleShortVersionString -string "$NEW" "$PLIST"
|
||||||
|
plutil -replace CFBundleVersion -string "$BUILD" "$PLIST"
|
||||||
|
# Edit only the version line so the rest of ime.json keeps its formatting.
|
||||||
|
sed -i '' -E "s/(\"version\"[[:space:]]*:[[:space:]]*\")[^\"]*\"/\1$NEW\"/" "$IME"
|
||||||
|
|
||||||
|
grep -q "\"version\": \"$NEW\"" "$IME" || { echo "failed to update $IME" >&2; exit 1; }
|
||||||
|
echo "$NEW"
|
||||||
@@ -13,6 +13,17 @@ rm -rf "$DEST/AutoZhuyin.app"
|
|||||||
cp -R build/AutoZhuyin.app "$DEST/"
|
cp -R build/AutoZhuyin.app "$DEST/"
|
||||||
"$DEST/AutoZhuyin.app/Contents/MacOS/AutoZhuyin" --register
|
"$DEST/AutoZhuyin.app/Contents/MacOS/AutoZhuyin" --register
|
||||||
|
|
||||||
|
SYSTEM_APP="/Library/Input Methods/AutoZhuyin.app"
|
||||||
|
if [ -d "$SYSTEM_APP" ]; then
|
||||||
|
OLD="$(plutil -extract CFBundleShortVersionString raw "$SYSTEM_APP/Contents/Info.plist" 2>/dev/null || echo '?')"
|
||||||
|
echo
|
||||||
|
echo "WARNING: a system-wide copy (v$OLD, installed from the .pkg) exists at:"
|
||||||
|
echo " $SYSTEM_APP"
|
||||||
|
echo "macOS runs that copy instead of this one. Install the new .pkg"
|
||||||
|
echo "(scripts/package.sh), or remove it with:"
|
||||||
|
echo " sudo rm -rf \"$SYSTEM_APP\" && pkill -x AutoZhuyin"
|
||||||
|
fi
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "Installed to $DEST/AutoZhuyin.app"
|
echo "Installed to $DEST/AutoZhuyin.app"
|
||||||
echo "First install: System Settings → Keyboard → Text Input → Edit… → + → 繁體中文 → 自動注音"
|
echo "First install: System Settings → Keyboard → Text Input → Edit… → + → 繁體中文 → 自動注音"
|
||||||
|
|||||||
Executable
+54
@@ -0,0 +1,54 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# One-step release: bump the version, run all tests, build both installers, then
|
||||||
|
# commit the version change and tag it.
|
||||||
|
#
|
||||||
|
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
|
||||||
|
# --push also pushes the commit and tag to origin
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
LEVEL=patch
|
||||||
|
PUSH=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
patch|minor|major) LEVEL="$arg" ;;
|
||||||
|
--push) PUSH=1 ;;
|
||||||
|
*) echo "usage: $0 [patch|minor|major] [--push]" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -n "$(git status --porcelain)" ]; then
|
||||||
|
echo "Working tree has uncommitted changes; commit them first." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}"
|
||||||
|
|
||||||
|
VERSION="$(scripts/bump-version.sh "$LEVEL")"
|
||||||
|
echo "==> Version $VERSION"
|
||||||
|
|
||||||
|
restore() {
|
||||||
|
echo "Release failed; reverting the version bump." >&2
|
||||||
|
git checkout -- Resources/Info.plist windows/autozhuyin/ime.json
|
||||||
|
}
|
||||||
|
trap restore ERR
|
||||||
|
|
||||||
|
echo "==> Swift tests"
|
||||||
|
xcrun swift test -q 2>&1 | tail -3
|
||||||
|
echo "==> Packaging (runs the Python tests too)"
|
||||||
|
scripts/package.sh >/dev/null
|
||||||
|
scripts/package-windows.sh >/dev/null
|
||||||
|
|
||||||
|
trap - ERR
|
||||||
|
git add Resources/Info.plist windows/autozhuyin/ime.json
|
||||||
|
git commit -q -m "Release v$VERSION"
|
||||||
|
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
|
||||||
|
echo "==> Tagged v$VERSION"
|
||||||
|
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
|
||||||
|
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
|
||||||
|
|
||||||
|
if [ "$PUSH" = 1 ]; then
|
||||||
|
git push origin HEAD "v$VERSION"
|
||||||
|
else
|
||||||
|
echo "Push with: git push origin HEAD v$VERSION"
|
||||||
|
fi
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
|
||||||
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
WIN = os.path.join(ROOT, "windows")
|
||||||
|
DATA = os.path.join(WIN, "autozhuyin", "data")
|
||||||
|
RESOURCES = os.path.join(ROOT, "Resources")
|
||||||
|
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
|
||||||
|
|
||||||
|
sys.path.insert(0, WIN)
|
||||||
|
|
||||||
|
from autozhuyin.core import bopomofo, features # noqa: E402
|
||||||
|
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
|
||||||
|
from autozhuyin.core.english import EnglishModel # noqa: E402
|
||||||
|
from autozhuyin.core.lexicon import Lexicon # noqa: E402
|
||||||
|
|
||||||
|
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
|
||||||
|
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
|
||||||
|
|
||||||
|
|
||||||
|
def load_models():
|
||||||
|
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
|
||||||
|
sys.exit("Run windows/tools/build_data.py first.")
|
||||||
|
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
|
||||||
|
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
|
||||||
|
os.path.join(DATA, "english-dict.txt"),
|
||||||
|
os.path.join(DATA, "english-bigram.json"))
|
||||||
|
return lexicon, english
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Writes training/parity-fixtures.json: segmentations produced by the Python core
|
||||||
|
with the current weights. The Swift tests (ParityTests) must produce the same ones,
|
||||||
|
which keeps the two implementations in sync. Re-run after changing features or weights.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
|
||||||
|
from synth import Generator # noqa: E402
|
||||||
|
|
||||||
|
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json")
|
||||||
|
|
||||||
|
HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up",
|
||||||
|
"u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
lexicon, english = load_models()
|
||||||
|
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
|
||||||
|
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
|
||||||
|
os.path.join(DATA, "english-dict.txt"), seed=7)
|
||||||
|
cases = [(k, f, p) for k in HAND_PICKED for f in (False, True) for p in (None, "literal")]
|
||||||
|
while len(cases) < 500:
|
||||||
|
ex = gen.example()
|
||||||
|
if len(ex["keys"]) <= 28:
|
||||||
|
cases.append((ex["keys"], ex["finishing"], ex["previous"]))
|
||||||
|
cases.append((gen.prefix_of(ex)["keys"], False, ex["previous"]))
|
||||||
|
fixtures = [{"keys": k, "finishing": f, "previous": p,
|
||||||
|
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f, p)]} for k, f, p in cases]
|
||||||
|
with open(OUT, "w", encoding="utf-8") as fh:
|
||||||
|
json.dump(fixtures, fh, ensure_ascii=False, indent=0)
|
||||||
|
print("wrote %d fixtures to %s" % (len(fixtures), OUT))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,147 @@
|
|||||||
|
"""Synthetic typing data for the Zhuyin / literal classifier.
|
||||||
|
|
||||||
|
Each example is what sits in the input buffer when the user commits it, with the
|
||||||
|
intended mode of every key:
|
||||||
|
{"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"}
|
||||||
|
labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter.
|
||||||
|
|
||||||
|
Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys;
|
||||||
|
English from the common-word list and the dictionary. Real typing commits English at
|
||||||
|
a space, so a buffer holds at most one English word next to Chinese.
|
||||||
|
"""
|
||||||
|
import random
|
||||||
|
|
||||||
|
from common import core
|
||||||
|
|
||||||
|
bopomofo = core.bopomofo
|
||||||
|
|
||||||
|
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
|
||||||
|
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
|
||||||
|
|
||||||
|
|
||||||
|
def reading_to_keys(reading):
|
||||||
|
"""'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable."""
|
||||||
|
out = []
|
||||||
|
for syl in reading.split("-"):
|
||||||
|
tone = " "
|
||||||
|
if syl and syl[-1] in TONE_KEY:
|
||||||
|
tone = TONE_KEY[syl[-1]]
|
||||||
|
syl = syl[:-1]
|
||||||
|
if not syl or syl not in bopomofo.VALID:
|
||||||
|
return None
|
||||||
|
out.append("".join(KEY_OF[c] for c in syl) + tone)
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_cjk(s):
|
||||||
|
return all(0x4E00 <= ord(c) <= 0x9FFF for c in s)
|
||||||
|
|
||||||
|
|
||||||
|
class Generator:
|
||||||
|
def __init__(self, lexicon_path, common_path, dict_path, seed=0):
|
||||||
|
self.rng = random.Random(seed)
|
||||||
|
phrases, weights = [], []
|
||||||
|
with open(lexicon_path, encoding="utf-8") as f:
|
||||||
|
for line in f:
|
||||||
|
cols = line.split()
|
||||||
|
if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]):
|
||||||
|
continue
|
||||||
|
n = cols[0].count("-") + 1
|
||||||
|
if n > 4:
|
||||||
|
continue
|
||||||
|
keys = reading_to_keys(cols[0])
|
||||||
|
if keys is None:
|
||||||
|
continue
|
||||||
|
phrases.append((cols[1], keys))
|
||||||
|
weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency
|
||||||
|
self.phrases = phrases
|
||||||
|
self.phrase_cum = _cumulative(weights)
|
||||||
|
self.common = [w.strip() for w in open(common_path, encoding="utf-8")
|
||||||
|
if w.strip() and not w.startswith("#")]
|
||||||
|
self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8")
|
||||||
|
if 3 <= len(w.strip()) <= 9]
|
||||||
|
|
||||||
|
# --- pieces ---
|
||||||
|
|
||||||
|
def chinese(self):
|
||||||
|
n = self.rng.choice([1, 1, 2, 2, 3])
|
||||||
|
picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n)
|
||||||
|
return "".join(keys for _, keys in picked)
|
||||||
|
|
||||||
|
def english(self):
|
||||||
|
w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words)
|
||||||
|
if self.rng.random() < 0.1:
|
||||||
|
w = w[:1].upper() + w[1:]
|
||||||
|
return w
|
||||||
|
|
||||||
|
def number(self):
|
||||||
|
r = self.rng.random()
|
||||||
|
if r < 0.4:
|
||||||
|
return str(self.rng.randint(0, 9999))
|
||||||
|
if r < 0.7:
|
||||||
|
return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30]))
|
||||||
|
return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9))
|
||||||
|
|
||||||
|
# --- examples ---
|
||||||
|
|
||||||
|
def example(self):
|
||||||
|
r = self.rng.random()
|
||||||
|
if r < 0.40:
|
||||||
|
parts, kind = [("z", self.chinese())], "zh"
|
||||||
|
elif r < 0.65:
|
||||||
|
parts, kind = [("l", self.english())], "en"
|
||||||
|
elif r < 0.80:
|
||||||
|
parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en"
|
||||||
|
elif r < 0.93:
|
||||||
|
parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh"
|
||||||
|
elif r < 0.96:
|
||||||
|
parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh"
|
||||||
|
else:
|
||||||
|
parts, kind = [("l", self.number())], "num"
|
||||||
|
keys = "".join(p for _, p in parts)
|
||||||
|
labels = "".join(m * len(p) for m, p in parts)
|
||||||
|
if parts[-1][0] == "l":
|
||||||
|
finishing = True # English / numbers end with a space
|
||||||
|
elif keys[-1] == " " and self.rng.random() < 0.6:
|
||||||
|
# The moment the space (first tone) is pressed: the buffer has no space yet and
|
||||||
|
# the input method decides "first tone" vs "English word + space".
|
||||||
|
keys, labels, finishing = keys[:-1], labels[:-1], True
|
||||||
|
kind += "/tone1"
|
||||||
|
else:
|
||||||
|
# Chinese ends with Enter, or space after a finished (toned) syllable.
|
||||||
|
finishing = keys[-1] != " " and self.rng.random() < 0.5
|
||||||
|
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind,
|
||||||
|
"previous": self.previous_mode(labels[0])}
|
||||||
|
|
||||||
|
def previous_mode(self, first_label):
|
||||||
|
"""Mode of the text committed just before: none, mostly the same, sometimes the other."""
|
||||||
|
r = self.rng.random()
|
||||||
|
if r < 0.35:
|
||||||
|
return None
|
||||||
|
same = core.ZHUYIN if first_label == "z" else core.LITERAL
|
||||||
|
other = core.LITERAL if first_label == "z" else core.ZHUYIN
|
||||||
|
return same if r < 0.80 else other
|
||||||
|
|
||||||
|
def prefix_of(self, ex):
|
||||||
|
"""The buffer while still typing: a random cut, not finishing."""
|
||||||
|
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
|
||||||
|
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
|
||||||
|
"kind": ex["kind"] + "/prefix", "previous": ex["previous"]}
|
||||||
|
|
||||||
|
|
||||||
|
def _cumulative(ws):
|
||||||
|
out, t = [], 0.0
|
||||||
|
for w in ws:
|
||||||
|
t += w
|
||||||
|
out.append(t)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def gold_segments(labels):
|
||||||
|
"""'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]"""
|
||||||
|
segs, start = [], 0
|
||||||
|
for i in range(1, len(labels) + 1):
|
||||||
|
if i == len(labels) or labels[i] != labels[start]:
|
||||||
|
segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL))
|
||||||
|
start = i
|
||||||
|
return segs
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Learns the classifier weights from synthetic typing data (averaged structured
|
||||||
|
perceptron over the same segmentation DP the input method runs) and reports
|
||||||
|
accuracy against the hand-tuned defaults.
|
||||||
|
|
||||||
|
Usage (from the repo root, after windows/tools/build_data.py):
|
||||||
|
python3 training/train.py # train, evaluate, write Resources/classifier-weights.json
|
||||||
|
python3 training/train.py --eval-only # just evaluate the current weights file
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import datetime
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
|
||||||
|
from synth import Generator, gold_segments # noqa: E402
|
||||||
|
|
||||||
|
MAX_KEYS = 28
|
||||||
|
FIXED = {"lit.upper"} # hard rule (Shift = English), not learned
|
||||||
|
|
||||||
|
|
||||||
|
def make_set(gen, n):
|
||||||
|
out = []
|
||||||
|
while len(out) < 2 * n:
|
||||||
|
ex = gen.example()
|
||||||
|
if len(ex["keys"]) > MAX_KEYS:
|
||||||
|
continue
|
||||||
|
out.append(ex)
|
||||||
|
if len(ex["keys"]) > 1:
|
||||||
|
out.append(gen.prefix_of(ex))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def labels_of(segs, n):
|
||||||
|
lab = [""] * n
|
||||||
|
for s, e, m in segs:
|
||||||
|
for i in range(s, e):
|
||||||
|
lab[i] = "z" if m == core.ZHUYIN else "l"
|
||||||
|
return "".join(lab)
|
||||||
|
|
||||||
|
|
||||||
|
def seg_features(clf, keys, segs, finishing, previous):
|
||||||
|
"""Summed features of a segmentation, or None if any segment is impossible."""
|
||||||
|
total = defaultdict(float)
|
||||||
|
n = len(keys)
|
||||||
|
for idx, (s, e, m) in enumerate(segs):
|
||||||
|
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing, previous if idx == 0 else None)
|
||||||
|
f = fz if m == core.ZHUYIN else fl
|
||||||
|
if f is None:
|
||||||
|
return None
|
||||||
|
for k, v in f.items():
|
||||||
|
total[k] += v
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(clf, weights, data):
|
||||||
|
by_kind = defaultdict(lambda: [0, 0])
|
||||||
|
for ex in data:
|
||||||
|
segs = clf.segment(list(ex["keys"]), ex["finishing"], ex["previous"], weights)
|
||||||
|
ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
|
||||||
|
kind = "prefix" if ex["kind"].endswith("/prefix") else ex["kind"]
|
||||||
|
for k in (kind, "after-other") if _after_other(ex) else (kind,):
|
||||||
|
by_kind[k][0] += ok
|
||||||
|
by_kind[k][1] += 1
|
||||||
|
commit = [v for k, v in by_kind.items() if k not in ("prefix", "after-other")]
|
||||||
|
summary = {
|
||||||
|
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
|
||||||
|
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
|
||||||
|
"by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())},
|
||||||
|
}
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def _after_other(ex):
|
||||||
|
"""Committed buffer whose first segment differs from the text typed just before."""
|
||||||
|
if ex["kind"].endswith("/prefix") or ex["previous"] is None:
|
||||||
|
return False
|
||||||
|
first = core.ZHUYIN if ex["labels"][0] == "z" else core.LITERAL
|
||||||
|
return ex["previous"] != first
|
||||||
|
|
||||||
|
|
||||||
|
def show(name, s):
|
||||||
|
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
|
||||||
|
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
|
||||||
|
|
||||||
|
|
||||||
|
def train(clf, data, epochs, lr, seed):
|
||||||
|
rng = random.Random(seed)
|
||||||
|
w = dict(clf.weights)
|
||||||
|
total = defaultdict(float)
|
||||||
|
steps = 0
|
||||||
|
for epoch in range(epochs):
|
||||||
|
rng.shuffle(data)
|
||||||
|
mistakes = 0
|
||||||
|
t0 = time.time()
|
||||||
|
for ex in data:
|
||||||
|
keys = list(ex["keys"])
|
||||||
|
pred = clf.segment(keys, ex["finishing"], ex["previous"], w)
|
||||||
|
if labels_of(pred, len(keys)) != ex["labels"]:
|
||||||
|
gold = gold_segments(ex["labels"])
|
||||||
|
fg = seg_features(clf, keys, gold, ex["finishing"], ex["previous"])
|
||||||
|
fp = seg_features(clf, keys, pred, ex["finishing"], ex["previous"])
|
||||||
|
if fg is not None and fp is not None:
|
||||||
|
mistakes += 1
|
||||||
|
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
|
||||||
|
for k in set(fg) | set(fp):
|
||||||
|
if k not in FIXED:
|
||||||
|
w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0))
|
||||||
|
steps += 1
|
||||||
|
for k, v in w.items():
|
||||||
|
total[k] += v
|
||||||
|
print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True)
|
||||||
|
return {k: round(v / steps, 4) for k, v in total.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)")
|
||||||
|
ap.add_argument("--test", type=int, default=1500)
|
||||||
|
ap.add_argument("--epochs", type=int, default=6)
|
||||||
|
ap.add_argument("--lr", type=float, default=0.1)
|
||||||
|
ap.add_argument("--eval-only", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
lexicon, english = load_models()
|
||||||
|
paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
|
||||||
|
os.path.join(DATA, "english-dict.txt"))
|
||||||
|
test = make_set(Generator(*paths, seed=2), args.test)
|
||||||
|
|
||||||
|
clf = core.Classifier(lexicon, english) # default (hand-tuned) weights
|
||||||
|
show("default", evaluate(clf, clf.weights, test))
|
||||||
|
if os.path.exists(WEIGHTS_PATH):
|
||||||
|
current = dict(clf.weights)
|
||||||
|
current.update(core.Classifier.load_weights(WEIGHTS_PATH))
|
||||||
|
show("current", evaluate(clf, current, test))
|
||||||
|
if args.eval_only:
|
||||||
|
return
|
||||||
|
|
||||||
|
data = make_set(Generator(*paths, seed=1), args.train)
|
||||||
|
print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True)
|
||||||
|
learned = train(clf, data, args.epochs, args.lr, seed=3)
|
||||||
|
result = evaluate(clf, learned, test)
|
||||||
|
show("learned", result)
|
||||||
|
|
||||||
|
with open(WEIGHTS_PATH, "w", encoding="utf-8") as f:
|
||||||
|
json.dump({
|
||||||
|
"about": "AutoZhuyin classifier weights, learned by training/train.py",
|
||||||
|
"trained": datetime.date.today().isoformat(),
|
||||||
|
"examples": len(data),
|
||||||
|
"test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)},
|
||||||
|
"weights": dict(sorted(learned.items())),
|
||||||
|
}, f, ensure_ascii=False, indent=2)
|
||||||
|
f.write("\n")
|
||||||
|
print("wrote", WEIGHTS_PATH)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -6,14 +6,16 @@ tests); this file only translates PIME key events and replies.
|
|||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
|
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
|
||||||
from textService import TextService
|
from textService import TextService
|
||||||
|
|
||||||
from .core.classifier import Classifier, LearnedPreferences
|
from .core.classifier import Classifier, LearnedPreferences
|
||||||
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, TAB, UP, CompositionEngine
|
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
|
||||||
from .core.english import EnglishModel
|
from .core.english import EnglishModel
|
||||||
from .core.lexicon import Lexicon
|
from .core.lexicon import Lexicon
|
||||||
|
from .core.shift_tap import ShiftTapDetector
|
||||||
|
|
||||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
_DATA = os.path.join(_HERE, "data")
|
_DATA = os.path.join(_HERE, "data")
|
||||||
@@ -52,7 +54,8 @@ def _shared_classifier():
|
|||||||
weights = json.load(f)
|
weights = json.load(f)
|
||||||
except (OSError, ValueError):
|
except (OSError, ValueError):
|
||||||
pass
|
pass
|
||||||
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned))
|
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
|
||||||
|
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained)
|
||||||
return _classifier
|
return _classifier
|
||||||
|
|
||||||
|
|
||||||
@@ -66,6 +69,7 @@ class AutoZhuyinTextService(TextService):
|
|||||||
def __init__(self, client):
|
def __init__(self, client):
|
||||||
TextService.__init__(self, client)
|
TextService.__init__(self, client)
|
||||||
self.engine = CompositionEngine(_shared_classifier())
|
self.engine = CompositionEngine(_shared_classifier())
|
||||||
|
self.shift_tap = ShiftTapDetector()
|
||||||
|
|
||||||
# --- lifecycle ---
|
# --- lifecycle ---
|
||||||
|
|
||||||
@@ -104,6 +108,13 @@ class AutoZhuyinTextService(TextService):
|
|||||||
return bool(self.engine.keys or self.engine.candidates)
|
return bool(self.engine.keys or self.engine.candidates)
|
||||||
|
|
||||||
def filterKeyDown(self, ev):
|
def filterKeyDown(self, ev):
|
||||||
|
if ev.keyCode == VK_SHIFT: # noqa: F405
|
||||||
|
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
|
||||||
|
self.shift_tap.key_pressed()
|
||||||
|
else:
|
||||||
|
self.shift_tap.shift_down(time.monotonic())
|
||||||
|
return False
|
||||||
|
self.shift_tap.key_pressed()
|
||||||
# Shortcuts (Ctrl/Alt) go to the app.
|
# Shortcuts (Ctrl/Alt) go to the app.
|
||||||
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
|
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
|
||||||
return False
|
return False
|
||||||
@@ -120,6 +131,21 @@ class AutoZhuyinTextService(TextService):
|
|||||||
self._update(commit)
|
self._update(commit)
|
||||||
return handled
|
return handled
|
||||||
|
|
||||||
|
def filterKeyUp(self, ev):
|
||||||
|
# Shift pressed and released alone: next word is English (or flip the current one).
|
||||||
|
if ev.keyCode == VK_SHIFT: # noqa: F405
|
||||||
|
return self.shift_tap.shift_up(time.monotonic())
|
||||||
|
return False
|
||||||
|
|
||||||
|
def onKeyUp(self, ev):
|
||||||
|
if ev.keyCode != VK_SHIFT: # noqa: F405
|
||||||
|
return False
|
||||||
|
self.engine.handle(SHIFT_TAP)
|
||||||
|
self._update(None)
|
||||||
|
if not self.engine.keys:
|
||||||
|
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
|
||||||
|
return True
|
||||||
|
|
||||||
# --- replies ---
|
# --- replies ---
|
||||||
|
|
||||||
def _update(self, commit):
|
def _update(self, commit):
|
||||||
@@ -127,7 +153,7 @@ class AutoZhuyinTextService(TextService):
|
|||||||
self.setCommitString(commit)
|
self.setCommitString(commit)
|
||||||
marked = self.engine.marked_text
|
marked = self.engine.marked_text
|
||||||
self.setCompositionString(marked)
|
self.setCompositionString(marked)
|
||||||
self.setCompositionCursor(len(marked))
|
self.setCompositionCursor(self.engine.cursor_offset)
|
||||||
if self.engine.showing_candidates:
|
if self.engine.showing_candidates:
|
||||||
page, cursor = self.engine.page()
|
page, cursor = self.engine.page()
|
||||||
self.setCandidateList(page)
|
self.setCandidateList(page)
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
|
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
|
||||||
from . import bopomofo
|
import json
|
||||||
|
|
||||||
|
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
|
||||||
|
|
||||||
ZHUYIN = "zhuyin"
|
ZHUYIN = "zhuyin"
|
||||||
LITERAL = "literal"
|
LITERAL = "literal"
|
||||||
@@ -28,67 +30,87 @@ class LearnedPreferences:
|
|||||||
|
|
||||||
|
|
||||||
class Classifier:
|
class Classifier:
|
||||||
zhuyin_prior = 0.5
|
|
||||||
context_weight = 1.5
|
|
||||||
|
|
||||||
def __init__(self, lexicon, english, learned=None):
|
def __init__(self, lexicon, english, learned=None, weights=None):
|
||||||
self.lexicon = lexicon
|
self.lexicon = lexicon
|
||||||
self.english = english
|
self.english = english
|
||||||
self.learned = learned or LearnedPreferences()
|
self.learned = learned or LearnedPreferences()
|
||||||
|
self.weights = dict(DEFAULT_WEIGHTS)
|
||||||
|
if weights:
|
||||||
|
self.weights.update(weights)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def load_weights(cls, path):
|
||||||
|
"""Reads a weights JSON file ({"weights": {...}}); returns {} if missing."""
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
return json.load(f).get("weights", {})
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def segment_features(self, seg, is_last, after_other, finishing, previous=None):
|
||||||
|
"""(zhuyin features or None, literal features or None) for one segment.
|
||||||
|
`previous` (mode of the text committed just before) only affects the first segment."""
|
||||||
|
seg = list(seg)
|
||||||
|
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing, previous),
|
||||||
|
literal_features(seg, self.english, is_last, after_other, finishing, previous))
|
||||||
|
|
||||||
|
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
|
||||||
|
fz, fl = self.segment_features(seg, is_last, after_other, finishing, previous)
|
||||||
|
zh = dot(weights, fz) if fz is not None else NEG_INF
|
||||||
|
lit = dot(weights, fl) if fl is not None else NEG_INF
|
||||||
|
if whole:
|
||||||
|
b = self.learned.bias("".join(seg))
|
||||||
|
if b > 0:
|
||||||
|
zh += b
|
||||||
|
else:
|
||||||
|
lit -= b
|
||||||
|
return zh, lit
|
||||||
|
|
||||||
def classify(self, keys, finishing=False, previous=None):
|
def classify(self, keys, finishing=False, previous=None):
|
||||||
"""Returns (mode, zhuyin_score, literal_score)."""
|
"""Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score)."""
|
||||||
keys = list(keys)
|
zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights)
|
||||||
zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior
|
|
||||||
lit = self.literal_score(keys, complete=finishing)
|
|
||||||
if previous == ZHUYIN:
|
|
||||||
zh += self.context_weight
|
|
||||||
elif previous == LITERAL:
|
|
||||||
lit += self.context_weight
|
|
||||||
b = self.learned.bias("".join(keys))
|
|
||||||
if b > 0:
|
|
||||||
zh += b
|
|
||||||
else:
|
|
||||||
lit -= b
|
|
||||||
return (ZHUYIN if zh >= lit else LITERAL), zh, lit
|
return (ZHUYIN if zh >= lit else LITERAL), zh, lit
|
||||||
|
|
||||||
def mode(self, keys, finishing=False, previous=None):
|
def mode(self, keys, finishing=False, previous=None):
|
||||||
return self.classify(keys, finishing, previous)[0]
|
return self.classify(keys, finishing, previous)[0]
|
||||||
|
|
||||||
def zhuyin_score(self, keys):
|
# --- segmentation ---
|
||||||
p = bopomofo.parse(keys)
|
|
||||||
if p.foreign_keys > 0 or not p.syllables:
|
|
||||||
return NEG_INF
|
|
||||||
s = -5.0 * p.stray_tones
|
|
||||||
last = len(p.syllables) - 1
|
|
||||||
for i, ps in enumerate(p.syllables):
|
|
||||||
if ps.reordered:
|
|
||||||
s -= 0.5
|
|
||||||
valid = bopomofo.is_valid(ps.syllable)
|
|
||||||
if ps.toned:
|
|
||||||
if valid:
|
|
||||||
s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5)
|
|
||||||
else:
|
|
||||||
s -= 5
|
|
||||||
elif i == last:
|
|
||||||
s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5
|
|
||||||
else:
|
|
||||||
s += -2.5 if valid else -5
|
|
||||||
return s
|
|
||||||
|
|
||||||
def literal_score(self, keys, complete):
|
def segment(self, keys, finishing=False, previous=None, weights=None):
|
||||||
if not keys or " " in keys:
|
"""Splits keys into alternating segments: [(start, end, mode)], e.g.
|
||||||
return NEG_INF
|
"ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
|
||||||
score = 0.0
|
split points; each segment is scored by features x weights."""
|
||||||
if any(c.isupper() for c in keys):
|
weights = weights or self.weights
|
||||||
score += 20
|
keys = list(keys)
|
||||||
letters = "".join(c for c in keys if c.isalpha())
|
n = len(keys)
|
||||||
digits = sum(1 for c in keys if c.isdigit())
|
if n == 0:
|
||||||
others = len(keys) - len(letters) - digits
|
return []
|
||||||
if not letters:
|
modes = (ZHUYIN, LITERAL)
|
||||||
return score + 0.8 * digits - 1.0 * others
|
best = [[NEG_INF, NEG_INF] for _ in range(n + 1)]
|
||||||
score += self.english.score(letters, complete or digits > 0)
|
back = [[-1, -1] for _ in range(n + 1)]
|
||||||
score -= 1.5 * digits
|
for i in range(1, n + 1):
|
||||||
apostrophes = keys.count("'")
|
for j in range(i):
|
||||||
score -= 1.5 * (others - apostrophes)
|
scores = self._scores(keys[j:i], i == n, j > 0, finishing,
|
||||||
return score
|
previous if j == 0 else None, j == 0 and i == n, weights)
|
||||||
|
for m in (0, 1):
|
||||||
|
if scores[m] == NEG_INF:
|
||||||
|
continue
|
||||||
|
if j == 0:
|
||||||
|
total = scores[m]
|
||||||
|
else:
|
||||||
|
if best[j][1 - m] == NEG_INF:
|
||||||
|
continue
|
||||||
|
total = best[j][1 - m] + scores[m]
|
||||||
|
if total > best[i][m]:
|
||||||
|
best[i][m] = total
|
||||||
|
back[i][m] = j
|
||||||
|
if best[n][0] == NEG_INF and best[n][1] == NEG_INF:
|
||||||
|
return [(0, n, self.mode(keys, finishing, previous))]
|
||||||
|
m = 0 if best[n][0] >= best[n][1] else 1
|
||||||
|
out, i = [], n
|
||||||
|
while i > 0:
|
||||||
|
j = back[i][m]
|
||||||
|
out.append((j, i, modes[m]))
|
||||||
|
i, m = j, 1 - m
|
||||||
|
return out[::-1]
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
|
|
||||||
Keys are given as one-character strings, or one of the names in SPECIAL_KEYS.
|
Keys are given as one-character strings, or one of the names in SPECIAL_KEYS.
|
||||||
handle() returns (handled, commit_text_or_None).
|
handle() returns (handled, commit_text_or_None).
|
||||||
|
The buffer is split into Zhuyin / literal segments, e.g. "ji3hello" -> 我 + hello.
|
||||||
"""
|
"""
|
||||||
import time
|
import time
|
||||||
|
|
||||||
@@ -11,13 +12,27 @@ from .lexicon import Converter
|
|||||||
|
|
||||||
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
|
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
|
||||||
UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>"
|
UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>"
|
||||||
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT}
|
SHIFT_TAP = "<shift-tap>" # Shift pressed and released on its own
|
||||||
|
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT, SHIFT_TAP}
|
||||||
|
|
||||||
FULLWIDTH = {
|
FULLWIDTH = {
|
||||||
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")",
|
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")",
|
||||||
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# An English word that is also a common one-syllable Chinese reading ("up " = ㄧㄣ 因)
|
||||||
|
# is not committed on space; the next word decides ("up jo4" = 因為, "up the" = up the).
|
||||||
|
HOLD_MIN_SCORE = -3.5
|
||||||
|
|
||||||
|
# "`" then one of these keys always gives the full-width symbol, whatever the context.
|
||||||
|
# "`" followed by anything else is just a backtick (so ``` still works).
|
||||||
|
BACKTICK = "`"
|
||||||
|
SYMBOLS = {
|
||||||
|
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
|
||||||
|
"'": "、", "\\": "、", '"': "「", "[": "「", "]": "」", "{": "『", "}": "』",
|
||||||
|
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
|
||||||
|
}
|
||||||
|
|
||||||
PASS = (False, None)
|
PASS = (False, None)
|
||||||
CONSUMED = (True, None)
|
CONSUMED = (True, None)
|
||||||
|
|
||||||
@@ -32,11 +47,22 @@ class CompositionEngine:
|
|||||||
self.now = now
|
self.now = now
|
||||||
self.keys = []
|
self.keys = []
|
||||||
self.forced_mode = None
|
self.forced_mode = None
|
||||||
self.fixed = {}
|
# English mode, toggled by a Shift tap when nothing is being typed: keys go straight
|
||||||
|
# to the app (no composition, no underline) until Shift is tapped again.
|
||||||
|
self.english_mode = False
|
||||||
|
self.backtick = False # "`" typed; the next key may be a forced symbol
|
||||||
|
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
|
||||||
self.candidates = []
|
self.candidates = []
|
||||||
self.candidate_index = 0
|
self.candidate_index = 0
|
||||||
|
self._candidate_segment_start = 0
|
||||||
|
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
|
||||||
|
self.cursor = None
|
||||||
|
# Index of a held space (see HOLD_MIN_SCORE), or None.
|
||||||
|
self.held = None
|
||||||
self._previous_mode = None
|
self._previous_mode = None
|
||||||
self._previous_time = float("-inf")
|
self._previous_time = float("-inf")
|
||||||
|
self._cache_key = None
|
||||||
|
self._cache_value = None
|
||||||
|
|
||||||
# --- derived state ---
|
# --- derived state ---
|
||||||
|
|
||||||
@@ -44,9 +70,49 @@ class CompositionEngine:
|
|||||||
def context(self):
|
def context(self):
|
||||||
return self._previous_mode if self.now() - self._previous_time < self.context_window else None
|
return self._previous_mode if self.now() - self._previous_time < self.context_window else None
|
||||||
|
|
||||||
|
def _segments(self, keys, finishing=False):
|
||||||
|
if not keys:
|
||||||
|
return []
|
||||||
|
if self.forced_mode:
|
||||||
|
return [(0, len(keys), self.forced_mode)]
|
||||||
|
ctx = self.context
|
||||||
|
held = self.held if self.held is not None and self.held < len(keys) and keys[self.held] == " " else None
|
||||||
|
cache_key = (tuple(keys), finishing, ctx, held)
|
||||||
|
if cache_key != self._cache_key:
|
||||||
|
self._cache_key = cache_key
|
||||||
|
if held is None:
|
||||||
|
self._cache_value = self.classifier.segment(keys, finishing, ctx)
|
||||||
|
else:
|
||||||
|
self._cache_value = self._held_segments(keys, held, finishing)
|
||||||
|
return self._cache_value
|
||||||
|
|
||||||
|
def _held_segments(self, keys, held, finishing):
|
||||||
|
"""The held word is Chinese if the words typed after it start with Chinese."""
|
||||||
|
rest = keys[held + 1:]
|
||||||
|
base = held + 1
|
||||||
|
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
|
||||||
|
if rest_segs and rest_segs[0][2] == ZHUYIN:
|
||||||
|
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
|
||||||
|
return [(0, base, LITERAL)] + rest_segs
|
||||||
|
|
||||||
|
def _is_ambiguous_word(self, keys):
|
||||||
|
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
|
||||||
|
if not all(k.isascii() and k.isalpha() for k in keys):
|
||||||
|
return False
|
||||||
|
p = bopomofo.parse(list(keys) + [" "])
|
||||||
|
if p.pending is not None or len(p.syllables) != 1:
|
||||||
|
return False
|
||||||
|
entries = self.classifier.lexicon.entries([s.syllable for s in p.syllables])
|
||||||
|
return bool(entries) and entries[0][1] >= HOLD_MIN_SCORE
|
||||||
|
|
||||||
|
@property
|
||||||
|
def segments(self):
|
||||||
|
return self._segments(self.keys)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def mode(self):
|
def mode(self):
|
||||||
return self.forced_mode or self.classifier.mode(self.keys, previous=self.context)
|
segs = self.segments
|
||||||
|
return segs[-1][2] if segs else ZHUYIN
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def showing_candidates(self):
|
def showing_candidates(self):
|
||||||
@@ -56,35 +122,86 @@ class CompositionEngine:
|
|||||||
def page_start(self):
|
def page_start(self):
|
||||||
return (self.candidate_index // self.page_size) * self.page_size
|
return (self.candidate_index // self.page_size) * self.page_size
|
||||||
|
|
||||||
def _parse(self):
|
def _convertible(self, seg, final):
|
||||||
return bopomofo.parse(self.keys)
|
"""(syllables converted to Chinese, pending Bopomofo) for a Zhuyin segment."""
|
||||||
|
start, end, _ = seg
|
||||||
|
p = bopomofo.parse(self.keys[start:end])
|
||||||
|
if final:
|
||||||
|
return [s.syllable for s in p.syllables], ""
|
||||||
|
return [s.syllable for s in p.completed], (p.pending.syllable.reading if p.pending else "")
|
||||||
|
|
||||||
def _valid_fixed(self, p):
|
def _conversion(self, seg, final):
|
||||||
n = len(p.completed)
|
syl, pending = self._convertible(seg, final)
|
||||||
return {k: v for k, v in self.fixed.items() if k[1] <= n}
|
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
|
||||||
|
return self.converter.convert(syl, fx), pending
|
||||||
|
|
||||||
|
def _render(self, seg, final):
|
||||||
|
start, end, mode = seg
|
||||||
|
if mode == LITERAL:
|
||||||
|
return "".join(self.keys[start:end])
|
||||||
|
converted, pending = self._conversion(seg, final)
|
||||||
|
return "".join(t for _, t in converted) + pending
|
||||||
|
|
||||||
|
# --- cursor ---
|
||||||
|
|
||||||
|
def _cursor_stops(self):
|
||||||
|
"""[(segment index, syllables before the stop)] for every Zhuyin syllable."""
|
||||||
|
segs = self.segments
|
||||||
|
out = []
|
||||||
|
for i, seg in enumerate(segs):
|
||||||
|
if seg[2] == ZHUYIN:
|
||||||
|
n = len(self._convertible(seg, i < len(segs) - 1)[0])
|
||||||
|
out.extend((i, k) for k in range(1, n + 1))
|
||||||
|
return out
|
||||||
|
|
||||||
|
@property
|
||||||
|
def cursor_offset(self):
|
||||||
|
"""Cursor position in marked_text (characters)."""
|
||||||
|
pieces = self.marked_pieces
|
||||||
|
stops = self._cursor_stops()
|
||||||
|
if self.cursor is None or not 0 <= self.cursor < len(stops):
|
||||||
|
return sum(len(t) for t, _ in pieces)
|
||||||
|
seg_index, local = stops[self.cursor]
|
||||||
|
offset = sum(len(t) for t, _ in pieces[:seg_index])
|
||||||
|
segs = self.segments
|
||||||
|
for (a, b), text in self._conversion(segs[seg_index], seg_index < len(segs) - 1)[0]:
|
||||||
|
if b <= local:
|
||||||
|
offset += len(text)
|
||||||
|
elif a < local:
|
||||||
|
offset += min(local - a, len(text))
|
||||||
|
return offset
|
||||||
|
|
||||||
|
def _move_cursor(self, delta):
|
||||||
|
stops = self._cursor_stops()
|
||||||
|
if not stops:
|
||||||
|
return
|
||||||
|
nxt = (len(stops) - 1 if self.cursor is None else self.cursor) + delta
|
||||||
|
self.cursor = None if nxt >= len(stops) - 1 else max(0, nxt)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def marked_pieces(self):
|
||||||
|
segs = self.segments
|
||||||
|
pieces = [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
|
||||||
|
if self.backtick:
|
||||||
|
pieces.append((BACKTICK, LITERAL))
|
||||||
|
return pieces
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def marked_text(self):
|
def marked_text(self):
|
||||||
if not self.keys:
|
return "".join(t for t, _ in self.marked_pieces)
|
||||||
return ""
|
|
||||||
if self.mode == LITERAL:
|
|
||||||
return "".join(self.keys)
|
|
||||||
p = self._parse()
|
|
||||||
done = self.converter.convert([s.syllable for s in p.completed], self._valid_fixed(p))
|
|
||||||
pending = p.pending.syllable.reading if p.pending else ""
|
|
||||||
return "".join(t for _, t in done) + pending
|
|
||||||
|
|
||||||
def _zhuyin_commit_text(self):
|
|
||||||
p = self._parse()
|
|
||||||
segs = self.converter.convert([s.syllable for s in p.syllables], self._valid_fixed(p))
|
|
||||||
return "".join(t for _, t in segs)
|
|
||||||
|
|
||||||
# --- key handling ---
|
# --- key handling ---
|
||||||
|
|
||||||
|
def _flip(self):
|
||||||
|
"""Forces the whole buffer into the other mode (a correction that is learned)."""
|
||||||
|
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
|
||||||
|
|
||||||
def wants(self, key):
|
def wants(self, key):
|
||||||
"""Whether handle(key) would consume the key, without changing state."""
|
"""Whether handle(key) would consume the key, without changing state."""
|
||||||
if self.candidates:
|
if self.candidates or key == SHIFT_TAP or self.backtick:
|
||||||
return True
|
return True
|
||||||
|
if self.english_mode and not self.keys:
|
||||||
|
return False # English mode: everything goes to the app as typed
|
||||||
if key in SPECIAL_KEYS:
|
if key in SPECIAL_KEYS:
|
||||||
return bool(self.keys)
|
return bool(self.keys)
|
||||||
if self.keys:
|
if self.keys:
|
||||||
@@ -98,17 +215,24 @@ class CompositionEngine:
|
|||||||
def handle(self, key):
|
def handle(self, key):
|
||||||
if self.candidates:
|
if self.candidates:
|
||||||
return self._handle_candidate_key(key)
|
return self._handle_candidate_key(key)
|
||||||
|
if self.backtick:
|
||||||
|
return self._after_backtick(key)
|
||||||
if key == BACKSPACE:
|
if key == BACKSPACE:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
|
self.cursor = None
|
||||||
self.keys.pop()
|
self.keys.pop()
|
||||||
self.fixed = self._valid_fixed(self._parse())
|
if self.held is not None and len(self.keys) <= self.held:
|
||||||
|
self.held = None
|
||||||
|
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
self.forced_mode = None
|
self.forced_mode = None
|
||||||
self.fixed = {}
|
self.fixed = {}
|
||||||
return CONSUMED
|
return CONSUMED
|
||||||
if key == ENTER:
|
if key == ENTER:
|
||||||
return (True, self._commit()) if self.keys else PASS
|
if not self.keys:
|
||||||
|
return PASS
|
||||||
|
return True, self._commit()
|
||||||
if key == ESCAPE:
|
if key == ESCAPE:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
@@ -117,36 +241,81 @@ class CompositionEngine:
|
|||||||
if key == TAB:
|
if key == TAB:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
|
self._flip()
|
||||||
|
return CONSUMED
|
||||||
|
if key == SHIFT_TAP:
|
||||||
|
if not self.keys:
|
||||||
|
# Toggle English mode; it stays on until the next Shift tap.
|
||||||
|
self.english_mode = not self.english_mode
|
||||||
|
self._previous_mode = LITERAL if self.english_mode else None
|
||||||
|
self._previous_time = self.now()
|
||||||
|
else:
|
||||||
|
self._flip()
|
||||||
return CONSUMED
|
return CONSUMED
|
||||||
if key == DOWN:
|
if key == DOWN:
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
self._open_candidates()
|
self._open_candidates()
|
||||||
return CONSUMED
|
return CONSUMED
|
||||||
if key in (UP, LEFT, RIGHT):
|
if key in (LEFT, RIGHT):
|
||||||
|
if not self.keys:
|
||||||
|
return PASS
|
||||||
|
self._move_cursor(-1 if key == LEFT else 1)
|
||||||
|
return CONSUMED
|
||||||
|
if key == UP:
|
||||||
return CONSUMED if self.keys else PASS
|
return CONSUMED if self.keys else PASS
|
||||||
return self._handle_character(key)
|
return self._handle_character(key)
|
||||||
|
|
||||||
|
def _after_backtick(self, key):
|
||||||
|
self.backtick = False
|
||||||
|
if key in (ESCAPE, BACKSPACE):
|
||||||
|
return CONSUMED
|
||||||
|
committed = self._commit() if self.keys else ""
|
||||||
|
if key in SYMBOLS:
|
||||||
|
return True, committed + SYMBOLS[key]
|
||||||
|
# Not a symbol: the backtick was meant literally; then handle the key as usual.
|
||||||
|
handled, more = self.handle(key)
|
||||||
|
if not handled and len(key) == 1:
|
||||||
|
handled, more = True, key
|
||||||
|
return handled, committed + BACKTICK + (more or "")
|
||||||
|
|
||||||
|
def _last_parse(self, segs):
|
||||||
|
if not segs or segs[-1][2] != ZHUYIN:
|
||||||
|
return None
|
||||||
|
start, end, _ = segs[-1]
|
||||||
|
return bopomofo.parse(self.keys[start:end])
|
||||||
|
|
||||||
def _handle_character(self, c):
|
def _handle_character(self, c):
|
||||||
|
self.cursor = None # typing always continues at the end
|
||||||
|
if self.english_mode and not self.keys:
|
||||||
|
return PASS # English mode: every key goes to the app as typed
|
||||||
|
if c == BACKTICK:
|
||||||
|
self.backtick = True
|
||||||
|
return CONSUMED
|
||||||
if c == " ":
|
if c == " ":
|
||||||
if not self.keys:
|
if not self.keys:
|
||||||
return PASS
|
return PASS
|
||||||
decided = self.forced_mode or self.classifier.mode(self.keys, finishing=True, previous=self.context)
|
segs = self._segments(self.keys, finishing=True)
|
||||||
if decided == LITERAL:
|
if segs[-1][2] == LITERAL:
|
||||||
return True, self._commit(LITERAL) + " "
|
if (self.held is None and len(segs) == 1 and not self.forced_mode
|
||||||
if self._parse().pending is not None:
|
and self._is_ambiguous_word(self.keys)):
|
||||||
|
self.held = len(self.keys)
|
||||||
|
self.keys.append(" ")
|
||||||
|
return CONSUMED
|
||||||
|
return True, self._commit(segs) + " "
|
||||||
|
p = self._last_parse(segs)
|
||||||
|
if p is not None and p.pending is not None:
|
||||||
self.keys.append(" ")
|
self.keys.append(" ")
|
||||||
return CONSUMED
|
return CONSUMED
|
||||||
return True, self._commit(ZHUYIN)
|
return True, self._commit(segs)
|
||||||
|
|
||||||
composable = bopomofo.is_zhuyin_key(c) or (c.isascii() and c.isalpha()) or (c == "'" and self.keys)
|
composable = bopomofo.is_zhuyin_key(c) or (c.isascii() and c.isalpha()) or (c == "'" and self.keys)
|
||||||
if composable:
|
if composable:
|
||||||
# ", . / ; -" are Zhuyin symbols but act as punctuation after an English word.
|
# ", . / ; -" are Zhuyin symbols but act as punctuation after an English word.
|
||||||
if self.keys and c in bopomofo.SYMBOLS and not c.isalnum() and self.forced_mode != ZHUYIN:
|
if self.keys and c in bopomofo.SYMBOLS and not c.isalnum() and self.forced_mode != ZHUYIN:
|
||||||
with_key = self.forced_mode or self.classifier.mode(self.keys + [c], previous=self.context)
|
with_key = self._segments(self.keys + [c])
|
||||||
if with_key == LITERAL and not all(k.isdigit() for k in self.keys):
|
if with_key[-1][2] == LITERAL and not all(k.isdigit() for k in self.keys):
|
||||||
return True, self._commit(LITERAL) + c
|
return True, self._commit() + c
|
||||||
self.keys.append(c)
|
self.keys.append(c)
|
||||||
return CONSUMED
|
return CONSUMED
|
||||||
|
|
||||||
@@ -157,14 +326,19 @@ class CompositionEngine:
|
|||||||
return PASS
|
return PASS
|
||||||
return True, committed + punct
|
return True, committed + punct
|
||||||
|
|
||||||
# --- candidates ---
|
# --- candidates (for the last Zhuyin segment) ---
|
||||||
|
|
||||||
def _open_candidates(self):
|
def _open_candidates(self):
|
||||||
p = self._parse()
|
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
|
||||||
if self.mode != ZHUYIN or not p.completed:
|
stops = self._cursor_stops()
|
||||||
|
if not stops:
|
||||||
return
|
return
|
||||||
syl = [s.syllable for s in p.completed]
|
seg_index, local = stops[min(len(stops) - 1 if self.cursor is None else self.cursor, len(stops) - 1)]
|
||||||
self.candidates = self.converter.candidates(syl, len(syl))
|
segs = self.segments
|
||||||
|
seg = segs[seg_index]
|
||||||
|
syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
|
||||||
|
self.candidates = self.converter.candidates(syl, local)
|
||||||
|
self._candidate_segment_start = seg[0]
|
||||||
self.candidate_index = 0
|
self.candidate_index = 0
|
||||||
|
|
||||||
def page(self):
|
def page(self):
|
||||||
@@ -176,8 +350,10 @@ class CompositionEngine:
|
|||||||
if not 0 <= index < len(self.candidates):
|
if not 0 <= index < len(self.candidates):
|
||||||
return
|
return
|
||||||
rng, text = self.candidates[index]
|
rng, text = self.candidates[index]
|
||||||
self.fixed = {k: v for k, v in self.fixed.items() if not (k[0] < rng[1] and rng[0] < k[1])}
|
here = {k: v for k, v in self.fixed.get(self._candidate_segment_start, {}).items()
|
||||||
self.fixed[rng] = text
|
if not (k[0] < rng[1] and rng[0] < k[1])}
|
||||||
|
here[rng] = text
|
||||||
|
self.fixed[self._candidate_segment_start] = here
|
||||||
self.candidates = []
|
self.candidates = []
|
||||||
|
|
||||||
def _handle_candidate_key(self, key):
|
def _handle_candidate_key(self, key):
|
||||||
@@ -195,12 +371,12 @@ class CompositionEngine:
|
|||||||
|
|
||||||
# --- commit ---
|
# --- commit ---
|
||||||
|
|
||||||
def _commit(self, decided=None):
|
def _commit(self, segs=None):
|
||||||
m = decided or self.mode
|
segs = segs if segs is not None else self.segments
|
||||||
text = "".join(self.keys) if m == LITERAL else self._zhuyin_commit_text()
|
text = "".join(self._render(s, True) for s in segs)
|
||||||
if self.forced_mode:
|
if self.forced_mode:
|
||||||
self.classifier.learned.record("".join(self.keys), self.forced_mode)
|
self.classifier.learned.record("".join(self.keys), self.forced_mode)
|
||||||
self._previous_mode = m
|
self._previous_mode = segs[-1][2] if segs else None
|
||||||
self._previous_time = self.now()
|
self._previous_time = self.now()
|
||||||
self.reset()
|
self.reset()
|
||||||
return text
|
return text
|
||||||
@@ -210,6 +386,9 @@ class CompositionEngine:
|
|||||||
|
|
||||||
def reset(self):
|
def reset(self):
|
||||||
self.keys = []
|
self.keys = []
|
||||||
|
self.held = None
|
||||||
|
self.backtick = False
|
||||||
|
self.cursor = None
|
||||||
self.forced_mode = None
|
self.forced_mode = None
|
||||||
self.fixed = {}
|
self.fixed = {}
|
||||||
self.candidates = []
|
self.candidates = []
|
||||||
|
|||||||
@@ -56,6 +56,29 @@ class EnglishModel:
|
|||||||
plaus = self.plausibility(w)
|
plaus = self.plausibility(w)
|
||||||
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||||
|
|
||||||
|
def word_features(self, word, complete):
|
||||||
|
"""Feature dict for the classifier (see features.py); score() is their
|
||||||
|
dot product with the default weights."""
|
||||||
|
w = word.lower()
|
||||||
|
n = float(len(w))
|
||||||
|
f = {}
|
||||||
|
if n <= 2:
|
||||||
|
f["lit.short"] = 1.0
|
||||||
|
if w in self.common:
|
||||||
|
f["lit.common"] = 1.0
|
||||||
|
f["lit.common_len"] = n
|
||||||
|
elif w in self.dictionary:
|
||||||
|
f["lit.dict"] = 1.0
|
||||||
|
f["lit.dict_len"] = n
|
||||||
|
elif not complete and _has_prefix(w, self.sorted_common):
|
||||||
|
f["lit.prefix_common_len"] = n
|
||||||
|
elif not complete and _has_prefix(w, self.sorted_words):
|
||||||
|
f["lit.prefix_dict_len"] = n
|
||||||
|
else:
|
||||||
|
plaus = self.plausibility(w)
|
||||||
|
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
|
||||||
|
return f
|
||||||
|
|
||||||
def plausibility(self, w):
|
def plausibility(self, w):
|
||||||
chars = ["^"] + list(w) + ["$"]
|
chars = ["^"] + list(w) + ["$"]
|
||||||
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
|
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
|
||||||
|
|||||||
@@ -0,0 +1,138 @@
|
|||||||
|
"""Segment features for the Zhuyin / literal classifier.
|
||||||
|
|
||||||
|
A segment's score in each mode is the dot product of its feature dict with a weight
|
||||||
|
dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights
|
||||||
|
(training/train.py → Resources/classifier-weights.json) replace them.
|
||||||
|
|
||||||
|
Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical
|
||||||
|
features; Tests/.../ParityTests checks this against training/parity-fixtures.json.
|
||||||
|
"""
|
||||||
|
from . import bopomofo
|
||||||
|
|
||||||
|
DEFAULT_WEIGHTS = {
|
||||||
|
# Zhuyin
|
||||||
|
"zh.bias": 0.5,
|
||||||
|
"zh.stray": -5.0,
|
||||||
|
"zh.reordered": -0.5,
|
||||||
|
"zh.toned_valid_lex": 2.8,
|
||||||
|
"zh.toned_valid_nolex": 0.5,
|
||||||
|
"zh.toned_invalid": -5.0,
|
||||||
|
"zh.pending_prefix": 0.0,
|
||||||
|
"zh.pending_invalid": -5.0,
|
||||||
|
"zh.untoned_mid_valid": -2.5,
|
||||||
|
"zh.untoned_mid_invalid": -5.0,
|
||||||
|
"zh.unconfirmed": -2.0,
|
||||||
|
"zh.lexscore": 0.0,
|
||||||
|
"zh.phrase": 0.0,
|
||||||
|
# Literal
|
||||||
|
"lit.bias": 0.0,
|
||||||
|
"lit.upper": 20.0,
|
||||||
|
"lit.num_digits": 0.8,
|
||||||
|
"lit.num_others": -1.0,
|
||||||
|
"lit.common": 3.0,
|
||||||
|
"lit.common_len": 1.0,
|
||||||
|
"lit.dict": 1.0,
|
||||||
|
"lit.dict_len": 0.6,
|
||||||
|
"lit.prefix_common_len": 0.4,
|
||||||
|
"lit.prefix_dict_len": 0.2,
|
||||||
|
"lit.unknown_score": 1.0,
|
||||||
|
"lit.short": 0.0,
|
||||||
|
"lit.digits": -1.5,
|
||||||
|
"lit.others": -1.5,
|
||||||
|
# Segmentation
|
||||||
|
"seg.switch": -3.0,
|
||||||
|
# Context: the first segment has the same mode as the previously committed text
|
||||||
|
"ctx.zh": 1.5,
|
||||||
|
"ctx.lit": 1.5,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def dot(weights, feats):
|
||||||
|
return sum(weights.get(k, 0.0) * v for k, v in feats.items())
|
||||||
|
|
||||||
|
|
||||||
|
def _add(f, k, v=1.0):
|
||||||
|
f[k] = f.get(k, 0.0) + v
|
||||||
|
|
||||||
|
|
||||||
|
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False, previous=None):
|
||||||
|
"""Features of `keys` read as Zhuyin, or None if impossible."""
|
||||||
|
parse = bopomofo.parse(keys)
|
||||||
|
if not is_last and parse.pending is not None:
|
||||||
|
return None # a Zhuyin segment followed by another must end with a tone
|
||||||
|
# Pressing space finishes an untoned last syllable as first tone; after a toned
|
||||||
|
# syllable it just commits, so it must not count as an extra (stray) tone.
|
||||||
|
scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse
|
||||||
|
if scored.foreign_keys > 0 or not scored.syllables:
|
||||||
|
return None
|
||||||
|
f = {"zh.bias": 1.0}
|
||||||
|
if scored.stray_tones:
|
||||||
|
_add(f, "zh.stray", scored.stray_tones)
|
||||||
|
last = len(scored.syllables) - 1
|
||||||
|
toned_run = []
|
||||||
|
for i, ps in enumerate(scored.syllables):
|
||||||
|
if ps.reordered:
|
||||||
|
_add(f, "zh.reordered")
|
||||||
|
valid = bopomofo.is_valid(ps.syllable)
|
||||||
|
if ps.toned:
|
||||||
|
if valid:
|
||||||
|
best = lexicon.best_score(ps.syllable)
|
||||||
|
if best is not None:
|
||||||
|
_add(f, "zh.toned_valid_lex")
|
||||||
|
_add(f, "zh.lexscore", best + 4.0)
|
||||||
|
else:
|
||||||
|
_add(f, "zh.toned_valid_nolex")
|
||||||
|
toned_run.append(ps.syllable)
|
||||||
|
else:
|
||||||
|
_add(f, "zh.toned_invalid")
|
||||||
|
elif i == last:
|
||||||
|
_add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid")
|
||||||
|
else:
|
||||||
|
_add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid")
|
||||||
|
# Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
|
||||||
|
covered = [False] * len(toned_run)
|
||||||
|
for i in range(len(toned_run)):
|
||||||
|
for n in (2, 3, 4):
|
||||||
|
if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]):
|
||||||
|
for k in range(i, i + n):
|
||||||
|
covered[k] = True
|
||||||
|
if any(covered):
|
||||||
|
_add(f, "zh.phrase", sum(covered))
|
||||||
|
if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables):
|
||||||
|
_add(f, "zh.unconfirmed")
|
||||||
|
if after_other:
|
||||||
|
_add(f, "seg.switch")
|
||||||
|
elif previous == "zhuyin":
|
||||||
|
_add(f, "ctx.zh")
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
def literal_features(keys, english, is_last=True, after_other=False, finishing=False, previous=None):
|
||||||
|
"""Features of `keys` output as typed, or None if impossible."""
|
||||||
|
if not keys or " " in keys:
|
||||||
|
return None
|
||||||
|
complete = (not is_last) or finishing
|
||||||
|
f = {"lit.bias": 1.0}
|
||||||
|
if after_other:
|
||||||
|
_add(f, "seg.switch")
|
||||||
|
elif previous == "literal":
|
||||||
|
_add(f, "ctx.lit")
|
||||||
|
if any(c.isupper() for c in keys):
|
||||||
|
_add(f, "lit.upper")
|
||||||
|
letters = "".join(c for c in keys if c.isalpha())
|
||||||
|
digits = sum(1 for c in keys if c.isdigit())
|
||||||
|
others = len(keys) - len(letters) - digits
|
||||||
|
if not letters:
|
||||||
|
if digits:
|
||||||
|
_add(f, "lit.num_digits", digits)
|
||||||
|
if others:
|
||||||
|
_add(f, "lit.num_others", others)
|
||||||
|
return f
|
||||||
|
for k, v in english.word_features(letters, complete or digits > 0).items():
|
||||||
|
_add(f, k, v)
|
||||||
|
if digits:
|
||||||
|
_add(f, "lit.digits", digits)
|
||||||
|
non_apostrophe = others - keys.count("'")
|
||||||
|
if non_apostrophe:
|
||||||
|
_add(f, "lit.others", non_apostrophe)
|
||||||
|
return f
|
||||||
@@ -16,7 +16,7 @@ class Lexicon:
|
|||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self.exact = {}
|
self.exact = {}
|
||||||
self.toneless = {}
|
self._toneless = None # built on first use (only untoned syllables need it)
|
||||||
self.max_phrase_length = 1
|
self.max_phrase_length = 1
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -42,9 +42,10 @@ class Lexicon:
|
|||||||
except ValueError:
|
except ValueError:
|
||||||
score = -6.0
|
score = -6.0
|
||||||
self.add(reading, phrase, score)
|
self.add(reading, phrase, score)
|
||||||
for table in (self.exact, self.toneless):
|
for v in self.exact.values():
|
||||||
for v in table.values():
|
if len(v) > 1:
|
||||||
v.sort(key=lambda e: -e[1])
|
v.sort(key=lambda e: -e[1])
|
||||||
|
self._toneless = None
|
||||||
|
|
||||||
def add(self, reading, phrase, score):
|
def add(self, reading, phrase, score):
|
||||||
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
|
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
|
||||||
@@ -53,15 +54,20 @@ class Lexicon:
|
|||||||
entries = self.exact.setdefault(reading, [])
|
entries = self.exact.setdefault(reading, [])
|
||||||
if not any(p == phrase for p, _ in entries):
|
if not any(p == phrase for p, _ in entries):
|
||||||
entries.append((phrase, score))
|
entries.append((phrase, score))
|
||||||
key = _strip_tones(reading)
|
self._toneless = None
|
||||||
tl = self.toneless.setdefault(key, [])
|
|
||||||
for i, (p, s) in enumerate(tl):
|
@property
|
||||||
if p == phrase:
|
def toneless(self):
|
||||||
if score > s:
|
"""Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
|
||||||
tl[i] = (p, score)
|
if self._toneless is None:
|
||||||
break
|
best = {}
|
||||||
else:
|
for reading, entries in self.exact.items():
|
||||||
tl.append((phrase, score))
|
bucket = best.setdefault(_strip_tones(reading), {})
|
||||||
|
for phrase, score in entries:
|
||||||
|
if score > bucket.get(phrase, float("-inf")):
|
||||||
|
bucket[phrase] = score
|
||||||
|
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
|
||||||
|
return self._toneless
|
||||||
|
|
||||||
def entries(self, syllables):
|
def entries(self, syllables):
|
||||||
if all(s.tone is not None for s in syllables):
|
if all(s.tone is not None for s in syllables):
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
"""Detects Shift pressed and released on its own. Port of ShiftTapDetector.swift."""
|
||||||
|
|
||||||
|
|
||||||
|
class ShiftTapDetector:
|
||||||
|
max_duration = 0.5
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._pressed_at = None
|
||||||
|
self._used = False
|
||||||
|
|
||||||
|
def shift_down(self, time):
|
||||||
|
if self._pressed_at is None: # ignore auto-repeat
|
||||||
|
self._pressed_at = time
|
||||||
|
self._used = False
|
||||||
|
|
||||||
|
def key_pressed(self):
|
||||||
|
"""Any other key while Shift is held: Shift was used as a modifier."""
|
||||||
|
if self._pressed_at is not None:
|
||||||
|
self._used = True
|
||||||
|
|
||||||
|
def shift_up(self, time):
|
||||||
|
"""Returns True when this release completes a Shift tap."""
|
||||||
|
start, self._pressed_at = self._pressed_at, None
|
||||||
|
return start is not None and not self._used and time - start <= self.max_duration
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "自動注音 AutoZhuyin",
|
"name": "自動注音 AutoZhuyin",
|
||||||
"version": "0.1.0",
|
"version": "0.7.0",
|
||||||
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
|
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
|
||||||
"locale": "zh-Hant-TW",
|
"locale": "zh-Hant-TW",
|
||||||
"fallbackLocale": "zh-TW",
|
"fallbackLocale": "zh-TW",
|
||||||
|
|||||||
@@ -26,10 +26,17 @@
|
|||||||
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
|
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
|
||||||
Enter 送出目前組字
|
Enter 送出目前組字
|
||||||
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
|
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
|
||||||
↓ 開啟候選字,按數字鍵 1–9 選字
|
← → 在組字區移動游標,選要改哪個字
|
||||||
|
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字
|
||||||
Esc 取消組字
|
Esc 取消組字
|
||||||
Shift+字母 強制英文
|
單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
|
||||||
|
再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
|
||||||
|
打到一半時按,同 Tab 整段翻轉
|
||||||
|
Shift+字母 打大寫字母(也會強制英文)
|
||||||
< > ? ! : 中文語境下自動變成全形 ,。?!:
|
< > ? ! : 中文語境下自動變成全形 ,。?!:
|
||||||
|
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
|
||||||
|
`' = 、 `[ `] = 「」 `( `) = () `! = !
|
||||||
|
` 後面接其他鍵就是普通的反引號
|
||||||
|
|
||||||
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
|
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
|
||||||
|
|
||||||
|
|||||||
+166
-5
@@ -11,7 +11,8 @@ sys.path.insert(0, os.path.dirname(HERE))
|
|||||||
|
|
||||||
from autozhuyin.core import bopomofo # noqa: E402
|
from autozhuyin.core import bopomofo # noqa: E402
|
||||||
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402
|
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402
|
||||||
from autozhuyin.core.engine import DOWN, ENTER, TAB, CompositionEngine # noqa: E402
|
from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BACKSPACE, CompositionEngine # noqa: E402
|
||||||
|
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
|
||||||
from autozhuyin.core.english import EnglishModel # noqa: E402
|
from autozhuyin.core.english import EnglishModel # noqa: E402
|
||||||
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
|
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
|
||||||
|
|
||||||
@@ -21,6 +22,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
|
|||||||
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
|
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
|
||||||
os.path.join(DATA, "english-dict.txt"),
|
os.path.join(DATA, "english-dict.txt"),
|
||||||
os.path.join(DATA, "english-bigram.json"))
|
os.path.join(DATA, "english-bigram.json"))
|
||||||
|
WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json"))
|
||||||
LOAD_TIME = time.time() - _start
|
LOAD_TIME = time.time() - _start
|
||||||
|
|
||||||
|
|
||||||
@@ -34,13 +36,13 @@ def convert(keys):
|
|||||||
|
|
||||||
|
|
||||||
def make_engine():
|
def make_engine():
|
||||||
return CompositionEngine(Classifier(LEX, ENG))
|
return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS))
|
||||||
|
|
||||||
|
|
||||||
def type_keys(s, e):
|
def type_keys(s, e):
|
||||||
out = ""
|
out = ""
|
||||||
for c in s:
|
for c in s:
|
||||||
key = ENTER if c == "\n" else TAB if c == "\t" else c
|
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN}.get(c, c)
|
||||||
handled, commit = e.handle(key)
|
handled, commit = e.handle(key)
|
||||||
if commit is not None:
|
if commit is not None:
|
||||||
out += commit
|
out += commit
|
||||||
@@ -86,7 +88,7 @@ class ConverterTests(unittest.TestCase):
|
|||||||
|
|
||||||
|
|
||||||
class ClassifierTests(unittest.TestCase):
|
class ClassifierTests(unittest.TestCase):
|
||||||
c = Classifier(LEX, ENG)
|
c = Classifier(LEX, ENG, weights=WEIGHTS)
|
||||||
|
|
||||||
def test_zhuyin(self):
|
def test_zhuyin(self):
|
||||||
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
|
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
|
||||||
@@ -101,7 +103,7 @@ class ClassifierTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_learning(self):
|
def test_learning(self):
|
||||||
learned = LearnedPreferences()
|
learned = LearnedPreferences()
|
||||||
c = Classifier(LEX, ENG, learned)
|
c = Classifier(LEX, ENG, learned, WEIGHTS)
|
||||||
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
|
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
|
||||||
learned.record("so", ZHUYIN)
|
learned.record("so", ZHUYIN)
|
||||||
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
|
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
|
||||||
@@ -150,9 +152,168 @@ class EngineTests(unittest.TestCase):
|
|||||||
self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
|
self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
|
||||||
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
|
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
|
||||||
|
|
||||||
|
def test_tone_then_space_stays_chinese(self):
|
||||||
|
self.assertEqual(len(type_keys("u4 ", make_engine())), 1)
|
||||||
|
|
||||||
def test_out_of_order_converts(self):
|
def test_out_of_order_converts(self):
|
||||||
self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
|
self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
|
||||||
|
|
||||||
|
|
||||||
|
class MixedSegmentTests(unittest.TestCase):
|
||||||
|
def run_keys(self, s):
|
||||||
|
e = make_engine()
|
||||||
|
return type_keys(s, e), e.marked_text
|
||||||
|
|
||||||
|
def test_chinese_then_english(self):
|
||||||
|
self.assertEqual(self.run_keys("ji3hello ")[0], "我hello ")
|
||||||
|
self.assertEqual(self.run_keys("su3cl3github\n")[0], "你好github")
|
||||||
|
self.assertEqual(self.run_keys("ji3hel")[1], "我hel")
|
||||||
|
|
||||||
|
def test_english_then_chinese(self):
|
||||||
|
self.assertEqual(self.run_keys("hellosu3cl3 ")[0], "hello你好")
|
||||||
|
self.assertEqual(self.run_keys("oksu3cl3\n")[0], "ok你好")
|
||||||
|
|
||||||
|
def test_three_segments(self):
|
||||||
|
self.assertEqual(self.run_keys("ji3emailsu3cl3\n")[0], "我email你好")
|
||||||
|
|
||||||
|
def test_single_language_unchanged(self):
|
||||||
|
self.assertEqual(self.run_keys("rup wu0 wu0 fu4cp3cl3\n")[0], "今天天氣很好")
|
||||||
|
self.assertEqual(self.run_keys("helloworld\n")[0], "helloworld")
|
||||||
|
self.assertEqual(self.run_keys("meeting ")[0], "meeting ")
|
||||||
|
self.assertEqual(len(self.run_keys("183\n")[0]), 1)
|
||||||
|
|
||||||
|
def test_speed(self):
|
||||||
|
e = make_engine()
|
||||||
|
start = time.time()
|
||||||
|
for c in "ji3emailsu3cl3rup wu0 wu0 fu4":
|
||||||
|
e.handle(c)
|
||||||
|
e.marked_text
|
||||||
|
per_key = (time.time() - start) / 30
|
||||||
|
print("\nper key: %.1f ms" % (per_key * 1000))
|
||||||
|
self.assertLess(per_key, 0.05)
|
||||||
|
|
||||||
|
|
||||||
|
class ShiftTapTests(unittest.TestCase):
|
||||||
|
def test_toggles_english_mode(self):
|
||||||
|
e = make_engine()
|
||||||
|
self.assertEqual(type_keys("⇧ru su ", e), "ru su ")
|
||||||
|
self.assertTrue(e.english_mode)
|
||||||
|
self.assertEqual(e.marked_text, "")
|
||||||
|
self.assertEqual(e.handle(ENTER), (False, None))
|
||||||
|
self.assertTrue(e.english_mode)
|
||||||
|
self.assertEqual(type_keys("⇧su3cl3\n", e), "你好")
|
||||||
|
self.assertFalse(e.english_mode)
|
||||||
|
|
||||||
|
def test_tap_twice_cancels(self):
|
||||||
|
self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好")
|
||||||
|
|
||||||
|
def test_mid_word_flips_and_learns(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("so", e)
|
||||||
|
e.handle(SHIFT_TAP)
|
||||||
|
self.assertEqual(e.mode, ZHUYIN)
|
||||||
|
self.assertFalse(e.english_mode)
|
||||||
|
type_keys("\n", e)
|
||||||
|
self.assertEqual(e.classifier.learned.weights["so"], 1)
|
||||||
|
|
||||||
|
def test_english_mode_not_learned(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("⇧hello ", e)
|
||||||
|
self.assertEqual(e.classifier.learned.weights, {})
|
||||||
|
|
||||||
|
def test_backspace_goes_to_app(self):
|
||||||
|
e = make_engine()
|
||||||
|
e.handle(SHIFT_TAP)
|
||||||
|
self.assertEqual(e.handle(BACKSPACE), (False, None))
|
||||||
|
self.assertTrue(e.english_mode)
|
||||||
|
|
||||||
|
def test_detector(self):
|
||||||
|
d = ShiftTapDetector()
|
||||||
|
d.shift_down(0)
|
||||||
|
self.assertTrue(d.shift_up(0.1))
|
||||||
|
d.shift_down(1)
|
||||||
|
d.key_pressed()
|
||||||
|
self.assertFalse(d.shift_up(1.1))
|
||||||
|
d.shift_down(2)
|
||||||
|
self.assertFalse(d.shift_up(3))
|
||||||
|
|
||||||
|
|
||||||
|
class CursorTests(unittest.TestCase):
|
||||||
|
def test_moves_and_clamps(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup wu0 wu0 fu4", e)
|
||||||
|
self.assertEqual(e.cursor_offset, 4)
|
||||||
|
type_keys("←", e)
|
||||||
|
self.assertEqual(e.cursor_offset, 3)
|
||||||
|
type_keys("←←←←", e)
|
||||||
|
self.assertEqual(e.cursor_offset, 1)
|
||||||
|
type_keys("→→→→", e)
|
||||||
|
self.assertEqual(e.cursor_offset, 4)
|
||||||
|
|
||||||
|
def test_choose_candidate_in_the_middle(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup wu0 wu0 fu4←←↓", e)
|
||||||
|
self.assertTrue(e.showing_candidates)
|
||||||
|
self.assertTrue(all(r[1] == 2 for r, _ in e.candidates))
|
||||||
|
e.select_candidate([t for _, t in e.candidates].index("添"))
|
||||||
|
self.assertEqual(type_keys("\n", e), "今添天氣")
|
||||||
|
|
||||||
|
def test_typing_returns_cursor_to_end(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("rup wu0 ←", e)
|
||||||
|
self.assertEqual(e.cursor_offset, 1)
|
||||||
|
type_keys("wu0 ", e)
|
||||||
|
self.assertEqual(e.cursor_offset, len(e.marked_text))
|
||||||
|
|
||||||
|
def test_candidates_for_chinese_before_english(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("su3cl3hello↓", e)
|
||||||
|
self.assertTrue(e.showing_candidates)
|
||||||
|
|
||||||
|
|
||||||
|
class FirstToneAfterEnglishTests(unittest.TestCase):
|
||||||
|
def test_first_tone_after_english(self):
|
||||||
|
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
|
||||||
|
e = make_engine()
|
||||||
|
self.assertEqual(type_keys("hello " + keys + " ", e), "hello ", keys)
|
||||||
|
self.assertEqual(len(e.marked_text), 1, keys)
|
||||||
|
|
||||||
|
|
||||||
|
class HeldWordTests(unittest.TestCase):
|
||||||
|
def test_next_word_decides(self):
|
||||||
|
cases = {"up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
|
||||||
|
"up\n": "up", "i love ": "i love "}
|
||||||
|
for keys, want in cases.items():
|
||||||
|
self.assertEqual(type_keys(keys, make_engine()), want, keys)
|
||||||
|
|
||||||
|
def test_held_word_stays_composing(self):
|
||||||
|
e = make_engine()
|
||||||
|
self.assertEqual(type_keys("up ", e), "")
|
||||||
|
self.assertEqual(e.marked_text, "up ")
|
||||||
|
e.handle(BACKSPACE)
|
||||||
|
self.assertEqual(e.marked_text, "up")
|
||||||
|
self.assertEqual(type_keys(" the ", e), "up the ")
|
||||||
|
|
||||||
|
|
||||||
|
class BacktickSymbolTests(unittest.TestCase):
|
||||||
|
def test_forced_symbols(self):
|
||||||
|
cases = {"`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
|
||||||
|
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"}
|
||||||
|
for keys, want in cases.items():
|
||||||
|
self.assertEqual(type_keys(keys, make_engine()), want, keys)
|
||||||
|
|
||||||
|
def test_plain_backtick(self):
|
||||||
|
e = make_engine()
|
||||||
|
self.assertEqual(type_keys("`", e), "")
|
||||||
|
self.assertEqual(e.marked_text, "`")
|
||||||
|
self.assertEqual(type_keys("ls` ", e), "`ls` ")
|
||||||
|
|
||||||
|
def test_escape_cancels(self):
|
||||||
|
e = make_engine()
|
||||||
|
type_keys("su3`", e)
|
||||||
|
e.handle(BACKSPACE)
|
||||||
|
self.assertEqual(e.marked_text, "你")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -36,6 +36,8 @@ class FakeTextService:
|
|||||||
def setCandidateList(self, c): self.reply["candidateList"] = c
|
def setCandidateList(self, c): self.reply["candidateList"] = c
|
||||||
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
|
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
|
||||||
|
|
||||||
|
def showMessage(self, message, duration=3): self.reply["showMessage"] = message
|
||||||
|
|
||||||
def setShowCandidates(self, show):
|
def setShowCandidates(self, show):
|
||||||
self.showCandidates = show
|
self.showCandidates = show
|
||||||
self.reply["showCandidates"] = show
|
self.reply["showCandidates"] = show
|
||||||
@@ -51,8 +53,8 @@ VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
|
|||||||
|
|
||||||
class FakeKeyEvent:
|
class FakeKeyEvent:
|
||||||
def __init__(self, ch, shift=False, ctrl=False):
|
def __init__(self, ch, shift=False, ctrl=False):
|
||||||
self.keyCode = VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA)
|
self.keyCode = 0x10 if ch == "⇧" else VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA)
|
||||||
self.charCode = 0 if ch in ("\n", "\t", "\b", "↓") else ord(ch)
|
self.charCode = 0 if ch in ("\n", "\t", "\b", "↓", "⇧") else ord(ch)
|
||||||
self.keyStates = [0] * 256
|
self.keyStates = [0] * 256
|
||||||
if shift:
|
if shift:
|
||||||
self.keyStates[0x10] = 0x80
|
self.keyStates[0x10] = 0x80
|
||||||
@@ -74,6 +76,17 @@ def press(svc, ch, **kw):
|
|||||||
return True, svc.reply
|
return True, svc.reply
|
||||||
|
|
||||||
|
|
||||||
|
def tap_shift(svc):
|
||||||
|
"""Shift down then up with nothing in between, as PIME delivers it."""
|
||||||
|
svc.reply = {}
|
||||||
|
down = FakeKeyEvent("⇧", shift=True)
|
||||||
|
assert not svc.filterKeyDown(down)
|
||||||
|
up = FakeKeyEvent("⇧")
|
||||||
|
if svc.filterKeyUp(up):
|
||||||
|
svc.onKeyUp(up)
|
||||||
|
return svc.reply
|
||||||
|
|
||||||
|
|
||||||
def type_text(svc, text):
|
def type_text(svc, text):
|
||||||
out = ""
|
out = ""
|
||||||
for ch in text:
|
for ch in text:
|
||||||
@@ -123,5 +136,24 @@ class WrapperTests(unittest.TestCase):
|
|||||||
self.assertEqual(out + reply.get("commitString", ""), "Hi ")
|
self.assertEqual(out + reply.get("commitString", ""), "Hi ")
|
||||||
|
|
||||||
|
|
||||||
|
class ShiftTapWrapperTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.svc = AutoZhuyinTextService(client=None)
|
||||||
|
self.svc.onActivate()
|
||||||
|
|
||||||
|
def test_tap_forces_english_lowercase(self):
|
||||||
|
reply = tap_shift(self.svc)
|
||||||
|
self.assertEqual(reply.get("showMessage"), "英文")
|
||||||
|
self.assertEqual(type_text(self.svc, "ru su "), "ru su ")
|
||||||
|
self.assertEqual(tap_shift(self.svc).get("showMessage"), "中英自動")
|
||||||
|
|
||||||
|
def test_shift_with_letter_is_not_a_tap(self):
|
||||||
|
self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True))
|
||||||
|
press(self.svc, "H", shift=True)
|
||||||
|
up = FakeKeyEvent("⇧")
|
||||||
|
self.assertFalse(self.svc.filterKeyUp(up))
|
||||||
|
self.assertFalse(self.svc.engine.english_mode)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Fills windows/autozhuyin/data/ from the shared Mac resources.
|
"""Fills windows/autozhuyin/data/ from the shared Mac resources.
|
||||||
|
|
||||||
- lexicon.txt, english-common.txt: copied from Resources/
|
- lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/
|
||||||
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
|
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
|
||||||
Webster's 2nd edition, public domain), since Windows has no such file
|
Webster's 2nd edition, public domain), since Windows has no such file
|
||||||
- english-bigram.json: precomputed letter-bigram table
|
- english-bigram.json: precomputed letter-bigram table
|
||||||
@@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402
|
|||||||
|
|
||||||
data = WIN / "autozhuyin" / "data"
|
data = WIN / "autozhuyin" / "data"
|
||||||
data.mkdir(parents=True, exist_ok=True)
|
data.mkdir(parents=True, exist_ok=True)
|
||||||
for name in ("lexicon.txt", "english-common.txt"):
|
for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"):
|
||||||
shutil.copy(ROOT / "Resources" / name, data / name)
|
shutil.copy(ROOT / "Resources" / name, data / name)
|
||||||
|
|
||||||
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")
|
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")
|
||||||
|
|||||||
Reference in New Issue
Block a user