Compare commits

...
7 Commits
Author SHA1 Message Date
ays953 42a2d3a2a3 Release v0.4.0 2026-11-11 09:57:34 +08:00
ays953andClaude Opus 5.5 20291bf515 Add Shift tap to force English for the next word
Pressing and releasing Shift on its own arms literal mode for the next word,
so lowercase English can be typed without it being read as Zhuyin; it resets
after the word is committed and a second tap cancels it. While typing, a
Shift tap flips the buffer like Tab. Forcing before typing is not recorded
as a learned correction; flipping is.

Mac: listens for flagsChanged events. Windows: Shift key-up via PIME's
filterKeyUp/onKeyUp, with a short "英文" tooltip. The tap detection lives in
the core (ShiftTapDetector / shift_tap.py) with tests on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:57:18 +08:00
ays953 21142c2d98 Release v0.3.0 2026-11-11 08:58:43 +08:00
ays953andClaude Opus 5.5 476858d841 Learn classifier weights from synthetic typing data
Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.

Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.

Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:58:08 +08:00
ays953 0a7242ddb3 Release v0.2.0 2026-11-10 16:21:42 +08:00
ays953andClaude Opus 5.5 d1f1112c65 Split mixed Chinese/English input without spaces; add release script
The classifier now segments a buffer into alternating Zhuyin and literal runs
by dynamic programming over split points, so "ji3hello" becomes 我hello and
"hellosu3cl3" becomes hello你好. Each switch costs a penalty, a Zhuyin run
followed by another must end with a tone, and an untoned trailing Zhuyin run
after English is penalized so partial words don't flicker into Bopomofo.
Implemented in both Swift and Python with the same tests; the Mac app colours
each segment separately.

scripts/bump-version.sh bumps the version in Info.plist and ime.json;
scripts/release.sh bumps, tests, packages, commits and tags in one step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:21:13 +08:00
ays953andClaude Opus 5.5 dbe08e00a7 Expand README with install, data, development and release docs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:14:49 +08:00
29 changed files with 6545 additions and 285 deletions
+184 -43
View File
@@ -1,12 +1,63 @@
# 自動注音 AutoZhuyin # 自動注音 AutoZhuyin
macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。 給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
``` ```
按鍵: ji3 hello world su3cl3␣ 按鍵: ji3 hello world su3cl3␣
輸出: 我hello world 你好 輸出: 我hello world 你好
``` ```
## 功能
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **單按 Shift 強制英文**:要打英文時先單按一下 Shift,接下來這個字照打、保持小寫
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:按 ↓ 選最後一個字或詞,選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝
到 Gitea 的 [Releases](https://git.ays.ddnsfree.com/ays953/AutoZhuyin/releases) 頁面下載最新版本。
### macOS
需求:macOS 13 以上,Apple Silicon 和 Intel 都可以(universal binary)。
1. 雙擊 `AutoZhuyin-<版本>.pkg` 安裝。安裝檔沒有簽章,如果被擋下來,按右鍵 →「打開」,或到「系統設定 → 隱私權與安全性」按「仍要打開」。
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
3. 清單裡沒出現的話,登出再登入一次。
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
### Windows
需求:Windows 10 / 11,x64、x86、ARM64 都可以。
1. 先安裝輸入法框架 [PIME](https://github.com/EasyIME/PIME/releases)(免費、開源)。
2. 解壓縮 `AutoZhuyin-Windows-<版本>.zip`,在 `install.bat` 上按右鍵 →「以系統管理員身分執行」。
3. 設定 → 時間與語言 → 語言與地區 → 中文(台灣) → 語言選項 → 新增鍵盤 → **自動注音 AutoZhuyin**。
4. 用 Win + 空白鍵 切換過去。清單裡沒出現的話,重新開機一次。
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
## 按鍵
| 按鍵 | 作用 |
|---|---|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 |
| **單按 Shift** | 沒在打字時:下一個字強制英文(保持小寫),送出後恢復自動判斷,再按一次取消。打到一半時:同 Tab,整段翻轉 |
| Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
## 運作方式 ## 運作方式
每按一個鍵,`Classifier` 會同時用兩種解讀幫目前的按鍵序列打分: 每按一個鍵,`Classifier` 會同時用兩種解讀幫目前的按鍵序列打分:
@@ -14,84 +65,174 @@ macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵
| 注音分數 | 英文分數 | | 注音分數 | 英文分數 |
|---|---| |---|---|
| 是否能拆成合法注音音節(`SyllableTable`) | 是否為常用英文字(`english-common.txt`) | | 是否能拆成合法注音音節(`SyllableTable`) | 是否為常用英文字(`english-common.txt`) |
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在系統字典 `/usr/share/dict/words` | | 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在英文字典裡 |
| 讀音是否存在於詞庫 | 字母 bigram 模型判斷「像不像英文」 | | 讀音是否存在於詞庫 | 字母 bigram 模型判斷「像不像英文」 |
| 前一段是中文 → 加分(語境) | 前一段是英文 → 加分(語境) | | 前一段是中文 → 加分(語境) | 前一段是英文 → 加分(語境) |
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 | | 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
| 一開始稍微偏向注音 | 純數字 → 數字 |
分數高的一方勝出。組字區即時顯示結果(英文以藍字顯示)。 每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。
## 按鍵 中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎:
- 每多切一刀會扣分(`seg.switch` 特徵)
- 後面還接著其他段的注音段,最後一個音節必須打了聲調
- 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
| 按鍵 | 作用 | 音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。
|---|---|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字 |
| Esc | 取消組字 |
| Shift + 字母 | 強制英文 |
| `< > ? ! :` | 中文語境下自動轉成全形 `,。?!:` |
## 建置與安裝 轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
需要 Xcode(使用 `/Applications/Xcode.app` 的工具鏈)。 ## 個人資料
| | macOS | Windows |
|---|---|---|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` |
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
Mac 版可以在輸入法選單選「清除學習紀錄」。
## 開發
### macOS
需要 Xcode。腳本會自動使用 `/Applications/Xcode.app` 的工具鏈。
```bash ```bash
swift test # 執行單元測試(需 DEVELOPER_DIR 指向 Xcode) export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
scripts/build.sh # 產生 build/AutoZhuyin.app xcrun swift test # 單元測試
scripts/install.sh # 安裝到 ~/Library/Input Methods 並重啟輸入法 scripts/build.sh # 產生 build/AutoZhuyin.app(universal)
scripts/install.sh # 裝到 ~/Library/Input Methods、重啟並向系統註冊
scripts/package.sh # 產生 build/AutoZhuyin-<版本>.pkg
``` ```
第一次安裝後:系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。 改完程式後執行 `scripts/install.sh`,再切換到其他輸入法後切回來,就會用到新版。
如果清單裡沒出現,登出再登入一次。 版本號在 `Resources/Info.plist` 的 `CFBundleShortVersionString`。
## 詞庫 ### Windows
`Resources/lexicon.txt` 由開源的小麥注音原始資料轉換而來(MIT 授權,約 17 萬筆), Windows 版建立在 [PIME](https://github.com/EasyIME/PIME) 上,核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣。不需要 Windows 也能開發和測試:
後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換只讀取資料檔,不執行小麥注音的程式碼。
要更新到小麥注音最新資料: ```bash
python3 windows/tools/build_data.py # 產生 windows/autozhuyin/data/
(cd windows && python3 -m unittest discover tests) # 核心測試 + 模擬 PIME 的外殼測試
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
```
- Windows 沒有內建英文字典,所以 `build_data.py` 會把 macOS 的 `/usr/share/dict/words`(韋氏字典,公有領域)轉成 `english-dict.txt` 一起打包,並預先算好 bigram 表。
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
- 版本號在 `windows/autozhuyin/ime.json`。
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
### 訓練判斷權重
中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法:
1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。
2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。
3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。
```bash
python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用)
python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘)
python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘
python3 training/train.py --eval-only # 只評估目前的權重
python3 training/make_parity.py # 權重或特徵改了之後都要重跑
```
目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 |
|---|---|---|
| 手調權重 | 97.58% | 86.15% |
| 訓練後 | **99.62%** | 87.81% |
送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
### 詞庫
`Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。
更新到小麥注音最新資料:
```bash ```bash
scripts/import-mcbopomofo.sh scripts/import-mcbopomofo.sh
scripts/install.sh
``` ```
自訂詞:`~/Library/Application Support/AutoZhuyin/user-phrases.txt`,格式同 `lexicon.txt`: ### 發布新版本
`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`
先 commit 所有修改,再執行:
```bash
scripts/release.sh # 修正版:0.1.0 → 0.1.1
scripts/release.sh minor # 新功能:0.1.1 → 0.2.0
scripts/release.sh major # 大改版:0.2.0 → 1.0.0
scripts/release.sh minor --push # 同時 push commit 和 tag
```
`release.sh` 會自動:
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
2. 跑 Swift 和 Python 全部測試
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip`
4. commit「Release v<版本>」並打上 `v<版本>` tag
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。
## 專案結構 ## 專案結構
``` ```
Sources/AutoZhuyinCore/ 純邏輯,可單元測試 Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試)
Bopomofo.swift 大千鍵盤對應、音節結構 Bopomofo.swift 大千鍵盤對應、音節結構
SyllableTable.swift 合法注音音節表 SyllableTable.swift 合法注音音節表
ZhuyinParser.swift 按鍵 → 音節 ZhuyinParser.swift 按鍵 → 音節(含順序錯誤修正)
Lexicon.swift 詞庫 Lexicon.swift 詞庫
Converter.swift 音節 → 中文(Viterbi) Converter.swift 音節 → 中文(Viterbi)
EnglishModel.swift 英文判斷 EnglishModel.swift 英文判斷
Classifier.swift 中英判斷 + 學習 Features.swift 判斷用的特徵與預設權重
Classifier.swift 中英判斷(特徵 × 權重)、切分、學習
CompositionEngine.swift 輸入法狀態機 CompositionEngine.swift 輸入法狀態機
Sources/AutoZhuyin/ InputMethodKit 外殼 Sources/AutoZhuyin/ macOS InputMethodKit 外殼
Tests/ 單元測試 Tests/ Swift 單元測試
Resources/ Info.plist、詞庫、英文常用字、訓練好的權重
scripts/ 建置、安裝、打包、詞庫轉換
pkg/postinstall PKG 安裝後腳本
windows/
autozhuyin/ PIME 輸入法(整個資料夾會被複製到 PIME)
core/ 核心邏輯(Python 版)
autozhuyin_ime.py PIME 外殼
ime.json PIME 輸入法設定
installer/ install.bat、uninstall.bat、給使用者的 README.txt
tests/ Python 測試
tools/ 產生資料檔、圖示
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
``` ```
## 已知限制
- Windows 版還沒在實機上測試過。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap) ## 下一步(Roadmap)
- [ ] 匯入完整詞庫,並用真實打字資料調整判斷權重 - [x] 匯入完整詞庫
- [x] 注音順序錯誤修正
- [x] Windows 版
- [x] 用資料訓練判斷權重(合成資料)
- [ ] 加入真實打字資料一起訓練
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」 - [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境 - [ ] 用 Core ML 小模型取代手調分數,參考整句語境
- [ ] 使用者詞頻學習(選過的候選字優先) - [ ] 使用者詞頻學習(選過的候選字優先)
- [ ] 偏好設定視窗 - [ ] 偏好設定視窗
## Windows 版 ## 致謝與授權
`windows/` 是給 Windows 用的版本,建立在開源輸入法框架 [PIME](https://github.com/EasyIME/PIME) 上。 - 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣;詞庫共用 `Resources/`。 - Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
```bash
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
```
zip 裡附有 `install.bat`、`uninstall.bat` 和給測試者看的 `README.txt`。使用者需要先安裝 PIME。
修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改。
+14 -14
View File
@@ -4,42 +4,42 @@
<dict> <dict>
<key>CFBundleDevelopmentRegion</key> <key>CFBundleDevelopmentRegion</key>
<string>zh_TW</string> <string>zh_TW</string>
<key>CFBundleDisplayName</key>
<string>自動注音</string>
<key>CFBundleExecutable</key> <key>CFBundleExecutable</key>
<string>AutoZhuyin</string> <string>AutoZhuyin</string>
<key>CFBundleIdentifier</key> <key>CFBundleIdentifier</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin</string> <string>com.autozhuyin.inputmethod.AutoZhuyin</string>
<key>CFBundleName</key> <key>CFBundleName</key>
<string>AutoZhuyin</string> <string>AutoZhuyin</string>
<key>CFBundleDisplayName</key>
<string>自動注音</string>
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.1.0</string> <string>0.4.0</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>1</string> <string>4</string>
<key>LSMinimumSystemVersion</key>
<string>13.0</string>
<key>LSBackgroundOnly</key>
<true/>
<key>LSUIElement</key>
<true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
<string>AutoZhuyinInputController</string> <string>AutoZhuyinInputController</string>
<key>LSBackgroundOnly</key>
<true/>
<key>LSMinimumSystemVersion</key>
<string>13.0</string>
<key>LSUIElement</key>
<true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
<key>TISInputSourceID</key> <key>TISInputSourceID</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin</string> <string>com.autozhuyin.inputmethod.AutoZhuyin</string>
<key>TISIntendedLanguage</key> <key>TISIntendedLanguage</key>
<string>zh-Hant</string> <string>zh-Hant</string>
<key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string>
<key>tsInputMethodCharacterRepertoireKey</key> <key>tsInputMethodCharacterRepertoireKey</key>
<array> <array>
<string>Hant</string> <string>Hant</string>
<string>Latn</string> <string>Latn</string>
</array> </array>
<key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string>
</dict> </dict>
</plist> </plist>
+39
View File
@@ -0,0 +1,39 @@
{
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11",
"examples": 40001,
"test_accuracy": {
"commit": 0.9962,
"typing": 0.8781
},
"weights": {
"lit.bias": 0.4276,
"lit.common": -0.3465,
"lit.common_len": 0.81,
"lit.dict": -1.7902,
"lit.dict_len": 0.6765,
"lit.digits": -2.7403,
"lit.num_digits": -0.0751,
"lit.num_others": -0.7433,
"lit.others": -1.5685,
"lit.prefix_common_len": 0.2082,
"lit.prefix_dict_len": 0.0003,
"lit.short": -0.2397,
"lit.unknown_score": 0.213,
"lit.upper": 20.0,
"seg.switch": -1.7203,
"zh.bias": 1.3522,
"zh.lexscore": 0.26,
"zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9295,
"zh.phrase": 0.4595,
"zh.reordered": -1.4348,
"zh.stray": -5.0,
"zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9823,
"zh.toned_valid_nolex": -1.8742,
"zh.unconfirmed": -0.7891,
"zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.8009
}
}
+29 -9
View File
@@ -26,22 +26,40 @@ enum Shared {
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
return Classifier(lexicon: lexicon, english: english, learned: learned) let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
}() }()
} }
@objc(AutoZhuyinInputController) @objc(AutoZhuyinInputController)
final class InputController: IMKInputController { final class InputController: IMKInputController {
private let engine = CompositionEngine(classifier: Shared.classifier) private let engine = CompositionEngine(classifier: Shared.classifier)
private var shiftTap = ShiftTapDetector()
// MARK: - Key events // MARK: - Key events
override func recognizedEvents(_ sender: Any!) -> Int { override func recognizedEvents(_ sender: Any!) -> Int {
Int(NSEvent.EventTypeMask.keyDown.rawValue) Int(NSEvent.EventTypeMask.keyDown.rawValue | NSEvent.EventTypeMask.flagsChanged.rawValue)
} }
override func handle(_ event: NSEvent!, client sender: Any!) -> Bool { override func handle(_ event: NSEvent!, client sender: Any!) -> Bool {
guard let event, event.type == .keyDown, let client = sender as? IMKTextInput else { return false } guard let event, let client = sender as? IMKTextInput else { return false }
// Shift pressed and released alone: next word is English (or flip the current one).
if event.type == .flagsChanged {
let flags = event.modifierFlags
let tapped = shiftTap.modifiersChanged(
shift: flags.contains(.shift),
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
time: event.timestamp)
if tapped {
_ = engine.handle(.shiftTap)
updateComposition(client)
}
return false
}
guard event.type == .keyDown else { return false }
shiftTap.keyPressed()
// Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first. // Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first.
if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty { if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty {
@@ -90,13 +108,15 @@ final class InputController: IMKInputController {
} }
private func updateComposition(_ client: IMKTextInput) { private func updateComposition(_ client: IMKTextInput) {
let text = engine.markedText // Each segment is underlined; literal (English) segments are blue as a hint.
let attrs: [NSAttributedString.Key: Any] = [ let marked = NSMutableAttributedString()
for piece in engine.markedPieces {
marked.append(NSAttributedString(string: piece.text, attributes: [
.underlineStyle: NSUnderlineStyle.single.rawValue, .underlineStyle: NSUnderlineStyle.single.rawValue,
// Literal (English) composition is shown in a different colour as a hint. .foregroundColor: piece.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
.foregroundColor: engine.mode == .literal ? NSColor.systemBlue : NSColor.textColor, ]))
] }
let marked = NSAttributedString(string: text, attributes: text.isEmpty ? [:] : attrs) let text = marked.string
client.setMarkedText(marked, client.setMarkedText(marked,
selectionRange: NSRange(location: (text as NSString).length, length: 0), selectionRange: NSRange(location: (text as NSString).length, length: 0),
replacementRange: NSRange(location: NSNotFound, length: 0)) replacementRange: NSRange(location: NSNotFound, length: 0))
+81 -52
View File
@@ -7,6 +7,17 @@ public enum InputMode: String, Codable {
case literal case literal
} }
/// A run of keys (indices into the buffer) interpreted in one mode.
public struct Segment: Equatable {
public var range: Range<Int>
public var mode: InputMode
public init(range: Range<Int>, mode: InputMode) {
self.range = range
self.mode = mode
}
}
public struct Classification { public struct Classification {
public var mode: InputMode public var mode: InputMode
public var zhuyinScore: Double public var zhuyinScore: Double
@@ -39,82 +50,100 @@ public final class LearnedPreferences {
} }
/// Decides whether a key sequence is Zhuyin or literal (English / numbers). /// Decides whether a key sequence is Zhuyin or literal (English / numbers).
/// Every segment is scored as features · weights (see Features.swift).
public final class Classifier { public final class Classifier {
public let lexicon: Lexicon public let lexicon: Lexicon
public let english: EnglishModel public let english: EnglishModel
public let learned: LearnedPreferences public let learned: LearnedPreferences
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector
/// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it.
public var zhuyinPrior = 0.5
/// How much the previous segment's mode pulls the current one. /// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5 public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) { public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) {
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned self.learned = learned
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
} }
/// - Parameters: /// (Zhuyin features or nil, literal features or nil) for one segment.
/// - finishing: the user pressed space. Zhuyin is scored as if the space were a public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool,
/// first-tone key; literal is scored as a completed word. finishing: Bool) -> (FeatureVector?, FeatureVector?) {
/// - previous: mode of the previously committed segment, if recent. (Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing),
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification { Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing))
var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior }
var lit = literalScore(keys, complete: finishing)
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
switch previous { switch previous {
case .zhuyin?: zh += contextWeight case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight case .literal?: lit += contextWeight
case nil: break case nil: break
} }
let learnedBias = learned.bias(for: String(keys)) if whole {
if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias } let b = learned.bias(for: String(seg))
return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit) if b > 0 { zh += b } else { lit -= b }
}
return [zh, lit]
} }
public func zhuyinScore(_ keys: [Character]) -> Double { /// The whole buffer as one segment.
let parse = ZhuyinParser.parse(keys) /// - Parameters:
if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity } /// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as
var s = -5.0 * Double(parse.strayTones) /// first tone; literal is scored as a completed word.
for (i, p) in parse.syllables.enumerated() { /// - previous: mode of the previously committed segment, if recent.
let isLast = i == parse.syllables.count - 1 public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
// Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin. let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true)
if p.reordered { s -= 0.5 } return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1])
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5)
} else {
s -= 5
}
} else if isLast {
s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5
} else {
// Untoned syllable in the middle: allowed (tone omission) but unusual.
s += valid ? -2.5 : -5
}
}
return s
} }
public func literalScore(_ keys: [Character], complete: Bool) -> Double { // MARK: - Segmentation
if keys.isEmpty || keys.contains(" ") { return -.infinity }
var score = 0.0
if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English
let letters = String(keys.filter { $0.isLetter }) /// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" →
let digits = keys.filter { $0.isNumber }.count /// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the
let others = keys.count - letters.count - digits /// cost of each switch is the "seg.switch" feature.
public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] {
let n = keys.count
guard n > 0 else { return [] }
let modes: [InputMode] = [.zhuyin, .literal]
// best[i][m]: best score for keys[0..<i] whose last segment has mode m.
var best = Array(repeating: [Double](repeating: -.infinity, count: 2), count: n + 1)
var back = Array(repeating: [Int](repeating: -1, count: 2), count: n + 1) // segment start j
if letters.isEmpty { for i in 1...n {
// Pure number like "2024" (with possible "." or ","). for j in 0..<i {
return score + 0.8 * Double(digits) - 1.0 * Double(others) let s = scores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, finishing: finishing,
previous: j == 0 ? previous : nil, whole: j == 0 && i == n)
for m in 0..<2 where s[m] > -.infinity {
// Adjacent segments must alternate modes; same-mode neighbours are one segment.
if j > 0 && best[j][1 - m] == -.infinity { continue }
let total = j == 0 ? s[m] : best[j][1 - m] + s[m]
if total > best[i][m] {
best[i][m] = total
back[i][m] = j
} }
score += english.score(letters, complete: complete || digits > 0) }
score -= 1.5 * Double(digits) }
// `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not. }
let apostrophes = keys.filter { $0 == "'" }.count
score -= 1.5 * Double(others - apostrophes) guard best[n][0] > -.infinity || best[n][1] > -.infinity else {
return score return [Segment(range: 0..<n, mode: classify(keys, finishing: finishing, previous: previous).mode)]
}
var m = best[n][0] >= best[n][1] ? 0 : 1
var i = n
var out: [Segment] = []
while i > 0 {
let j = back[i][m]
out.append(Segment(range: j..<i, mode: modes[m]))
i = j
m = 1 - m
}
return out.reversed()
} }
} }
+101 -48
View File
@@ -4,6 +4,8 @@ public enum KeyInput: Equatable {
case character(Character) case character(Character)
case backspace, enter, escape, tab case backspace, enter, escape, tab
case up, down, left, right case up, down, left, right
/// Shift pressed and released on its own.
case shiftTap
} }
public struct KeyResult: Equatable { public struct KeyResult: Equatable {
@@ -16,22 +18,33 @@ public struct KeyResult: Equatable {
public static let consumed = KeyResult(handled: true, commit: nil) public static let consumed = KeyResult(handled: true, commit: nil)
} }
/// A piece of the composition as shown to the user.
public struct MarkedPiece: Equatable {
public var text: String
public var mode: InputMode
}
/// The whole input-method state machine, independent of InputMethodKit. /// The whole input-method state machine, independent of InputMethodKit.
/// ///
/// Keystrokes accumulate in a buffer. On every key the classifier decides whether the /// Keystrokes accumulate in a buffer. On every key the classifier splits the buffer
/// buffer is Zhuyin or literal text; the composition (marked text) shows the result /// into Zhuyin and literal segments (e.g. "ji3hello" → 我 + hello); the composition
/// live. Space / Enter / punctuation commit it. Tab flips the decision and the /// (marked text) shows the result live. Space / Enter / punctuation commit it. Tab
/// correction is learned. /// forces the whole buffer into the other mode and the correction is learned.
public final class CompositionEngine { public final class CompositionEngine {
public let classifier: Classifier public let classifier: Classifier
public let converter: Converter public let converter: Converter
public private(set) var keys: [Character] = [] public private(set) var keys: [Character] = []
/// Set by Tab: overrides the classifier for the current buffer. /// Set by Tab / Shift tap: overrides the classifier for the whole buffer.
public private(set) var forcedMode: InputMode? public private(set) var forcedMode: InputMode?
/// User-picked phrases, by syllable range. /// True when forcedMode corrects the classifier (learned on commit); false when the
public private(set) var fixed: [Range<Int>: String] = [:] /// user forced English before typing (Shift tap on an empty buffer), which is not a
/// correction and is not learned.
private var forcedIsCorrection = false
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
private var fixed: [Int: [Range<Int>: String]] = [:]
public private(set) var candidates: [Candidate] = [] public private(set) var candidates: [Candidate] = []
private var candidateSegmentStart = 0
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
@@ -52,33 +65,41 @@ public final class CompositionEngine {
now().timeIntervalSince(previousCommitTime) < contextWindow ? previousMode : nil now().timeIntervalSince(previousCommitTime) < contextWindow ? previousMode : nil
} }
public var mode: InputMode { private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
forcedMode ?? classifier.classify(keys, previous: context).mode guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
return classifier.segment(keys, finishing: finishing, previous: context)
} }
public var parse: ZhuyinParse { ZhuyinParser.parse(keys) } /// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) }
/// Mode of the last segment (what the user is typing right now).
public var mode: InputMode { segments.last?.mode ?? .zhuyin }
/// What to show underlined in the app while composing. /// What to show underlined in the app while composing.
public var markedText: String { public var markedText: String { markedPieces.map(\.text).joined() }
guard !keys.isEmpty else { return "" }
switch mode { public var markedPieces: [MarkedPiece] {
case .literal: let segs = segments
return String(keys) return segs.enumerated().map { i, seg in
case .zhuyin: MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
let p = parse
let done = converter.convert(p.completed.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
return done + (p.pending?.syllable.reading ?? "")
} }
} }
private func validFixed(_ p: ZhuyinParse) -> [Range<Int>: String] { /// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
fixed.filter { $0.key.upperBound <= p.completed.count } private func render(_ seg: Segment, final: Bool) -> String {
let segKeys = Array(keys[seg.range])
guard seg.mode == .zhuyin else { return String(segKeys) }
let p = ZhuyinParser.parse(segKeys)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:])
if final {
let all = p.syllables.map(\.syllable)
return converter.convert(all, fixed: fixedHere.filter { $0.key.upperBound <= all.count }).map(\.text).joined()
} }
let done = p.completed.map(\.syllable)
private func zhuyinCommitText() -> String { let text = converter.convert(done, fixed: fixedHere.filter { $0.key.upperBound <= done.count }).map(\.text).joined()
let p = parse return text + (p.pending?.syllable.reading ?? "")
// An unfinished trailing syllable is converted as untoned.
return converter.convert(p.syllables.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
} }
// MARK: - Key handling // MARK: - Key handling
@@ -92,8 +113,12 @@ public final class CompositionEngine {
case .backspace: case .backspace:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
keys.removeLast() keys.removeLast()
fixed = validFixed(parse) fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] } if keys.isEmpty {
// A Shift-tap "next word is English" stays armed; a correction is dropped.
if forcedIsCorrection { forcedMode = nil; forcedIsCorrection = false }
fixed = [:]
}
return .consumed return .consumed
case .enter: case .enter:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
@@ -104,7 +129,16 @@ public final class CompositionEngine {
return .consumed return .consumed
case .tab: case .tab:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin flip()
return .consumed
case .shiftTap:
if keys.isEmpty {
// Next word is English, typed as-is (lowercase stays lowercase). Tap again to cancel.
forcedMode = forcedMode == .literal ? nil : .literal
forcedIsCorrection = false
} else {
flip()
}
return .consumed return .consumed
case .down: case .down:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
@@ -115,18 +149,27 @@ public final class CompositionEngine {
} }
} }
/// Forces the whole buffer into the other mode (a correction that is learned).
private func flip() {
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
forcedIsCorrection = true
}
/// True while the user has forced English before typing anything (Shift tap).
public var isEnglishArmed: Bool { keys.isEmpty && forcedMode == .literal }
private func handleCharacter(_ c: Character) -> KeyResult { private func handleCharacter(_ c: Character) -> KeyResult {
if c == " " { if c == " " {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
let decided = forcedMode ?? classifier.classify(keys, finishing: true, previous: context).mode let segs = segments(of: keys, finishing: true)
if decided == .literal { if segs.last?.mode == .literal {
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + " ") return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
} }
if parse.pending != nil { if parse(ofLast: segs)?.pending != nil {
keys.append(" ") // first tone keys.append(" ") // first tone
return .consumed return .consumed
} }
return KeyResult(handled: true, commit: commitCurrent(as: .zhuyin)) return KeyResult(handled: true, commit: commitCurrent(segs))
} }
let composable = DachenLayout.isZhuyinKey(c) || (c.isASCII && c.isLetter) || (c == "'" && !keys.isEmpty) let composable = DachenLayout.isZhuyinKey(c) || (c.isASCII && c.isLetter) || (c == "'" && !keys.isEmpty)
@@ -134,9 +177,9 @@ public final class CompositionEngine {
// ", . / ; -" are Zhuyin symbols (ㄝ ㄡ ㄥ ㄤ ㄦ) but act as punctuation after an // ", . / ; -" are Zhuyin symbols (ㄝ ㄡ ㄥ ㄤ ㄦ) but act as punctuation after an
// English word. Decide with the key included: "2;" is ㄉㄤ (檔…), "hello," is English. // English word. Decide with the key included: "2;" is ㄉㄤ (檔…), "hello," is English.
if !keys.isEmpty, DachenLayout.symbols[c] != nil, !c.isLetter, !c.isNumber, forcedMode != .zhuyin { if !keys.isEmpty, DachenLayout.symbols[c] != nil, !c.isLetter, !c.isNumber, forcedMode != .zhuyin {
let withKey = forcedMode ?? classifier.classify(keys + [c], previous: context).mode let withKey = segments(of: keys + [c])
if withKey == .literal && !keys.allSatisfy(\.isNumber) { if withKey.last?.mode == .literal && !keys.allSatisfy(\.isNumber) {
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + String(c)) return KeyResult(handled: true, commit: commitCurrent() + String(c))
} }
} }
keys.append(c) keys.append(c)
@@ -151,25 +194,34 @@ public final class CompositionEngine {
return KeyResult(handled: true, commit: committed + punct) return KeyResult(handled: true, commit: committed + punct)
} }
private func parse(ofLast segs: [Segment]) -> ZhuyinParse? {
guard let last = segs.last, last.mode == .zhuyin else { return nil }
return ZhuyinParser.parse(Array(keys[last.range]))
}
static let fullwidth: [Character: String] = [ static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
] ]
// MARK: - Candidates // MARK: - Candidates (for the last Zhuyin segment)
private func openCandidates() { private func openCandidates() {
let p = parse let segs = segments
guard mode == .zhuyin, !p.completed.isEmpty else { return } guard let last = segs.last, last.mode == .zhuyin else { return }
candidates = converter.candidates(p.completed.map(\.syllable), endingAt: p.completed.count) let done = ZhuyinParser.parse(Array(keys[last.range])).completed.map(\.syllable)
guard !done.isEmpty else { return }
candidates = converter.candidates(done, endingAt: done.count)
candidateSegmentStart = last.range.lowerBound
candidateIndex = 0 candidateIndex = 0
} }
public func selectCandidate(_ index: Int) { public func selectCandidate(_ index: Int) {
guard candidates.indices.contains(index) else { return } guard candidates.indices.contains(index) else { return }
let c = candidates[index] let c = candidates[index]
fixed = fixed.filter { !$0.key.overlaps(c.range) } var here = (fixed[candidateSegmentStart] ?? [:]).filter { !$0.key.overlaps(c.range) }
fixed[c.range] = c.text here[c.range] = c.text
fixed[candidateSegmentStart] = here
candidates = [] candidates = []
} }
@@ -197,13 +249,13 @@ public final class CompositionEngine {
// MARK: - Commit // MARK: - Commit
@discardableResult @discardableResult
private func commitCurrent(as decided: InputMode? = nil) -> String { private func commitCurrent(_ given: [Segment]? = nil) -> String {
let m = decided ?? mode let segs = given ?? segments
let text = m == .literal ? String(keys) : zhuyinCommitText() let text = segs.map { render($0, final: true) }.joined()
if let forced = forcedMode { if let forced = forcedMode, forcedIsCorrection {
classifier.learned.record(String(keys), as: forced) classifier.learned.record(String(keys), as: forced)
} }
previousMode = m previousMode = segs.last?.mode
previousCommitTime = now() previousCommitTime = now()
reset() reset()
return text return text
@@ -217,6 +269,7 @@ public final class CompositionEngine {
public func reset() { public func reset() {
keys = [] keys = []
forcedMode = nil forcedMode = nil
forcedIsCorrection = false
fixed = [:] fixed = [:]
candidates = [] candidates = []
} }
+24
View File
@@ -53,6 +53,30 @@ public final class EnglishModel {
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
} }
/// Feature dict for the classifier (see Features.swift); `score` is their dot
/// product with the default weights.
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
let w = word.lowercased()
let n = Double(w.count)
var f: FeatureVector = [:]
if n <= 2 { f["lit.short"] = 1 }
if common.contains(w) {
f["lit.common"] = 1
f["lit.common_len"] = n
} else if dictionary.contains(w) {
f["lit.dict"] = 1
f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) {
f["lit.prefix_dict_len"] = n
} else {
let plaus = plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
return f
}
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish). /// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
public func plausibility(_ w: String) -> Double { public func plausibility(_ w: String) -> Double {
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
+138
View File
@@ -0,0 +1,138 @@
import Foundation
/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode
/// is the dot product of its features with the weights. `defaultWeights` reproduces the
/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json
/// (training/train.py).
///
/// Python twin: windows/autozhuyin/core/features.py — both must produce identical
/// features (checked by ParityTests against training/parity-fixtures.json).
public typealias FeatureVector = [String: Double]
public enum Features {
public static let defaultWeights: FeatureVector = [
// Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
// Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
// Segmentation
"seg.switch": -3.0,
]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
public static func loadWeights(from url: URL?) -> FeatureVector {
guard let url, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let w = obj["weights"] as? [String: Double] else { return [:] }
return w
}
public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double {
f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value }
}
/// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil }
// Space finishes an untoned last syllable as first tone; after a toned syllable it
// just commits, so it must not count as an extra (stray) tone.
let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse
if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil }
var f: FeatureVector = ["zh.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) }
let last = scored.syllables.count - 1
var tonedRun: [Syllable] = []
for (i, p) in scored.syllables.enumerated() {
if p.reordered { add("zh.reordered") }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
if let best = lexicon.bestScore(for: p.syllable) {
add("zh.toned_valid_lex")
add("zh.lexscore", best + 4)
} else {
add("zh.toned_valid_nolex")
}
tonedRun.append(p.syllable)
} else {
add("zh.toned_invalid")
}
} else if i == last {
add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid")
} else {
add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid")
}
}
// Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
var covered = [Bool](repeating: false, count: tonedRun.count)
for i in tonedRun.indices {
for n in 2...4 where i + n <= tonedRun.count {
if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty {
for k in i..<(i + n) { covered[k] = true }
}
}
}
let coveredCount = covered.filter { $0 }.count
if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) }
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed")
}
if afterOther { add("seg.switch") }
return f
}
/// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther { add("seg.switch") }
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
if digits > 0 { add("lit.num_digits", Double(digits)) }
if others > 0 { add("lit.num_others", Double(others)) }
return f
}
for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) }
if digits > 0 { add("lit.digits", Double(digits)) }
let nonApostrophe = others - keys.filter { $0 == "'" }.count
if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) }
return f
}
}
@@ -0,0 +1,32 @@
import Foundation
/// Detects Shift pressed and released on its own (no other key in between, released
/// within `maxDuration`), as opposed to Shift used for capitals or shortcuts.
///
/// Python twin: windows/autozhuyin/core/shift_tap.py
public struct ShiftTapDetector {
public var maxDuration: TimeInterval = 0.5
private var pressedAt: TimeInterval?
private var used = false
public init() {}
/// Call when the modifier state changes. Returns true when a Shift tap completed.
public mutating func modifiersChanged(shift: Bool, otherModifiers: Bool, time: TimeInterval) -> Bool {
if shift && !otherModifiers {
if pressedAt == nil {
pressedAt = time
used = false
}
return false
}
defer { pressedAt = nil }
guard let start = pressedAt, !shift, !otherModifiers, !used else { return false }
return time - start <= maxDuration
}
/// Call for every other key press: Shift was used as a modifier, not tapped.
public mutating func keyPressed() {
if pressedAt != nil { used = true }
}
}
@@ -7,6 +7,7 @@ private let resources = URL(fileURLWithPath: #filePath)
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase { final class ParserTests: XCTestCase {
func testParsesNiHao() { func testParsesNiHao() {
@@ -84,7 +85,7 @@ final class ConverterTests: XCTestCase {
} }
final class ClassifierTests: XCTestCase { final class ClassifierTests: XCTestCase {
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish) let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
func mode(_ s: String, finishing: Bool = false) -> InputMode { func mode(_ s: String, finishing: Bool = false) -> InputMode {
classifier.classify(Array(s), finishing: finishing).mode classifier.classify(Array(s), finishing: finishing).mode
@@ -106,7 +107,7 @@ final class ClassifierTests: XCTestCase {
func testLearningFlipsDecision() { func testLearningFlipsDecision() {
let learned = LearnedPreferences() let learned = LearnedPreferences()
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned) let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal) XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
learned.record("so", as: .zhuyin) learned.record("so", as: .zhuyin)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin) XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
@@ -115,7 +116,7 @@ final class ClassifierTests: XCTestCase {
final class EngineTests: XCTestCase { final class EngineTests: XCTestCase {
func makeEngine() -> CompositionEngine { func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
} }
/// Types a string and returns everything committed. /// Types a string and returns everything committed.
@@ -188,7 +189,7 @@ final class EngineTests: XCTestCase {
final class DigitInitialTests: XCTestCase { final class DigitInitialTests: XCTestCase {
func engine() -> CompositionEngine { func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
} }
func run(_ s: String) -> (out: String, marked: String) { func run(_ s: String) -> (out: String, marked: String) {
@@ -219,3 +220,134 @@ final class DigitInitialTests: XCTestCase {
XCTAssertEqual(run("hello, ").out, "hello, ") XCTAssertEqual(run("hello, ").out, "hello, ")
} }
} }
final class MixedSegmentTests: XCTestCase {
func run(_ s: String) -> (out: String, marked: String) {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return (out, e.markedText)
}
func testChineseThenEnglish() {
XCTAssertEqual(run("ji3hello ").out, "我hello ")
XCTAssertEqual(run("su3cl3github\n").out, "你好github")
XCTAssertEqual(run("ji3hel").marked, "我hel")
}
func testEnglishThenChinese() {
XCTAssertEqual(run("hellosu3cl3 ").out, "hello你好")
XCTAssertEqual(run("oksu3cl3\n").out, "ok你好")
}
func testThreeSegments() {
XCTAssertEqual(run("ji3emailsu3cl3\n").out, "我email你好")
}
func testSingleLanguageUnchanged() {
XCTAssertEqual(run("rup wu0 wu0 fu4cp3cl3\n").out, "今天天氣很好")
XCTAssertEqual(run("helloworld\n").out, "helloworld")
XCTAssertEqual(run("meeting ").out, "meeting ")
XCTAssertEqual(run("183\n").out.count, 1)
}
}
/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py).
final class ParityTests: XCTestCase {
func testMatchesPythonSegmentation() throws {
let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json")
let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]]
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
var mismatches: [String] = []
for fx in fixtures {
let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
}
if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") }
}
XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
}
func testToneThenSpaceStaysChinese() {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out.count, 1, out)
}
}
final class ShiftTapTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "⇧" ? .shiftTap : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testShiftTapForcesNextWordEnglishLowercase() {
let e = engine()
// "ru" alone would be ㄐㄧ (機); after a Shift tap it stays English, lowercase.
XCTAssertEqual(type("⇧ru ", into: e), "ru ")
// Automatic detection resumes after the word.
XCTAssertFalse(e.isEnglishArmed)
XCTAssertEqual(type("su3cl3\n", into: e), "你好")
}
func testShiftTapTwiceCancels() {
XCTAssertEqual(type("⇧⇧su3cl3\n", into: engine()), "你好")
}
func testShiftTapMidWordFlips() {
let e = engine()
_ = type("so", into: e)
XCTAssertEqual(e.mode, .literal)
_ = e.handle(.shiftTap)
XCTAssertEqual(e.mode, .zhuyin)
_ = type("\n", into: e)
XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned
}
func testForcingBeforeTypingIsNotLearned() {
let e = engine()
_ = type("⇧hello ", into: e)
XCTAssertNil(e.classifier.learned.weights["hello"])
}
func testArmingSurvivesBackspace() {
let e = engine()
_ = type("⇧r", into: e)
_ = e.handle(.backspace)
XCTAssertTrue(e.isEnglishArmed)
XCTAssertEqual(type("ru ", into: e), "ru ")
}
func testDetector() {
var d = ShiftTapDetector()
XCTAssertFalse(d.modifiersChanged(shift: true, otherModifiers: false, time: 0))
XCTAssertTrue(d.modifiersChanged(shift: false, otherModifiers: false, time: 0.1))
// Shift used for a capital letter: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 1)
d.keyPressed()
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 1.1))
// Held too long: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 2)
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 3))
// Shift + Command: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 4)
_ = d.modifiersChanged(shift: true, otherModifiers: true, time: 4.1)
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 4.2))
}
}
+1 -1
View File
@@ -18,7 +18,7 @@ rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources" mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
+32
View File
@@ -0,0 +1,32 @@
#!/bin/bash
# Increments the version in every place it lives:
# Resources/Info.plist CFBundleShortVersionString (x.y.z) and CFBundleVersion (build number)
# windows/autozhuyin/ime.json "version"
#
# Usage: scripts/bump-version.sh [patch|minor|major] (default: patch)
# Prints the new version.
set -euo pipefail
cd "$(dirname "$0")/.."
LEVEL="${1:-patch}"
PLIST=Resources/Info.plist
IME=windows/autozhuyin/ime.json
CURRENT="$(plutil -extract CFBundleShortVersionString raw "$PLIST")"
IFS=. read -r MAJOR MINOR PATCH <<< "$CURRENT"
case "$LEVEL" in
major) MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0 ;;
minor) MINOR=$((MINOR + 1)); PATCH=0 ;;
patch) PATCH=$((PATCH + 1)) ;;
*) echo "usage: $0 [patch|minor|major]" >&2; exit 2 ;;
esac
NEW="$MAJOR.$MINOR.$PATCH"
BUILD=$(( $(plutil -extract CFBundleVersion raw "$PLIST") + 1 ))
plutil -replace CFBundleShortVersionString -string "$NEW" "$PLIST"
plutil -replace CFBundleVersion -string "$BUILD" "$PLIST"
# Edit only the version line so the rest of ime.json keeps its formatting.
sed -i '' -E "s/(\"version\"[[:space:]]*:[[:space:]]*\")[^\"]*\"/\1$NEW\"/" "$IME"
grep -q "\"version\": \"$NEW\"" "$IME" || { echo "failed to update $IME" >&2; exit 1; }
echo "$NEW"
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# One-step release: bump the version, run all tests, build both installers, then
# commit the version change and tag it.
#
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
# --push also pushes the commit and tag to origin
set -euo pipefail
cd "$(dirname "$0")/.."
LEVEL=patch
PUSH=0
for arg in "$@"; do
case "$arg" in
patch|minor|major) LEVEL="$arg" ;;
--push) PUSH=1 ;;
*) echo "usage: $0 [patch|minor|major] [--push]" >&2; exit 2 ;;
esac
done
if [ -n "$(git status --porcelain)" ]; then
echo "Working tree has uncommitted changes; commit them first." >&2
exit 1
fi
export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}"
VERSION="$(scripts/bump-version.sh "$LEVEL")"
echo "==> Version $VERSION"
restore() {
echo "Release failed; reverting the version bump." >&2
git checkout -- Resources/Info.plist windows/autozhuyin/ime.json
}
trap restore ERR
echo "==> Swift tests"
xcrun swift test -q 2>&1 | tail -3
echo "==> Packaging (runs the Python tests too)"
scripts/package.sh >/dev/null
scripts/package-windows.sh >/dev/null
trap - ERR
git add Resources/Info.plist windows/autozhuyin/ime.json
git commit -q -m "Release v$VERSION"
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
echo "==> Tagged v$VERSION"
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
if [ "$PUSH" = 1 ]; then
git push origin HEAD "v$VERSION"
else
echo "Push with: git push origin HEAD v$VERSION"
fi
+30
View File
@@ -0,0 +1,30 @@
"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
import os
import sys
import types
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
WIN = os.path.join(ROOT, "windows")
DATA = os.path.join(WIN, "autozhuyin", "data")
RESOURCES = os.path.join(ROOT, "Resources")
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
sys.path.insert(0, WIN)
from autozhuyin.core import bopomofo, features # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
def load_models():
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
sys.exit("Run windows/tools/build_data.py first.")
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
return lexicon, english
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Writes training/parity-fixtures.json: segmentations produced by the Python core
with the current weights. The Swift tests (ParityTests) must produce the same ones,
which keeps the two implementations in sync. Re-run after changing features or weights.
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator # noqa: E402
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json")
HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up",
"u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"]
def main():
lexicon, english = load_models()
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f) for k in HAND_PICKED for f in (False, True)]
while len(cases) < 400:
ex = gen.example()
if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"]))
cases.append((gen.prefix_of(ex)["keys"], False))
fixtures = [{"keys": k, "finishing": f,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases]
with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT))
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+132
View File
@@ -0,0 +1,132 @@
"""Synthetic typing data for the Zhuyin / literal classifier.
Each example is what sits in the input buffer when the user commits it, with the
intended mode of every key:
{"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"}
labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter.
Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys;
English from the common-word list and the dictionary. Real typing commits English at
a space, so a buffer holds at most one English word next to Chinese.
"""
import random
from common import core
bopomofo = core.bopomofo
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
def reading_to_keys(reading):
"""'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable."""
out = []
for syl in reading.split("-"):
tone = " "
if syl and syl[-1] in TONE_KEY:
tone = TONE_KEY[syl[-1]]
syl = syl[:-1]
if not syl or syl not in bopomofo.VALID:
return None
out.append("".join(KEY_OF[c] for c in syl) + tone)
return "".join(out)
def _is_cjk(s):
return all(0x4E00 <= ord(c) <= 0x9FFF for c in s)
class Generator:
def __init__(self, lexicon_path, common_path, dict_path, seed=0):
self.rng = random.Random(seed)
phrases, weights = [], []
with open(lexicon_path, encoding="utf-8") as f:
for line in f:
cols = line.split()
if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]):
continue
n = cols[0].count("-") + 1
if n > 4:
continue
keys = reading_to_keys(cols[0])
if keys is None:
continue
phrases.append((cols[1], keys))
weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency
self.phrases = phrases
self.phrase_cum = _cumulative(weights)
self.common = [w.strip() for w in open(common_path, encoding="utf-8")
if w.strip() and not w.startswith("#")]
self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8")
if 3 <= len(w.strip()) <= 9]
# --- pieces ---
def chinese(self):
n = self.rng.choice([1, 1, 2, 2, 3])
picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n)
return "".join(keys for _, keys in picked)
def english(self):
w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words)
if self.rng.random() < 0.1:
w = w[:1].upper() + w[1:]
return w
def number(self):
r = self.rng.random()
if r < 0.4:
return str(self.rng.randint(0, 9999))
if r < 0.7:
return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30]))
return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9))
# --- examples ---
def example(self):
r = self.rng.random()
if r < 0.40:
parts, kind = [("z", self.chinese())], "zh"
elif r < 0.65:
parts, kind = [("l", self.english())], "en"
elif r < 0.80:
parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en"
elif r < 0.93:
parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh"
elif r < 0.96:
parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh"
else:
parts, kind = [("l", self.number())], "num"
keys = "".join(p for _, p in parts)
labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l":
finishing = True # English / numbers end with a space
else:
# Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind}
def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix"}
def _cumulative(ws):
out, t = [], 0.0
for w in ws:
t += w
out.append(t)
return out
def gold_segments(labels):
"""'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]"""
segs, start = [], 0
for i in range(1, len(labels) + 1):
if i == len(labels) or labels[i] != labels[start]:
segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL))
start = i
return segs
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""Learns the classifier weights from synthetic typing data (averaged structured
perceptron over the same segmentation DP the input method runs) and reports
accuracy against the hand-tuned defaults.
Usage (from the repo root, after windows/tools/build_data.py):
python3 training/train.py # train, evaluate, write Resources/classifier-weights.json
python3 training/train.py --eval-only # just evaluate the current weights file
"""
import argparse
import datetime
import json
import os
import random
import sys
import time
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator, gold_segments # noqa: E402
MAX_KEYS = 28
FIXED = {"lit.upper"} # hard rule (Shift = English), not learned
def make_set(gen, n):
out = []
while len(out) < 2 * n:
ex = gen.example()
if len(ex["keys"]) > MAX_KEYS:
continue
out.append(ex)
if len(ex["keys"]) > 1:
out.append(gen.prefix_of(ex))
return out
def labels_of(segs, n):
lab = [""] * n
for s, e, m in segs:
for i in range(s, e):
lab[i] = "z" if m == core.ZHUYIN else "l"
return "".join(lab)
def seg_features(clf, keys, segs, finishing):
"""Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float)
n = len(keys)
for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing)
f = fz if m == core.ZHUYIN else fl
if f is None:
return None
for k, v in f.items():
total[k] += v
return total
def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0])
for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]]
bucket[0] += ok
bucket[1] += 1
commit = [v for k, v in by_kind.items() if k != "prefix"]
summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
"by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())},
}
return summary
def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
def train(clf, data, epochs, lr, seed):
rng = random.Random(seed)
w = dict(clf.weights)
total = defaultdict(float)
steps = 0
for epoch in range(epochs):
rng.shuffle(data)
mistakes = 0
t0 = time.time()
for ex in data:
keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], None, w)
if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"])
fp = seg_features(clf, keys, pred, ex["finishing"])
if fg is not None and fp is not None:
mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
for k in set(fg) | set(fp):
if k not in FIXED:
w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0))
steps += 1
for k, v in w.items():
total[k] += v
print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True)
return {k: round(v / steps, 4) for k, v in total.items()}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)")
ap.add_argument("--test", type=int, default=1500)
ap.add_argument("--epochs", type=int, default=6)
ap.add_argument("--lr", type=float, default=0.1)
ap.add_argument("--eval-only", action="store_true")
args = ap.parse_args()
lexicon, english = load_models()
paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"))
test = make_set(Generator(*paths, seed=2), args.test)
clf = core.Classifier(lexicon, english) # default (hand-tuned) weights
show("default", evaluate(clf, clf.weights, test))
if os.path.exists(WEIGHTS_PATH):
current = dict(clf.weights)
current.update(core.Classifier.load_weights(WEIGHTS_PATH))
show("current", evaluate(clf, current, test))
if args.eval_only:
return
data = make_set(Generator(*paths, seed=1), args.train)
print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True)
learned = train(clf, data, args.epochs, args.lr, seed=3)
result = evaluate(clf, learned, test)
show("learned", result)
with open(WEIGHTS_PATH, "w", encoding="utf-8") as f:
json.dump({
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": datetime.date.today().isoformat(),
"examples": len(data),
"test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)},
"weights": dict(sorted(learned.items())),
}, f, ensure_ascii=False, indent=2)
f.write("\n")
print("wrote", WEIGHTS_PATH)
if __name__ == "__main__":
main()
+28 -2
View File
@@ -6,14 +6,16 @@ tests); this file only translates PIME key events and replies.
""" """
import json import json
import os import os
import time
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME) from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
from textService import TextService from textService import TextService
from .core.classifier import Classifier, LearnedPreferences from .core.classifier import Classifier, LearnedPreferences
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, TAB, UP, CompositionEngine from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
from .core.english import EnglishModel from .core.english import EnglishModel
from .core.lexicon import Lexicon from .core.lexicon import Lexicon
from .core.shift_tap import ShiftTapDetector
_HERE = os.path.dirname(os.path.abspath(__file__)) _HERE = os.path.dirname(os.path.abspath(__file__))
_DATA = os.path.join(_HERE, "data") _DATA = os.path.join(_HERE, "data")
@@ -52,7 +54,8 @@ def _shared_classifier():
weights = json.load(f) weights = json.load(f)
except (OSError, ValueError): except (OSError, ValueError):
pass pass
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned)) trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained)
return _classifier return _classifier
@@ -66,6 +69,7 @@ class AutoZhuyinTextService(TextService):
def __init__(self, client): def __init__(self, client):
TextService.__init__(self, client) TextService.__init__(self, client)
self.engine = CompositionEngine(_shared_classifier()) self.engine = CompositionEngine(_shared_classifier())
self.shift_tap = ShiftTapDetector()
# --- lifecycle --- # --- lifecycle ---
@@ -104,6 +108,13 @@ class AutoZhuyinTextService(TextService):
return bool(self.engine.keys or self.engine.candidates) return bool(self.engine.keys or self.engine.candidates)
def filterKeyDown(self, ev): def filterKeyDown(self, ev):
if ev.keyCode == VK_SHIFT: # noqa: F405
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
self.shift_tap.key_pressed()
else:
self.shift_tap.shift_down(time.monotonic())
return False
self.shift_tap.key_pressed()
# Shortcuts (Ctrl/Alt) go to the app. # Shortcuts (Ctrl/Alt) go to the app.
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405 if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
return False return False
@@ -120,6 +131,21 @@ class AutoZhuyinTextService(TextService):
self._update(commit) self._update(commit)
return handled return handled
def filterKeyUp(self, ev):
# Shift pressed and released alone: next word is English (or flip the current one).
if ev.keyCode == VK_SHIFT: # noqa: F405
return self.shift_tap.shift_up(time.monotonic())
return False
def onKeyUp(self, ev):
if ev.keyCode != VK_SHIFT: # noqa: F405
return False
self.engine.handle(SHIFT_TAP)
self._update(None)
if self.engine.english_armed:
self.showMessage("英文", 1)
return True
# --- replies --- # --- replies ---
def _update(self, commit): def _update(self, commit):
+71 -45
View File
@@ -1,5 +1,7 @@
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift.""" """Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
from . import bopomofo import json
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
ZHUYIN = "zhuyin" ZHUYIN = "zhuyin"
LITERAL = "literal" LITERAL = "literal"
@@ -28,67 +30,91 @@ class LearnedPreferences:
class Classifier: class Classifier:
zhuyin_prior = 0.5
context_weight = 1.5 context_weight = 1.5
def __init__(self, lexicon, english, learned=None): def __init__(self, lexicon, english, learned=None, weights=None):
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned or LearnedPreferences() self.learned = learned or LearnedPreferences()
self.weights = dict(DEFAULT_WEIGHTS)
if weights:
self.weights.update(weights)
def classify(self, keys, finishing=False, previous=None): @classmethod
"""Returns (mode, zhuyin_score, literal_score).""" def load_weights(cls, path):
keys = list(keys) """Reads a weights JSON file ({"weights": {...}}); returns {} if missing."""
zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior try:
lit = self.literal_score(keys, complete=finishing) with open(path, encoding="utf-8") as f:
return json.load(f).get("weights", {})
except (OSError, ValueError):
return {}
def segment_features(self, seg, is_last, after_other, finishing):
"""(zhuyin features or None, literal features or None) for one segment."""
seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing),
literal_features(seg, self.english, is_last, after_other, finishing))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing)
zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF
if previous == ZHUYIN: if previous == ZHUYIN:
zh += self.context_weight zh += self.context_weight
elif previous == LITERAL: elif previous == LITERAL:
lit += self.context_weight lit += self.context_weight
b = self.learned.bias("".join(keys)) if whole:
b = self.learned.bias("".join(seg))
if b > 0: if b > 0:
zh += b zh += b
else: else:
lit -= b lit -= b
return zh, lit
def classify(self, keys, finishing=False, previous=None):
"""Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score)."""
zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights)
return (ZHUYIN if zh >= lit else LITERAL), zh, lit return (ZHUYIN if zh >= lit else LITERAL), zh, lit
def mode(self, keys, finishing=False, previous=None): def mode(self, keys, finishing=False, previous=None):
return self.classify(keys, finishing, previous)[0] return self.classify(keys, finishing, previous)[0]
def zhuyin_score(self, keys): # --- segmentation ---
p = bopomofo.parse(keys)
if p.foreign_keys > 0 or not p.syllables:
return NEG_INF
s = -5.0 * p.stray_tones
last = len(p.syllables) - 1
for i, ps in enumerate(p.syllables):
if ps.reordered:
s -= 0.5
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5)
else:
s -= 5
elif i == last:
s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5
else:
s += -2.5 if valid else -5
return s
def literal_score(self, keys, complete): def segment(self, keys, finishing=False, previous=None, weights=None):
if not keys or " " in keys: """Splits keys into alternating segments: [(start, end, mode)], e.g.
return NEG_INF "ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
score = 0.0 split points; each segment is scored by features x weights."""
if any(c.isupper() for c in keys): weights = weights or self.weights
score += 20 keys = list(keys)
letters = "".join(c for c in keys if c.isalpha()) n = len(keys)
digits = sum(1 for c in keys if c.isdigit()) if n == 0:
others = len(keys) - len(letters) - digits return []
if not letters: modes = (ZHUYIN, LITERAL)
return score + 0.8 * digits - 1.0 * others best = [[NEG_INF, NEG_INF] for _ in range(n + 1)]
score += self.english.score(letters, complete or digits > 0) back = [[-1, -1] for _ in range(n + 1)]
score -= 1.5 * digits for i in range(1, n + 1):
apostrophes = keys.count("'") for j in range(i):
score -= 1.5 * (others - apostrophes) scores = self._scores(keys[j:i], i == n, j > 0, finishing,
return score previous if j == 0 else None, j == 0 and i == n, weights)
for m in (0, 1):
if scores[m] == NEG_INF:
continue
if j == 0:
total = scores[m]
else:
if best[j][1 - m] == NEG_INF:
continue
total = best[j][1 - m] + scores[m]
if total > best[i][m]:
best[i][m] = total
back[i][m] = j
if best[n][0] == NEG_INF and best[n][1] == NEG_INF:
return [(0, n, self.mode(keys, finishing, previous))]
m = 0 if best[n][0] >= best[n][1] else 1
out, i = [], n
while i > 0:
j = back[i][m]
out.append((j, i, modes[m]))
i, m = j, 1 - m
return out[::-1]
+102 -43
View File
@@ -2,6 +2,7 @@
Keys are given as one-character strings, or one of the names in SPECIAL_KEYS. Keys are given as one-character strings, or one of the names in SPECIAL_KEYS.
handle() returns (handled, commit_text_or_None). handle() returns (handled, commit_text_or_None).
The buffer is split into Zhuyin / literal segments, e.g. "ji3hello" -> 我 + hello.
""" """
import time import time
@@ -11,7 +12,8 @@ from .lexicon import Converter
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>" BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>" UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>"
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT} SHIFT_TAP = "<shift-tap>" # Shift pressed and released on its own
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT, SHIFT_TAP}
FULLWIDTH = { FULLWIDTH = {
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")",
@@ -32,11 +34,17 @@ class CompositionEngine:
self.now = now self.now = now
self.keys = [] self.keys = []
self.forced_mode = None self.forced_mode = None
self.fixed = {} # True when forced_mode corrects the classifier (learned on commit); False when the
# user forced English before typing (Shift tap on an empty buffer).
self._forced_is_correction = False
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = [] self.candidates = []
self.candidate_index = 0 self.candidate_index = 0
self._candidate_segment_start = 0
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None
self._cache_value = None
# --- derived state --- # --- derived state ---
@@ -44,9 +52,26 @@ class CompositionEngine:
def context(self): def context(self):
return self._previous_mode if self.now() - self._previous_time < self.context_window else None return self._previous_mode if self.now() - self._previous_time < self.context_window else None
def _segments(self, keys, finishing=False):
if not keys:
return []
if self.forced_mode:
return [(0, len(keys), self.forced_mode)]
ctx = self.context
cache_key = (tuple(keys), finishing, ctx)
if cache_key != self._cache_key:
self._cache_key = cache_key
self._cache_value = self.classifier.segment(keys, finishing, ctx)
return self._cache_value
@property
def segments(self):
return self._segments(self.keys)
@property @property
def mode(self): def mode(self):
return self.forced_mode or self.classifier.mode(self.keys, previous=self.context) segs = self.segments
return segs[-1][2] if segs else ZHUYIN
@property @property
def showing_candidates(self): def showing_candidates(self):
@@ -56,34 +81,43 @@ class CompositionEngine:
def page_start(self): def page_start(self):
return (self.candidate_index // self.page_size) * self.page_size return (self.candidate_index // self.page_size) * self.page_size
def _parse(self): def _render(self, seg, final):
return bopomofo.parse(self.keys) start, end, mode = seg
seg_keys = self.keys[start:end]
if mode == LITERAL:
return "".join(seg_keys)
p = bopomofo.parse(seg_keys)
fixed_here = self.fixed.get(start, {})
syl = [s.syllable for s in (p.syllables if final else p.completed)]
fx = {k: v for k, v in fixed_here.items() if k[1] <= len(syl)}
text = "".join(t for _, t in self.converter.convert(syl, fx))
if not final and p.pending:
text += p.pending.syllable.reading
return text
def _valid_fixed(self, p): @property
n = len(p.completed) def marked_pieces(self):
return {k: v for k, v in self.fixed.items() if k[1] <= n} segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
@property @property
def marked_text(self): def marked_text(self):
if not self.keys: return "".join(t for t, _ in self.marked_pieces)
return ""
if self.mode == LITERAL:
return "".join(self.keys)
p = self._parse()
done = self.converter.convert([s.syllable for s in p.completed], self._valid_fixed(p))
pending = p.pending.syllable.reading if p.pending else ""
return "".join(t for _, t in done) + pending
def _zhuyin_commit_text(self):
p = self._parse()
segs = self.converter.convert([s.syllable for s in p.syllables], self._valid_fixed(p))
return "".join(t for _, t in segs)
# --- key handling --- # --- key handling ---
@property
def english_armed(self):
"""True while the user has forced English before typing anything (Shift tap)."""
return not self.keys and self.forced_mode == LITERAL
def _flip(self):
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
self._forced_is_correction = True
def wants(self, key): def wants(self, key):
"""Whether handle(key) would consume the key, without changing state.""" """Whether handle(key) would consume the key, without changing state."""
if self.candidates: if self.candidates or key == SHIFT_TAP:
return True return True
if key in SPECIAL_KEYS: if key in SPECIAL_KEYS:
return bool(self.keys) return bool(self.keys)
@@ -102,9 +136,12 @@ class CompositionEngine:
if not self.keys: if not self.keys:
return PASS return PASS
self.keys.pop() self.keys.pop()
self.fixed = self._valid_fixed(self._parse()) self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys: if not self.keys:
# A Shift-tap "next word is English" stays armed; a correction is dropped.
if self._forced_is_correction:
self.forced_mode = None self.forced_mode = None
self._forced_is_correction = False
self.fixed = {} self.fixed = {}
return CONSUMED return CONSUMED
if key == ENTER: if key == ENTER:
@@ -117,7 +154,15 @@ class CompositionEngine:
if key == TAB: if key == TAB:
if not self.keys: if not self.keys:
return PASS return PASS
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN self._flip()
return CONSUMED
if key == SHIFT_TAP:
if not self.keys:
# Next word is English, typed as-is. Tap again to cancel.
self.forced_mode = None if self.forced_mode == LITERAL else LITERAL
self._forced_is_correction = False
else:
self._flip()
return CONSUMED return CONSUMED
if key == DOWN: if key == DOWN:
if not self.keys: if not self.keys:
@@ -128,25 +173,32 @@ class CompositionEngine:
return CONSUMED if self.keys else PASS return CONSUMED if self.keys else PASS
return self._handle_character(key) return self._handle_character(key)
def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN:
return None
start, end, _ = segs[-1]
return bopomofo.parse(self.keys[start:end])
def _handle_character(self, c): def _handle_character(self, c):
if c == " ": if c == " ":
if not self.keys: if not self.keys:
return PASS return PASS
decided = self.forced_mode or self.classifier.mode(self.keys, finishing=True, previous=self.context) segs = self._segments(self.keys, finishing=True)
if decided == LITERAL: if segs[-1][2] == LITERAL:
return True, self._commit(LITERAL) + " " return True, self._commit(segs) + " "
if self._parse().pending is not None: p = self._last_parse(segs)
if p is not None and p.pending is not None:
self.keys.append(" ") self.keys.append(" ")
return CONSUMED return CONSUMED
return True, self._commit(ZHUYIN) return True, self._commit(segs)
composable = bopomofo.is_zhuyin_key(c) or (c.isascii() and c.isalpha()) or (c == "'" and self.keys) composable = bopomofo.is_zhuyin_key(c) or (c.isascii() and c.isalpha()) or (c == "'" and self.keys)
if composable: if composable:
# ", . / ; -" are Zhuyin symbols but act as punctuation after an English word. # ", . / ; -" are Zhuyin symbols but act as punctuation after an English word.
if self.keys and c in bopomofo.SYMBOLS and not c.isalnum() and self.forced_mode != ZHUYIN: if self.keys and c in bopomofo.SYMBOLS and not c.isalnum() and self.forced_mode != ZHUYIN:
with_key = self.forced_mode or self.classifier.mode(self.keys + [c], previous=self.context) with_key = self._segments(self.keys + [c])
if with_key == LITERAL and not all(k.isdigit() for k in self.keys): if with_key[-1][2] == LITERAL and not all(k.isdigit() for k in self.keys):
return True, self._commit(LITERAL) + c return True, self._commit() + c
self.keys.append(c) self.keys.append(c)
return CONSUMED return CONSUMED
@@ -157,14 +209,18 @@ class CompositionEngine:
return PASS return PASS
return True, committed + punct return True, committed + punct
# --- candidates --- # --- candidates (for the last Zhuyin segment) ---
def _open_candidates(self): def _open_candidates(self):
p = self._parse() segs = self.segments
if self.mode != ZHUYIN or not p.completed: if not segs or segs[-1][2] != ZHUYIN:
return
start, end, _ = segs[-1]
syl = [s.syllable for s in bopomofo.parse(self.keys[start:end]).completed]
if not syl:
return return
syl = [s.syllable for s in p.completed]
self.candidates = self.converter.candidates(syl, len(syl)) self.candidates = self.converter.candidates(syl, len(syl))
self._candidate_segment_start = start
self.candidate_index = 0 self.candidate_index = 0
def page(self): def page(self):
@@ -176,8 +232,10 @@ class CompositionEngine:
if not 0 <= index < len(self.candidates): if not 0 <= index < len(self.candidates):
return return
rng, text = self.candidates[index] rng, text = self.candidates[index]
self.fixed = {k: v for k, v in self.fixed.items() if not (k[0] < rng[1] and rng[0] < k[1])} here = {k: v for k, v in self.fixed.get(self._candidate_segment_start, {}).items()
self.fixed[rng] = text if not (k[0] < rng[1] and rng[0] < k[1])}
here[rng] = text
self.fixed[self._candidate_segment_start] = here
self.candidates = [] self.candidates = []
def _handle_candidate_key(self, key): def _handle_candidate_key(self, key):
@@ -195,12 +253,12 @@ class CompositionEngine:
# --- commit --- # --- commit ---
def _commit(self, decided=None): def _commit(self, segs=None):
m = decided or self.mode segs = segs if segs is not None else self.segments
text = "".join(self.keys) if m == LITERAL else self._zhuyin_commit_text() text = "".join(self._render(s, True) for s in segs)
if self.forced_mode: if self.forced_mode and self._forced_is_correction:
self.classifier.learned.record("".join(self.keys), self.forced_mode) self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = m self._previous_mode = segs[-1][2] if segs else None
self._previous_time = self.now() self._previous_time = self.now()
self.reset() self.reset()
return text return text
@@ -211,5 +269,6 @@ class CompositionEngine:
def reset(self): def reset(self):
self.keys = [] self.keys = []
self.forced_mode = None self.forced_mode = None
self._forced_is_correction = False
self.fixed = {} self.fixed = {}
self.candidates = [] self.candidates = []
+23
View File
@@ -56,6 +56,29 @@ class EnglishModel:
plaus = self.plausibility(w) plaus = self.plausibility(w)
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
def word_features(self, word, complete):
"""Feature dict for the classifier (see features.py); score() is their
dot product with the default weights."""
w = word.lower()
n = float(len(w))
f = {}
if n <= 2:
f["lit.short"] = 1.0
if w in self.common:
f["lit.common"] = 1.0
f["lit.common_len"] = n
elif w in self.dictionary:
f["lit.dict"] = 1.0
f["lit.dict_len"] = n
elif not complete and _has_prefix(w, self.sorted_common):
f["lit.prefix_common_len"] = n
elif not complete and _has_prefix(w, self.sorted_words):
f["lit.prefix_dict_len"] = n
else:
plaus = self.plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
return f
def plausibility(self, w): def plausibility(self, w):
chars = ["^"] + list(w) + ["$"] chars = ["^"] + list(w) + ["$"]
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars))) total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
+131
View File
@@ -0,0 +1,131 @@
"""Segment features for the Zhuyin / literal classifier.
A segment's score in each mode is the dot product of its feature dict with a weight
dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights
(training/train.py → Resources/classifier-weights.json) replace them.
Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical
features; Tests/.../ParityTests checks this against training/parity-fixtures.json.
"""
from . import bopomofo
DEFAULT_WEIGHTS = {
# Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
# Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
# Segmentation
"seg.switch": -3.0,
}
def dot(weights, feats):
return sum(weights.get(k, 0.0) * v for k, v in feats.items())
def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False):
"""Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None:
return None # a Zhuyin segment followed by another must end with a tone
# Pressing space finishes an untoned last syllable as first tone; after a toned
# syllable it just commits, so it must not count as an extra (stray) tone.
scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse
if scored.foreign_keys > 0 or not scored.syllables:
return None
f = {"zh.bias": 1.0}
if scored.stray_tones:
_add(f, "zh.stray", scored.stray_tones)
last = len(scored.syllables) - 1
toned_run = []
for i, ps in enumerate(scored.syllables):
if ps.reordered:
_add(f, "zh.reordered")
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
best = lexicon.best_score(ps.syllable)
if best is not None:
_add(f, "zh.toned_valid_lex")
_add(f, "zh.lexscore", best + 4.0)
else:
_add(f, "zh.toned_valid_nolex")
toned_run.append(ps.syllable)
else:
_add(f, "zh.toned_invalid")
elif i == last:
_add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid")
else:
_add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid")
# Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
covered = [False] * len(toned_run)
for i in range(len(toned_run)):
for n in (2, 3, 4):
if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]):
for k in range(i, i + n):
covered[k] = True
if any(covered):
_add(f, "zh.phrase", sum(covered))
if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables):
_add(f, "zh.unconfirmed")
if after_other:
_add(f, "seg.switch")
return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False):
"""Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys:
return None
complete = (not is_last) or finishing
f = {"lit.bias": 1.0}
if after_other:
_add(f, "seg.switch")
if any(c.isupper() for c in keys):
_add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
if digits:
_add(f, "lit.num_digits", digits)
if others:
_add(f, "lit.num_others", others)
return f
for k, v in english.word_features(letters, complete or digits > 0).items():
_add(f, k, v)
if digits:
_add(f, "lit.digits", digits)
non_apostrophe = others - keys.count("'")
if non_apostrophe:
_add(f, "lit.others", non_apostrophe)
return f
+24
View File
@@ -0,0 +1,24 @@
"""Detects Shift pressed and released on its own. Port of ShiftTapDetector.swift."""
class ShiftTapDetector:
max_duration = 0.5
def __init__(self):
self._pressed_at = None
self._used = False
def shift_down(self, time):
if self._pressed_at is None: # ignore auto-repeat
self._pressed_at = time
self._used = False
def key_pressed(self):
"""Any other key while Shift is held: Shift was used as a modifier."""
if self._pressed_at is not None:
self._used = True
def shift_up(self, time):
"""Returns True when this release completes a Shift tap."""
start, self._pressed_at = self._pressed_at, None
return start is not None and not self._used and time - start <= self.max_duration
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.1.0", "version": "0.4.0",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+3 -1
View File
@@ -28,7 +28,9 @@
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正 Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
↓ 開啟候選字,按數字鍵 1–9 選字 ↓ 開啟候選字,按數字鍵 1–9 選字
Esc 取消組字 Esc 取消組字
Shift+字母 強制英文 單按 Shift 下一個字強制英文(保持小寫),送出後恢復自動判斷;
打到一半時按,同 Tab 整段翻轉。會出現「英文」小提示
Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!: < > ? ! : 中文語境下自動變成全形 ,。?!:
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
+85 -5
View File
@@ -11,7 +11,8 @@ sys.path.insert(0, os.path.dirname(HERE))
from autozhuyin.core import bopomofo # noqa: E402 from autozhuyin.core import bopomofo # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402 from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import DOWN, ENTER, TAB, CompositionEngine # noqa: E402 from autozhuyin.core.engine import DOWN, ENTER, SHIFT_TAP, TAB, BACKSPACE, CompositionEngine # noqa: E402
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402 from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402 from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
@@ -21,6 +22,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"), ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json")) os.path.join(DATA, "english-bigram.json"))
WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json"))
LOAD_TIME = time.time() - _start LOAD_TIME = time.time() - _start
@@ -34,13 +36,13 @@ def convert(keys):
def make_engine(): def make_engine():
return CompositionEngine(Classifier(LEX, ENG)) return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS))
def type_keys(s, e): def type_keys(s, e):
out = "" out = ""
for c in s: for c in s:
key = ENTER if c == "\n" else TAB if c == "\t" else c key = ENTER if c == "\n" else TAB if c == "\t" else SHIFT_TAP if c == "⇧" else c
handled, commit = e.handle(key) handled, commit = e.handle(key)
if commit is not None: if commit is not None:
out += commit out += commit
@@ -86,7 +88,7 @@ class ConverterTests(unittest.TestCase):
class ClassifierTests(unittest.TestCase): class ClassifierTests(unittest.TestCase):
c = Classifier(LEX, ENG) c = Classifier(LEX, ENG, weights=WEIGHTS)
def test_zhuyin(self): def test_zhuyin(self):
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]: for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
@@ -101,7 +103,7 @@ class ClassifierTests(unittest.TestCase):
def test_learning(self): def test_learning(self):
learned = LearnedPreferences() learned = LearnedPreferences()
c = Classifier(LEX, ENG, learned) c = Classifier(LEX, ENG, learned, WEIGHTS)
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL) self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
learned.record("so", ZHUYIN) learned.record("so", ZHUYIN)
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN) self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
@@ -150,9 +152,87 @@ class EngineTests(unittest.TestCase):
self.assertEqual(type_keys("2j/ \n", make_engine()), "東") self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5") self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
def test_tone_then_space_stays_chinese(self):
self.assertEqual(len(type_keys("u4 ", make_engine())), 1)
def test_out_of_order_converts(self): def test_out_of_order_converts(self):
self.assertEqual(type_keys("uvl3\n", make_engine()), "小") self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
class MixedSegmentTests(unittest.TestCase):
def run_keys(self, s):
e = make_engine()
return type_keys(s, e), e.marked_text
def test_chinese_then_english(self):
self.assertEqual(self.run_keys("ji3hello ")[0], "我hello ")
self.assertEqual(self.run_keys("su3cl3github\n")[0], "你好github")
self.assertEqual(self.run_keys("ji3hel")[1], "我hel")
def test_english_then_chinese(self):
self.assertEqual(self.run_keys("hellosu3cl3 ")[0], "hello你好")
self.assertEqual(self.run_keys("oksu3cl3\n")[0], "ok你好")
def test_three_segments(self):
self.assertEqual(self.run_keys("ji3emailsu3cl3\n")[0], "我email你好")
def test_single_language_unchanged(self):
self.assertEqual(self.run_keys("rup wu0 wu0 fu4cp3cl3\n")[0], "今天天氣很好")
self.assertEqual(self.run_keys("helloworld\n")[0], "helloworld")
self.assertEqual(self.run_keys("meeting ")[0], "meeting ")
self.assertEqual(len(self.run_keys("183\n")[0]), 1)
def test_speed(self):
e = make_engine()
start = time.time()
for c in "ji3emailsu3cl3rup wu0 wu0 fu4":
e.handle(c)
e.marked_text
per_key = (time.time() - start) / 30
print("\nper key: %.1f ms" % (per_key * 1000))
self.assertLess(per_key, 0.05)
class ShiftTapTests(unittest.TestCase):
def test_forces_next_word_english_lowercase(self):
e = make_engine()
self.assertEqual(type_keys("⇧ru ", e), "ru ")
self.assertFalse(e.english_armed)
self.assertEqual(type_keys("su3cl3\n", e), "你好")
def test_tap_twice_cancels(self):
self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好")
def test_mid_word_flips_and_learns(self):
e = make_engine()
type_keys("so", e)
e.handle(SHIFT_TAP)
self.assertEqual(e.mode, ZHUYIN)
type_keys("\n", e)
self.assertEqual(e.classifier.learned.weights["so"], 1)
def test_forcing_before_typing_not_learned(self):
e = make_engine()
type_keys("⇧hello ", e)
self.assertNotIn("hello", e.classifier.learned.weights)
def test_arming_survives_backspace(self):
e = make_engine()
type_keys("⇧r", e)
e.handle(BACKSPACE)
self.assertTrue(e.english_armed)
self.assertEqual(type_keys("ru ", e), "ru ")
def test_detector(self):
d = ShiftTapDetector()
d.shift_down(0)
self.assertTrue(d.shift_up(0.1))
d.shift_down(1)
d.key_pressed()
self.assertFalse(d.shift_up(1.1))
d.shift_down(2)
self.assertFalse(d.shift_up(3))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+33 -2
View File
@@ -36,6 +36,8 @@ class FakeTextService:
def setCandidateList(self, c): self.reply["candidateList"] = c def setCandidateList(self, c): self.reply["candidateList"] = c
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
def showMessage(self, message, duration=3): self.reply["showMessage"] = message
def setShowCandidates(self, show): def setShowCandidates(self, show):
self.showCandidates = show self.showCandidates = show
self.reply["showCandidates"] = show self.reply["showCandidates"] = show
@@ -51,8 +53,8 @@ VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
class FakeKeyEvent: class FakeKeyEvent:
def __init__(self, ch, shift=False, ctrl=False): def __init__(self, ch, shift=False, ctrl=False):
self.keyCode = VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA) self.keyCode = 0x10 if ch == "⇧" else VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA)
self.charCode = 0 if ch in ("\n", "\t", "\b", "↓") else ord(ch) self.charCode = 0 if ch in ("\n", "\t", "\b", "↓", "⇧") else ord(ch)
self.keyStates = [0] * 256 self.keyStates = [0] * 256
if shift: if shift:
self.keyStates[0x10] = 0x80 self.keyStates[0x10] = 0x80
@@ -74,6 +76,17 @@ def press(svc, ch, **kw):
return True, svc.reply return True, svc.reply
def tap_shift(svc):
"""Shift down then up with nothing in between, as PIME delivers it."""
svc.reply = {}
down = FakeKeyEvent("⇧", shift=True)
assert not svc.filterKeyDown(down)
up = FakeKeyEvent("⇧")
if svc.filterKeyUp(up):
svc.onKeyUp(up)
return svc.reply
def type_text(svc, text): def type_text(svc, text):
out = "" out = ""
for ch in text: for ch in text:
@@ -123,5 +136,23 @@ class WrapperTests(unittest.TestCase):
self.assertEqual(out + reply.get("commitString", ""), "Hi ") self.assertEqual(out + reply.get("commitString", ""), "Hi ")
class ShiftTapWrapperTests(unittest.TestCase):
def setUp(self):
self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate()
def test_tap_forces_english_lowercase(self):
reply = tap_shift(self.svc)
self.assertEqual(reply.get("showMessage"), "英文")
self.assertEqual(type_text(self.svc, "ru "), "ru ")
def test_shift_with_letter_is_not_a_tap(self):
self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True))
press(self.svc, "H", shift=True)
up = FakeKeyEvent("⇧")
self.assertFalse(self.svc.filterKeyUp(up))
self.assertFalse(self.svc.engine.english_armed)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Fills windows/autozhuyin/data/ from the shared Mac resources. """Fills windows/autozhuyin/data/ from the shared Mac resources.
- lexicon.txt, english-common.txt: copied from Resources/ - lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words, - english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
Webster's 2nd edition, public domain), since Windows has no such file Webster's 2nd edition, public domain), since Windows has no such file
- english-bigram.json: precomputed letter-bigram table - english-bigram.json: precomputed letter-bigram table
@@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402
data = WIN / "autozhuyin" / "data" data = WIN / "autozhuyin" / "data"
data.mkdir(parents=True, exist_ok=True) data.mkdir(parents=True, exist_ok=True)
for name in ("lexicon.txt", "english-common.txt"): for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"):
shutil.copy(ROOT / "Resources" / name, data / name) shutil.copy(ROOT / "Resources" / name, data / name)
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words") word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")