Compare commits

...
5 Commits
Author SHA1 Message Date
ays953 21142c2d98 Release v0.3.0 2026-11-11 08:58:43 +08:00
ays953andClaude Opus 5.5 476858d841 Learn classifier weights from synthetic typing data
Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.

Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.

Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:58:08 +08:00
ays953 0a7242ddb3 Release v0.2.0 2026-11-10 16:21:42 +08:00
ays953andClaude Opus 5.5 d1f1112c65 Split mixed Chinese/English input without spaces; add release script
The classifier now segments a buffer into alternating Zhuyin and literal runs
by dynamic programming over split points, so "ji3hello" becomes 我hello and
"hellosu3cl3" becomes hello你好. Each switch costs a penalty, a Zhuyin run
followed by another must end with a tone, and an untoned trailing Zhuyin run
after English is penalized so partial words don't flicker into Bopomofo.
Implemented in both Swift and Python with the same tests; the Mac app colours
each segment separately.

scripts/bump-version.sh bumps the version in Info.plist and ime.json;
scripts/release.sh bumps, tests, packages, commits and tags in one step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:21:13 +08:00
ays953andClaude Opus 5.5 dbe08e00a7 Expand README with install, data, development and release docs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:14:49 +08:00
25 changed files with 6230 additions and 269 deletions
+188 -49
View File
@@ -1,12 +1,61 @@
# 自動注音 AutoZhuyin
macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
```
按鍵: ji3 hello world su3cl3␣
輸出: 我hello world 你好
```
## 功能
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:按 ↓ 選最後一個字或詞,選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝
到 Gitea 的 [Releases](https://git.ays.ddnsfree.com/ays953/AutoZhuyin/releases) 頁面下載最新版本。
### macOS
需求:macOS 13 以上,Apple Silicon 和 Intel 都可以(universal binary)。
1. 雙擊 `AutoZhuyin-<版本>.pkg` 安裝。安裝檔沒有簽章,如果被擋下來,按右鍵 →「打開」,或到「系統設定 → 隱私權與安全性」按「仍要打開」。
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
3. 清單裡沒出現的話,登出再登入一次。
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
### Windows
需求:Windows 10 / 11,x64、x86、ARM64 都可以。
1. 先安裝輸入法框架 [PIME](https://github.com/EasyIME/PIME/releases)(免費、開源)。
2. 解壓縮 `AutoZhuyin-Windows-<版本>.zip`,在 `install.bat` 上按右鍵 →「以系統管理員身分執行」。
3. 設定 → 時間與語言 → 語言與地區 → 中文(台灣) → 語言選項 → 新增鍵盤 → **自動注音 AutoZhuyin**。
4. 用 Win + 空白鍵 切換過去。清單裡沒出現的話,重新開機一次。
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
## 按鍵
| 按鍵 | 作用 |
|---|---|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 |
| Shift + 字母 | 強制英文 |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
## 運作方式
每按一個鍵,`Classifier` 會同時用兩種解讀幫目前的按鍵序列打分:
@@ -14,84 +63,174 @@ macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵
| 注音分數 | 英文分數 |
|---|---|
| 是否能拆成合法注音音節(`SyllableTable`) | 是否為常用英文字(`english-common.txt`) |
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在系統字典 `/usr/share/dict/words` |
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在英文字典裡 |
| 讀音是否存在於詞庫 | 字母 bigram 模型判斷「像不像英文」 |
| 前一段是中文 → 加分(語境) | 前一段是英文 → 加分(語境) |
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
| 一開始稍微偏向注音 | 純數字 → 數字 |
分數高的一方勝出。組字區即時顯示結果(英文以藍字顯示)。
每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。
## 按鍵
中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎:
- 每多切一刀會扣分(`seg.switch` 特徵)
- 後面還接著其他段的注音段,最後一個音節必須打了聲調
- 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
| 按鍵 | 作用 |
|---|---|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字 |
| Esc | 取消組字 |
| Shift + 字母 | 強制英文 |
| `< > ? ! :` | 中文語境下自動轉成全形 `,。?!:` |
音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。
## 建置與安裝
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
需要 Xcode(使用 `/Applications/Xcode.app` 的工具鏈)。
## 個人資料
| | macOS | Windows |
|---|---|---|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` |
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
Mac 版可以在輸入法選單選「清除學習紀錄」。
## 開發
### macOS
需要 Xcode。腳本會自動使用 `/Applications/Xcode.app` 的工具鏈。
```bash
swift test # 執行單元測試(需 DEVELOPER_DIR 指向 Xcode)
scripts/build.sh # 產生 build/AutoZhuyin.app
scripts/install.sh # 安裝到 ~/Library/Input Methods 並重啟輸入法
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
xcrun swift test # 單元測試
scripts/build.sh # 產生 build/AutoZhuyin.app(universal)
scripts/install.sh # 裝到 ~/Library/Input Methods、重啟並向系統註冊
scripts/package.sh # 產生 build/AutoZhuyin-<版本>.pkg
```
第一次安裝後:系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
如果清單裡沒出現,登出再登入一次。
改完程式後執行 `scripts/install.sh`,再切換到其他輸入法後切回來,就會用到新版。
版本號在 `Resources/Info.plist` 的 `CFBundleShortVersionString`。
## 詞庫
### Windows
`Resources/lexicon.txt` 由開源的小麥注音原始資料轉換而來(MIT 授權,約 17 萬筆),
後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換只讀取資料檔,不執行小麥注音的程式碼。
要更新到小麥注音最新資料:
Windows 版建立在 [PIME](https://github.com/EasyIME/PIME) 上,核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣。不需要 Windows 也能開發和測試:
```bash
python3 windows/tools/build_data.py # 產生 windows/autozhuyin/data/
(cd windows && python3 -m unittest discover tests) # 核心測試 + 模擬 PIME 的外殼測試
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
```
- Windows 沒有內建英文字典,所以 `build_data.py` 會把 macOS 的 `/usr/share/dict/words`(韋氏字典,公有領域)轉成 `english-dict.txt` 一起打包,並預先算好 bigram 表。
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
- 版本號在 `windows/autozhuyin/ime.json`。
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
### 訓練判斷權重
中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法:
1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。
2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。
3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。
```bash
python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用)
python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘)
python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘
python3 training/train.py --eval-only # 只評估目前的權重
python3 training/make_parity.py # 權重或特徵改了之後都要重跑
```
目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 |
|---|---|---|
| 手調權重 | 97.58% | 86.15% |
| 訓練後 | **99.62%** | 87.81% |
送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
### 詞庫
`Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。
更新到小麥注音最新資料:
```bash
scripts/import-mcbopomofo.sh
scripts/install.sh
```
自訂詞:`~/Library/Application Support/AutoZhuyin/user-phrases.txt`,格式同 `lexicon.txt`:
`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`
### 發布新版本
先 commit 所有修改,再執行:
```bash
scripts/release.sh # 修正版:0.1.0 → 0.1.1
scripts/release.sh minor # 新功能:0.1.1 → 0.2.0
scripts/release.sh major # 大改版:0.2.0 → 1.0.0
scripts/release.sh minor --push # 同時 push commit 和 tag
```
`release.sh` 會自動:
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
2. 跑 Swift 和 Python 全部測試
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip`
4. commit「Release v<版本>」並打上 `v<版本>` tag
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。
## 專案結構
```
Sources/AutoZhuyinCore/ 純邏輯,可單元測試
Bopomofo.swift 大千鍵盤對應、音節結構
SyllableTable.swift 合法注音音節表
ZhuyinParser.swift 按鍵 → 音節
Lexicon.swift 詞庫
Converter.swift 音節 → 中文(Viterbi)
EnglishModel.swift 英文判斷
Classifier.swift 中英判斷 + 學習
CompositionEngine.swift 輸入法狀態機
Sources/AutoZhuyin/ InputMethodKit 外殼
Tests/ 單元測試
Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試)
Bopomofo.swift 大千鍵盤對應、音節結構
SyllableTable.swift 合法注音音節表
ZhuyinParser.swift 按鍵 → 音節(含順序錯誤修正)
Lexicon.swift 詞庫
Converter.swift 音節 → 中文(Viterbi)
EnglishModel.swift 英文判斷
Features.swift 判斷用的特徵與預設權重
Classifier.swift 中英判斷(特徵 × 權重)、切分、學習
CompositionEngine.swift 輸入法狀態機
Sources/AutoZhuyin/ macOS InputMethodKit 外殼
Tests/ Swift 單元測試
Resources/ Info.plist、詞庫、英文常用字、訓練好的權重
scripts/ 建置、安裝、打包、詞庫轉換
pkg/postinstall PKG 安裝後腳本
windows/
autozhuyin/ PIME 輸入法(整個資料夾會被複製到 PIME)
core/ 核心邏輯(Python 版)
autozhuyin_ime.py PIME 外殼
ime.json PIME 輸入法設定
installer/ install.bat、uninstall.bat、給使用者的 README.txt
tests/ Python 測試
tools/ 產生資料檔、圖示
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
```
## 已知限制
- Windows 版還沒在實機上測試過。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap)
- [ ] 匯入完整詞庫,並用真實打字資料調整判斷權重
- [x] 匯入完整詞庫
- [x] 注音順序錯誤修正
- [x] Windows 版
- [x] 用資料訓練判斷權重(合成資料)
- [ ] 加入真實打字資料一起訓練
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境
- [ ] 使用者詞頻學習(選過的候選字優先)
- [ ] 偏好設定視窗
## Windows 版
## 致謝與授權
`windows/` 是給 Windows 用的版本,建立在開源輸入法框架 [PIME](https://github.com/EasyIME/PIME) 上。
核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣;詞庫共用 `Resources/`。
```bash
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
```
zip 裡附有 `install.bat`、`uninstall.bat` 和給測試者看的 `README.txt`。使用者需要先安裝 PIME。
修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改。
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
+14 -14
View File
@@ -4,42 +4,42 @@
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>zh_TW</string>
<key>CFBundleDisplayName</key>
<string>自動注音</string>
<key>CFBundleExecutable</key>
<string>AutoZhuyin</string>
<key>CFBundleIdentifier</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
<key>CFBundleName</key>
<string>AutoZhuyin</string>
<key>CFBundleDisplayName</key>
<string>自動注音</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>0.1.0</string>
<string>0.3.0</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>LSMinimumSystemVersion</key>
<string>13.0</string>
<key>LSBackgroundOnly</key>
<true/>
<key>LSUIElement</key>
<true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
<string>3</string>
<key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key>
<string>AutoZhuyinInputController</string>
<key>LSBackgroundOnly</key>
<true/>
<key>LSMinimumSystemVersion</key>
<string>13.0</string>
<key>LSUIElement</key>
<true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
<key>TISInputSourceID</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
<key>TISIntendedLanguage</key>
<string>zh-Hant</string>
<key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string>
<key>tsInputMethodCharacterRepertoireKey</key>
<array>
<string>Hant</string>
<string>Latn</string>
</array>
<key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string>
</dict>
</plist>
+39
View File
@@ -0,0 +1,39 @@
{
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11",
"examples": 40001,
"test_accuracy": {
"commit": 0.9962,
"typing": 0.8781
},
"weights": {
"lit.bias": 0.4276,
"lit.common": -0.3465,
"lit.common_len": 0.81,
"lit.dict": -1.7902,
"lit.dict_len": 0.6765,
"lit.digits": -2.7403,
"lit.num_digits": -0.0751,
"lit.num_others": -0.7433,
"lit.others": -1.5685,
"lit.prefix_common_len": 0.2082,
"lit.prefix_dict_len": 0.0003,
"lit.short": -0.2397,
"lit.unknown_score": 0.213,
"lit.upper": 20.0,
"seg.switch": -1.7203,
"zh.bias": 1.3522,
"zh.lexscore": 0.26,
"zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9295,
"zh.phrase": 0.4595,
"zh.reordered": -1.4348,
"zh.stray": -5.0,
"zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9823,
"zh.toned_valid_nolex": -1.8742,
"zh.unconfirmed": -0.7891,
"zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.8009
}
}
+11 -8
View File
@@ -26,7 +26,8 @@ enum Shared {
let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey)
}
return Classifier(lexicon: lexicon, english: english, learned: learned)
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
}()
}
@@ -90,13 +91,15 @@ final class InputController: IMKInputController {
}
private func updateComposition(_ client: IMKTextInput) {
let text = engine.markedText
let attrs: [NSAttributedString.Key: Any] = [
.underlineStyle: NSUnderlineStyle.single.rawValue,
// Literal (English) composition is shown in a different colour as a hint.
.foregroundColor: engine.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
]
let marked = NSAttributedString(string: text, attributes: text.isEmpty ? [:] : attrs)
// Each segment is underlined; literal (English) segments are blue as a hint.
let marked = NSMutableAttributedString()
for piece in engine.markedPieces {
marked.append(NSAttributedString(string: piece.text, attributes: [
.underlineStyle: NSUnderlineStyle.single.rawValue,
.foregroundColor: piece.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
]))
}
let text = marked.string
client.setMarkedText(marked,
selectionRange: NSRange(location: (text as NSString).length, length: 0),
replacementRange: NSRange(location: NSNotFound, length: 0))
+81 -52
View File
@@ -7,6 +7,17 @@ public enum InputMode: String, Codable {
case literal
}
/// A run of keys (indices into the buffer) interpreted in one mode.
public struct Segment: Equatable {
public var range: Range<Int>
public var mode: InputMode
public init(range: Range<Int>, mode: InputMode) {
self.range = range
self.mode = mode
}
}
public struct Classification {
public var mode: InputMode
public var zhuyinScore: Double
@@ -39,82 +50,100 @@ public final class LearnedPreferences {
}
/// Decides whether a key sequence is Zhuyin or literal (English / numbers).
/// Every segment is scored as features · weights (see Features.swift).
public final class Classifier {
public let lexicon: Lexicon
public let english: EnglishModel
public let learned: LearnedPreferences
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector
/// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it.
public var zhuyinPrior = 0.5
/// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) {
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) {
self.lexicon = lexicon
self.english = english
self.learned = learned
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
}
/// - Parameters:
/// - finishing: the user pressed space. Zhuyin is scored as if the space were a
/// first-tone key; literal is scored as a completed word.
/// - previous: mode of the previously committed segment, if recent.
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior
var lit = literalScore(keys, complete: finishing)
/// (Zhuyin features or nil, literal features or nil) for one segment.
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool,
finishing: Bool) -> (FeatureVector?, FeatureVector?) {
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing),
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing))
}
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
let learnedBias = learned.bias(for: String(keys))
if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias }
return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit)
if whole {
let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b }
}
return [zh, lit]
}
public func zhuyinScore(_ keys: [Character]) -> Double {
let parse = ZhuyinParser.parse(keys)
if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity }
var s = -5.0 * Double(parse.strayTones)
for (i, p) in parse.syllables.enumerated() {
let isLast = i == parse.syllables.count - 1
// Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin.
if p.reordered { s -= 0.5 }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5)
} else {
s -= 5
/// The whole buffer as one segment.
/// - Parameters:
/// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as
/// first tone; literal is scored as a completed word.
/// - previous: mode of the previously committed segment, if recent.
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true)
return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1])
}
// MARK: - Segmentation
/// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" →
/// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the
/// cost of each switch is the "seg.switch" feature.
public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] {
let n = keys.count
guard n > 0 else { return [] }
let modes: [InputMode] = [.zhuyin, .literal]
// best[i][m]: best score for keys[0..<i] whose last segment has mode m.
var best = Array(repeating: [Double](repeating: -.infinity, count: 2), count: n + 1)
var back = Array(repeating: [Int](repeating: -1, count: 2), count: n + 1) // segment start j
for i in 1...n {
for j in 0..<i {
let s = scores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, finishing: finishing,
previous: j == 0 ? previous : nil, whole: j == 0 && i == n)
for m in 0..<2 where s[m] > -.infinity {
// Adjacent segments must alternate modes; same-mode neighbours are one segment.
if j > 0 && best[j][1 - m] == -.infinity { continue }
let total = j == 0 ? s[m] : best[j][1 - m] + s[m]
if total > best[i][m] {
best[i][m] = total
back[i][m] = j
}
}
} else if isLast {
s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5
} else {
// Untoned syllable in the middle: allowed (tone omission) but unusual.
s += valid ? -2.5 : -5
}
}
return s
}
public func literalScore(_ keys: [Character], complete: Bool) -> Double {
if keys.isEmpty || keys.contains(" ") { return -.infinity }
var score = 0.0
if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
// Pure number like "2024" (with possible "." or ",").
return score + 0.8 * Double(digits) - 1.0 * Double(others)
guard best[n][0] > -.infinity || best[n][1] > -.infinity else {
return [Segment(range: 0..<n, mode: classify(keys, finishing: finishing, previous: previous).mode)]
}
score += english.score(letters, complete: complete || digits > 0)
score -= 1.5 * Double(digits)
// `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not.
let apostrophes = keys.filter { $0 == "'" }.count
score -= 1.5 * Double(others - apostrophes)
return score
var m = best[n][0] >= best[n][1] ? 0 : 1
var i = n
var out: [Segment] = []
while i > 0 {
let j = back[i][m]
out.append(Segment(range: j..<i, mode: modes[m]))
i = j
m = 1 - m
}
return out.reversed()
}
}
+70 -46
View File
@@ -16,22 +16,29 @@ public struct KeyResult: Equatable {
public static let consumed = KeyResult(handled: true, commit: nil)
}
/// A piece of the composition as shown to the user.
public struct MarkedPiece: Equatable {
public var text: String
public var mode: InputMode
}
/// The whole input-method state machine, independent of InputMethodKit.
///
/// Keystrokes accumulate in a buffer. On every key the classifier decides whether the
/// buffer is Zhuyin or literal text; the composition (marked text) shows the result
/// live. Space / Enter / punctuation commit it. Tab flips the decision and the
/// correction is learned.
/// Keystrokes accumulate in a buffer. On every key the classifier splits the buffer
/// into Zhuyin and literal segments (e.g. "ji3hello" → 我 + hello); the composition
/// (marked text) shows the result live. Space / Enter / punctuation commit it. Tab
/// forces the whole buffer into the other mode and the correction is learned.
public final class CompositionEngine {
public let classifier: Classifier
public let converter: Converter
public private(set) var keys: [Character] = []
/// Set by Tab: overrides the classifier for the current buffer.
/// Set by Tab: overrides the classifier for the whole buffer.
public private(set) var forcedMode: InputMode?
/// User-picked phrases, by syllable range.
public private(set) var fixed: [Range<Int>: String] = [:]
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
private var fixed: [Int: [Range<Int>: String]] = [:]
public private(set) var candidates: [Candidate] = []
private var candidateSegmentStart = 0
public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty }
@@ -52,33 +59,41 @@ public final class CompositionEngine {
now().timeIntervalSince(previousCommitTime) < contextWindow ? previousMode : nil
}
public var mode: InputMode {
forcedMode ?? classifier.classify(keys, previous: context).mode
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
return classifier.segment(keys, finishing: finishing, previous: context)
}
public var parse: ZhuyinParse { ZhuyinParser.parse(keys) }
/// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) }
/// Mode of the last segment (what the user is typing right now).
public var mode: InputMode { segments.last?.mode ?? .zhuyin }
/// What to show underlined in the app while composing.
public var markedText: String {
guard !keys.isEmpty else { return "" }
switch mode {
case .literal:
return String(keys)
case .zhuyin:
let p = parse
let done = converter.convert(p.completed.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
return done + (p.pending?.syllable.reading ?? "")
public var markedText: String { markedPieces.map(\.text).joined() }
public var markedPieces: [MarkedPiece] {
let segs = segments
return segs.enumerated().map { i, seg in
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
}
}
private func validFixed(_ p: ZhuyinParse) -> [Range<Int>: String] {
fixed.filter { $0.key.upperBound <= p.completed.count }
}
private func zhuyinCommitText() -> String {
let p = parse
// An unfinished trailing syllable is converted as untoned.
return converter.convert(p.syllables.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
private func render(_ seg: Segment, final: Bool) -> String {
let segKeys = Array(keys[seg.range])
guard seg.mode == .zhuyin else { return String(segKeys) }
let p = ZhuyinParser.parse(segKeys)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:])
if final {
let all = p.syllables.map(\.syllable)
return converter.convert(all, fixed: fixedHere.filter { $0.key.upperBound <= all.count }).map(\.text).joined()
}
let done = p.completed.map(\.syllable)
let text = converter.convert(done, fixed: fixedHere.filter { $0.key.upperBound <= done.count }).map(\.text).joined()
return text + (p.pending?.syllable.reading ?? "")
}
// MARK: - Key handling
@@ -92,7 +107,7 @@ public final class CompositionEngine {
case .backspace:
guard !keys.isEmpty else { return .passThrough }
keys.removeLast()
fixed = validFixed(parse)
fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] }
return .consumed
case .enter:
@@ -118,15 +133,15 @@ public final class CompositionEngine {
private func handleCharacter(_ c: Character) -> KeyResult {
if c == " " {
guard !keys.isEmpty else { return .passThrough }
let decided = forcedMode ?? classifier.classify(keys, finishing: true, previous: context).mode
if decided == .literal {
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + " ")
let segs = segments(of: keys, finishing: true)
if segs.last?.mode == .literal {
return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
}
if parse.pending != nil {
if parse(ofLast: segs)?.pending != nil {
keys.append(" ") // first tone
return .consumed
}
return KeyResult(handled: true, commit: commitCurrent(as: .zhuyin))
return KeyResult(handled: true, commit: commitCurrent(segs))
}
let composable = DachenLayout.isZhuyinKey(c) || (c.isASCII && c.isLetter) || (c == "'" && !keys.isEmpty)
@@ -134,9 +149,9 @@ public final class CompositionEngine {
// ", . / ; -" are Zhuyin symbols (ㄝ ㄡ ㄥ ㄤ ㄦ) but act as punctuation after an
// English word. Decide with the key included: "2;" is ㄉㄤ (檔…), "hello," is English.
if !keys.isEmpty, DachenLayout.symbols[c] != nil, !c.isLetter, !c.isNumber, forcedMode != .zhuyin {
let withKey = forcedMode ?? classifier.classify(keys + [c], previous: context).mode
if withKey == .literal && !keys.allSatisfy(\.isNumber) {
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + String(c))
let withKey = segments(of: keys + [c])
if withKey.last?.mode == .literal && !keys.allSatisfy(\.isNumber) {
return KeyResult(handled: true, commit: commitCurrent() + String(c))
}
}
keys.append(c)
@@ -151,25 +166,34 @@ public final class CompositionEngine {
return KeyResult(handled: true, commit: committed + punct)
}
private func parse(ofLast segs: [Segment]) -> ZhuyinParse? {
guard let last = segs.last, last.mode == .zhuyin else { return nil }
return ZhuyinParser.parse(Array(keys[last.range]))
}
static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
]
// MARK: - Candidates
// MARK: - Candidates (for the last Zhuyin segment)
private func openCandidates() {
let p = parse
guard mode == .zhuyin, !p.completed.isEmpty else { return }
candidates = converter.candidates(p.completed.map(\.syllable), endingAt: p.completed.count)
let segs = segments
guard let last = segs.last, last.mode == .zhuyin else { return }
let done = ZhuyinParser.parse(Array(keys[last.range])).completed.map(\.syllable)
guard !done.isEmpty else { return }
candidates = converter.candidates(done, endingAt: done.count)
candidateSegmentStart = last.range.lowerBound
candidateIndex = 0
}
public func selectCandidate(_ index: Int) {
guard candidates.indices.contains(index) else { return }
let c = candidates[index]
fixed = fixed.filter { !$0.key.overlaps(c.range) }
fixed[c.range] = c.text
var here = (fixed[candidateSegmentStart] ?? [:]).filter { !$0.key.overlaps(c.range) }
here[c.range] = c.text
fixed[candidateSegmentStart] = here
candidates = []
}
@@ -197,13 +221,13 @@ public final class CompositionEngine {
// MARK: - Commit
@discardableResult
private func commitCurrent(as decided: InputMode? = nil) -> String {
let m = decided ?? mode
let text = m == .literal ? String(keys) : zhuyinCommitText()
private func commitCurrent(_ given: [Segment]? = nil) -> String {
let segs = given ?? segments
let text = segs.map { render($0, final: true) }.joined()
if let forced = forcedMode {
classifier.learned.record(String(keys), as: forced)
}
previousMode = m
previousMode = segs.last?.mode
previousCommitTime = now()
reset()
return text
+24
View File
@@ -53,6 +53,30 @@ public final class EnglishModel {
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
/// Feature dict for the classifier (see Features.swift); `score` is their dot
/// product with the default weights.
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
let w = word.lowercased()
let n = Double(w.count)
var f: FeatureVector = [:]
if n <= 2 { f["lit.short"] = 1 }
if common.contains(w) {
f["lit.common"] = 1
f["lit.common_len"] = n
} else if dictionary.contains(w) {
f["lit.dict"] = 1
f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) {
f["lit.prefix_dict_len"] = n
} else {
let plaus = plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
return f
}
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
public func plausibility(_ w: String) -> Double {
let chars = ["^"] + w.map(String.init) + ["$"]
+138
View File
@@ -0,0 +1,138 @@
import Foundation
/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode
/// is the dot product of its features with the weights. `defaultWeights` reproduces the
/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json
/// (training/train.py).
///
/// Python twin: windows/autozhuyin/core/features.py — both must produce identical
/// features (checked by ParityTests against training/parity-fixtures.json).
public typealias FeatureVector = [String: Double]
public enum Features {
public static let defaultWeights: FeatureVector = [
// Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
// Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
// Segmentation
"seg.switch": -3.0,
]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
public static func loadWeights(from url: URL?) -> FeatureVector {
guard let url, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let w = obj["weights"] as? [String: Double] else { return [:] }
return w
}
public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double {
f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value }
}
/// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil }
// Space finishes an untoned last syllable as first tone; after a toned syllable it
// just commits, so it must not count as an extra (stray) tone.
let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse
if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil }
var f: FeatureVector = ["zh.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) }
let last = scored.syllables.count - 1
var tonedRun: [Syllable] = []
for (i, p) in scored.syllables.enumerated() {
if p.reordered { add("zh.reordered") }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
if let best = lexicon.bestScore(for: p.syllable) {
add("zh.toned_valid_lex")
add("zh.lexscore", best + 4)
} else {
add("zh.toned_valid_nolex")
}
tonedRun.append(p.syllable)
} else {
add("zh.toned_invalid")
}
} else if i == last {
add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid")
} else {
add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid")
}
}
// Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
var covered = [Bool](repeating: false, count: tonedRun.count)
for i in tonedRun.indices {
for n in 2...4 where i + n <= tonedRun.count {
if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty {
for k in i..<(i + n) { covered[k] = true }
}
}
}
let coveredCount = covered.filter { $0 }.count
if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) }
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed")
}
if afterOther { add("seg.switch") }
return f
}
/// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther { add("seg.switch") }
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
if digits > 0 { add("lit.num_digits", Double(digits)) }
if others > 0 { add("lit.num_others", Double(others)) }
return f
}
for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) }
if digits > 0 { add("lit.digits", Double(digits)) }
let nonApostrophe = others - keys.filter { $0 == "'" }.count
if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) }
return f
}
}
@@ -7,6 +7,7 @@ private let resources = URL(fileURLWithPath: #filePath)
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase {
func testParsesNiHao() {
@@ -84,7 +85,7 @@ final class ConverterTests: XCTestCase {
}
final class ClassifierTests: XCTestCase {
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish)
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
func mode(_ s: String, finishing: Bool = false) -> InputMode {
classifier.classify(Array(s), finishing: finishing).mode
@@ -106,7 +107,7 @@ final class ClassifierTests: XCTestCase {
func testLearningFlipsDecision() {
let learned = LearnedPreferences()
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned)
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
learned.record("so", as: .zhuyin)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
@@ -115,7 +116,7 @@ final class ClassifierTests: XCTestCase {
final class EngineTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
/// Types a string and returns everything committed.
@@ -188,7 +189,7 @@ final class EngineTests: XCTestCase {
final class DigitInitialTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func run(_ s: String) -> (out: String, marked: String) {
@@ -219,3 +220,64 @@ final class DigitInitialTests: XCTestCase {
XCTAssertEqual(run("hello, ").out, "hello, ")
}
}
final class MixedSegmentTests: XCTestCase {
func run(_ s: String) -> (out: String, marked: String) {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return (out, e.markedText)
}
func testChineseThenEnglish() {
XCTAssertEqual(run("ji3hello ").out, "我hello ")
XCTAssertEqual(run("su3cl3github\n").out, "你好github")
XCTAssertEqual(run("ji3hel").marked, "我hel")
}
func testEnglishThenChinese() {
XCTAssertEqual(run("hellosu3cl3 ").out, "hello你好")
XCTAssertEqual(run("oksu3cl3\n").out, "ok你好")
}
func testThreeSegments() {
XCTAssertEqual(run("ji3emailsu3cl3\n").out, "我email你好")
}
func testSingleLanguageUnchanged() {
XCTAssertEqual(run("rup wu0 wu0 fu4cp3cl3\n").out, "今天天氣很好")
XCTAssertEqual(run("helloworld\n").out, "helloworld")
XCTAssertEqual(run("meeting ").out, "meeting ")
XCTAssertEqual(run("183\n").out.count, 1)
}
}
/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py).
final class ParityTests: XCTestCase {
func testMatchesPythonSegmentation() throws {
let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json")
let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]]
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
var mismatches: [String] = []
for fx in fixtures {
let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
}
if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") }
}
XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
}
func testToneThenSpaceStaysChinese() {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out.count, 1, out)
}
}
+1 -1
View File
@@ -18,7 +18,7 @@ rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff
+32
View File
@@ -0,0 +1,32 @@
#!/bin/bash
# Increments the version in every place it lives:
# Resources/Info.plist CFBundleShortVersionString (x.y.z) and CFBundleVersion (build number)
# windows/autozhuyin/ime.json "version"
#
# Usage: scripts/bump-version.sh [patch|minor|major] (default: patch)
# Prints the new version.
set -euo pipefail
cd "$(dirname "$0")/.."
LEVEL="${1:-patch}"
PLIST=Resources/Info.plist
IME=windows/autozhuyin/ime.json
CURRENT="$(plutil -extract CFBundleShortVersionString raw "$PLIST")"
IFS=. read -r MAJOR MINOR PATCH <<< "$CURRENT"
case "$LEVEL" in
major) MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0 ;;
minor) MINOR=$((MINOR + 1)); PATCH=0 ;;
patch) PATCH=$((PATCH + 1)) ;;
*) echo "usage: $0 [patch|minor|major]" >&2; exit 2 ;;
esac
NEW="$MAJOR.$MINOR.$PATCH"
BUILD=$(( $(plutil -extract CFBundleVersion raw "$PLIST") + 1 ))
plutil -replace CFBundleShortVersionString -string "$NEW" "$PLIST"
plutil -replace CFBundleVersion -string "$BUILD" "$PLIST"
# Edit only the version line so the rest of ime.json keeps its formatting.
sed -i '' -E "s/(\"version\"[[:space:]]*:[[:space:]]*\")[^\"]*\"/\1$NEW\"/" "$IME"
grep -q "\"version\": \"$NEW\"" "$IME" || { echo "failed to update $IME" >&2; exit 1; }
echo "$NEW"
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# One-step release: bump the version, run all tests, build both installers, then
# commit the version change and tag it.
#
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
# --push also pushes the commit and tag to origin
set -euo pipefail
cd "$(dirname "$0")/.."
LEVEL=patch
PUSH=0
for arg in "$@"; do
case "$arg" in
patch|minor|major) LEVEL="$arg" ;;
--push) PUSH=1 ;;
*) echo "usage: $0 [patch|minor|major] [--push]" >&2; exit 2 ;;
esac
done
if [ -n "$(git status --porcelain)" ]; then
echo "Working tree has uncommitted changes; commit them first." >&2
exit 1
fi
export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}"
VERSION="$(scripts/bump-version.sh "$LEVEL")"
echo "==> Version $VERSION"
restore() {
echo "Release failed; reverting the version bump." >&2
git checkout -- Resources/Info.plist windows/autozhuyin/ime.json
}
trap restore ERR
echo "==> Swift tests"
xcrun swift test -q 2>&1 | tail -3
echo "==> Packaging (runs the Python tests too)"
scripts/package.sh >/dev/null
scripts/package-windows.sh >/dev/null
trap - ERR
git add Resources/Info.plist windows/autozhuyin/ime.json
git commit -q -m "Release v$VERSION"
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
echo "==> Tagged v$VERSION"
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
if [ "$PUSH" = 1 ]; then
git push origin HEAD "v$VERSION"
else
echo "Push with: git push origin HEAD v$VERSION"
fi
+30
View File
@@ -0,0 +1,30 @@
"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
import os
import sys
import types
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
WIN = os.path.join(ROOT, "windows")
DATA = os.path.join(WIN, "autozhuyin", "data")
RESOURCES = os.path.join(ROOT, "Resources")
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
sys.path.insert(0, WIN)
from autozhuyin.core import bopomofo, features # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
def load_models():
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
sys.exit("Run windows/tools/build_data.py first.")
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
return lexicon, english
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Writes training/parity-fixtures.json: segmentations produced by the Python core
with the current weights. The Swift tests (ParityTests) must produce the same ones,
which keeps the two implementations in sync. Re-run after changing features or weights.
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator # noqa: E402
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json")
HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up",
"u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"]
def main():
lexicon, english = load_models()
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f) for k in HAND_PICKED for f in (False, True)]
while len(cases) < 400:
ex = gen.example()
if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"]))
cases.append((gen.prefix_of(ex)["keys"], False))
fixtures = [{"keys": k, "finishing": f,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases]
with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT))
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+132
View File
@@ -0,0 +1,132 @@
"""Synthetic typing data for the Zhuyin / literal classifier.
Each example is what sits in the input buffer when the user commits it, with the
intended mode of every key:
{"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"}
labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter.
Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys;
English from the common-word list and the dictionary. Real typing commits English at
a space, so a buffer holds at most one English word next to Chinese.
"""
import random
from common import core
bopomofo = core.bopomofo
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
def reading_to_keys(reading):
"""'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable."""
out = []
for syl in reading.split("-"):
tone = " "
if syl and syl[-1] in TONE_KEY:
tone = TONE_KEY[syl[-1]]
syl = syl[:-1]
if not syl or syl not in bopomofo.VALID:
return None
out.append("".join(KEY_OF[c] for c in syl) + tone)
return "".join(out)
def _is_cjk(s):
return all(0x4E00 <= ord(c) <= 0x9FFF for c in s)
class Generator:
def __init__(self, lexicon_path, common_path, dict_path, seed=0):
self.rng = random.Random(seed)
phrases, weights = [], []
with open(lexicon_path, encoding="utf-8") as f:
for line in f:
cols = line.split()
if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]):
continue
n = cols[0].count("-") + 1
if n > 4:
continue
keys = reading_to_keys(cols[0])
if keys is None:
continue
phrases.append((cols[1], keys))
weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency
self.phrases = phrases
self.phrase_cum = _cumulative(weights)
self.common = [w.strip() for w in open(common_path, encoding="utf-8")
if w.strip() and not w.startswith("#")]
self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8")
if 3 <= len(w.strip()) <= 9]
# --- pieces ---
def chinese(self):
n = self.rng.choice([1, 1, 2, 2, 3])
picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n)
return "".join(keys for _, keys in picked)
def english(self):
w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words)
if self.rng.random() < 0.1:
w = w[:1].upper() + w[1:]
return w
def number(self):
r = self.rng.random()
if r < 0.4:
return str(self.rng.randint(0, 9999))
if r < 0.7:
return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30]))
return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9))
# --- examples ---
def example(self):
r = self.rng.random()
if r < 0.40:
parts, kind = [("z", self.chinese())], "zh"
elif r < 0.65:
parts, kind = [("l", self.english())], "en"
elif r < 0.80:
parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en"
elif r < 0.93:
parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh"
elif r < 0.96:
parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh"
else:
parts, kind = [("l", self.number())], "num"
keys = "".join(p for _, p in parts)
labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l":
finishing = True # English / numbers end with a space
else:
# Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind}
def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix"}
def _cumulative(ws):
out, t = [], 0.0
for w in ws:
t += w
out.append(t)
return out
def gold_segments(labels):
"""'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]"""
segs, start = [], 0
for i in range(1, len(labels) + 1):
if i == len(labels) or labels[i] != labels[start]:
segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL))
start = i
return segs
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""Learns the classifier weights from synthetic typing data (averaged structured
perceptron over the same segmentation DP the input method runs) and reports
accuracy against the hand-tuned defaults.
Usage (from the repo root, after windows/tools/build_data.py):
python3 training/train.py # train, evaluate, write Resources/classifier-weights.json
python3 training/train.py --eval-only # just evaluate the current weights file
"""
import argparse
import datetime
import json
import os
import random
import sys
import time
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator, gold_segments # noqa: E402
MAX_KEYS = 28
FIXED = {"lit.upper"} # hard rule (Shift = English), not learned
def make_set(gen, n):
out = []
while len(out) < 2 * n:
ex = gen.example()
if len(ex["keys"]) > MAX_KEYS:
continue
out.append(ex)
if len(ex["keys"]) > 1:
out.append(gen.prefix_of(ex))
return out
def labels_of(segs, n):
lab = [""] * n
for s, e, m in segs:
for i in range(s, e):
lab[i] = "z" if m == core.ZHUYIN else "l"
return "".join(lab)
def seg_features(clf, keys, segs, finishing):
"""Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float)
n = len(keys)
for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing)
f = fz if m == core.ZHUYIN else fl
if f is None:
return None
for k, v in f.items():
total[k] += v
return total
def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0])
for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]]
bucket[0] += ok
bucket[1] += 1
commit = [v for k, v in by_kind.items() if k != "prefix"]
summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
"by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())},
}
return summary
def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
def train(clf, data, epochs, lr, seed):
rng = random.Random(seed)
w = dict(clf.weights)
total = defaultdict(float)
steps = 0
for epoch in range(epochs):
rng.shuffle(data)
mistakes = 0
t0 = time.time()
for ex in data:
keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], None, w)
if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"])
fp = seg_features(clf, keys, pred, ex["finishing"])
if fg is not None and fp is not None:
mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
for k in set(fg) | set(fp):
if k not in FIXED:
w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0))
steps += 1
for k, v in w.items():
total[k] += v
print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True)
return {k: round(v / steps, 4) for k, v in total.items()}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)")
ap.add_argument("--test", type=int, default=1500)
ap.add_argument("--epochs", type=int, default=6)
ap.add_argument("--lr", type=float, default=0.1)
ap.add_argument("--eval-only", action="store_true")
args = ap.parse_args()
lexicon, english = load_models()
paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"))
test = make_set(Generator(*paths, seed=2), args.test)
clf = core.Classifier(lexicon, english) # default (hand-tuned) weights
show("default", evaluate(clf, clf.weights, test))
if os.path.exists(WEIGHTS_PATH):
current = dict(clf.weights)
current.update(core.Classifier.load_weights(WEIGHTS_PATH))
show("current", evaluate(clf, current, test))
if args.eval_only:
return
data = make_set(Generator(*paths, seed=1), args.train)
print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True)
learned = train(clf, data, args.epochs, args.lr, seed=3)
result = evaluate(clf, learned, test)
show("learned", result)
with open(WEIGHTS_PATH, "w", encoding="utf-8") as f:
json.dump({
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": datetime.date.today().isoformat(),
"examples": len(data),
"test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)},
"weights": dict(sorted(learned.items())),
}, f, ensure_ascii=False, indent=2)
f.write("\n")
print("wrote", WEIGHTS_PATH)
if __name__ == "__main__":
main()
+2 -1
View File
@@ -52,7 +52,8 @@ def _shared_classifier():
weights = json.load(f)
except (OSError, ValueError):
pass
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned))
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained)
return _classifier
+75 -49
View File
@@ -1,5 +1,7 @@
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
from . import bopomofo
import json
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
ZHUYIN = "zhuyin"
LITERAL = "literal"
@@ -28,67 +30,91 @@ class LearnedPreferences:
class Classifier:
zhuyin_prior = 0.5
context_weight = 1.5
def __init__(self, lexicon, english, learned=None):
def __init__(self, lexicon, english, learned=None, weights=None):
self.lexicon = lexicon
self.english = english
self.learned = learned or LearnedPreferences()
self.weights = dict(DEFAULT_WEIGHTS)
if weights:
self.weights.update(weights)
def classify(self, keys, finishing=False, previous=None):
"""Returns (mode, zhuyin_score, literal_score)."""
keys = list(keys)
zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior
lit = self.literal_score(keys, complete=finishing)
@classmethod
def load_weights(cls, path):
"""Reads a weights JSON file ({"weights": {...}}); returns {} if missing."""
try:
with open(path, encoding="utf-8") as f:
return json.load(f).get("weights", {})
except (OSError, ValueError):
return {}
def segment_features(self, seg, is_last, after_other, finishing):
"""(zhuyin features or None, literal features or None) for one segment."""
seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing),
literal_features(seg, self.english, is_last, after_other, finishing))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing)
zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
b = self.learned.bias("".join(keys))
if b > 0:
zh += b
else:
lit -= b
if whole:
b = self.learned.bias("".join(seg))
if b > 0:
zh += b
else:
lit -= b
return zh, lit
def classify(self, keys, finishing=False, previous=None):
"""Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score)."""
zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights)
return (ZHUYIN if zh >= lit else LITERAL), zh, lit
def mode(self, keys, finishing=False, previous=None):
return self.classify(keys, finishing, previous)[0]
def zhuyin_score(self, keys):
p = bopomofo.parse(keys)
if p.foreign_keys > 0 or not p.syllables:
return NEG_INF
s = -5.0 * p.stray_tones
last = len(p.syllables) - 1
for i, ps in enumerate(p.syllables):
if ps.reordered:
s -= 0.5
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5)
else:
s -= 5
elif i == last:
s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5
else:
s += -2.5 if valid else -5
return s
# --- segmentation ---
def literal_score(self, keys, complete):
if not keys or " " in keys:
return NEG_INF
score = 0.0
if any(c.isupper() for c in keys):
score += 20
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
return score + 0.8 * digits - 1.0 * others
score += self.english.score(letters, complete or digits > 0)
score -= 1.5 * digits
apostrophes = keys.count("'")
score -= 1.5 * (others - apostrophes)
return score
def segment(self, keys, finishing=False, previous=None, weights=None):
"""Splits keys into alternating segments: [(start, end, mode)], e.g.
"ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
split points; each segment is scored by features x weights."""
weights = weights or self.weights
keys = list(keys)
n = len(keys)
if n == 0:
return []
modes = (ZHUYIN, LITERAL)
best = [[NEG_INF, NEG_INF] for _ in range(n + 1)]
back = [[-1, -1] for _ in range(n + 1)]
for i in range(1, n + 1):
for j in range(i):
scores = self._scores(keys[j:i], i == n, j > 0, finishing,
previous if j == 0 else None, j == 0 and i == n, weights)
for m in (0, 1):
if scores[m] == NEG_INF:
continue
if j == 0:
total = scores[m]
else:
if best[j][1 - m] == NEG_INF:
continue
total = best[j][1 - m] + scores[m]
if total > best[i][m]:
best[i][m] = total
back[i][m] = j
if best[n][0] == NEG_INF and best[n][1] == NEG_INF:
return [(0, n, self.mode(keys, finishing, previous))]
m = 0 if best[n][0] >= best[n][1] else 1
out, i = [], n
while i > 0:
j = back[i][m]
out.append((j, i, modes[m]))
i, m = j, 1 - m
return out[::-1]
+73 -39
View File
@@ -2,6 +2,7 @@
Keys are given as one-character strings, or one of the names in SPECIAL_KEYS.
handle() returns (handled, commit_text_or_None).
The buffer is split into Zhuyin / literal segments, e.g. "ji3hello" -> 我 + hello.
"""
import time
@@ -32,11 +33,14 @@ class CompositionEngine:
self.now = now
self.keys = []
self.forced_mode = None
self.fixed = {}
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = []
self.candidate_index = 0
self._candidate_segment_start = 0
self._previous_mode = None
self._previous_time = float("-inf")
self._cache_key = None
self._cache_value = None
# --- derived state ---
@@ -44,9 +48,26 @@ class CompositionEngine:
def context(self):
return self._previous_mode if self.now() - self._previous_time < self.context_window else None
def _segments(self, keys, finishing=False):
if not keys:
return []
if self.forced_mode:
return [(0, len(keys), self.forced_mode)]
ctx = self.context
cache_key = (tuple(keys), finishing, ctx)
if cache_key != self._cache_key:
self._cache_key = cache_key
self._cache_value = self.classifier.segment(keys, finishing, ctx)
return self._cache_value
@property
def segments(self):
return self._segments(self.keys)
@property
def mode(self):
return self.forced_mode or self.classifier.mode(self.keys, previous=self.context)
segs = self.segments
return segs[-1][2] if segs else ZHUYIN
@property
def showing_candidates(self):
@@ -56,28 +77,28 @@ class CompositionEngine:
def page_start(self):
return (self.candidate_index // self.page_size) * self.page_size
def _parse(self):
return bopomofo.parse(self.keys)
def _render(self, seg, final):
start, end, mode = seg
seg_keys = self.keys[start:end]
if mode == LITERAL:
return "".join(seg_keys)
p = bopomofo.parse(seg_keys)
fixed_here = self.fixed.get(start, {})
syl = [s.syllable for s in (p.syllables if final else p.completed)]
fx = {k: v for k, v in fixed_here.items() if k[1] <= len(syl)}
text = "".join(t for _, t in self.converter.convert(syl, fx))
if not final and p.pending:
text += p.pending.syllable.reading
return text
def _valid_fixed(self, p):
n = len(p.completed)
return {k: v for k, v in self.fixed.items() if k[1] <= n}
@property
def marked_pieces(self):
segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
@property
def marked_text(self):
if not self.keys:
return ""
if self.mode == LITERAL:
return "".join(self.keys)
p = self._parse()
done = self.converter.convert([s.syllable for s in p.completed], self._valid_fixed(p))
pending = p.pending.syllable.reading if p.pending else ""
return "".join(t for _, t in done) + pending
def _zhuyin_commit_text(self):
p = self._parse()
segs = self.converter.convert([s.syllable for s in p.syllables], self._valid_fixed(p))
return "".join(t for _, t in segs)
return "".join(t for t, _ in self.marked_pieces)
# --- key handling ---
@@ -102,7 +123,7 @@ class CompositionEngine:
if not self.keys:
return PASS
self.keys.pop()
self.fixed = self._valid_fixed(self._parse())
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys:
self.forced_mode = None
self.fixed = {}
@@ -128,25 +149,32 @@ class CompositionEngine:
return CONSUMED if self.keys else PASS
return self._handle_character(key)
def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN:
return None
start, end, _ = segs[-1]
return bopomofo.parse(self.keys[start:end])
def _handle_character(self, c):
if c == " ":
if not self.keys:
return PASS
decided = self.forced_mode or self.classifier.mode(self.keys, finishing=True, previous=self.context)
if decided == LITERAL:
return True, self._commit(LITERAL) + " "
if self._parse().pending is not None:
segs = self._segments(self.keys, finishing=True)
if segs[-1][2] == LITERAL:
return True, self._commit(segs) + " "
p = self._last_parse(segs)
if p is not None and p.pending is not None:
self.keys.append(" ")
return CONSUMED
return True, self._commit(ZHUYIN)
return True, self._commit(segs)
composable = bopomofo.is_zhuyin_key(c) or (c.isascii() and c.isalpha()) or (c == "'" and self.keys)
if composable:
# ", . / ; -" are Zhuyin symbols but act as punctuation after an English word.
if self.keys and c in bopomofo.SYMBOLS and not c.isalnum() and self.forced_mode != ZHUYIN:
with_key = self.forced_mode or self.classifier.mode(self.keys + [c], previous=self.context)
if with_key == LITERAL and not all(k.isdigit() for k in self.keys):
return True, self._commit(LITERAL) + c
with_key = self._segments(self.keys + [c])
if with_key[-1][2] == LITERAL and not all(k.isdigit() for k in self.keys):
return True, self._commit() + c
self.keys.append(c)
return CONSUMED
@@ -157,14 +185,18 @@ class CompositionEngine:
return PASS
return True, committed + punct
# --- candidates ---
# --- candidates (for the last Zhuyin segment) ---
def _open_candidates(self):
p = self._parse()
if self.mode != ZHUYIN or not p.completed:
segs = self.segments
if not segs or segs[-1][2] != ZHUYIN:
return
start, end, _ = segs[-1]
syl = [s.syllable for s in bopomofo.parse(self.keys[start:end]).completed]
if not syl:
return
syl = [s.syllable for s in p.completed]
self.candidates = self.converter.candidates(syl, len(syl))
self._candidate_segment_start = start
self.candidate_index = 0
def page(self):
@@ -176,8 +208,10 @@ class CompositionEngine:
if not 0 <= index < len(self.candidates):
return
rng, text = self.candidates[index]
self.fixed = {k: v for k, v in self.fixed.items() if not (k[0] < rng[1] and rng[0] < k[1])}
self.fixed[rng] = text
here = {k: v for k, v in self.fixed.get(self._candidate_segment_start, {}).items()
if not (k[0] < rng[1] and rng[0] < k[1])}
here[rng] = text
self.fixed[self._candidate_segment_start] = here
self.candidates = []
def _handle_candidate_key(self, key):
@@ -195,12 +229,12 @@ class CompositionEngine:
# --- commit ---
def _commit(self, decided=None):
m = decided or self.mode
text = "".join(self.keys) if m == LITERAL else self._zhuyin_commit_text()
def _commit(self, segs=None):
segs = segs if segs is not None else self.segments
text = "".join(self._render(s, True) for s in segs)
if self.forced_mode:
self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = m
self._previous_mode = segs[-1][2] if segs else None
self._previous_time = self.now()
self.reset()
return text
+23
View File
@@ -56,6 +56,29 @@ class EnglishModel:
plaus = self.plausibility(w)
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
def word_features(self, word, complete):
"""Feature dict for the classifier (see features.py); score() is their
dot product with the default weights."""
w = word.lower()
n = float(len(w))
f = {}
if n <= 2:
f["lit.short"] = 1.0
if w in self.common:
f["lit.common"] = 1.0
f["lit.common_len"] = n
elif w in self.dictionary:
f["lit.dict"] = 1.0
f["lit.dict_len"] = n
elif not complete and _has_prefix(w, self.sorted_common):
f["lit.prefix_common_len"] = n
elif not complete and _has_prefix(w, self.sorted_words):
f["lit.prefix_dict_len"] = n
else:
plaus = self.plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
return f
def plausibility(self, w):
chars = ["^"] + list(w) + ["$"]
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
+131
View File
@@ -0,0 +1,131 @@
"""Segment features for the Zhuyin / literal classifier.
A segment's score in each mode is the dot product of its feature dict with a weight
dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights
(training/train.py → Resources/classifier-weights.json) replace them.
Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical
features; Tests/.../ParityTests checks this against training/parity-fixtures.json.
"""
from . import bopomofo
DEFAULT_WEIGHTS = {
# Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
# Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
# Segmentation
"seg.switch": -3.0,
}
def dot(weights, feats):
return sum(weights.get(k, 0.0) * v for k, v in feats.items())
def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False):
"""Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None:
return None # a Zhuyin segment followed by another must end with a tone
# Pressing space finishes an untoned last syllable as first tone; after a toned
# syllable it just commits, so it must not count as an extra (stray) tone.
scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse
if scored.foreign_keys > 0 or not scored.syllables:
return None
f = {"zh.bias": 1.0}
if scored.stray_tones:
_add(f, "zh.stray", scored.stray_tones)
last = len(scored.syllables) - 1
toned_run = []
for i, ps in enumerate(scored.syllables):
if ps.reordered:
_add(f, "zh.reordered")
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
best = lexicon.best_score(ps.syllable)
if best is not None:
_add(f, "zh.toned_valid_lex")
_add(f, "zh.lexscore", best + 4.0)
else:
_add(f, "zh.toned_valid_nolex")
toned_run.append(ps.syllable)
else:
_add(f, "zh.toned_invalid")
elif i == last:
_add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid")
else:
_add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid")
# Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
covered = [False] * len(toned_run)
for i in range(len(toned_run)):
for n in (2, 3, 4):
if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]):
for k in range(i, i + n):
covered[k] = True
if any(covered):
_add(f, "zh.phrase", sum(covered))
if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables):
_add(f, "zh.unconfirmed")
if after_other:
_add(f, "seg.switch")
return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False):
"""Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys:
return None
complete = (not is_last) or finishing
f = {"lit.bias": 1.0}
if after_other:
_add(f, "seg.switch")
if any(c.isupper() for c in keys):
_add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
if digits:
_add(f, "lit.num_digits", digits)
if others:
_add(f, "lit.num_others", others)
return f
for k, v in english.word_features(letters, complete or digits > 0).items():
_add(f, k, v)
if digits:
_add(f, "lit.digits", digits)
non_apostrophe = others - keys.count("'")
if non_apostrophe:
_add(f, "lit.others", non_apostrophe)
return f
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "自動注音 AutoZhuyin",
"version": "0.1.0",
"version": "0.3.0",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW",
+41 -3
View File
@@ -21,6 +21,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json"))
LOAD_TIME = time.time() - _start
@@ -34,7 +35,7 @@ def convert(keys):
def make_engine():
return CompositionEngine(Classifier(LEX, ENG))
return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS))
def type_keys(s, e):
@@ -86,7 +87,7 @@ class ConverterTests(unittest.TestCase):
class ClassifierTests(unittest.TestCase):
c = Classifier(LEX, ENG)
c = Classifier(LEX, ENG, weights=WEIGHTS)
def test_zhuyin(self):
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
@@ -101,7 +102,7 @@ class ClassifierTests(unittest.TestCase):
def test_learning(self):
learned = LearnedPreferences()
c = Classifier(LEX, ENG, learned)
c = Classifier(LEX, ENG, learned, WEIGHTS)
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
learned.record("so", ZHUYIN)
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
@@ -150,9 +151,46 @@ class EngineTests(unittest.TestCase):
self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
def test_tone_then_space_stays_chinese(self):
self.assertEqual(len(type_keys("u4 ", make_engine())), 1)
def test_out_of_order_converts(self):
self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
class MixedSegmentTests(unittest.TestCase):
def run_keys(self, s):
e = make_engine()
return type_keys(s, e), e.marked_text
def test_chinese_then_english(self):
self.assertEqual(self.run_keys("ji3hello ")[0], "我hello ")
self.assertEqual(self.run_keys("su3cl3github\n")[0], "你好github")
self.assertEqual(self.run_keys("ji3hel")[1], "我hel")
def test_english_then_chinese(self):
self.assertEqual(self.run_keys("hellosu3cl3 ")[0], "hello你好")
self.assertEqual(self.run_keys("oksu3cl3\n")[0], "ok你好")
def test_three_segments(self):
self.assertEqual(self.run_keys("ji3emailsu3cl3\n")[0], "我email你好")
def test_single_language_unchanged(self):
self.assertEqual(self.run_keys("rup wu0 wu0 fu4cp3cl3\n")[0], "今天天氣很好")
self.assertEqual(self.run_keys("helloworld\n")[0], "helloworld")
self.assertEqual(self.run_keys("meeting ")[0], "meeting ")
self.assertEqual(len(self.run_keys("183\n")[0]), 1)
def test_speed(self):
e = make_engine()
start = time.time()
for c in "ji3emailsu3cl3rup wu0 wu0 fu4":
e.handle(c)
e.marked_text
per_key = (time.time() - start) / 30
print("\nper key: %.1f ms" % (per_key * 1000))
self.assertLess(per_key, 0.05)
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Fills windows/autozhuyin/data/ from the shared Mac resources.
- lexicon.txt, english-common.txt: copied from Resources/
- lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
Webster's 2nd edition, public domain), since Windows has no such file
- english-bigram.json: precomputed letter-bigram table
@@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402
data = WIN / "autozhuyin" / "data"
data.mkdir(parents=True, exist_ok=True)
for name in ("lexicon.txt", "english-common.txt"):
for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"):
shutil.copy(ROOT / "Resources" / name, data / name)
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")