Compare commits

..
20 Commits
Author SHA1 Message Date
ays953 1dacb8f164 Release v0.6.2 2026-11-11 12:13:15 +08:00
ays953andClaude Opus 5.5 bfeed22916 Cut input method startup from ~0.95 s to ~0.15 s
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
  parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
  Windows build) instead of retraining the bigram model on every launch, and
  uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:13:03 +08:00
ays953 bb9cb0c4e0 Release v0.6.1 2026-11-11 12:00:10 +08:00
ays953andClaude Opus 5.5 2c5c70c066 Learn the previous-word context; fix first-tone characters after English
After an English word, 23 of 50 common first-tone characters (機, 吃, 書…)
were read as English when space was pressed: the fixed +1.5 context bonus
was far too strong for the trained weight scale, and the training data never
contained the "space just pressed on a first-tone syllable" decision.

Context is now the ctx.zh / ctx.lit features, trained with the rest; the
synthetic data includes the previous mode and the first-tone space decision.
Test accuracy: commit 98.22% -> 99.52%, typing 88.73% -> 91.49%, and when the
language differs from the previous word 92.0% -> 99.3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:59:57 +08:00
ays953 17f7d6f50d Release v0.6.0 2026-11-11 11:44:38 +08:00
ays953andClaude Opus 5.5 7e173a1da8 Make the Shift tap a persistent English-mode toggle
A Shift tap with nothing composed now switches English mode on until the
next tap, instead of forcing only the next word: space, Enter and other keys
no longer drop back to automatic detection. In English mode every key goes
straight to the app, so there is no composition or underline. A Shift tap
while typing still flips the buffer like Tab. Windows shows "英文" /
"中英自動" when toggling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:44:25 +08:00
ays953 fa0b78e5a4 Release v0.5.1 2026-11-11 11:36:12 +08:00
ays953andClaude Opus 5.5 e928d08fdc Type forced English directly, without an underlined composition
After a Shift tap, letters, digits and word punctuation go straight to the
app instead of into marked text, so nothing is underlined. Space, Enter, Tab,
Escape or another symbol ends the forced word and automatic detection
resumes; backspace is handled by the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:35:59 +08:00
ays953 82f20a6551 Release v0.5.0 2026-11-11 10:37:37 +08:00
ays953andClaude Opus 5.5 e6e5004630 Add a composition cursor so candidates can be chosen for any character
Left/Right move a cursor between Zhuyin syllables in the composition; Down
opens candidates for the phrase ending at the cursor instead of always the
last syllable. Typing or backspace returns the cursor to the end. Candidates
also work for Chinese that is followed by English in the same buffer. The Mac
caret and PIME composition cursor follow it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:37:24 +08:00
ays953 988ed471be Release v0.4.1 2026-11-11 10:18:14 +08:00
ays953andClaude Opus 5.5 6c36c01f0e Stop declaring Latin repertoire so password fields bypass the IME
With "Latn" in tsInputMethodCharacterRepertoireKey macOS treated AutoZhuyin
as ASCII-capable and kept it active in secure text fields, so typed passwords
showed up as underlined composition text. Apple's Zhuyin and vChewing declare
only Hant (not ASCII-capable); macOS then switches to an ASCII keyboard layout
in password fields and the input method never sees the keystrokes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:18:01 +08:00
ays953andClaude Opus 5.5 0fc42a4b8c install.sh: warn when a system-wide .pkg copy shadows the per-user install
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:05:24 +08:00
ays953 42a2d3a2a3 Release v0.4.0 2026-11-11 09:57:34 +08:00
ays953andClaude Opus 5.5 20291bf515 Add Shift tap to force English for the next word
Pressing and releasing Shift on its own arms literal mode for the next word,
so lowercase English can be typed without it being read as Zhuyin; it resets
after the word is committed and a second tap cancels it. While typing, a
Shift tap flips the buffer like Tab. Forcing before typing is not recorded
as a learned correction; flipping is.

Mac: listens for flagsChanged events. Windows: Shift key-up via PIME's
filterKeyUp/onKeyUp, with a short "英文" tooltip. The tap detection lives in
the core (ShiftTapDetector / shift_tap.py) with tests on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:57:18 +08:00
ays953 21142c2d98 Release v0.3.0 2026-11-11 08:58:43 +08:00
ays953andClaude Opus 5.5 476858d841 Learn classifier weights from synthetic typing data
Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.

Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.

Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:58:08 +08:00
ays953 0a7242ddb3 Release v0.2.0 2026-11-10 16:21:42 +08:00
ays953andClaude Opus 5.5 d1f1112c65 Split mixed Chinese/English input without spaces; add release script
The classifier now segments a buffer into alternating Zhuyin and literal runs
by dynamic programming over split points, so "ji3hello" becomes 我hello and
"hellosu3cl3" becomes hello你好. Each switch costs a penalty, a Zhuyin run
followed by another must end with a tone, and an untoned trailing Zhuyin run
after English is penalized so partial words don't flicker into Bopomofo.
Implemented in both Swift and Python with the same tests; the Mac app colours
each segment separately.

scripts/bump-version.sh bumps the version in Info.plist and ime.json;
scripts/release.sh bumps, tests, packages, commits and tags in one step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:21:13 +08:00
ays953andClaude Opus 5.5 dbe08e00a7 Expand README with install, data, development and release docs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:14:49 +08:00
32 changed files with 8811 additions and 355 deletions
+191 -49
View File
@@ -1,12 +1,64 @@
# 自動注音 AutoZhuyin
macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
```
按鍵: ji3 hello world su3cl3␣
輸出: 我hello world 你好
```
## 功能
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝
到 Gitea 的 [Releases](https://git.ays.ddnsfree.com/ays953/AutoZhuyin/releases) 頁面下載最新版本。
### macOS
需求:macOS 13 以上,Apple Silicon 和 Intel 都可以(universal binary)。
1. 雙擊 `AutoZhuyin-<版本>.pkg` 安裝。安裝檔沒有簽章,如果被擋下來,按右鍵 →「打開」,或到「系統設定 → 隱私權與安全性」按「仍要打開」。
2. 系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
3. 清單裡沒出現的話,登出再登入一次。
PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
### Windows
需求:Windows 10 / 11,x64、x86、ARM64 都可以。
1. 先安裝輸入法框架 [PIME](https://github.com/EasyIME/PIME/releases)(免費、開源)。
2. 解壓縮 `AutoZhuyin-Windows-<版本>.zip`,在 `install.bat` 上按右鍵 →「以系統管理員身分執行」。
3. 設定 → 時間與語言 → 語言與地區 → 中文(台灣) → 語言選項 → 新增鍵盤 → **自動注音 AutoZhuyin**。
4. 用 Win + 空白鍵 切換過去。清單裡沒出現的話,重新開機一次。
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
## 按鍵
| 按鍵 | 作用 |
|---|---|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ← → | 在組字區移動游標,選要改哪個字 |
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 |
| **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
| Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
## 運作方式
每按一個鍵,`Classifier` 會同時用兩種解讀幫目前的按鍵序列打分:
@@ -14,84 +66,174 @@ macOS 繁體中文輸入法:**不用按切換鍵**,直接用大千注音鍵
| 注音分數 | 英文分數 |
|---|---|
| 是否能拆成合法注音音節(`SyllableTable`) | 是否為常用英文字(`english-common.txt`) |
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在系統字典 `/usr/share/dict/words` |
| 每個音節有沒有聲調鍵(3 4 6 7 空白) | 是否在英文字典裡 |
| 讀音是否存在於詞庫 | 字母 bigram 模型判斷「像不像英文」 |
| 前一段是中文 → 加分(語境) | 前一段是英文 → 加分(語境) |
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
| 一開始稍微偏向注音 | 純數字 → 數字 |
分數高的一方勝出。組字區即時顯示結果(英文以藍字顯示)。
每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。
## 按鍵
中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎:
- 每多切一刀會扣分(`seg.switch` 特徵)
- 後面還接著其他段的注音段,最後一個音節必須打了聲調
- 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
| 按鍵 | 作用 |
|---|---|
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字 |
| Esc | 取消組字 |
| Shift + 字母 | 強制英文 |
| `< > ? ! :` | 中文語境下自動轉成全形 `,。?!:` |
音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。
## 建置與安裝
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
需要 Xcode(使用 `/Applications/Xcode.app` 的工具鏈)。
## 個人資料
| | macOS | Windows |
|---|---|---|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` |
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
Mac 版可以在輸入法選單選「清除學習紀錄」。
## 開發
### macOS
需要 Xcode。腳本會自動使用 `/Applications/Xcode.app` 的工具鏈。
```bash
swift test # 執行單元測試(需 DEVELOPER_DIR 指向 Xcode)
scripts/build.sh # 產生 build/AutoZhuyin.app
scripts/install.sh # 安裝到 ~/Library/Input Methods 並重啟輸入法
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
xcrun swift test # 單元測試
scripts/build.sh # 產生 build/AutoZhuyin.app(universal)
scripts/install.sh # 裝到 ~/Library/Input Methods、重啟並向系統註冊
scripts/package.sh # 產生 build/AutoZhuyin-<版本>.pkg
```
第一次安裝後:系統設定 → 鍵盤 → 文字輸入 → 編輯… → + → 繁體中文 → **自動注音**。
如果清單裡沒出現,登出再登入一次。
改完程式後執行 `scripts/install.sh`,再切換到其他輸入法後切回來,就會用到新版。
版本號在 `Resources/Info.plist` 的 `CFBundleShortVersionString`。
## 詞庫
### Windows
`Resources/lexicon.txt` 由開源的小麥注音原始資料轉換而來(MIT 授權,約 17 萬筆),
後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換只讀取資料檔,不執行小麥注音的程式碼。
要更新到小麥注音最新資料:
Windows 版建立在 [PIME](https://github.com/EasyIME/PIME) 上,核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣。不需要 Windows 也能開發和測試:
```bash
python3 windows/tools/build_data.py # 產生 windows/autozhuyin/data/
(cd windows && python3 -m unittest discover tests) # 核心測試 + 模擬 PIME 的外殼測試
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
```
- Windows 沒有內建英文字典,所以 `build_data.py` 會把 macOS 的 `/usr/share/dict/words`(韋氏字典,公有領域)轉成 `english-dict.txt` 一起打包,並預先算好 bigram 表。
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
- 版本號在 `windows/autozhuyin/ime.json`。
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
### 訓練判斷權重
中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法:
1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。
2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。
3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。
```bash
python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用)
python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘)
python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘
python3 training/train.py --eval-only # 只評估目前的權重
python3 training/make_parity.py # 權重或特徵改了之後都要重跑
```
目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 | 跟前一個字語言不同時 |
|---|---|---|---|
| 手調權重 | 97.84% | 88.19% | 96.2% |
| 訓練後 | **99.52%** | **91.49%** | **99.3%** |
訓練資料也模擬了「前一個字是中文或英文」和「按下空白(一聲)的那一刻」,所以「前一個字」的影響(`ctx.zh` / `ctx.lit`)也是訓練出來的,不是手調的。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
### 詞庫
`Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。
更新到小麥注音最新資料:
```bash
scripts/import-mcbopomofo.sh
scripts/install.sh
```
自訂詞:`~/Library/Application Support/AutoZhuyin/user-phrases.txt`,格式同 `lexicon.txt`:
`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`
### 發布新版本
先 commit 所有修改,再執行:
```bash
scripts/release.sh # 修正版:0.1.0 → 0.1.1
scripts/release.sh minor # 新功能:0.1.1 → 0.2.0
scripts/release.sh major # 大改版:0.2.0 → 1.0.0
scripts/release.sh minor --push # 同時 push commit 和 tag
```
`release.sh` 會自動:
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
2. 跑 Swift 和 Python 全部測試
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip`
4. commit「Release v<版本>」並打上 `v<版本>` tag
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。
## 專案結構
```
Sources/AutoZhuyinCore/ 純邏輯,可單元測試
Bopomofo.swift 大千鍵盤對應、音節結構
SyllableTable.swift 合法注音音節表
ZhuyinParser.swift 按鍵 → 音節
Lexicon.swift 詞庫
Converter.swift 音節 → 中文(Viterbi)
EnglishModel.swift 英文判斷
Classifier.swift 中英判斷 + 學習
CompositionEngine.swift 輸入法狀態機
Sources/AutoZhuyin/ InputMethodKit 外殼
Tests/ 單元測試
Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試)
Bopomofo.swift 大千鍵盤對應、音節結構
SyllableTable.swift 合法注音音節表
ZhuyinParser.swift 按鍵 → 音節(含順序錯誤修正)
Lexicon.swift 詞庫
Converter.swift 音節 → 中文(Viterbi)
EnglishModel.swift 英文判斷
Features.swift 判斷用的特徵與預設權重
Classifier.swift 中英判斷(特徵 × 權重)、切分、學習
CompositionEngine.swift 輸入法狀態機
Sources/AutoZhuyin/ macOS InputMethodKit 外殼
Tests/ Swift 單元測試
Resources/ Info.plist、詞庫、英文常用字、訓練好的權重
scripts/ 建置、安裝、打包、詞庫轉換
pkg/postinstall PKG 安裝後腳本
windows/
autozhuyin/ PIME 輸入法(整個資料夾會被複製到 PIME)
core/ 核心邏輯(Python 版)
autozhuyin_ime.py PIME 外殼
ime.json PIME 輸入法設定
installer/ install.bat、uninstall.bat、給使用者的 README.txt
tests/ Python 測試
tools/ 產生資料檔、圖示
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
```
## 已知限制
- Windows 版還沒在實機上測試過。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap)
- [ ] 匯入完整詞庫,並用真實打字資料調整判斷權重
- [x] 匯入完整詞庫
- [x] 注音順序錯誤修正
- [x] Windows 版
- [x] 用資料訓練判斷權重(合成資料)
- [ ] 加入真實打字資料一起訓練
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境
- [ ] 使用者詞頻學習(選過的候選字優先)
- [ ] 偏好設定視窗
## Windows 版
## 致謝與授權
`windows/` 是給 Windows 用的版本,建立在開源輸入法框架 [PIME](https://github.com/EasyIME/PIME) 上。
核心邏輯用 Python 重寫(`windows/autozhuyin/core/`),行為和測試案例都跟 Mac 版一樣;詞庫共用 `Resources/`。
```bash
scripts/package-windows.sh # 產生 build/AutoZhuyin-Windows-<版本>.zip(會先跑測試)
```
zip 裡附有 `install.bat`、`uninstall.bat` 和給測試者看的 `README.txt`。使用者需要先安裝 PIME。
修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改。
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
+14 -15
View File
@@ -4,42 +4,41 @@
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>zh_TW</string>
<key>CFBundleDisplayName</key>
<string>自動注音</string>
<key>CFBundleExecutable</key>
<string>AutoZhuyin</string>
<key>CFBundleIdentifier</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
<key>CFBundleName</key>
<string>AutoZhuyin</string>
<key>CFBundleDisplayName</key>
<string>自動注音</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>0.1.0</string>
<string>0.6.2</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>LSMinimumSystemVersion</key>
<string>13.0</string>
<key>LSBackgroundOnly</key>
<true/>
<key>LSUIElement</key>
<true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
<string>10</string>
<key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key>
<string>AutoZhuyinInputController</string>
<key>LSBackgroundOnly</key>
<true/>
<key>LSMinimumSystemVersion</key>
<string>13.0</string>
<key>LSUIElement</key>
<true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
<key>TISInputSourceID</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin</string>
<key>TISIntendedLanguage</key>
<string>zh-Hant</string>
<key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string>
<key>tsInputMethodCharacterRepertoireKey</key>
<array>
<string>Hant</string>
<string>Latn</string>
</array>
<key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string>
</dict>
</plist>
+41
View File
@@ -0,0 +1,41 @@
{
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11",
"examples": 40000,
"test_accuracy": {
"commit": 0.9952,
"typing": 0.9149
},
"weights": {
"ctx.lit": 0.0916,
"ctx.zh": 0.1349,
"lit.bias": 0.3982,
"lit.common": -0.2906,
"lit.common_len": 0.8046,
"lit.dict": -1.6662,
"lit.dict_len": 0.6441,
"lit.digits": -2.9689,
"lit.num_digits": -0.018,
"lit.num_others": -0.5554,
"lit.others": -2.0002,
"lit.prefix_common_len": 0.2831,
"lit.prefix_dict_len": 0.0299,
"lit.short": -0.2482,
"lit.unknown_score": 0.2479,
"lit.upper": 20.0,
"seg.switch": -1.7596,
"zh.bias": 1.3423,
"zh.lexscore": 0.2917,
"zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9162,
"zh.phrase": 0.4499,
"zh.reordered": -1.3079,
"zh.stray": -5.0,
"zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9018,
"zh.toned_valid_nolex": -1.6341,
"zh.unconfirmed": -0.7952,
"zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.7479
}
}
+42 -14
View File
@@ -11,37 +11,61 @@ enum Shared {
static let classifier: Classifier = {
let bundle = Bundle.main
let start = Date()
let lexicon = Lexicon()
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
let text = try? String(contentsOf: url, encoding: .utf8) {
lexicon.load(text)
let data = try? Data(contentsOf: url) {
lexicon.load(data: data)
}
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first {
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt")
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) }
}
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"))
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey)
}
return Classifier(lexicon: lexicon, english: english, learned: learned)
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained)
}()
}
@objc(AutoZhuyinInputController)
final class InputController: IMKInputController {
private let engine = CompositionEngine(classifier: Shared.classifier)
private var shiftTap = ShiftTapDetector()
// MARK: - Key events
override func recognizedEvents(_ sender: Any!) -> Int {
Int(NSEvent.EventTypeMask.keyDown.rawValue)
Int(NSEvent.EventTypeMask.keyDown.rawValue | NSEvent.EventTypeMask.flagsChanged.rawValue)
}
override func handle(_ event: NSEvent!, client sender: Any!) -> Bool {
guard let event, event.type == .keyDown, let client = sender as? IMKTextInput else { return false }
guard let event, let client = sender as? IMKTextInput else { return false }
// Shift pressed and released alone: next word is English (or flip the current one).
if event.type == .flagsChanged {
let flags = event.modifierFlags
let tapped = shiftTap.modifiersChanged(
shift: flags.contains(.shift),
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
time: event.timestamp)
if tapped {
_ = engine.handle(.shiftTap)
updateComposition(client)
}
return false
}
guard event.type == .keyDown else { return false }
shiftTap.keyPressed()
// Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first.
if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty {
@@ -90,15 +114,19 @@ final class InputController: IMKInputController {
}
private func updateComposition(_ client: IMKTextInput) {
let text = engine.markedText
let attrs: [NSAttributedString.Key: Any] = [
.underlineStyle: NSUnderlineStyle.single.rawValue,
// Literal (English) composition is shown in a different colour as a hint.
.foregroundColor: engine.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
]
let marked = NSAttributedString(string: text, attributes: text.isEmpty ? [:] : attrs)
// Each segment is underlined; literal (English) segments are blue as a hint.
let marked = NSMutableAttributedString()
for piece in engine.markedPieces {
marked.append(NSAttributedString(string: piece.text, attributes: [
.underlineStyle: NSUnderlineStyle.single.rawValue,
.foregroundColor: piece.mode == .literal ? NSColor.systemBlue : NSColor.textColor,
]))
}
let text = marked.string
// Caret at the candidate cursor (UTF-16 offset of the first `cursorOffset` characters).
let caret = String(text.prefix(engine.cursorOffset)).utf16.count
client.setMarkedText(marked,
selectionRange: NSRange(location: (text as NSString).length, length: 0),
selectionRange: NSRange(location: caret, length: 0),
replacementRange: NSRange(location: NSNotFound, length: 0))
if engine.isShowingCandidates {
+82 -57
View File
@@ -7,6 +7,17 @@ public enum InputMode: String, Codable {
case literal
}
/// A run of keys (indices into the buffer) interpreted in one mode.
public struct Segment: Equatable {
public var range: Range<Int>
public var mode: InputMode
public init(range: Range<Int>, mode: InputMode) {
self.range = range
self.mode = mode
}
}
public struct Classification {
public var mode: InputMode
public var zhuyinScore: Double
@@ -39,82 +50,96 @@ public final class LearnedPreferences {
}
/// Decides whether a key sequence is Zhuyin or literal (English / numbers).
/// Every segment is scored as features · weights (see Features.swift).
public final class Classifier {
public let lexicon: Lexicon
public let english: EnglishModel
public let learned: LearnedPreferences
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector
/// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it.
public var zhuyinPrior = 0.5
/// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) {
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) {
self.lexicon = lexicon
self.english = english
self.learned = learned
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
}
/// (Zhuyin features or nil, literal features or nil) for one segment. `previous` (mode of
/// the text committed just before) only affects the first segment.
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode? = nil) -> (FeatureVector?, FeatureVector?) {
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous),
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous))
}
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
if whole {
let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b }
}
return [zh, lit]
}
/// The whole buffer as one segment.
/// - Parameters:
/// - finishing: the user pressed space. Zhuyin is scored as if the space were a
/// first-tone key; literal is scored as a completed word.
/// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as
/// first tone; literal is scored as a completed word.
/// - previous: mode of the previously committed segment, if recent.
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior
var lit = literalScore(keys, complete: finishing)
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
let learnedBias = learned.bias(for: String(keys))
if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias }
return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit)
let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true)
return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1])
}
public func zhuyinScore(_ keys: [Character]) -> Double {
let parse = ZhuyinParser.parse(keys)
if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity }
var s = -5.0 * Double(parse.strayTones)
for (i, p) in parse.syllables.enumerated() {
let isLast = i == parse.syllables.count - 1
// Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin.
if p.reordered { s -= 0.5 }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5)
} else {
s -= 5
// MARK: - Segmentation
/// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" →
/// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the
/// cost of each switch is the "seg.switch" feature.
public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] {
let n = keys.count
guard n > 0 else { return [] }
let modes: [InputMode] = [.zhuyin, .literal]
// best[i][m]: best score for keys[0..<i] whose last segment has mode m.
var best = Array(repeating: [Double](repeating: -.infinity, count: 2), count: n + 1)
var back = Array(repeating: [Int](repeating: -1, count: 2), count: n + 1) // segment start j
for i in 1...n {
for j in 0..<i {
let s = scores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, finishing: finishing,
previous: j == 0 ? previous : nil, whole: j == 0 && i == n)
for m in 0..<2 where s[m] > -.infinity {
// Adjacent segments must alternate modes; same-mode neighbours are one segment.
if j > 0 && best[j][1 - m] == -.infinity { continue }
let total = j == 0 ? s[m] : best[j][1 - m] + s[m]
if total > best[i][m] {
best[i][m] = total
back[i][m] = j
}
}
} else if isLast {
s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5
} else {
// Untoned syllable in the middle: allowed (tone omission) but unusual.
s += valid ? -2.5 : -5
}
}
return s
}
public func literalScore(_ keys: [Character], complete: Bool) -> Double {
if keys.isEmpty || keys.contains(" ") { return -.infinity }
var score = 0.0
if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
// Pure number like "2024" (with possible "." or ",").
return score + 0.8 * Double(digits) - 1.0 * Double(others)
guard best[n][0] > -.infinity || best[n][1] > -.infinity else {
return [Segment(range: 0..<n, mode: classify(keys, finishing: finishing, previous: previous).mode)]
}
score += english.score(letters, complete: complete || digits > 0)
score -= 1.5 * Double(digits)
// `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not.
let apostrophes = keys.filter { $0 == "'" }.count
score -= 1.5 * Double(others - apostrophes)
return score
var m = best[n][0] >= best[n][1] ? 0 : 1
var i = n
var out: [Segment] = []
while i > 0 {
let j = back[i][m]
out.append(Segment(range: j..<i, mode: modes[m]))
i = j
m = 1 - m
}
return out.reversed()
}
}
+157 -46
View File
@@ -4,6 +4,8 @@ public enum KeyInput: Equatable {
case character(Character)
case backspace, enter, escape, tab
case up, down, left, right
/// Shift pressed and released on its own.
case shiftTap
}
public struct KeyResult: Equatable {
@@ -16,22 +18,36 @@ public struct KeyResult: Equatable {
public static let consumed = KeyResult(handled: true, commit: nil)
}
/// A piece of the composition as shown to the user.
public struct MarkedPiece: Equatable {
public var text: String
public var mode: InputMode
}
/// The whole input-method state machine, independent of InputMethodKit.
///
/// Keystrokes accumulate in a buffer. On every key the classifier decides whether the
/// buffer is Zhuyin or literal text; the composition (marked text) shows the result
/// live. Space / Enter / punctuation commit it. Tab flips the decision and the
/// correction is learned.
/// Keystrokes accumulate in a buffer. On every key the classifier splits the buffer
/// into Zhuyin and literal segments (e.g. "ji3hello" → 我 + hello); the composition
/// (marked text) shows the result live. Space / Enter / punctuation commit it. Tab
/// forces the whole buffer into the other mode and the correction is learned.
public final class CompositionEngine {
public let classifier: Classifier
public let converter: Converter
public private(set) var keys: [Character] = []
/// Set by Tab: overrides the classifier for the current buffer.
/// Set by Tab / a Shift tap while typing: overrides the classifier for the whole
/// buffer. It is a correction, learned on commit.
public private(set) var forcedMode: InputMode?
/// User-picked phrases, by syllable range.
public private(set) var fixed: [Range<Int>: String] = [:]
/// English mode, toggled by a Shift tap when nothing is being typed. Keys go straight
/// to the app (no composition, no underline) until Shift is tapped again.
public private(set) var isEnglishMode = false
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
private var fixed: [Int: [Range<Int>: String]] = [:]
public private(set) var candidates: [Candidate] = []
private var candidateSegmentStart = 0
/// Composition cursor for choosing candidates: index into `cursorStops`, nil = at the end.
/// ← → move it; ↓ shows candidates for the phrase ending at the cursor.
private var cursor: Int?
public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty }
@@ -52,33 +68,88 @@ public final class CompositionEngine {
now().timeIntervalSince(previousCommitTime) < contextWindow ? previousMode : nil
}
public var mode: InputMode {
forcedMode ?? classifier.classify(keys, previous: context).mode
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
return classifier.segment(keys, finishing: finishing, previous: context)
}
public var parse: ZhuyinParse { ZhuyinParser.parse(keys) }
/// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) }
/// Mode of the last segment (what the user is typing right now).
public var mode: InputMode { segments.last?.mode ?? .zhuyin }
/// What to show underlined in the app while composing.
public var markedText: String {
guard !keys.isEmpty else { return "" }
switch mode {
case .literal:
return String(keys)
case .zhuyin:
let p = parse
let done = converter.convert(p.completed.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
return done + (p.pending?.syllable.reading ?? "")
public var markedText: String { markedPieces.map(\.text).joined() }
public var markedPieces: [MarkedPiece] {
let segs = segments
return segs.enumerated().map { i, seg in
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
}
}
private func validFixed(_ p: ZhuyinParse) -> [Range<Int>: String] {
fixed.filter { $0.key.upperBound <= p.completed.count }
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
private func render(_ seg: Segment, final: Bool) -> String {
guard seg.mode == .zhuyin else { return String(keys[seg.range]) }
let (converted, pending) = conversion(of: seg, final: final)
return converted.map(\.text).joined() + pending
}
private func zhuyinCommitText() -> String {
let p = parse
// An unfinished trailing syllable is converted as untoned.
return converter.convert(p.syllables.map(\.syllable), fixed: validFixed(p)).map(\.text).joined()
/// Syllables of a Zhuyin segment that are converted to Chinese (all of them when
/// final, otherwise the finished ones).
private func convertibleSyllables(_ seg: Segment, final: Bool) -> (syllables: [Syllable], pending: String) {
let p = ZhuyinParser.parse(Array(keys[seg.range]))
if final { return (p.syllables.map(\.syllable), "") }
return (p.completed.map(\.syllable), p.pending?.syllable.reading ?? "")
}
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
let (syl, pending) = convertibleSyllables(seg, final: final)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
return (converter.convert(syl, fixed: fixedHere), pending)
}
// MARK: - Cursor
/// A place the cursor can stop: after the `local`-th syllable of Zhuyin segment `segIndex`.
private struct CursorStop { let segIndex: Int; let local: Int }
private var cursorStops: [CursorStop] {
let segs = segments
var out: [CursorStop] = []
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin {
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
for k in stride(from: 1, through: n, by: 1) { out.append(CursorStop(segIndex: i, local: k)) }
}
return out
}
/// Cursor position in `markedText`, in Characters.
public var cursorOffset: Int {
let pieces = markedPieces
let total = pieces.reduce(0) { $0 + $1.text.count }
let stops = cursorStops
guard let c = cursor, stops.indices.contains(c) else { return total }
let stop = stops[c]
var offset = pieces[..<stop.segIndex].reduce(0) { $0 + $1.text.count }
let segs = segments
for conv in conversion(of: segs[stop.segIndex], final: stop.segIndex < segs.count - 1).segments {
if conv.range.upperBound <= stop.local {
offset += conv.text.count
} else if conv.range.lowerBound < stop.local {
offset += min(stop.local - conv.range.lowerBound, conv.text.count)
}
}
return offset
}
private func moveCursor(by delta: Int) {
let stops = cursorStops
guard !stops.isEmpty else { return }
let next = (cursor ?? stops.count - 1) + delta
cursor = next >= stops.count - 1 ? nil : max(0, next)
}
// MARK: - Key handling
@@ -91,8 +162,9 @@ public final class CompositionEngine {
return handleCharacter(c)
case .backspace:
guard !keys.isEmpty else { return .passThrough }
cursor = nil
keys.removeLast()
fixed = validFixed(parse)
fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] }
return .consumed
case .enter:
@@ -104,29 +176,55 @@ public final class CompositionEngine {
return .consumed
case .tab:
guard !keys.isEmpty else { return .passThrough }
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
flip()
return .consumed
case .shiftTap:
if keys.isEmpty {
// Toggle English mode; it stays on until the next Shift tap.
isEnglishMode.toggle()
previousMode = isEnglishMode ? .literal : nil
previousCommitTime = now()
} else {
flip()
}
return .consumed
case .down:
guard !keys.isEmpty else { return .passThrough }
openCandidates()
return .consumed
case .up, .left, .right:
case .left:
guard !keys.isEmpty else { return .passThrough }
moveCursor(by: -1)
return .consumed
case .right:
guard !keys.isEmpty else { return .passThrough }
moveCursor(by: 1)
return .consumed
case .up:
return keys.isEmpty ? .passThrough : .consumed
}
}
/// Forces the whole buffer into the other mode (a correction that is learned).
private func flip() {
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
}
private func handleCharacter(_ c: Character) -> KeyResult {
cursor = nil // typing always continues at the end
// English mode: every key goes to the app as typed.
if isEnglishMode && keys.isEmpty { return .passThrough }
if c == " " {
guard !keys.isEmpty else { return .passThrough }
let decided = forcedMode ?? classifier.classify(keys, finishing: true, previous: context).mode
if decided == .literal {
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + " ")
let segs = segments(of: keys, finishing: true)
if segs.last?.mode == .literal {
return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
}
if parse.pending != nil {
if parse(ofLast: segs)?.pending != nil {
keys.append(" ") // first tone
return .consumed
}
return KeyResult(handled: true, commit: commitCurrent(as: .zhuyin))
return KeyResult(handled: true, commit: commitCurrent(segs))
}
let composable = DachenLayout.isZhuyinKey(c) || (c.isASCII && c.isLetter) || (c == "'" && !keys.isEmpty)
@@ -134,9 +232,9 @@ public final class CompositionEngine {
// ", . / ; -" are Zhuyin symbols (ㄝ ㄡ ㄥ ㄤ ㄦ) but act as punctuation after an
// English word. Decide with the key included: "2;" is ㄉㄤ (檔…), "hello," is English.
if !keys.isEmpty, DachenLayout.symbols[c] != nil, !c.isLetter, !c.isNumber, forcedMode != .zhuyin {
let withKey = forcedMode ?? classifier.classify(keys + [c], previous: context).mode
if withKey == .literal && !keys.allSatisfy(\.isNumber) {
return KeyResult(handled: true, commit: commitCurrent(as: .literal) + String(c))
let withKey = segments(of: keys + [c])
if withKey.last?.mode == .literal && !keys.allSatisfy(\.isNumber) {
return KeyResult(handled: true, commit: commitCurrent() + String(c))
}
}
keys.append(c)
@@ -151,25 +249,37 @@ public final class CompositionEngine {
return KeyResult(handled: true, commit: committed + punct)
}
private func parse(ofLast segs: [Segment]) -> ZhuyinParse? {
guard let last = segs.last, last.mode == .zhuyin else { return nil }
return ZhuyinParser.parse(Array(keys[last.range]))
}
static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
]
// MARK: - Candidates
// MARK: - Candidates (for the last Zhuyin segment)
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
private func openCandidates() {
let p = parse
guard mode == .zhuyin, !p.completed.isEmpty else { return }
candidates = converter.candidates(p.completed.map(\.syllable), endingAt: p.completed.count)
let stops = cursorStops
guard !stops.isEmpty else { return }
let stop = stops[min(cursor ?? stops.count - 1, stops.count - 1)]
let segs = segments
let seg = segs[stop.segIndex]
let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
candidates = converter.candidates(syl, endingAt: stop.local)
candidateSegmentStart = seg.range.lowerBound
candidateIndex = 0
}
public func selectCandidate(_ index: Int) {
guard candidates.indices.contains(index) else { return }
let c = candidates[index]
fixed = fixed.filter { !$0.key.overlaps(c.range) }
fixed[c.range] = c.text
var here = (fixed[candidateSegmentStart] ?? [:]).filter { !$0.key.overlaps(c.range) }
here[c.range] = c.text
fixed[candidateSegmentStart] = here
candidates = []
}
@@ -197,13 +307,13 @@ public final class CompositionEngine {
// MARK: - Commit
@discardableResult
private func commitCurrent(as decided: InputMode? = nil) -> String {
let m = decided ?? mode
let text = m == .literal ? String(keys) : zhuyinCommitText()
private func commitCurrent(_ given: [Segment]? = nil) -> String {
let segs = given ?? segments
let text = segs.map { render($0, final: true) }.joined()
if let forced = forcedMode {
classifier.learned.record(String(keys), as: forced)
}
previousMode = m
previousMode = segs.last?.mode
previousCommitTime = now()
reset()
return text
@@ -216,6 +326,7 @@ public final class CompositionEngine {
public func reset() {
keys = []
cursor = nil
forcedMode = nil
fixed = [:]
candidates = []
+74 -15
View File
@@ -8,33 +8,63 @@ import Foundation
/// - a letter-bigram model trained on those words, for words not in either list.
public final class EnglishModel {
private var common = Set<String>()
private var dictionary = Set<String>()
private var sortedWords: [String] = []
private var sortedCommon: [String] = []
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
/// binary searches, so no 230k-entry Set has to be built at startup.
private var sortedDict: [String] = []
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
private var unseenBigram = -12.0
public init(commonWords: [String], dictionaryWords: [String] = []) {
/// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
public init(commonWords: [String], dictionaryWords: [String] = [],
bigram: (table: [String: Double], unseen: Double)? = nil) {
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
sortedCommon = common.sorted()
sortedWords = common.union(dictionary).sorted()
trainBigram()
var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
dict = Array(Set(dict)).sorted()
}
sortedDict = dict
if let bigram {
self.bigram = bigram.table
unseenBigram = bigram.unseen
} else {
trainBigram()
}
}
/// Loads common words from a bundle resource file and the system dictionary if present.
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) {
/// Loads the bundled data files. Falls back to the system word list and to training
/// the bigram table when the precomputed files are missing.
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
func lines(_ url: URL?) -> [String] {
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
return text.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.hasPrefix("#") }
.filter { !$0.isEmpty && !$0.hasPrefix("#") }
}
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary))
var bigram: (table: [String: Double], unseen: Double)?
if let url = bigramURL, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
bigram = (table, unseen)
}
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
?? URL(fileURLWithPath: "/usr/share/dict/words")
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
}
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
private func inDictionary(_ w: String) -> Bool {
let i = lowerBound(w, in: sortedDict)
return i < sortedDict.count && sortedDict[i] == w
}
/// Some word strictly longer than `p` starts with it (common words or dictionary).
private func isPrefixOfWord(_ p: String) -> Bool {
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
}
/// - Parameter complete: true when the user ended the word (space / punctuation),
/// false while still typing (prefixes of words get partial credit).
@@ -43,16 +73,40 @@ public final class EnglishModel {
let n = Double(w.count)
guard !w.isEmpty else { return 0 }
if common.contains(w) { return 3 + n }
if dictionary.contains(w) { return 1 + 0.6 * n }
if inDictionary(w) { return 1 + 0.6 * n }
if !complete {
if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
if hasPrefix(w, in: sortedWords) { return 0.2 * n }
if isPrefixOfWord(w) { return 0.2 * n }
}
// Unknown word: judge by how English-like its letter sequence is.
let plaus = plausibility(w)
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
/// Feature dict for the classifier (see Features.swift); `score` is their dot
/// product with the default weights.
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
let w = word.lowercased()
let n = Double(w.count)
var f: FeatureVector = [:]
if n <= 2 { f["lit.short"] = 1 }
if common.contains(w) {
f["lit.common"] = 1
f["lit.common_len"] = n
} else if inDictionary(w) {
f["lit.dict"] = 1
f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n
} else if !complete && isPrefixOfWord(w) {
f["lit.prefix_dict_len"] = n
} else {
let plaus = plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
return f
}
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
public func plausibility(_ w: String) -> Double {
let chars = ["^"] + w.map(String.init) + ["$"]
@@ -61,12 +115,17 @@ public final class EnglishModel {
return total / Double(chars.count - 1)
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
private func lowerBound(_ p: String, in words: [String]) -> Int {
var lo = 0, hi = words.count
while lo < hi {
let mid = (lo + hi) / 2
if words[mid] < p { lo = mid + 1 } else { hi = mid }
}
return lo
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
var lo = lowerBound(p, in: words)
// words[lo] is the first word >= p; skip an exact match, then test the next one.
if lo < words.count && words[lo] == p { lo += 1 }
return lo < words.count && words[lo].hasPrefix(p)
@@ -75,7 +134,7 @@ public final class EnglishModel {
private func trainBigram() {
var pair: [String: Double] = [:]
var first: [String: Double] = [:]
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
let weight = common.contains(w) ? 5.0 : 1.0
let chars = ["^"] + w.map(String.init) + ["$"]
for i in 1..<chars.count {
+151
View File
@@ -0,0 +1,151 @@
import Foundation
/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode
/// is the dot product of its features with the weights. `defaultWeights` reproduces the
/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json
/// (training/train.py).
///
/// Python twin: windows/autozhuyin/core/features.py — both must produce identical
/// features (checked by ParityTests against training/parity-fixtures.json).
public typealias FeatureVector = [String: Double]
public enum Features {
public static let defaultWeights: FeatureVector = [
// Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
// Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
// Segmentation
"seg.switch": -3.0,
// Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
public static func loadWeights(from url: URL?) -> FeatureVector {
guard let url, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let w = obj["weights"] as? [String: Double] else { return [:] }
return w
}
public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double {
f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value }
}
/// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil }
// Space finishes an untoned last syllable as first tone; after a toned syllable it
// just commits, so it must not count as an extra (stray) tone.
let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse
if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil }
var f: FeatureVector = ["zh.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) }
let last = scored.syllables.count - 1
var tonedRun: [Syllable] = []
for (i, p) in scored.syllables.enumerated() {
if p.reordered { add("zh.reordered") }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
if let best = lexicon.bestScore(for: p.syllable) {
add("zh.toned_valid_lex")
add("zh.lexscore", best + 4)
} else {
add("zh.toned_valid_nolex")
}
tonedRun.append(p.syllable)
} else {
add("zh.toned_invalid")
}
} else if i == last {
add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid")
} else {
add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid")
}
}
// Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
var covered = [Bool](repeating: false, count: tonedRun.count)
for i in tonedRun.indices {
for n in 2...4 where i + n <= tonedRun.count {
if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty {
for k in i..<(i + n) { covered[k] = true }
}
}
}
let coveredCount = covered.filter { $0 }.count
if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) }
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed")
}
if afterOther {
add("seg.switch")
} else if previous == .zhuyin {
add("ctx.zh")
}
return f
}
/// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther {
add("seg.switch")
} else if previous == .literal {
add("ctx.lit")
}
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
if digits > 0 { add("lit.num_digits", Double(digits)) }
if others > 0 { add("lit.num_others", Double(others)) }
return f
}
for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) }
if digits > 0 { add("lit.digits", Double(digits)) }
let nonApostrophe = others - keys.filter { $0 == "'" }.count
if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) }
return f
}
}
+138 -28
View File
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
/// File format, one entry per line (`#` starts a comment):
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
/// The phrase/reading columns may be in either order; the score is optional.
///
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
/// call `prepareTonelessIndex()` from a background queue to have it ready.
public final class Lexicon {
private var exact: [String: [LexiconEntry]] = [:]
private var toneless: [String: [LexiconEntry]] = [:]
private var tonelessIndex: [String: [LexiconEntry]]?
private let lock = NSLock()
public private(set) var maxPhraseLength = 1
public init() {}
public convenience init(contentsOf url: URL) throws {
self.init()
load(try String(contentsOf: url, encoding: .utf8))
load(data: try Data(contentsOf: url))
}
public func load(_ text: String) {
for line in text.split(whereSeparator: \.isNewline) {
let trimmed = line.trimmingCharacters(in: .whitespaces)
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
guard cols.count >= 2 else { continue }
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
guard Lexicon.isBopomofo(reading) else { continue }
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
add(reading: reading, phrase: phrase, score: score)
}
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
load(data: Data(text.utf8))
}
public func add(reading rawReading: String, phrase: String, score: Double) {
let entry = LexiconEntry(phrase: phrase, score: score)
public func load(data: Data) {
lock.lock()
defer { lock.unlock() }
exact.reserveCapacity(exact.count + data.count / 30)
data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
let bytes = raw.bindMemory(to: UInt8.self)
let n = bytes.count
var i = 0
while i < n {
var end = i
while end < n && bytes[end] != 0x0A { end += 1 }
parseLine(bytes, i, end)
i = end + 1
}
}
for (key, entries) in exact where entries.count > 1 {
exact[key] = entries.sorted { $0.score > $1.score }
}
tonelessIndex = nil // rebuilt on demand
}
/// Splits one line into up to three space/tab separated fields.
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
var fields: [Range<Int>] = []
var i = start
while i < end && fields.count < 3 {
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
if i >= end { break }
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
let s = i
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
fields.append(s..<i)
}
guard fields.count >= 2 else { return }
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
guard Lexicon.isBopomofo(b, r) else { return }
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
var prefixNeutral = b[r.lowerBound] == 0xCB
var k = r.lowerBound
while !prefixNeutral && k + 2 < r.upperBound {
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
k += 1
}
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
}
/// Parses "-3.9151"-style decimals without creating a String.
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
var i = r.lowerBound
var sign = 1.0
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
var value = 0.0, scale = 1.0, digits = 0, fraction = false
while i < r.upperBound {
let c = b[i]
if c >= 0x30 && c <= 0x39 {
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
digits += 1
} else if c == 0x2E && !fraction {
fraction = true
} else {
return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
}
i += 1
}
return digits > 0 ? sign * value : nil
}
/// Adds one entry (call `load` for bulk data; this does not re-sort).
public func add(reading: String, phrase: String, score: Double) {
lock.lock()
defer { lock.unlock() }
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
exact[key]?.sort { $0.score > $1.score }
tonelessIndex = nil
}
@discardableResult
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
var reading = rawReading
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
let syllables = rawReading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
if normalise {
reading = reading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
}.joined(separator: "-")
}
let reading = syllables.joined(separator: "-")
maxPhraseLength = max(maxPhraseLength, syllables.count)
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) {
exact[reading, default: []].append(entry)
maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
if exact[reading] == nil {
exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
} else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
}
let key = Lexicon.stripTones(reading)
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) {
toneless[key]![i].score = max(toneless[key]![i].score, score)
} else {
toneless[key, default: []].append(entry)
return reading
}
/// Builds the index that ignores tones (for untoned syllables). Safe to call from a
/// background queue; lookups build it themselves if it is not ready yet.
public func prepareTonelessIndex() {
lock.lock()
defer { lock.unlock() }
_ = buildTonelessIfNeeded()
}
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
if let index = tonelessIndex { return index }
var best: [String: [String: Double]] = [:]
for (reading, entries) in exact {
let key = Lexicon.stripTones(reading)
for e in entries {
if let s = best[key]?[e.phrase], s >= e.score { continue }
best[key, default: [:]][e.phrase] = e.score
}
}
var index: [String: [LexiconEntry]] = [:]
index.reserveCapacity(best.count)
for (key, phrases) in best {
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
}
tonelessIndex = index
return index
}
/// Entries for a sequence of readings. Untoned syllables match any tone.
public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
lock.lock()
defer { lock.unlock() }
if syllables.allSatisfy({ $0.tone != nil }) {
return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
}
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? []
return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
}
public func bestScore(for syllable: Syllable) -> Double? {
@@ -81,4 +182,13 @@ public final class Lexicon {
guard let first = s.unicodeScalars.first else { return false }
return (0x3105...0x312F).contains(first.value) || first == "˙"
}
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
guard r.count >= 2 else { return false }
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
let c = b[r.lowerBound + 2]
return c >= 0x85 && c <= 0xAF
}
}
@@ -0,0 +1,32 @@
import Foundation
/// Detects Shift pressed and released on its own (no other key in between, released
/// within `maxDuration`), as opposed to Shift used for capitals or shortcuts.
///
/// Python twin: windows/autozhuyin/core/shift_tap.py
public struct ShiftTapDetector {
public var maxDuration: TimeInterval = 0.5
private var pressedAt: TimeInterval?
private var used = false
public init() {}
/// Call when the modifier state changes. Returns true when a Shift tap completed.
public mutating func modifiersChanged(shift: Bool, otherModifiers: Bool, time: TimeInterval) -> Bool {
if shift && !otherModifiers {
if pressedAt == nil {
pressedAt = time
used = false
}
return false
}
defer { pressedAt = nil }
guard let start = pressedAt, !shift, !otherModifiers, !used else { return false }
return time - start <= maxDuration
}
/// Call for every other key press: Shift was used as a modifier, not tapped.
public mutating func keyPressed() {
if pressedAt != nil { used = true }
}
}
@@ -6,7 +6,11 @@ private let resources = URL(fileURLWithPath: #filePath)
.appendingPathComponent("Resources")
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"))
private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase {
func testParsesNiHao() {
@@ -84,7 +88,7 @@ final class ConverterTests: XCTestCase {
}
final class ClassifierTests: XCTestCase {
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish)
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
func mode(_ s: String, finishing: Bool = false) -> InputMode {
classifier.classify(Array(s), finishing: finishing).mode
@@ -106,7 +110,7 @@ final class ClassifierTests: XCTestCase {
func testLearningFlipsDecision() {
let learned = LearnedPreferences()
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned)
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
learned.record("so", as: .zhuyin)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
@@ -115,7 +119,7 @@ final class ClassifierTests: XCTestCase {
final class EngineTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
/// Types a string and returns everything committed.
@@ -188,7 +192,7 @@ final class EngineTests: XCTestCase {
final class DigitInitialTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish))
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func run(_ s: String) -> (out: String, marked: String) {
@@ -219,3 +223,219 @@ final class DigitInitialTests: XCTestCase {
XCTAssertEqual(run("hello, ").out, "hello, ")
}
}
final class MixedSegmentTests: XCTestCase {
func run(_ s: String) -> (out: String, marked: String) {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return (out, e.markedText)
}
func testChineseThenEnglish() {
XCTAssertEqual(run("ji3hello ").out, "我hello ")
XCTAssertEqual(run("su3cl3github\n").out, "你好github")
XCTAssertEqual(run("ji3hel").marked, "我hel")
}
func testEnglishThenChinese() {
XCTAssertEqual(run("hellosu3cl3 ").out, "hello你好")
XCTAssertEqual(run("oksu3cl3\n").out, "ok你好")
}
func testThreeSegments() {
XCTAssertEqual(run("ji3emailsu3cl3\n").out, "我email你好")
}
func testSingleLanguageUnchanged() {
XCTAssertEqual(run("rup wu0 wu0 fu4cp3cl3\n").out, "今天天氣很好")
XCTAssertEqual(run("helloworld\n").out, "helloworld")
XCTAssertEqual(run("meeting ").out, "meeting ")
XCTAssertEqual(run("183\n").out.count, 1)
}
}
/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py).
final class ParityTests: XCTestCase {
func testMatchesPythonSegmentation() throws {
let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json")
let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]]
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
var mismatches: [String] = []
for fx in fixtures {
let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let previous = (fx["previous"] as? String).flatMap(InputMode.init(rawValue:))
let got = c.segment(keys, finishing: fx["finishing"] as! Bool, previous: previous).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
}
if got != expected {
mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!) previous=\(String(describing: previous)): \(got) vs \(expected)")
}
}
XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
}
func testToneThenSpaceStaysChinese() {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out.count, 1, out)
}
}
final class ShiftTapTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "⇧" ? .shiftTap : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testShiftTapTogglesEnglishMode() {
let e = engine()
// "ru" alone would be ㄐㄧ (機); in English mode keys pass straight through, lowercase.
XCTAssertEqual(type("⇧ru su ", into: e), "ru su ")
XCTAssertTrue(e.isEnglishMode) // still English after spaces
XCTAssertEqual(e.markedText, "") // never composed: no underline
XCTAssertEqual(type("\n", into: e), "\n") // Enter goes to the app, mode stays
XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(type("⇧su3cl3\n", into: e), "你好") // tap again: automatic again
XCTAssertFalse(e.isEnglishMode)
}
func testShiftTapTwiceCancels() {
XCTAssertEqual(type("⇧⇧su3cl3\n", into: engine()), "你好")
}
func testShiftTapMidWordFlips() {
let e = engine()
_ = type("so", into: e)
XCTAssertEqual(e.mode, .literal)
_ = e.handle(.shiftTap)
XCTAssertEqual(e.mode, .zhuyin)
XCTAssertFalse(e.isEnglishMode) // a flip, not English mode
_ = type("\n", into: e)
XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned
}
func testEnglishModeIsNotLearned() {
let e = engine()
_ = type("⇧hello ", into: e)
XCTAssertTrue(e.classifier.learned.weights.isEmpty)
}
func testBackspaceInEnglishModeGoesToApp() {
let e = engine()
_ = type("⇧", into: e)
XCTAssertFalse(e.handle(.backspace).handled)
XCTAssertTrue(e.isEnglishMode)
}
func testDetector() {
var d = ShiftTapDetector()
XCTAssertFalse(d.modifiersChanged(shift: true, otherModifiers: false, time: 0))
XCTAssertTrue(d.modifiersChanged(shift: false, otherModifiers: false, time: 0.1))
// Shift used for a capital letter: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 1)
d.keyPressed()
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 1.1))
// Held too long: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 2)
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 3))
// Shift + Command: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 4)
_ = d.modifiersChanged(shift: true, otherModifiers: true, time: 4.1)
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 4.2))
}
}
final class CursorTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testCursorMovesAndClamps() {
let e = engine()
_ = type("rup wu0 wu0 fu4", into: e) // 今天天氣
XCTAssertEqual(e.cursorOffset, 4)
_ = type("←", into: e)
XCTAssertEqual(e.cursorOffset, 3)
_ = type("←←←←", into: e)
XCTAssertEqual(e.cursorOffset, 1) // stops after the first character
_ = type("→→→→", into: e)
XCTAssertEqual(e.cursorOffset, 4)
}
func testChooseCandidateInTheMiddle() {
let e = engine()
_ = type("rup wu0 wu0 fu4", into: e)
_ = type("←←↓", into: e) // cursor after the 2nd syllable (天)
XCTAssertTrue(e.isShowingCandidates)
XCTAssertTrue(e.candidates.allSatisfy { $0.range.upperBound == 2 })
let i = e.candidates.firstIndex { $0.text == "添" }!
e.selectCandidate(i)
XCTAssertEqual(type("\n", into: e), "今添天氣")
}
func testTypingReturnsCursorToEnd() {
let e = engine()
_ = type("rup wu0 ←", into: e)
XCTAssertEqual(e.cursorOffset, 1)
_ = type("wu0 ", into: e)
XCTAssertEqual(e.cursorOffset, e.markedText.count)
}
func testCandidatesForChineseBeforeEnglish() {
let e = engine()
_ = type("su3cl3hello", into: e) // 你好hello
_ = type("↓", into: e) // last Chinese syllable, not the English word
XCTAssertTrue(e.isShowingCandidates)
}
}
final class FirstToneAfterEnglishTests: XCTestCase {
/// After an English word, a first-tone syllable + space must stay Chinese (機, 吃, 書…).
func testFirstToneAfterEnglish() {
for keys in ["ru", "t", "gj", "vup", "tj", "el"] {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out, "hello ", keys)
XCTAssertEqual(e.markedText.count, 1, keys) // still composing one Chinese character
}
}
}
final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws {
var t = Date()
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
}
}
+4 -1
View File
@@ -18,7 +18,10 @@ rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff
+32
View File
@@ -0,0 +1,32 @@
#!/bin/bash
# Increments the version in every place it lives:
# Resources/Info.plist CFBundleShortVersionString (x.y.z) and CFBundleVersion (build number)
# windows/autozhuyin/ime.json "version"
#
# Usage: scripts/bump-version.sh [patch|minor|major] (default: patch)
# Prints the new version.
set -euo pipefail
cd "$(dirname "$0")/.."
LEVEL="${1:-patch}"
PLIST=Resources/Info.plist
IME=windows/autozhuyin/ime.json
CURRENT="$(plutil -extract CFBundleShortVersionString raw "$PLIST")"
IFS=. read -r MAJOR MINOR PATCH <<< "$CURRENT"
case "$LEVEL" in
major) MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0 ;;
minor) MINOR=$((MINOR + 1)); PATCH=0 ;;
patch) PATCH=$((PATCH + 1)) ;;
*) echo "usage: $0 [patch|minor|major]" >&2; exit 2 ;;
esac
NEW="$MAJOR.$MINOR.$PATCH"
BUILD=$(( $(plutil -extract CFBundleVersion raw "$PLIST") + 1 ))
plutil -replace CFBundleShortVersionString -string "$NEW" "$PLIST"
plutil -replace CFBundleVersion -string "$BUILD" "$PLIST"
# Edit only the version line so the rest of ime.json keeps its formatting.
sed -i '' -E "s/(\"version\"[[:space:]]*:[[:space:]]*\")[^\"]*\"/\1$NEW\"/" "$IME"
grep -q "\"version\": \"$NEW\"" "$IME" || { echo "failed to update $IME" >&2; exit 1; }
echo "$NEW"
+11
View File
@@ -13,6 +13,17 @@ rm -rf "$DEST/AutoZhuyin.app"
cp -R build/AutoZhuyin.app "$DEST/"
"$DEST/AutoZhuyin.app/Contents/MacOS/AutoZhuyin" --register
SYSTEM_APP="/Library/Input Methods/AutoZhuyin.app"
if [ -d "$SYSTEM_APP" ]; then
OLD="$(plutil -extract CFBundleShortVersionString raw "$SYSTEM_APP/Contents/Info.plist" 2>/dev/null || echo '?')"
echo
echo "WARNING: a system-wide copy (v$OLD, installed from the .pkg) exists at:"
echo " $SYSTEM_APP"
echo "macOS runs that copy instead of this one. Install the new .pkg"
echo "(scripts/package.sh), or remove it with:"
echo " sudo rm -rf \"$SYSTEM_APP\" && pkill -x AutoZhuyin"
fi
echo
echo "Installed to $DEST/AutoZhuyin.app"
echo "First install: System Settings → Keyboard → Text Input → Edit… → + → 繁體中文 → 自動注音"
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# One-step release: bump the version, run all tests, build both installers, then
# commit the version change and tag it.
#
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
# --push also pushes the commit and tag to origin
set -euo pipefail
cd "$(dirname "$0")/.."
LEVEL=patch
PUSH=0
for arg in "$@"; do
case "$arg" in
patch|minor|major) LEVEL="$arg" ;;
--push) PUSH=1 ;;
*) echo "usage: $0 [patch|minor|major] [--push]" >&2; exit 2 ;;
esac
done
if [ -n "$(git status --porcelain)" ]; then
echo "Working tree has uncommitted changes; commit them first." >&2
exit 1
fi
export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}"
VERSION="$(scripts/bump-version.sh "$LEVEL")"
echo "==> Version $VERSION"
restore() {
echo "Release failed; reverting the version bump." >&2
git checkout -- Resources/Info.plist windows/autozhuyin/ime.json
}
trap restore ERR
echo "==> Swift tests"
xcrun swift test -q 2>&1 | tail -3
echo "==> Packaging (runs the Python tests too)"
scripts/package.sh >/dev/null
scripts/package-windows.sh >/dev/null
trap - ERR
git add Resources/Info.plist windows/autozhuyin/ime.json
git commit -q -m "Release v$VERSION"
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
echo "==> Tagged v$VERSION"
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip"
if [ "$PUSH" = 1 ]; then
git push origin HEAD "v$VERSION"
else
echo "Push with: git push origin HEAD v$VERSION"
fi
+30
View File
@@ -0,0 +1,30 @@
"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
import os
import sys
import types
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
WIN = os.path.join(ROOT, "windows")
DATA = os.path.join(WIN, "autozhuyin", "data")
RESOURCES = os.path.join(ROOT, "Resources")
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
sys.path.insert(0, WIN)
from autozhuyin.core import bopomofo, features # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
def load_models():
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
sys.exit("Run windows/tools/build_data.py first.")
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
return lexicon, english
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Writes training/parity-fixtures.json: segmentations produced by the Python core
with the current weights. The Swift tests (ParityTests) must produce the same ones,
which keeps the two implementations in sync. Re-run after changing features or weights.
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator # noqa: E402
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json")
HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up",
"u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"]
def main():
lexicon, english = load_models()
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f, p) for k in HAND_PICKED for f in (False, True) for p in (None, "literal")]
while len(cases) < 500:
ex = gen.example()
if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"], ex["previous"]))
cases.append((gen.prefix_of(ex)["keys"], False, ex["previous"]))
fixtures = [{"keys": k, "finishing": f, "previous": p,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f, p)]} for k, f, p in cases]
with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT))
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+147
View File
@@ -0,0 +1,147 @@
"""Synthetic typing data for the Zhuyin / literal classifier.
Each example is what sits in the input buffer when the user commits it, with the
intended mode of every key:
{"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"}
labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter.
Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys;
English from the common-word list and the dictionary. Real typing commits English at
a space, so a buffer holds at most one English word next to Chinese.
"""
import random
from common import core
bopomofo = core.bopomofo
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
def reading_to_keys(reading):
"""'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable."""
out = []
for syl in reading.split("-"):
tone = " "
if syl and syl[-1] in TONE_KEY:
tone = TONE_KEY[syl[-1]]
syl = syl[:-1]
if not syl or syl not in bopomofo.VALID:
return None
out.append("".join(KEY_OF[c] for c in syl) + tone)
return "".join(out)
def _is_cjk(s):
return all(0x4E00 <= ord(c) <= 0x9FFF for c in s)
class Generator:
def __init__(self, lexicon_path, common_path, dict_path, seed=0):
self.rng = random.Random(seed)
phrases, weights = [], []
with open(lexicon_path, encoding="utf-8") as f:
for line in f:
cols = line.split()
if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]):
continue
n = cols[0].count("-") + 1
if n > 4:
continue
keys = reading_to_keys(cols[0])
if keys is None:
continue
phrases.append((cols[1], keys))
weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency
self.phrases = phrases
self.phrase_cum = _cumulative(weights)
self.common = [w.strip() for w in open(common_path, encoding="utf-8")
if w.strip() and not w.startswith("#")]
self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8")
if 3 <= len(w.strip()) <= 9]
# --- pieces ---
def chinese(self):
n = self.rng.choice([1, 1, 2, 2, 3])
picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n)
return "".join(keys for _, keys in picked)
def english(self):
w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words)
if self.rng.random() < 0.1:
w = w[:1].upper() + w[1:]
return w
def number(self):
r = self.rng.random()
if r < 0.4:
return str(self.rng.randint(0, 9999))
if r < 0.7:
return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30]))
return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9))
# --- examples ---
def example(self):
r = self.rng.random()
if r < 0.40:
parts, kind = [("z", self.chinese())], "zh"
elif r < 0.65:
parts, kind = [("l", self.english())], "en"
elif r < 0.80:
parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en"
elif r < 0.93:
parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh"
elif r < 0.96:
parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh"
else:
parts, kind = [("l", self.number())], "num"
keys = "".join(p for _, p in parts)
labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l":
finishing = True # English / numbers end with a space
elif keys[-1] == " " and self.rng.random() < 0.6:
# The moment the space (first tone) is pressed: the buffer has no space yet and
# the input method decides "first tone" vs "English word + space".
keys, labels, finishing = keys[:-1], labels[:-1], True
kind += "/tone1"
else:
# Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind,
"previous": self.previous_mode(labels[0])}
def previous_mode(self, first_label):
"""Mode of the text committed just before: none, mostly the same, sometimes the other."""
r = self.rng.random()
if r < 0.35:
return None
same = core.ZHUYIN if first_label == "z" else core.LITERAL
other = core.LITERAL if first_label == "z" else core.ZHUYIN
return same if r < 0.80 else other
def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix", "previous": ex["previous"]}
def _cumulative(ws):
out, t = [], 0.0
for w in ws:
t += w
out.append(t)
return out
def gold_segments(labels):
"""'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]"""
segs, start = [], 0
for i in range(1, len(labels) + 1):
if i == len(labels) or labels[i] != labels[start]:
segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL))
start = i
return segs
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""Learns the classifier weights from synthetic typing data (averaged structured
perceptron over the same segmentation DP the input method runs) and reports
accuracy against the hand-tuned defaults.
Usage (from the repo root, after windows/tools/build_data.py):
python3 training/train.py # train, evaluate, write Resources/classifier-weights.json
python3 training/train.py --eval-only # just evaluate the current weights file
"""
import argparse
import datetime
import json
import os
import random
import sys
import time
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator, gold_segments # noqa: E402
MAX_KEYS = 28
FIXED = {"lit.upper"} # hard rule (Shift = English), not learned
def make_set(gen, n):
out = []
while len(out) < 2 * n:
ex = gen.example()
if len(ex["keys"]) > MAX_KEYS:
continue
out.append(ex)
if len(ex["keys"]) > 1:
out.append(gen.prefix_of(ex))
return out
def labels_of(segs, n):
lab = [""] * n
for s, e, m in segs:
for i in range(s, e):
lab[i] = "z" if m == core.ZHUYIN else "l"
return "".join(lab)
def seg_features(clf, keys, segs, finishing, previous):
"""Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float)
n = len(keys)
for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing, previous if idx == 0 else None)
f = fz if m == core.ZHUYIN else fl
if f is None:
return None
for k, v in f.items():
total[k] += v
return total
def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0])
for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], ex["previous"], weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
kind = "prefix" if ex["kind"].endswith("/prefix") else ex["kind"]
for k in (kind, "after-other") if _after_other(ex) else (kind,):
by_kind[k][0] += ok
by_kind[k][1] += 1
commit = [v for k, v in by_kind.items() if k not in ("prefix", "after-other")]
summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
"by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())},
}
return summary
def _after_other(ex):
"""Committed buffer whose first segment differs from the text typed just before."""
if ex["kind"].endswith("/prefix") or ex["previous"] is None:
return False
first = core.ZHUYIN if ex["labels"][0] == "z" else core.LITERAL
return ex["previous"] != first
def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
def train(clf, data, epochs, lr, seed):
rng = random.Random(seed)
w = dict(clf.weights)
total = defaultdict(float)
steps = 0
for epoch in range(epochs):
rng.shuffle(data)
mistakes = 0
t0 = time.time()
for ex in data:
keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], ex["previous"], w)
if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"], ex["previous"])
fp = seg_features(clf, keys, pred, ex["finishing"], ex["previous"])
if fg is not None and fp is not None:
mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
for k in set(fg) | set(fp):
if k not in FIXED:
w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0))
steps += 1
for k, v in w.items():
total[k] += v
print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True)
return {k: round(v / steps, 4) for k, v in total.items()}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)")
ap.add_argument("--test", type=int, default=1500)
ap.add_argument("--epochs", type=int, default=6)
ap.add_argument("--lr", type=float, default=0.1)
ap.add_argument("--eval-only", action="store_true")
args = ap.parse_args()
lexicon, english = load_models()
paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"))
test = make_set(Generator(*paths, seed=2), args.test)
clf = core.Classifier(lexicon, english) # default (hand-tuned) weights
show("default", evaluate(clf, clf.weights, test))
if os.path.exists(WEIGHTS_PATH):
current = dict(clf.weights)
current.update(core.Classifier.load_weights(WEIGHTS_PATH))
show("current", evaluate(clf, current, test))
if args.eval_only:
return
data = make_set(Generator(*paths, seed=1), args.train)
print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True)
learned = train(clf, data, args.epochs, args.lr, seed=3)
result = evaluate(clf, learned, test)
show("learned", result)
with open(WEIGHTS_PATH, "w", encoding="utf-8") as f:
json.dump({
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": datetime.date.today().isoformat(),
"examples": len(data),
"test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)},
"weights": dict(sorted(learned.items())),
}, f, ensure_ascii=False, indent=2)
f.write("\n")
print("wrote", WEIGHTS_PATH)
if __name__ == "__main__":
main()
+29 -3
View File
@@ -6,14 +6,16 @@ tests); this file only translates PIME key events and replies.
"""
import json
import os
import time
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
from textService import TextService
from .core.classifier import Classifier, LearnedPreferences
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, TAB, UP, CompositionEngine
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
from .core.english import EnglishModel
from .core.lexicon import Lexicon
from .core.shift_tap import ShiftTapDetector
_HERE = os.path.dirname(os.path.abspath(__file__))
_DATA = os.path.join(_HERE, "data")
@@ -52,7 +54,8 @@ def _shared_classifier():
weights = json.load(f)
except (OSError, ValueError):
pass
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned))
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained)
return _classifier
@@ -66,6 +69,7 @@ class AutoZhuyinTextService(TextService):
def __init__(self, client):
TextService.__init__(self, client)
self.engine = CompositionEngine(_shared_classifier())
self.shift_tap = ShiftTapDetector()
# --- lifecycle ---
@@ -104,6 +108,13 @@ class AutoZhuyinTextService(TextService):
return bool(self.engine.keys or self.engine.candidates)
def filterKeyDown(self, ev):
if ev.keyCode == VK_SHIFT: # noqa: F405
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
self.shift_tap.key_pressed()
else:
self.shift_tap.shift_down(time.monotonic())
return False
self.shift_tap.key_pressed()
# Shortcuts (Ctrl/Alt) go to the app.
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
return False
@@ -120,6 +131,21 @@ class AutoZhuyinTextService(TextService):
self._update(commit)
return handled
def filterKeyUp(self, ev):
# Shift pressed and released alone: next word is English (or flip the current one).
if ev.keyCode == VK_SHIFT: # noqa: F405
return self.shift_tap.shift_up(time.monotonic())
return False
def onKeyUp(self, ev):
if ev.keyCode != VK_SHIFT: # noqa: F405
return False
self.engine.handle(SHIFT_TAP)
self._update(None)
if not self.engine.keys:
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
return True
# --- replies ---
def _update(self, commit):
@@ -127,7 +153,7 @@ class AutoZhuyinTextService(TextService):
self.setCommitString(commit)
marked = self.engine.marked_text
self.setCompositionString(marked)
self.setCompositionCursor(len(marked))
self.setCompositionCursor(self.engine.cursor_offset)
if self.engine.showing_candidates:
page, cursor = self.engine.page()
self.setCandidateList(page)
+75 -53
View File
@@ -1,5 +1,7 @@
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
from . import bopomofo
import json
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
ZHUYIN = "zhuyin"
LITERAL = "literal"
@@ -28,67 +30,87 @@ class LearnedPreferences:
class Classifier:
zhuyin_prior = 0.5
context_weight = 1.5
def __init__(self, lexicon, english, learned=None):
def __init__(self, lexicon, english, learned=None, weights=None):
self.lexicon = lexicon
self.english = english
self.learned = learned or LearnedPreferences()
self.weights = dict(DEFAULT_WEIGHTS)
if weights:
self.weights.update(weights)
@classmethod
def load_weights(cls, path):
"""Reads a weights JSON file ({"weights": {...}}); returns {} if missing."""
try:
with open(path, encoding="utf-8") as f:
return json.load(f).get("weights", {})
except (OSError, ValueError):
return {}
def segment_features(self, seg, is_last, after_other, finishing, previous=None):
"""(zhuyin features or None, literal features or None) for one segment.
`previous` (mode of the text committed just before) only affects the first segment."""
seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing, previous),
literal_features(seg, self.english, is_last, after_other, finishing, previous))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing, previous)
zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF
if whole:
b = self.learned.bias("".join(seg))
if b > 0:
zh += b
else:
lit -= b
return zh, lit
def classify(self, keys, finishing=False, previous=None):
"""Returns (mode, zhuyin_score, literal_score)."""
keys = list(keys)
zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior
lit = self.literal_score(keys, complete=finishing)
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
b = self.learned.bias("".join(keys))
if b > 0:
zh += b
else:
lit -= b
"""Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score)."""
zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights)
return (ZHUYIN if zh >= lit else LITERAL), zh, lit
def mode(self, keys, finishing=False, previous=None):
return self.classify(keys, finishing, previous)[0]
def zhuyin_score(self, keys):
p = bopomofo.parse(keys)
if p.foreign_keys > 0 or not p.syllables:
return NEG_INF
s = -5.0 * p.stray_tones
last = len(p.syllables) - 1
for i, ps in enumerate(p.syllables):
if ps.reordered:
s -= 0.5
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5)
else:
s -= 5
elif i == last:
s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5
else:
s += -2.5 if valid else -5
return s
# --- segmentation ---
def literal_score(self, keys, complete):
if not keys or " " in keys:
return NEG_INF
score = 0.0
if any(c.isupper() for c in keys):
score += 20
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
return score + 0.8 * digits - 1.0 * others
score += self.english.score(letters, complete or digits > 0)
score -= 1.5 * digits
apostrophes = keys.count("'")
score -= 1.5 * (others - apostrophes)
return score
def segment(self, keys, finishing=False, previous=None, weights=None):
"""Splits keys into alternating segments: [(start, end, mode)], e.g.
"ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
split points; each segment is scored by features x weights."""
weights = weights or self.weights
keys = list(keys)
n = len(keys)
if n == 0:
return []
modes = (ZHUYIN, LITERAL)
best = [[NEG_INF, NEG_INF] for _ in range(n + 1)]
back = [[-1, -1] for _ in range(n + 1)]
for i in range(1, n + 1):
for j in range(i):
scores = self._scores(keys[j:i], i == n, j > 0, finishing,
previous if j == 0 else None, j == 0 and i == n, weights)
for m in (0, 1):
if scores[m] == NEG_INF:
continue
if j == 0:
total = scores[m]
else:
if best[j][1 - m] == NEG_INF:
continue
total = best[j][1 - m] + scores[m]
if total > best[i][m]:
best[i][m] = total
back[i][m] = j
if best[n][0] == NEG_INF and best[n][1] == NEG_INF:
return [(0, n, self.mode(keys, finishing, previous))]
m = 0 if best[n][0] >= best[n][1] else 1
out, i = [], n
while i > 0:
j = back[i][m]
out.append((j, i, modes[m]))
i, m = j, 1 - m
return out[::-1]
+155 -45
View File
@@ -2,6 +2,7 @@
Keys are given as one-character strings, or one of the names in SPECIAL_KEYS.
handle() returns (handled, commit_text_or_None).
The buffer is split into Zhuyin / literal segments, e.g. "ji3hello" -> 我 + hello.
"""
import time
@@ -11,7 +12,8 @@ from .lexicon import Converter
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>"
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT}
SHIFT_TAP = "<shift-tap>" # Shift pressed and released on its own
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT, SHIFT_TAP}
FULLWIDTH = {
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")",
@@ -32,11 +34,19 @@ class CompositionEngine:
self.now = now
self.keys = []
self.forced_mode = None
self.fixed = {}
# English mode, toggled by a Shift tap when nothing is being typed: keys go straight
# to the app (no composition, no underline) until Shift is tapped again.
self.english_mode = False
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = []
self.candidate_index = 0
self._candidate_segment_start = 0
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end.
self.cursor = None
self._previous_mode = None
self._previous_time = float("-inf")
self._cache_key = None
self._cache_value = None
# --- derived state ---
@@ -44,9 +54,26 @@ class CompositionEngine:
def context(self):
return self._previous_mode if self.now() - self._previous_time < self.context_window else None
def _segments(self, keys, finishing=False):
if not keys:
return []
if self.forced_mode:
return [(0, len(keys), self.forced_mode)]
ctx = self.context
cache_key = (tuple(keys), finishing, ctx)
if cache_key != self._cache_key:
self._cache_key = cache_key
self._cache_value = self.classifier.segment(keys, finishing, ctx)
return self._cache_value
@property
def segments(self):
return self._segments(self.keys)
@property
def mode(self):
return self.forced_mode or self.classifier.mode(self.keys, previous=self.context)
segs = self.segments
return segs[-1][2] if segs else ZHUYIN
@property
def showing_candidates(self):
@@ -56,35 +83,83 @@ class CompositionEngine:
def page_start(self):
return (self.candidate_index // self.page_size) * self.page_size
def _parse(self):
return bopomofo.parse(self.keys)
def _convertible(self, seg, final):
"""(syllables converted to Chinese, pending Bopomofo) for a Zhuyin segment."""
start, end, _ = seg
p = bopomofo.parse(self.keys[start:end])
if final:
return [s.syllable for s in p.syllables], ""
return [s.syllable for s in p.completed], (p.pending.syllable.reading if p.pending else "")
def _valid_fixed(self, p):
n = len(p.completed)
return {k: v for k, v in self.fixed.items() if k[1] <= n}
def _conversion(self, seg, final):
syl, pending = self._convertible(seg, final)
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
return self.converter.convert(syl, fx), pending
def _render(self, seg, final):
start, end, mode = seg
if mode == LITERAL:
return "".join(self.keys[start:end])
converted, pending = self._conversion(seg, final)
return "".join(t for _, t in converted) + pending
# --- cursor ---
def _cursor_stops(self):
"""[(segment index, syllables before the stop)] for every Zhuyin syllable."""
segs = self.segments
out = []
for i, seg in enumerate(segs):
if seg[2] == ZHUYIN:
n = len(self._convertible(seg, i < len(segs) - 1)[0])
out.extend((i, k) for k in range(1, n + 1))
return out
@property
def cursor_offset(self):
"""Cursor position in marked_text (characters)."""
pieces = self.marked_pieces
stops = self._cursor_stops()
if self.cursor is None or not 0 <= self.cursor < len(stops):
return sum(len(t) for t, _ in pieces)
seg_index, local = stops[self.cursor]
offset = sum(len(t) for t, _ in pieces[:seg_index])
segs = self.segments
for (a, b), text in self._conversion(segs[seg_index], seg_index < len(segs) - 1)[0]:
if b <= local:
offset += len(text)
elif a < local:
offset += min(local - a, len(text))
return offset
def _move_cursor(self, delta):
stops = self._cursor_stops()
if not stops:
return
nxt = (len(stops) - 1 if self.cursor is None else self.cursor) + delta
self.cursor = None if nxt >= len(stops) - 1 else max(0, nxt)
@property
def marked_pieces(self):
segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
@property
def marked_text(self):
if not self.keys:
return ""
if self.mode == LITERAL:
return "".join(self.keys)
p = self._parse()
done = self.converter.convert([s.syllable for s in p.completed], self._valid_fixed(p))
pending = p.pending.syllable.reading if p.pending else ""
return "".join(t for _, t in done) + pending
def _zhuyin_commit_text(self):
p = self._parse()
segs = self.converter.convert([s.syllable for s in p.syllables], self._valid_fixed(p))
return "".join(t for _, t in segs)
return "".join(t for t, _ in self.marked_pieces)
# --- key handling ---
def _flip(self):
"""Forces the whole buffer into the other mode (a correction that is learned)."""
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
def wants(self, key):
"""Whether handle(key) would consume the key, without changing state."""
if self.candidates:
if self.candidates or key == SHIFT_TAP:
return True
if self.english_mode and not self.keys:
return False # English mode: everything goes to the app as typed
if key in SPECIAL_KEYS:
return bool(self.keys)
if self.keys:
@@ -101,14 +176,17 @@ class CompositionEngine:
if key == BACKSPACE:
if not self.keys:
return PASS
self.cursor = None
self.keys.pop()
self.fixed = self._valid_fixed(self._parse())
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys:
self.forced_mode = None
self.fixed = {}
return CONSUMED
if key == ENTER:
return (True, self._commit()) if self.keys else PASS
if not self.keys:
return PASS
return True, self._commit()
if key == ESCAPE:
if not self.keys:
return PASS
@@ -117,36 +195,60 @@ class CompositionEngine:
if key == TAB:
if not self.keys:
return PASS
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
self._flip()
return CONSUMED
if key == SHIFT_TAP:
if not self.keys:
# Toggle English mode; it stays on until the next Shift tap.
self.english_mode = not self.english_mode
self._previous_mode = LITERAL if self.english_mode else None
self._previous_time = self.now()
else:
self._flip()
return CONSUMED
if key == DOWN:
if not self.keys:
return PASS
self._open_candidates()
return CONSUMED
if key in (UP, LEFT, RIGHT):
if key in (LEFT, RIGHT):
if not self.keys:
return PASS
self._move_cursor(-1 if key == LEFT else 1)
return CONSUMED
if key == UP:
return CONSUMED if self.keys else PASS
return self._handle_character(key)
def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN:
return None
start, end, _ = segs[-1]
return bopomofo.parse(self.keys[start:end])
def _handle_character(self, c):
self.cursor = None # typing always continues at the end
if self.english_mode and not self.keys:
return PASS # English mode: every key goes to the app as typed
if c == " ":
if not self.keys:
return PASS
decided = self.forced_mode or self.classifier.mode(self.keys, finishing=True, previous=self.context)
if decided == LITERAL:
return True, self._commit(LITERAL) + " "
if self._parse().pending is not None:
segs = self._segments(self.keys, finishing=True)
if segs[-1][2] == LITERAL:
return True, self._commit(segs) + " "
p = self._last_parse(segs)
if p is not None and p.pending is not None:
self.keys.append(" ")
return CONSUMED
return True, self._commit(ZHUYIN)
return True, self._commit(segs)
composable = bopomofo.is_zhuyin_key(c) or (c.isascii() and c.isalpha()) or (c == "'" and self.keys)
if composable:
# ", . / ; -" are Zhuyin symbols but act as punctuation after an English word.
if self.keys and c in bopomofo.SYMBOLS and not c.isalnum() and self.forced_mode != ZHUYIN:
with_key = self.forced_mode or self.classifier.mode(self.keys + [c], previous=self.context)
if with_key == LITERAL and not all(k.isdigit() for k in self.keys):
return True, self._commit(LITERAL) + c
with_key = self._segments(self.keys + [c])
if with_key[-1][2] == LITERAL and not all(k.isdigit() for k in self.keys):
return True, self._commit() + c
self.keys.append(c)
return CONSUMED
@@ -157,14 +259,19 @@ class CompositionEngine:
return PASS
return True, committed + punct
# --- candidates ---
# --- candidates (for the last Zhuyin segment) ---
def _open_candidates(self):
p = self._parse()
if self.mode != ZHUYIN or not p.completed:
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
stops = self._cursor_stops()
if not stops:
return
syl = [s.syllable for s in p.completed]
self.candidates = self.converter.candidates(syl, len(syl))
seg_index, local = stops[min(len(stops) - 1 if self.cursor is None else self.cursor, len(stops) - 1)]
segs = self.segments
seg = segs[seg_index]
syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
self.candidates = self.converter.candidates(syl, local)
self._candidate_segment_start = seg[0]
self.candidate_index = 0
def page(self):
@@ -176,8 +283,10 @@ class CompositionEngine:
if not 0 <= index < len(self.candidates):
return
rng, text = self.candidates[index]
self.fixed = {k: v for k, v in self.fixed.items() if not (k[0] < rng[1] and rng[0] < k[1])}
self.fixed[rng] = text
here = {k: v for k, v in self.fixed.get(self._candidate_segment_start, {}).items()
if not (k[0] < rng[1] and rng[0] < k[1])}
here[rng] = text
self.fixed[self._candidate_segment_start] = here
self.candidates = []
def _handle_candidate_key(self, key):
@@ -195,12 +304,12 @@ class CompositionEngine:
# --- commit ---
def _commit(self, decided=None):
m = decided or self.mode
text = "".join(self.keys) if m == LITERAL else self._zhuyin_commit_text()
def _commit(self, segs=None):
segs = segs if segs is not None else self.segments
text = "".join(self._render(s, True) for s in segs)
if self.forced_mode:
self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = m
self._previous_mode = segs[-1][2] if segs else None
self._previous_time = self.now()
self.reset()
return text
@@ -210,6 +319,7 @@ class CompositionEngine:
def reset(self):
self.keys = []
self.cursor = None
self.forced_mode = None
self.fixed = {}
self.candidates = []
+23
View File
@@ -56,6 +56,29 @@ class EnglishModel:
plaus = self.plausibility(w)
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
def word_features(self, word, complete):
"""Feature dict for the classifier (see features.py); score() is their
dot product with the default weights."""
w = word.lower()
n = float(len(w))
f = {}
if n <= 2:
f["lit.short"] = 1.0
if w in self.common:
f["lit.common"] = 1.0
f["lit.common_len"] = n
elif w in self.dictionary:
f["lit.dict"] = 1.0
f["lit.dict_len"] = n
elif not complete and _has_prefix(w, self.sorted_common):
f["lit.prefix_common_len"] = n
elif not complete and _has_prefix(w, self.sorted_words):
f["lit.prefix_dict_len"] = n
else:
plaus = self.plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
return f
def plausibility(self, w):
chars = ["^"] + list(w) + ["$"]
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
+138
View File
@@ -0,0 +1,138 @@
"""Segment features for the Zhuyin / literal classifier.
A segment's score in each mode is the dot product of its feature dict with a weight
dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights
(training/train.py → Resources/classifier-weights.json) replace them.
Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical
features; Tests/.../ParityTests checks this against training/parity-fixtures.json.
"""
from . import bopomofo
DEFAULT_WEIGHTS = {
# Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
# Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
# Segmentation
"seg.switch": -3.0,
# Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
}
def dot(weights, feats):
return sum(weights.get(k, 0.0) * v for k, v in feats.items())
def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None:
return None # a Zhuyin segment followed by another must end with a tone
# Pressing space finishes an untoned last syllable as first tone; after a toned
# syllable it just commits, so it must not count as an extra (stray) tone.
scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse
if scored.foreign_keys > 0 or not scored.syllables:
return None
f = {"zh.bias": 1.0}
if scored.stray_tones:
_add(f, "zh.stray", scored.stray_tones)
last = len(scored.syllables) - 1
toned_run = []
for i, ps in enumerate(scored.syllables):
if ps.reordered:
_add(f, "zh.reordered")
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
best = lexicon.best_score(ps.syllable)
if best is not None:
_add(f, "zh.toned_valid_lex")
_add(f, "zh.lexscore", best + 4.0)
else:
_add(f, "zh.toned_valid_nolex")
toned_run.append(ps.syllable)
else:
_add(f, "zh.toned_invalid")
elif i == last:
_add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid")
else:
_add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid")
# Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
covered = [False] * len(toned_run)
for i in range(len(toned_run)):
for n in (2, 3, 4):
if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]):
for k in range(i, i + n):
covered[k] = True
if any(covered):
_add(f, "zh.phrase", sum(covered))
if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables):
_add(f, "zh.unconfirmed")
if after_other:
_add(f, "seg.switch")
elif previous == "zhuyin":
_add(f, "ctx.zh")
return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys:
return None
complete = (not is_last) or finishing
f = {"lit.bias": 1.0}
if after_other:
_add(f, "seg.switch")
elif previous == "literal":
_add(f, "ctx.lit")
if any(c.isupper() for c in keys):
_add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
if digits:
_add(f, "lit.num_digits", digits)
if others:
_add(f, "lit.num_others", others)
return f
for k, v in english.word_features(letters, complete or digits > 0).items():
_add(f, k, v)
if digits:
_add(f, "lit.digits", digits)
non_apostrophe = others - keys.count("'")
if non_apostrophe:
_add(f, "lit.others", non_apostrophe)
return f
+18 -12
View File
@@ -16,7 +16,7 @@ class Lexicon:
def __init__(self):
self.exact = {}
self.toneless = {}
self._toneless = None # built on first use (only untoned syllables need it)
self.max_phrase_length = 1
@classmethod
@@ -42,9 +42,10 @@ class Lexicon:
except ValueError:
score = -6.0
self.add(reading, phrase, score)
for table in (self.exact, self.toneless):
for v in table.values():
for v in self.exact.values():
if len(v) > 1:
v.sort(key=lambda e: -e[1])
self._toneless = None
def add(self, reading, phrase, score):
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
@@ -53,15 +54,20 @@ class Lexicon:
entries = self.exact.setdefault(reading, [])
if not any(p == phrase for p, _ in entries):
entries.append((phrase, score))
key = _strip_tones(reading)
tl = self.toneless.setdefault(key, [])
for i, (p, s) in enumerate(tl):
if p == phrase:
if score > s:
tl[i] = (p, score)
break
else:
tl.append((phrase, score))
self._toneless = None
@property
def toneless(self):
"""Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
if self._toneless is None:
best = {}
for reading, entries in self.exact.items():
bucket = best.setdefault(_strip_tones(reading), {})
for phrase, score in entries:
if score > bucket.get(phrase, float("-inf")):
bucket[phrase] = score
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
return self._toneless
def entries(self, syllables):
if all(s.tone is not None for s in syllables):
+24
View File
@@ -0,0 +1,24 @@
"""Detects Shift pressed and released on its own. Port of ShiftTapDetector.swift."""
class ShiftTapDetector:
max_duration = 0.5
def __init__(self):
self._pressed_at = None
self._used = False
def shift_down(self, time):
if self._pressed_at is None: # ignore auto-repeat
self._pressed_at = time
self._used = False
def key_pressed(self):
"""Any other key while Shift is held: Shift was used as a modifier."""
if self._pressed_at is not None:
self._used = True
def shift_up(self, time):
"""Returns True when this release completes a Shift tap."""
start, self._pressed_at = self._pressed_at, None
return start is not None and not self._used and time - start <= self.max_duration
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "自動注音 AutoZhuyin",
"version": "0.1.0",
"version": "0.6.2",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW",
+6 -2
View File
@@ -26,9 +26,13 @@
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
Enter 送出目前組字
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
↓ 開啟候選字,按數字鍵 1–9 選字
← → 在組字區移動游標,選要改哪個字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字
Esc 取消組字
Shift+字母 強制英文
單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!:
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
+130 -5
View File
@@ -11,7 +11,8 @@ sys.path.insert(0, os.path.dirname(HERE))
from autozhuyin.core import bopomofo # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import DOWN, ENTER, TAB, CompositionEngine # noqa: E402
from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BACKSPACE, CompositionEngine # noqa: E402
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
@@ -21,6 +22,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json"))
LOAD_TIME = time.time() - _start
@@ -34,13 +36,13 @@ def convert(keys):
def make_engine():
return CompositionEngine(Classifier(LEX, ENG))
return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS))
def type_keys(s, e):
out = ""
for c in s:
key = ENTER if c == "\n" else TAB if c == "\t" else c
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN}.get(c, c)
handled, commit = e.handle(key)
if commit is not None:
out += commit
@@ -86,7 +88,7 @@ class ConverterTests(unittest.TestCase):
class ClassifierTests(unittest.TestCase):
c = Classifier(LEX, ENG)
c = Classifier(LEX, ENG, weights=WEIGHTS)
def test_zhuyin(self):
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
@@ -101,7 +103,7 @@ class ClassifierTests(unittest.TestCase):
def test_learning(self):
learned = LearnedPreferences()
c = Classifier(LEX, ENG, learned)
c = Classifier(LEX, ENG, learned, WEIGHTS)
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
learned.record("so", ZHUYIN)
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
@@ -150,9 +152,132 @@ class EngineTests(unittest.TestCase):
self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
def test_tone_then_space_stays_chinese(self):
self.assertEqual(len(type_keys("u4 ", make_engine())), 1)
def test_out_of_order_converts(self):
self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
class MixedSegmentTests(unittest.TestCase):
def run_keys(self, s):
e = make_engine()
return type_keys(s, e), e.marked_text
def test_chinese_then_english(self):
self.assertEqual(self.run_keys("ji3hello ")[0], "我hello ")
self.assertEqual(self.run_keys("su3cl3github\n")[0], "你好github")
self.assertEqual(self.run_keys("ji3hel")[1], "我hel")
def test_english_then_chinese(self):
self.assertEqual(self.run_keys("hellosu3cl3 ")[0], "hello你好")
self.assertEqual(self.run_keys("oksu3cl3\n")[0], "ok你好")
def test_three_segments(self):
self.assertEqual(self.run_keys("ji3emailsu3cl3\n")[0], "我email你好")
def test_single_language_unchanged(self):
self.assertEqual(self.run_keys("rup wu0 wu0 fu4cp3cl3\n")[0], "今天天氣很好")
self.assertEqual(self.run_keys("helloworld\n")[0], "helloworld")
self.assertEqual(self.run_keys("meeting ")[0], "meeting ")
self.assertEqual(len(self.run_keys("183\n")[0]), 1)
def test_speed(self):
e = make_engine()
start = time.time()
for c in "ji3emailsu3cl3rup wu0 wu0 fu4":
e.handle(c)
e.marked_text
per_key = (time.time() - start) / 30
print("\nper key: %.1f ms" % (per_key * 1000))
self.assertLess(per_key, 0.05)
class ShiftTapTests(unittest.TestCase):
def test_toggles_english_mode(self):
e = make_engine()
self.assertEqual(type_keys("⇧ru su ", e), "ru su ")
self.assertTrue(e.english_mode)
self.assertEqual(e.marked_text, "")
self.assertEqual(e.handle(ENTER), (False, None))
self.assertTrue(e.english_mode)
self.assertEqual(type_keys("⇧su3cl3\n", e), "你好")
self.assertFalse(e.english_mode)
def test_tap_twice_cancels(self):
self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好")
def test_mid_word_flips_and_learns(self):
e = make_engine()
type_keys("so", e)
e.handle(SHIFT_TAP)
self.assertEqual(e.mode, ZHUYIN)
self.assertFalse(e.english_mode)
type_keys("\n", e)
self.assertEqual(e.classifier.learned.weights["so"], 1)
def test_english_mode_not_learned(self):
e = make_engine()
type_keys("⇧hello ", e)
self.assertEqual(e.classifier.learned.weights, {})
def test_backspace_goes_to_app(self):
e = make_engine()
e.handle(SHIFT_TAP)
self.assertEqual(e.handle(BACKSPACE), (False, None))
self.assertTrue(e.english_mode)
def test_detector(self):
d = ShiftTapDetector()
d.shift_down(0)
self.assertTrue(d.shift_up(0.1))
d.shift_down(1)
d.key_pressed()
self.assertFalse(d.shift_up(1.1))
d.shift_down(2)
self.assertFalse(d.shift_up(3))
class CursorTests(unittest.TestCase):
def test_moves_and_clamps(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4", e)
self.assertEqual(e.cursor_offset, 4)
type_keys("←", e)
self.assertEqual(e.cursor_offset, 3)
type_keys("←←←←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("→→→→", e)
self.assertEqual(e.cursor_offset, 4)
def test_choose_candidate_in_the_middle(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4←←↓", e)
self.assertTrue(e.showing_candidates)
self.assertTrue(all(r[1] == 2 for r, _ in e.candidates))
e.select_candidate([t for _, t in e.candidates].index("添"))
self.assertEqual(type_keys("\n", e), "今添天氣")
def test_typing_returns_cursor_to_end(self):
e = make_engine()
type_keys("rup wu0 ←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("wu0 ", e)
self.assertEqual(e.cursor_offset, len(e.marked_text))
def test_candidates_for_chinese_before_english(self):
e = make_engine()
type_keys("su3cl3hello↓", e)
self.assertTrue(e.showing_candidates)
class FirstToneAfterEnglishTests(unittest.TestCase):
def test_first_tone_after_english(self):
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
e = make_engine()
self.assertEqual(type_keys("hello " + keys + " ", e), "hello ", keys)
self.assertEqual(len(e.marked_text), 1, keys)
if __name__ == "__main__":
unittest.main()
+34 -2
View File
@@ -36,6 +36,8 @@ class FakeTextService:
def setCandidateList(self, c): self.reply["candidateList"] = c
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
def showMessage(self, message, duration=3): self.reply["showMessage"] = message
def setShowCandidates(self, show):
self.showCandidates = show
self.reply["showCandidates"] = show
@@ -51,8 +53,8 @@ VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
class FakeKeyEvent:
def __init__(self, ch, shift=False, ctrl=False):
self.keyCode = VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA)
self.charCode = 0 if ch in ("\n", "\t", "\b", "↓") else ord(ch)
self.keyCode = 0x10 if ch == "⇧" else VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA)
self.charCode = 0 if ch in ("\n", "\t", "\b", "↓", "⇧") else ord(ch)
self.keyStates = [0] * 256
if shift:
self.keyStates[0x10] = 0x80
@@ -74,6 +76,17 @@ def press(svc, ch, **kw):
return True, svc.reply
def tap_shift(svc):
"""Shift down then up with nothing in between, as PIME delivers it."""
svc.reply = {}
down = FakeKeyEvent("⇧", shift=True)
assert not svc.filterKeyDown(down)
up = FakeKeyEvent("⇧")
if svc.filterKeyUp(up):
svc.onKeyUp(up)
return svc.reply
def type_text(svc, text):
out = ""
for ch in text:
@@ -123,5 +136,24 @@ class WrapperTests(unittest.TestCase):
self.assertEqual(out + reply.get("commitString", ""), "Hi ")
class ShiftTapWrapperTests(unittest.TestCase):
def setUp(self):
self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate()
def test_tap_forces_english_lowercase(self):
reply = tap_shift(self.svc)
self.assertEqual(reply.get("showMessage"), "英文")
self.assertEqual(type_text(self.svc, "ru su "), "ru su ")
self.assertEqual(tap_shift(self.svc).get("showMessage"), "中英自動")
def test_shift_with_letter_is_not_a_tap(self):
self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True))
press(self.svc, "H", shift=True)
up = FakeKeyEvent("⇧")
self.assertFalse(self.svc.filterKeyUp(up))
self.assertFalse(self.svc.engine.english_mode)
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Fills windows/autozhuyin/data/ from the shared Mac resources.
- lexicon.txt, english-common.txt: copied from Resources/
- lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
Webster's 2nd edition, public domain), since Windows has no such file
- english-bigram.json: precomputed letter-bigram table
@@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402
data = WIN / "autozhuyin" / "data"
data.mkdir(parents=True, exist_ok=True)
for name in ("lexicon.txt", "english-common.txt"):
for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"):
shutil.copy(ROOT / "Resources" / name, data / name)
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")