Compare commits

...
24 Commits
Author SHA1 Message Date
ays953 38e0758a13 Release v0.11.0 2026-11-13 16:34:15 +08:00
ays953andClaude Opus 5.5 fd87da8061 AI candidate selection: rerank with a Traditional Chinese language model (macOS)
The converter's K best paths are rescored with CKIP gpt2-tiny-chinese (GPL-3.0,
character-level, ~12M parameters, 22 MB as float16):
score = lexicon log-prob + 0.6 x LM log-prob, with the recently committed text
as context. Inference is plain Accelerate (Apple Silicon and Intel).

Modes off / saver (K=3, 15 ms) / standard (K=10, 40 ms) / quality (K=30, 120 ms)
in the input menu. A rerank over its time budget falls back to the lexicon; three
in a row pause AI for a minute. With auto-pause on, AI pauses when the Mac is hot
or the CPU is busy and drops to saver in Low Power Mode. The model loads in the
background and is not loaded at all when off.

Evaluation (training/lm/eval_rerank.py): chat 96.6% -> 97.5% chars,
Wikipedia 92.9% -> 94.7% chars (about 25-30% fewer wrong characters).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:33:58 +08:00
ays953 acafd8f96e Release v0.10.1 2026-11-13 15:30:18 +08:00
ays953andClaude Opus 5.5 fa5b266ae4 Stop the cursor after the first Chinese character
Moving left no longer reaches the start of the buffer (or any position before the
first Zhuyin syllable), where there is nothing to pick candidates for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:30:03 +08:00
ays953 6448bf4ca2 Release v0.10.0 2026-11-13 14:31:10 +08:00
ays953andClaude Opus 5.5 2cebf0e700 Remember picked candidates (user history)
Candidates chosen with the arrow-down list are recorded on commit as
toneless reading -> phrase with a count. The converter lifts a picked phrase just
above the best lexicon entry for that reading (more for more picks), so it
comes first next time without overriding longer phrases. Up to 2000 pairs,
least recently used evicted. Stored as user-history.json on every platform; the
Mac menu's clear item clears it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:30:55 +08:00
ays953 815dcf979d Release v0.9.1 2026-11-13 12:06:24 +08:00
ays953andClaude Opus 5.5 8e82105af9 Hold the first word on space when it also reads as Zhuyin
At the start of a composition (not right after English), a word that reads as
valid Zhuyin is no longer committed as English on space. The next word decides,
by comparing "Chinese all the way" with "English word + Chinese" using the
classifier's features: el ji3 = 高我, no su3cl3 = no 你好, up the = up the.
After English text only common one-syllable readings (up, url) are held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:10 +08:00
ays953 9cac3467a0 Release v0.9.0 2026-11-13 11:59:53 +08:00
ays953andClaude Opus 5.5 49e0b52fd5 Add a Linux version (IBus) for Bazzite / SteamOS desktop mode
linux/autozhuyin_ibus.py is an IBus engine on top of the shared Python core:
preedit with the cursor and blue English segments, lookup table, Shift tap for
English mode, candidate clicks. install.sh puts everything in the home folder
(works on read-only systems) and an autostart entry registers the engine with
ibus-daemon and selects it. Tested with a stand-in IBus module.

Also fixes Windows: wants() did not claim the backtick, so backtick + punctuation
never reached the engine under PIME.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:59:40 +08:00
ays953 fcc0f9d2db Release v0.8.0 2026-11-13 09:56:18 +08:00
ays953andClaude Opus 5.5 31d9991e16 Insert and delete syllables at the composition cursor
The ← → cursor is now an editing position (key index), not just a candidate
stop. It can move to the start; typing Zhuyin there composes a syllable at the
cursor and splices it in on its tone key; Backspace removes the character before
the cursor. Picked phrases before the edit point are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:56:06 +08:00
ays953 6b4ec39dc1 Release v0.7.1 2026-11-11 17:01:37 +08:00
ays953andClaude Opus 5.5 955b0ca5c3 Ship third-party notices with both installers
THIRD_PARTY_LICENSES.txt carries McBopomofo's full MIT notice (required when the
converted lexicon is distributed) plus notes on the public-domain word list and
PIME. It goes into AutoZhuyin.app/Contents/Resources and the Windows zip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:01:26 +08:00
ays953 251a16fa08 Release v0.7.0 2026-11-11 16:29:06 +08:00
ays953andClaude Opus 5.5 7cb1f62dfa Backtick + punctuation key forces a full-width symbol
` then , . / ; : ' [ ] ( ) ! ~ - = and their shifted keys always give the
full-width symbol (,。?;:、「」()!~——=), whatever the context. Backtick
followed by any other key is a plain backtick, so ``` still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:28:54 +08:00
ays953 3670305f0f Release v0.6.3 2026-11-11 16:10:59 +08:00
ays953andClaude Opus 5.5 ef9f91a6b7 Hold ambiguous English words (up = 因) until the next word decides
"up " also reads as ㄧㄣ (因). Instead of committing English on space, keep the
word composing: "up jo4" becomes 因為, "up the" stays English. Only applies when
the reading is a common single syllable (lexicon score >= -3.5): up, url.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:10:47 +08:00
ays953 1dacb8f164 Release v0.6.2 2026-11-11 12:13:15 +08:00
ays953andClaude Opus 5.5 bfeed22916 Cut input method startup from ~0.95 s to ~0.15 s
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
  parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
  Windows build) instead of retraining the bigram model on every launch, and
  uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:13:03 +08:00
ays953 bb9cb0c4e0 Release v0.6.1 2026-11-11 12:00:10 +08:00
ays953andClaude Opus 5.5 2c5c70c066 Learn the previous-word context; fix first-tone characters after English
After an English word, 23 of 50 common first-tone characters (機, 吃, 書…)
were read as English when space was pressed: the fixed +1.5 context bonus
was far too strong for the trained weight scale, and the training data never
contained the "space just pressed on a first-tone syllable" decision.

Context is now the ctx.zh / ctx.lit features, trained with the rest; the
synthetic data includes the previous mode and the first-tone space decision.
Test accuracy: commit 98.22% -> 99.52%, typing 88.73% -> 91.49%, and when the
language differs from the previous word 92.0% -> 99.3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:59:57 +08:00
ays953 17f7d6f50d Release v0.6.0 2026-11-11 11:44:38 +08:00
ays953andClaude Opus 5.5 7e173a1da8 Make the Shift tap a persistent English-mode toggle
A Shift tap with nothing composed now switches English mode on until the
next tap, instead of forcing only the next word: space, Enter and other keys
no longer drop back to automatic detection. In English mode every key goes
straight to the app, so there is no composition or underline. A Shift tap
while typing still flips the buffer like Tab. Windows shows "英文" /
"中英自動" when toggling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:44:25 +08:00
46 changed files with 9176 additions and 3793 deletions
+108 -21
View File
@@ -1,6 +1,6 @@
# 自動注音 AutoZhuyin # 自動注音 AutoZhuyin
給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。 給繁體中文使用者的輸入法(macOS、Windows、Linux):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
``` ```
按鍵: ji3 hello world su3cl3␣ 按鍵: ji3 hello world su3cl3␣
@@ -12,11 +12,14 @@
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示) - **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分 - **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對 - **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **單按 Shift 強制英文**:要打英文時先單按一下 Shift,接下來這個字照打、保持小寫,而且直接送出、不顯示底線 - **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小) - **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」 - **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來 - **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,` - **記住選字**:從候選字選過的字詞會被記住,下次打同樣的讀音會優先出現(選越多次越優先)
- **AI 選字(macOS)**:用中研院 CKIP 的繁中小型語言模型看前後文挑同音字(他跑「得」很快、請你「再」說一次),完全在本機運算;可調整或關閉,系統忙碌時自動暫停
- **中間補字、刪字**:游標移到前面後直接打注音,會插在游標處;Backspace 刪掉游標前的那個字
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權) - **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝 ## 下載與安裝
@@ -44,6 +47,16 @@ PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。 SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
### Linux(Bazzite、SteamOS 桌面模式、Fedora…)
需求:系統內建 IBus 輸入法框架(Bazzite、Fedora 預設就有)。只在桌面模式可用,Steam 遊戲模式不支援系統輸入法。
1. 解壓縮 `AutoZhuyin-Linux-<版本>.tar.gz`,在終端機進到資料夾執行 `./install.sh`。不需要 sudo,只會裝在家目錄(`~/.local/share/autozhuyin`),所以唯讀系統也能用。
2. KDE(Bazzite / SteamOS 預設):系統設定 → 鍵盤 → 虛擬鍵盤,選 **IBus Wayland** → 套用,登出再登入。GNOME 不用設定。
3. 之後每次登入會自動切換到自動注音(`~/.config/autostart/autozhuyin.desktop`)。
暫時改回一般鍵盤:`ibus engine xkb:us::eng`;切回:`ibus engine autozhuyin`;移除:`./uninstall.sh`。
## 按鍵 ## 按鍵
| 按鍵 | 作用 | | 按鍵 | 作用 |
@@ -51,13 +64,14 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 | | 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 | | Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 | | **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ← → | 在組字區移動游標,選要改哪個字 | | ← → | 在組字區移動游標(最左停在第一個中文字後面)。游標在中間時:直接打注音會**插入**新字,Backspace 刪掉游標前的字,↓ 選游標前的字 |
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 | | ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 | | Esc | 取消組字 |
| **單按 Shift** | 沒在打字時:下一個字強制英文(保持小寫),字母直接送出、沒有底線;按空白、Enter 或其他符號後恢復自動判斷,再按一次 Shift 取消。打到一半時:同 Tab,整段翻轉 | | **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
| Shift + 字母 | 打大寫字母(也會強制英文) | | Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` | | `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) | | `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
| `` ` `` + 標點鍵 | **強制全形標點**,不管前後是中文還是英文:`` `, `` `` `< `` = `,`、`` `. `` `` `> `` = `。`、`` `/ `` = `?`、`` `; `` = `;`、`` `: `` = `:`、`` `' `` `` `\ `` = `、`、`` `[ `` `` `] `` = `「」`、`` `{ `` `` `} `` = `『』`、`` `( `` `` `) `` = `()`、`` `! `` = `!`、`` `~ `` = `~`、`` `- `` = `——`、`` `= `` = `=`。`` ` `` 後面接其他鍵就是普通的反引號(` ``` ` 照常可用) |
## 運作方式 ## 運作方式
@@ -83,15 +97,48 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。 轉換用 Viterbi 演算法,在詞庫中找出整串音節機率最高的詞組組合。
### AI 選字(macOS)
詞庫只知道每個詞本身有多常見,不看前後文。AI 選字讓語言模型當裁判:
1. 詞庫先找出前 K 名的整句組合(例如「請你在說一次」「請你再說一次」)。
2. 模型 [CKIP GPT2-tiny](https://huggingface.co/ckiplab/gpt2-tiny-chinese)(中研院詞庫小組,繁體中文,約 1200 萬參數,22 MB)計算每個組合出現的機率,並參考剛送出的前文。
3. 分數 = 詞庫分數 + 0.6 × 模型分數,取最高的。選過的候選字仍然固定不變。
輸入法選單可以調整:
| 模式 | 比較幾個組合 | 每次運算上限 |
|---|---|---|
| 關閉 | — | 不載入模型、不佔記憶體 |
| 省電 | 3 | 15 毫秒 |
| 標準(預設) | 10 | 40 毫秒 |
| 高品質 | 30 | 120 毫秒 |
不會拖慢打字的機制:
- 每次運算超過上限就直接放棄,改用詞庫結果;連續 3 次超時就暫停一分鐘。
- 「系統忙碌時自動暫停 AI」(預設開啟):Mac 過熱,或 CPU 負載(每核心 1 分鐘平均)超過 0.9 時暫停;低耗電模式下降到省電模式。選單最下面會顯示目前狀態。
- 模型在背景載入,載入完成前照常用詞庫。
實測(`training/lm/eval_rerank.py`,標準模式):
| 測試資料 | 只用詞庫 | AI 選字 |
|---|---|---|
| 日常對話 60 句 | 字正確率 96.6%,整句全對 73% | 97.5%,80% |
| 維基百科 714 句 | 92.9%,31% | 94.7%,44% |
錯字大約少了 25–30%。Windows、Linux 版目前還沒有 AI 選字。
## 個人資料 ## 個人資料
| | macOS | Windows | | | macOS | Windows | Linux |
|---|---|---| |---|---|---|---|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | | 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | `~/.config/autozhuyin/learned.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | | 選字紀錄 | `~/Library/Application Support/AutoZhuyin/user-history.json` | `%APPDATA%\PIME\autozhuyin\user-history.json` | `~/.config/autozhuyin/user-history.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | `~/.config/autozhuyin/user-phrases.txt` |
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。 自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
Mac 版可以在輸入法選單選「清除學習紀錄」。 選字紀錄最多保留 2000 筆,太久沒用的會先淘汰。被選過的字只會排到同讀音第一順位,不會蓋過整個詞組(例如選過「添」之後,ㄊㄧㄢ 單打會出「添」,但「今天」還是「今天」)。
Mac 版可以在輸入法選單選「清除學習紀錄」(中英判斷和選字紀錄一起清除);Windows、Linux 刪除上面的檔案即可。
## 開發 ## 開發
@@ -124,6 +171,18 @@ scripts/package-windows.sh # 產生 build/AutoZhuyin-Wi
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。 - `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
- 版本號在 `windows/autozhuyin/ime.json`。 - 版本號在 `windows/autozhuyin/ime.json`。
### Linux
Linux 版是 IBus 輸入法(`linux/autozhuyin_ibus.py`),直接使用 Windows 版的 Python 核心(`windows/autozhuyin/core/`)和資料。
```bash
python3 -m unittest discover -s linux/tests # 用假的 IBus 模組測試外殼
scripts/package-linux.sh # 產生 build/AutoZhuyin-Linux-<版本>.tar.gz(會先跑測試)
```
- 安裝在使用者目錄,不寫入 `/usr`:開機時由 autostart 啟動 `autozhuyin_ibus.py --activate`,它會把引擎動態註冊給 ibus-daemon 並切換過去。
- 也支援由 ibus-daemon 啟動(`--ibus`,配合 `/usr/share/ibus/component` 的 XML),之後要做 RPM 時可以用。
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。 > 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
### 訓練判斷權重 ### 訓練判斷權重
@@ -144,12 +203,12 @@ python3 training/make_parity.py # 權重或特徵改了之後都要重
目前的結果(10,000 筆測試資料): 目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 | | | 送出時整段判斷正確 | 打字過程中顯示正確 | 跟前一個字語言不同時 |
|---|---|---| |---|---|---|---|
| 手調權重 | 97.58% | 86.15% | | 手調權重 | 97.84% | 88.19% | 96.2% |
| 訓練後 | **99.62%** | 87.81% | | 訓練後 | **99.52%** | **91.49%** | **99.3%** |
送出時的錯誤率從 2.4% 降到 0.4%。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。 訓練資料也模擬了「前一個字是中文或英文」和「按下空白(一聲)的那一刻」,所以「前一個字」的影響(`ctx.zh` / `ctx.lit`)也是訓練出來的,不是手調的。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。 新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
@@ -163,6 +222,22 @@ python3 training/make_parity.py # 權重或特徵改了之後都要重
scripts/import-mcbopomofo.sh scripts/import-mcbopomofo.sh
``` ```
### AI 選字模型
```bash
scripts/fetch-lm.sh # 下載 CKIP gpt2-tiny-chinese,轉成 build/lm/lm.bin(float16)+ lm-vocab.txt
```
`build.sh` 會自動執行並把模型打包進 app。模型推論寫在 `Sources/AutoZhuyinCore/LanguageModel.swift`(Accelerate,Apple Silicon 和 Intel 都可用),重排在 `Reranker.swift`。
評估效果(需要 numpy;`build/lm-eval/` 放測試句子,一行一句):
```bash
python training/lm/eval_rerank.py build/lm-eval/chat.txt build/lm-eval/wiki.txt
```
`training/lm/gpt2.py` 是 numpy 版的參考實作,Swift 版的測試會比對它算出的分數。
### 發布新版本 ### 發布新版本
先 commit 所有修改,再執行: 先 commit 所有修改,再執行:
@@ -177,12 +252,12 @@ scripts/release.sh minor --push # 同時 push commit 和 tag
`release.sh` 會自動: `release.sh` 會自動:
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`) 1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
2. 跑 Swift 和 Python 全部測試 2. 跑 Swift 和 Python 全部測試
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip` 3. 產生 `build/AutoZhuyin-<版本>.pkg`、`build/AutoZhuyin-Windows-<版本>.zip`、`build/AutoZhuyin-Linux-<版本>.tar.gz`
4. commit「Release v<版本>」並打上 `v<版本>` tag 4. commit「Release v<版本>」並打上 `v<版本>` tag
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。 任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。 最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg`、`.zip` 和 `.tar.gz`。
## 專案結構 ## 專案結構
@@ -210,14 +285,20 @@ windows/
installer/ install.bat、uninstall.bat、給使用者的 README.txt installer/ install.bat、uninstall.bat、給使用者的 README.txt
tests/ Python 測試 tests/ Python 測試
tools/ 產生資料檔、圖示 tools/ 產生資料檔、圖示
linux/
autozhuyin_ibus.py IBus 外殼(共用 windows/autozhuyin/core)
installer/ install.sh、uninstall.sh、給使用者的 README.txt
tests/ 用假 IBus 模組的外殼測試
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料 training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝檔)
``` ```
## 已知限制 ## 已知限制
- Windows 版還沒在實機上測試過。 - Windows 版、Linux 版還沒在實機上測試過。
- Linux 版只支援 IBus。KDE 上有些 Flatpak 程式(例如 Flatpak 版 Chrome)在 IBus Wayland 下可能打不出字。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。 - 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。 - 很短的字串本質上就有歧義。開頭第一個詞如果也能讀成注音,按空白後會先保留(顯示成英文),看下一個詞再決定:`up jo4` = 因為、`el ji3` = 高我、`no su3cl3` = no 你好、`up the` = up the。接在英文後面時只保留 `up`(因)這種很常見的讀音,不會拖慢打英文。還是判斷錯的話按 Tab 修正,修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。 - 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap) ## 下一步(Roadmap)
@@ -225,11 +306,14 @@ training/ 合成資料、訓練權重、Swift/Python 一致
- [x] 匯入完整詞庫 - [x] 匯入完整詞庫
- [x] 注音順序錯誤修正 - [x] 注音順序錯誤修正
- [x] Windows 版 - [x] Windows 版
- [x] Linux 版(IBus)
- [ ] Linux Fcitx5 版(KDE 上相容性較好)
- [x] 用資料訓練判斷權重(合成資料) - [x] 用資料訓練判斷權重(合成資料)
- [ ] 加入真實打字資料一起訓練 - [ ] 加入真實打字資料一起訓練
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」 - [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境 - [x] AI 選字:語言模型參考前後文挑同音字(macOS)
- [ ] 使用者詞頻學習(選過的候選字優先) - [ ] AI 選字移植到 Windows、Linux
- [x] 使用者詞頻學習(選過的候選字優先)
- [ ] 偏好設定視窗 - [ ] 偏好設定視窗
## 致謝與授權 ## 致謝與授權
@@ -237,3 +321,6 @@ training/ 合成資料、訓練權重、Swift/Python 一致
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License) - 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼) - Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`) - 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
- AI 選字語言模型:[CKIP GPT2 Tiny Chinese](https://github.com/ckiplab/ckip-transformers)(中研院詞庫小組,GPL-3.0,全文在 `licenses/GPL-3.0.txt`,Mac 版一併放在 app 的 Resources)
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
+2 -2
View File
@@ -15,9 +15,9 @@
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.5.1</string> <string>0.11.0</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>7</string> <string>19</string>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
+28 -26
View File
@@ -1,39 +1,41 @@
{ {
"about": "AutoZhuyin classifier weights, learned by training/train.py", "about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11", "trained": "2026-11-11",
"examples": 40001, "examples": 40000,
"test_accuracy": { "test_accuracy": {
"commit": 0.9962, "commit": 0.9952,
"typing": 0.8781 "typing": 0.9149
}, },
"weights": { "weights": {
"lit.bias": 0.4276, "ctx.lit": 0.0916,
"lit.common": -0.3465, "ctx.zh": 0.1349,
"lit.common_len": 0.81, "lit.bias": 0.3982,
"lit.dict": -1.7902, "lit.common": -0.2906,
"lit.dict_len": 0.6765, "lit.common_len": 0.8046,
"lit.digits": -2.7403, "lit.dict": -1.6662,
"lit.num_digits": -0.0751, "lit.dict_len": 0.6441,
"lit.num_others": -0.7433, "lit.digits": -2.9689,
"lit.others": -1.5685, "lit.num_digits": -0.018,
"lit.prefix_common_len": 0.2082, "lit.num_others": -0.5554,
"lit.prefix_dict_len": 0.0003, "lit.others": -2.0002,
"lit.short": -0.2397, "lit.prefix_common_len": 0.2831,
"lit.unknown_score": 0.213, "lit.prefix_dict_len": 0.0299,
"lit.short": -0.2482,
"lit.unknown_score": 0.2479,
"lit.upper": 20.0, "lit.upper": 20.0,
"seg.switch": -1.7203, "seg.switch": -1.7596,
"zh.bias": 1.3522, "zh.bias": 1.3423,
"zh.lexscore": 0.26, "zh.lexscore": 0.2917,
"zh.pending_invalid": -5.0, "zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9295, "zh.pending_prefix": -0.9162,
"zh.phrase": 0.4595, "zh.phrase": 0.4499,
"zh.reordered": -1.4348, "zh.reordered": -1.3079,
"zh.stray": -5.0, "zh.stray": -5.0,
"zh.toned_invalid": -5.0, "zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9823, "zh.toned_valid_lex": -0.9018,
"zh.toned_valid_nolex": -1.8742, "zh.toned_valid_nolex": -1.6341,
"zh.unconfirmed": -0.7891, "zh.unconfirmed": -0.7952,
"zh.untoned_mid_invalid": -5.0, "zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.8009 "zh.untoned_mid_valid": -2.7479
} }
} }
+97
View File
@@ -0,0 +1,97 @@
import Foundation
import AutoZhuyinCore
/// "AI 選字" settings, model loading and system-load detection for the Mac app.
enum AI {
static let modeKey = "AIMode"
static let autoPauseKey = "AIAutoPause"
static let reranker: Reranker = {
let saved = UserDefaults.standard.string(forKey: modeKey).flatMap(Reranker.Mode.init(rawValue:))
let r = Reranker(mode: saved ?? .standard)
r.systemState = { SystemMonitor.shared.state() }
return r
}()
/// The model file is missing from the app bundle (built without scripts/fetch-lm.sh).
private(set) static var modelMissing = false
private static var loading = false
static var autoPause: Bool {
get { UserDefaults.standard.object(forKey: autoPauseKey) as? Bool ?? true }
set { UserDefaults.standard.set(newValue, forKey: autoPauseKey); SystemMonitor.shared.invalidate() }
}
static func setMode(_ mode: Reranker.Mode) {
reranker.mode = mode
UserDefaults.standard.set(mode.rawValue, forKey: modeKey)
if mode == .off {
reranker.model = nil // free the memory
} else {
loadModelIfNeeded()
}
}
/// Loads the model in the background (only when AI is on).
static func loadModelIfNeeded() {
guard reranker.mode != .off, reranker.model == nil, !loading, !modelMissing else { return }
guard let bin = Bundle.main.url(forResource: "lm", withExtension: "bin"),
let vocab = Bundle.main.url(forResource: "lm-vocab", withExtension: "txt") else {
modelMissing = true
return
}
loading = true
DispatchQueue.global(qos: .utility).async {
let start = Date()
let model = try? GPT2LanguageModel(modelURL: bin, vocabURL: vocab)
NSLog("AutoZhuyin: language model %@ in %.0f ms", model == nil ? "failed" : "loaded",
Date().timeIntervalSince(start) * 1000)
DispatchQueue.main.async {
loading = false
if model == nil { modelMissing = true }
if reranker.mode != .off { reranker.model = model }
}
}
}
static var statusText: String {
if modelMissing { return "AI 模型未安裝" }
switch reranker.status {
case .off: return "AI 選字已關閉"
case .loading: return "AI 模型載入中…"
case .active:
return SystemMonitor.shared.state() == .reduced ? "AI 使用中(省電:低耗電模式)" : "AI 使用中"
case .pausedBusy: return "AI 已暫停:系統忙碌或過熱"
case .pausedSlow: return "AI 已暫停:運算太慢,一分鐘後再試"
}
}
}
/// Decides when AI should step back: paused when the Mac is hot or the CPU is busy,
/// reduced (fewest candidates) in Low Power Mode. Checked at most every 2 seconds.
final class SystemMonitor {
static let shared = SystemMonitor()
private var cached: (time: Date, state: Reranker.SystemState) = (.distantPast, .normal)
/// CPU load (1-minute average per core) above which AI pauses.
var busyLoad = 0.9
func invalidate() { cached.time = .distantPast }
func state() -> Reranker.SystemState {
if Date().timeIntervalSince(cached.time) < 2 { return cached.state }
cached = (Date(), compute())
return cached.state
}
private func compute() -> Reranker.SystemState {
guard AI.autoPause else { return .normal }
let info = ProcessInfo.processInfo
if info.thermalState == .serious || info.thermalState == .critical { return .paused }
var load = [Double](repeating: 0, count: 3)
if getloadavg(&load, 3) > 0, load[0] / Double(max(1, info.activeProcessorCount)) > busyLoad {
return .paused
}
if info.isLowPowerModeEnabled { return .reduced }
return .normal
}
}
+51 -8
View File
@@ -9,31 +9,52 @@ enum Shared {
static let learnedKey = "LearnedPreferences" static let learnedKey = "LearnedPreferences"
/// ~/Library/Application Support/AutoZhuyin (user phrases, picked-candidate history).
static let supportDir: URL? = FileManager.default
.urls(for: .applicationSupportDirectory, in: .userDomainMask).first?
.appendingPathComponent("AutoZhuyin")
static let classifier: Classifier = { static let classifier: Classifier = {
let bundle = Bundle.main let bundle = Bundle.main
let start = Date()
let lexicon = Lexicon() let lexicon = Lexicon()
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"), if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
let text = try? String(contentsOf: url, encoding: .utf8) { let data = try? Data(contentsOf: url) {
lexicon.load(text) lexicon.load(data: data)
} }
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt // Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first { if let userFile = supportDir?.appendingPathComponent("user-phrases.txt"),
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt") let text = try? String(contentsOf: userFile, encoding: .utf8) {
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) } lexicon.load(text)
} }
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt")) let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:] let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
let historyFile = supportDir?.appendingPathComponent("user-history.json")
let history = UserHistory(data: historyFile.map(UserHistory.load(from:)) ?? [:]) { data in
guard let url = historyFile, let json = try? JSONEncoder().encode(data) else { return }
try? FileManager.default.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
try? json.write(to: url, options: .atomic)
}
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json")) let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained) NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained, history: history)
}() }()
} }
@objc(AutoZhuyinInputController) @objc(AutoZhuyinInputController)
final class InputController: IMKInputController { final class InputController: IMKInputController {
private let engine = CompositionEngine(classifier: Shared.classifier) private lazy var engine: CompositionEngine = {
let e = CompositionEngine(classifier: Shared.classifier)
e.reranker = AI.reranker
return e
}()
private var shiftTap = ShiftTapDetector() private var shiftTap = ShiftTapDetector()
// MARK: - Key events // MARK: - Key events
@@ -165,11 +186,33 @@ final class InputController: IMKInputController {
override func menu() -> NSMenu! { override func menu() -> NSMenu! {
let menu = NSMenu() let menu = NSMenu()
let modes: [(Reranker.Mode, String, Selector)] = [
(.off, "AI 選字:關閉", #selector(aiOff(_:))),
(.saver, "AI 選字:省電", #selector(aiSaver(_:))),
(.standard, "AI 選字:標準", #selector(aiStandard(_:))),
(.quality, "AI 選字:高品質", #selector(aiQuality(_:))),
]
for (mode, title, action) in modes {
let item = menu.addItem(withTitle: title, action: action, keyEquivalent: "")
item.state = AI.reranker.mode == mode ? .on : .off
}
let pause = menu.addItem(withTitle: "系統忙碌時自動暫停 AI", action: #selector(toggleAutoPause(_:)), keyEquivalent: "")
pause.state = AI.autoPause ? .on : .off
let status = menu.addItem(withTitle: AI.statusText, action: nil, keyEquivalent: "")
status.isEnabled = false
menu.addItem(.separator())
menu.addItem(withTitle: "清除學習紀錄", action: #selector(clearLearned(_:)), keyEquivalent: "") menu.addItem(withTitle: "清除學習紀錄", action: #selector(clearLearned(_:)), keyEquivalent: "")
return menu return menu
} }
@objc private func aiOff(_ sender: Any?) { AI.setMode(.off) }
@objc private func aiSaver(_ sender: Any?) { AI.setMode(.saver) }
@objc private func aiStandard(_ sender: Any?) { AI.setMode(.standard) }
@objc private func aiQuality(_ sender: Any?) { AI.setMode(.quality) }
@objc private func toggleAutoPause(_ sender: Any?) { AI.autoPause.toggle() }
@objc private func clearLearned(_ sender: Any?) { @objc private func clearLearned(_ sender: Any?) {
Shared.classifier.learned.clear() Shared.classifier.learned.clear()
Shared.classifier.history.clear()
} }
} }
+1
View File
@@ -22,5 +22,6 @@ else {
Shared.server = server Shared.server = server
Shared.candidates = IMKCandidates(server: server, panelType: kIMKSingleColumnScrollingCandidatePanel) Shared.candidates = IMKCandidates(server: server, panelType: kIMKSingleColumnScrollingCandidatePanel)
_ = Shared.classifier // load the lexicon now rather than on the first keystroke _ = Shared.classifier // load the lexicon now rather than on the first keystroke
AI.loadModelIfNeeded() // "AI 選字" language model, in the background
NSApplication.shared.run() NSApplication.shared.run()
+14 -15
View File
@@ -55,37 +55,36 @@ public final class Classifier {
public let lexicon: Lexicon public let lexicon: Lexicon
public let english: EnglishModel public let english: EnglishModel
public let learned: LearnedPreferences public let learned: LearnedPreferences
/// Picked candidates (shared by every engine using this classifier).
public let history: UserHistory
/// Feature weights: the hand-tuned defaults overlaid with trained ones. /// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector public private(set) var weights: FeatureVector
/// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(), public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
weights: FeatureVector = [:]) { weights: FeatureVector = [:], history: UserHistory = UserHistory()) {
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned self.learned = learned
self.history = history
self.weights = Features.defaultWeights.merging(weights) { _, new in new } self.weights = Features.defaultWeights.merging(weights) { _, new in new }
} }
/// (Zhuyin features or nil, literal features or nil) for one segment. /// (Zhuyin features or nil, literal features or nil) for one segment. `previous` (mode of
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, /// the text committed just before) only affects the first segment.
finishing: Bool) -> (FeatureVector?, FeatureVector?) { public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther, finishing: finishing), previous: InputMode? = nil) -> (FeatureVector?, FeatureVector?) {
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther, finishing: finishing)) (Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous),
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous))
} }
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool, private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] { previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther, finishing: finishing) let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
if whole { if whole {
let b = learned.bias(for: String(seg)) let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b } if b > 0 { zh += b } else { lit -= b }
+310 -80
View File
@@ -35,22 +35,39 @@ public final class CompositionEngine {
public let converter: Converter public let converter: Converter
public private(set) var keys: [Character] = [] public private(set) var keys: [Character] = []
/// Set by Tab / Shift tap: overrides the classifier for the whole buffer. /// Set by Tab / a Shift tap while typing: overrides the classifier for the whole
/// buffer. It is a correction, learned on commit.
public private(set) var forcedMode: InputMode? public private(set) var forcedMode: InputMode?
/// True when forcedMode corrects the classifier (learned on commit); false when the /// English mode, toggled by a Shift tap when nothing is being typed. Keys go straight
/// user forced English before typing (Shift tap on an empty buffer), which is not a /// to the app (no composition, no underline) until Shift is tapped again.
/// correction and is not learned. public private(set) var isEnglishMode = false
private var forcedIsCorrection = false
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text. /// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
private var fixed: [Int: [Range<Int>: String]] = [:] private var fixed: [Int: [Range<Int>: String]] = [:]
public private(set) var candidates: [Candidate] = [] public private(set) var candidates: [Candidate] = []
private var candidateSegmentStart = 0 private var candidateSegmentStart = 0
/// Composition cursor for choosing candidates: index into `cursorStops`, nil = at the end. /// Composition cursor: a key index into `keys` (nil = at the end). ← → move it between
/// ← → move it; ↓ shows candidates for the phrase ending at the cursor. /// syllables; ↓ shows candidates for the phrase before it; typing there inserts a
/// syllable (kept in `inserting` until its tone key, then spliced into `keys`).
private var cursor: Int? private var cursor: Int?
private var inserting: [Character] = []
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
/// Index of a held space: the first word of a composition that also reads as Zhuyin is
/// not committed as English on space; the next word decides ("up jo4" = 因為,
/// "up the" = up the, "no su3" = no 你). Right after English text only common
/// one-syllable readings are held (up = 因, url = 教), so English typing isn't delayed.
private var held: Int?
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
public private(set) var pendingBacktick = false
static let holdMinScore = -3.5
/// "AI 選字" (language-model reranking); nil = lexicon only.
public var reranker: Reranker?
/// The last characters committed, used as context for the reranker.
private var recentText = ""
private var previousMode: InputMode? private var previousMode: InputMode?
private var previousCommitTime = Date.distantPast private var previousCommitTime = Date.distantPast
/// How long the previous segment's mode keeps influencing the next one. /// How long the previous segment's mode keeps influencing the next one.
@@ -59,7 +76,7 @@ public final class CompositionEngine {
public init(classifier: Classifier) { public init(classifier: Classifier) {
self.classifier = classifier self.classifier = classifier
self.converter = Converter(lexicon: classifier.lexicon) self.converter = Converter(lexicon: classifier.lexicon, history: classifier.history)
} }
// MARK: - Derived state // MARK: - Derived state
@@ -71,9 +88,52 @@ public final class CompositionEngine {
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] { private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
guard !keys.isEmpty else { return [] } guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] } if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
if let h = held, h < keys.count, keys[h] == " " {
return heldSegments(keys, held: h, finishing: finishing)
}
return classifier.segment(keys, finishing: finishing, previous: context) return classifier.segment(keys, finishing: finishing, previous: context)
} }
/// The held word is Chinese if the words typed after it start with Chinese.
private func heldSegments(_ keys: [Character], held h: Int, finishing: Bool) -> [Segment] {
let base = h + 1
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
}
if let first = rest.first, first.mode == .zhuyin,
heldReadsAsChinese(keys, held: h, end: first.range.upperBound, finishing: finishing) {
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
}
return [Segment(range: 0..<base, mode: .literal)] + rest
}
/// Chinese all the way to `end` vs. the held English word followed by Chinese.
private func heldReadsAsChinese(_ keys: [Character], held h: Int, end: Int, finishing: Bool) -> Bool {
let c = classifier, w = classifier.weights, ctx = context
let isLast = end == keys.count
guard let zhWhole = c.segmentFeatures(Array(keys[..<end]), isLast: isLast, afterOther: false,
finishing: finishing, previous: ctx).0 else { return false }
guard let word = c.segmentFeatures(Array(keys[..<h]), isLast: false, afterOther: false,
finishing: true, previous: ctx).1,
let zhRest = c.segmentFeatures(Array(keys[(h + 1)..<end]), isLast: isLast, afterOther: true,
finishing: finishing, previous: nil).0 else { return true }
return Features.dot(w, zhWhole) >= Features.dot(w, word) + Features.dot(w, zhRest)
}
/// Whether keys + space also read as a common Chinese syllable (see `held`).
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
let p = ZhuyinParser.parse(keys + [" "])
guard p.pending == nil, !p.syllables.isEmpty, p.strayTones == 0, p.foreignKeys == 0 else { return false }
var best: [Double] = []
for ps in p.syllables {
guard let e = classifier.lexicon.entries(for: [ps.syllable]).first else { return false }
best.append(e.score)
}
if context != .literal { return true } // first word: hold anything that reads as Zhuyin
return best.count == 1 && best[0] >= CompositionEngine.holdMinScore
}
/// Current split of the buffer into Zhuyin / literal runs. /// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) } public var segments: [Segment] { segments(of: keys) }
@@ -84,6 +144,29 @@ public final class CompositionEngine {
public var markedText: String { markedPieces.map(\.text).joined() } public var markedText: String { markedPieces.map(\.text).joined() }
public var markedPieces: [MarkedPiece] { public var markedPieces: [MarkedPiece] {
var pieces = segmentPieces
if !inserting.isEmpty {
// Show the syllable being inserted at the cursor.
var at = baseCursorOffset, out: [MarkedPiece] = [], done = false
for piece in pieces {
if !done && at <= piece.text.count {
let i = piece.text.index(piece.text.startIndex, offsetBy: at)
out.append(MarkedPiece(text: String(piece.text[..<i]), mode: piece.mode))
out.append(MarkedPiece(text: insertingText, mode: .zhuyin))
out.append(MarkedPiece(text: String(piece.text[i...]), mode: piece.mode))
done = true
} else {
out.append(piece)
at -= piece.text.count
}
}
pieces = out.filter { !$0.text.isEmpty }
}
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
return pieces
}
private var segmentPieces: [MarkedPiece] {
let segs = segments let segs = segments
return segs.enumerated().map { i, seg in return segs.enumerated().map { i, seg in
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode) MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
@@ -108,92 +191,218 @@ public final class CompositionEngine {
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) { private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
let (syl, pending) = convertibleSyllables(seg, final: final) let (syl, pending) = convertibleSyllables(seg, final: final)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count } let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
if let reranker, let chosen = reranker.choose(converter, syl, fixed: fixedHere, context: rerankContext) {
return (chosen, pending)
}
return (converter.convert(syl, fixed: fixedHere), pending) return (converter.convert(syl, fixed: fixedHere), pending)
} }
private var rerankContext: String {
now().timeIntervalSince(previousCommitTime) < contextWindow ? recentText : ""
}
// MARK: - Cursor // MARK: - Cursor
/// A place the cursor can stop: after the `local`-th syllable of Zhuyin segment `segIndex`. /// Absolute key index after each syllable of a Zhuyin segment.
private func syllableEnds(_ seg: Segment) -> [Int] {
ZhuyinParser.parse(Array(keys[seg.range])).syllables.map { seg.range.lowerBound + $0.keyRange.upperBound }
}
/// Key indices the cursor can stop at: after every Zhuyin syllable and at every segment
/// boundary, but never before the first Chinese character (there would be nothing to
/// pick candidates for).
private var cursorPositions: [Int] {
var out: Set<Int> = [keys.count]
var ends: [Int] = []
for seg in segments {
out.insert(seg.range.upperBound)
if seg.mode == .zhuyin { ends += syllableEnds(seg) }
}
out.formUnion(ends)
let first = ends.min() ?? keys.count
return out.filter { $0 >= first }.sorted()
}
/// A place to open candidates: after the `local`-th syllable of Zhuyin segment `segIndex`.
private struct CursorStop { let segIndex: Int; let local: Int } private struct CursorStop { let segIndex: Int; let local: Int }
private var cursorStops: [CursorStop] { private func stop(at pos: Int) -> CursorStop? {
let segs = segments let segs = segments
var out: [CursorStop] = [] for (i, seg) in segs.enumerated() where seg.mode == .zhuyin
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin { && seg.range.lowerBound < pos && pos <= seg.range.upperBound {
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
for k in stride(from: 1, through: n, by: 1) { out.append(CursorStop(segIndex: i, local: k)) } let local = syllableEnds(seg).filter { $0 <= pos }.count
return local > 0 ? CursorStop(segIndex: i, local: min(local, n)) : nil
} }
return out return nil
}
/// Where ↓ opens candidates: before the cursor, or the last syllable.
private var candidateStop: CursorStop? {
if let c = cursor { return stop(at: c) }
let segs = segments
for i in segs.indices.reversed() where segs[i].mode == .zhuyin {
let n = convertibleSyllables(segs[i], final: i < segs.count - 1).syllables.count
if n > 0 { return CursorStop(segIndex: i, local: n) }
}
return nil
}
/// Cursor position in the rendered segments (without the syllable being inserted).
private var baseCursorOffset: Int {
let pieces = segmentPieces
guard let c = cursor else { return pieces.reduce(0) { $0 + $1.text.count } }
let segs = segments
for (i, seg) in segs.enumerated() where seg.range.lowerBound < c && c <= seg.range.upperBound {
var offset = pieces[..<i].reduce(0) { $0 + $1.text.count }
if seg.mode == .literal { return offset + c - seg.range.lowerBound }
let local = stop(at: c)?.local ?? 0
for conv in conversion(of: seg, final: i < segs.count - 1).segments {
if conv.range.upperBound <= local {
offset += conv.text.count
} else if conv.range.lowerBound < local {
offset += min(local - conv.range.lowerBound, conv.text.count)
}
}
return offset
}
return 0
} }
/// Cursor position in `markedText`, in Characters. /// Cursor position in `markedText`, in Characters.
public var cursorOffset: Int { public var cursorOffset: Int { baseCursorOffset + insertingText.count }
let pieces = markedPieces
let total = pieces.reduce(0) { $0 + $1.text.count } private var insertingText: String {
let stops = cursorStops inserting.isEmpty ? "" : ZhuyinParser.parse(inserting).syllables.map(\.syllable.reading).joined()
guard let c = cursor, stops.indices.contains(c) else { return total }
let stop = stops[c]
var offset = pieces[..<stop.segIndex].reduce(0) { $0 + $1.text.count }
let segs = segments
for conv in conversion(of: segs[stop.segIndex], final: stop.segIndex < segs.count - 1).segments {
if conv.range.upperBound <= stop.local {
offset += conv.text.count
} else if conv.range.lowerBound < stop.local {
offset += min(stop.local - conv.range.lowerBound, conv.text.count)
}
}
return offset
} }
private func moveCursor(by delta: Int) { private func moveCursor(by delta: Int) {
let stops = cursorStops inserting = []
guard !stops.isEmpty else { return } let stops = cursorPositions
let next = (cursor ?? stops.count - 1) + delta let here = cursor ?? keys.count
cursor = next >= stops.count - 1 ? nil : max(0, next) let next = delta < 0 ? (stops.last { $0 < here } ?? here) : (stops.first { $0 > here } ?? here)
cursor = next >= keys.count ? nil : next
}
// MARK: - Editing in the middle
/// Keeps picked phrases that end at or before key index `pos` (they don't move).
private func trimFixed(at pos: Int) {
var kept: [Int: [Range<Int>: String]] = [:]
for seg in segments where seg.mode == .zhuyin {
guard let picks = fixed[seg.range.lowerBound] else { continue }
if seg.range.upperBound <= pos {
kept[seg.range.lowerBound] = picks
} else if seg.range.lowerBound < pos {
let ends = syllableEnds(seg)
let here = picks.filter { $0.key.upperBound <= ends.count && ends[$0.key.upperBound - 1] <= pos }
if !here.isEmpty { kept[seg.range.lowerBound] = here }
}
}
fixed = kept
}
/// Inserts keys at the cursor and moves the cursor after them.
private func splice(_ newKeys: [Character]) {
guard let pos = cursor else { keys += newKeys; return }
trimFixed(at: pos)
keys.insert(contentsOf: newKeys, at: pos)
if let h = held, h >= pos { held = h + newKeys.count }
let next = pos + newKeys.count
cursor = next >= keys.count ? nil : next
}
/// Backspace in the middle: removes the syllable (or literal key) before the cursor.
private func deleteBeforeCursor(_ pos: Int) {
var start = pos - 1
for seg in segments where seg.mode == .zhuyin && seg.range.lowerBound < pos && pos <= seg.range.upperBound {
for ps in ZhuyinParser.parse(Array(keys[seg.range])).syllables
where seg.range.lowerBound + ps.keyRange.upperBound == pos {
start = seg.range.lowerBound + ps.keyRange.lowerBound
}
}
trimFixed(at: start)
keys.removeSubrange(start..<pos)
if let h = held {
if h >= start && h < pos { held = nil } else if h >= pos { held = h - (pos - start) }
}
cursor = start
if keys.isEmpty { reset() }
}
/// Typing with the cursor in the middle: compose one syllable, splice it in on its tone.
private func insertKey(_ c: Character) -> KeyResult {
if c == " " {
if inserting.isEmpty { return KeyResult(handled: true, commit: commitCurrent()) }
splice(inserting + [" "])
inserting = []
return .consumed
}
inserting.append(c)
let p = ZhuyinParser.parse(inserting)
if p.strayTones > 0 || p.foreignKeys > 0 {
inserting.removeLast() // a tone with nothing to attach to
} else if p.syllables.count == 1, p.syllables[0].toned {
splice(inserting)
inserting = []
} else if p.syllables.count > 1 {
// A new syllable started without a tone: splice the first one as it is.
let first = p.syllables[0].keyRange.upperBound
splice(Array(inserting[..<first]))
inserting = Array(inserting[first...])
}
return .consumed
} }
// MARK: - Key handling // MARK: - Key handling
public func handle(_ key: KeyInput) -> KeyResult { public func handle(_ key: KeyInput) -> KeyResult {
if isShowingCandidates { return handleCandidateKey(key) } if isShowingCandidates { return handleCandidateKey(key) }
if pendingBacktick { return afterBacktick(key) }
if !inserting.isEmpty && (key == .backspace || key == .escape) {
if key == .backspace { inserting.removeLast() } else { inserting = [] }
return .consumed
}
switch key { switch key {
case .character(let c): case .character(let c):
return handleCharacter(c) return handleCharacter(c)
case .backspace: case .backspace:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
cursor = nil if let c = cursor {
keys.removeLast() if c > 0 { deleteBeforeCursor(c) }
fixed = fixed.filter { $0.key < keys.count } return .consumed
if keys.isEmpty {
// A Shift-tap "next word is English" stays armed; a correction is dropped.
if forcedIsCorrection { forcedMode = nil; forcedIsCorrection = false }
fixed = [:]
} }
keys.removeLast()
if let h = held, keys.count <= h { held = nil }
fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] }
return .consumed return .consumed
case .enter: case .enter:
guard !keys.isEmpty else { endDirectEnglish(); return .passThrough } guard !keys.isEmpty else { return .passThrough }
if !inserting.isEmpty { splice(inserting); inserting = [] }
return KeyResult(handled: true, commit: commitCurrent()) return KeyResult(handled: true, commit: commitCurrent())
case .escape: case .escape:
guard !keys.isEmpty else { endDirectEnglish(); return .passThrough } guard !keys.isEmpty else { return .passThrough }
reset() reset()
return .consumed return .consumed
case .tab: case .tab:
guard !keys.isEmpty else { endDirectEnglish(); return .passThrough } guard !keys.isEmpty else { return .passThrough }
flip() flip()
return .consumed return .consumed
case .shiftTap: case .shiftTap:
if keys.isEmpty { if keys.isEmpty {
// Next word is English, typed as-is (lowercase stays lowercase). Tap again to cancel. // Toggle English mode; it stays on until the next Shift tap.
forcedMode = forcedMode == .literal ? nil : .literal isEnglishMode.toggle()
forcedIsCorrection = false previousMode = isEnglishMode ? .literal : nil
previousCommitTime = now()
} else { } else {
flip() flip()
} }
return .consumed return .consumed
case .down: case .down:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
openCandidates() if inserting.isEmpty { openCandidates() }
return .consumed return .consumed
case .left: case .left:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
@@ -211,41 +420,29 @@ public final class CompositionEngine {
/// Forces the whole buffer into the other mode (a correction that is learned). /// Forces the whole buffer into the other mode (a correction that is learned).
private func flip() { private func flip() {
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
forcedIsCorrection = true
} }
/// True while the user has forced English for the next word (Shift tap). In this
/// mode keys go straight to the app with no composition (no underline).
public var isEnglishArmed: Bool { keys.isEmpty && forcedMode == .literal && !forcedIsCorrection }
/// Leaves forced-English mode once the word is done.
private func endDirectEnglish() {
guard isEnglishArmed else { return }
forcedMode = nil
previousMode = .literal
previousCommitTime = now()
}
private static let englishWordPunctuation: Set<Character> = ["'", ".", ",", "-", "/", ";"]
private func handleCharacter(_ c: Character) -> KeyResult { private func handleCharacter(_ c: Character) -> KeyResult {
cursor = nil // typing always continues at the end if cursor != nil, !keys.isEmpty, c == " " || DachenLayout.isZhuyinKey(c) {
if isEnglishArmed { return insertKey(c)
// Forced English: insert directly. Space ends the word; other symbols end it too. }
if c == " " { if !inserting.isEmpty { splice(inserting); inserting = [] }
endDirectEnglish() cursor = nil // anything else continues at the end
return KeyResult(handled: true, commit: " ") // English mode: every key goes to the app as typed.
} if isEnglishMode && keys.isEmpty { return .passThrough }
if c.isASCII && (c.isLetter || c.isNumber || CompositionEngine.englishWordPunctuation.contains(c)) { if c == "`" {
return KeyResult(handled: true, commit: String(c)) pendingBacktick = true
} return .consumed
endDirectEnglish()
return .passThrough
} }
if c == " " { if c == " " {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
let segs = segments(of: keys, finishing: true) let segs = segments(of: keys, finishing: true)
if segs.last?.mode == .literal { if segs.last?.mode == .literal {
if held == nil, segs.count == 1, forcedMode == nil, isAmbiguousWord(keys) {
held = keys.count
keys.append(" ")
return .consumed
}
return KeyResult(handled: true, commit: commitCurrent(segs) + " ") return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
} }
if parse(ofLast: segs)?.pending != nil { if parse(ofLast: segs)?.pending != nil {
@@ -282,6 +479,26 @@ public final class CompositionEngine {
return ZhuyinParser.parse(Array(keys[last.range])) return ZhuyinParser.parse(Array(keys[last.range]))
} }
private func afterBacktick(_ key: KeyInput) -> KeyResult {
pendingBacktick = false
if key == .escape || key == .backspace { return .consumed }
let committed = keys.isEmpty ? "" : commitCurrent()
if case .character(let c) = key, let symbol = CompositionEngine.symbols[c] {
return KeyResult(handled: true, commit: committed + symbol)
}
// Not a symbol: the backtick was meant literally; then handle the key as usual.
var r = handle(key)
if !r.handled, case .character(let c) = key { r = KeyResult(handled: true, commit: String(c)) }
return KeyResult(handled: r.handled, commit: committed + "`" + (r.commit ?? ""))
}
/// "`" + key → full-width symbol.
static let symbols: [Character: String] = [
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", "\"": "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
]
static let fullwidth: [Character: String] = [ static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
@@ -291,9 +508,7 @@ public final class CompositionEngine {
/// Opens candidates for the phrase ending at the cursor (default: the last syllable). /// Opens candidates for the phrase ending at the cursor (default: the last syllable).
private func openCandidates() { private func openCandidates() {
let stops = cursorStops guard let stop = candidateStop else { return }
guard !stops.isEmpty else { return }
let stop = stops[min(cursor ?? stops.count - 1, stops.count - 1)]
let segs = segments let segs = segments
let seg = segs[stop.segIndex] let seg = segs[stop.segIndex]
let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
@@ -338,7 +553,9 @@ public final class CompositionEngine {
private func commitCurrent(_ given: [Segment]? = nil) -> String { private func commitCurrent(_ given: [Segment]? = nil) -> String {
let segs = given ?? segments let segs = given ?? segments
let text = segs.map { render($0, final: true) }.joined() let text = segs.map { render($0, final: true) }.joined()
if let forced = forcedMode, forcedIsCorrection { rememberPicks(segs)
recentText = String((rerankContext + text).suffix(20))
if let forced = forcedMode {
classifier.learned.record(String(keys), as: forced) classifier.learned.record(String(keys), as: forced)
} }
previousMode = segs.last?.mode previousMode = segs.last?.mode
@@ -347,6 +564,17 @@ public final class CompositionEngine {
return text return text
} }
/// Candidates the user picked are remembered and come first next time.
private func rememberPicks(_ segs: [Segment]) {
for seg in segs where seg.mode == .zhuyin {
guard let picks = fixed[seg.range.lowerBound], !picks.isEmpty else { continue }
let syl = convertibleSyllables(seg, final: true).syllables
for (range, text) in picks where range.upperBound <= syl.count {
classifier.history.record(Array(syl[range]), phrase: text)
}
}
}
/// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty. /// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty.
public func commitAll() -> String { public func commitAll() -> String {
keys.isEmpty ? "" : commitCurrent() keys.isEmpty ? "" : commitCurrent()
@@ -354,9 +582,11 @@ public final class CompositionEngine {
public func reset() { public func reset() {
keys = [] keys = []
inserting = []
held = nil
pendingBacktick = false
cursor = nil cursor = nil
forcedMode = nil forcedMode = nil
forcedIsCorrection = false
fixed = [:] fixed = [:]
candidates = [] candidates = []
} }
+39 -3
View File
@@ -13,12 +13,22 @@ public struct Candidate: Equatable {
/// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon). /// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon).
public struct Converter { public struct Converter {
public let lexicon: Lexicon public let lexicon: Lexicon
/// Picked candidates come first.
public let history: UserHistory?
/// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted. /// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted.
static let unknownScore = -30.0 static let unknownScore = -30.0
/// Penalty for matching an untoned syllable against all tones. /// Penalty for matching an untoned syllable against all tones.
static let tonelessPenalty = -1.5 static let tonelessPenalty = -1.5
public init(lexicon: Lexicon) { self.lexicon = lexicon } public init(lexicon: Lexicon, history: UserHistory? = nil) {
self.lexicon = lexicon
self.history = history
}
private func entries(_ syllables: [Syllable]) -> [LexiconEntry] {
let e = lexicon.entries(for: syllables)
return history?.adjust(syllables, e) ?? e
}
/// - Parameter fixed: user-chosen phrases by syllable range; they are always kept. /// - Parameter fixed: user-chosen phrases by syllable range; they are always kept.
public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] { public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] {
@@ -51,13 +61,39 @@ public struct Converter {
return segments.reversed() return segments.reversed()
} }
/// The `k` best paths (log10 score, segments), best first — for language-model reranking.
public func kBest(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:], k: Int) -> [(score: Double, segments: [ConvertedSegment])] {
let n = syllables.count
guard n > 0, k > 0 else { return [] }
let maxLen = max(1, lexicon.maxPhraseLength)
typealias Path = (score: Double, segments: [ConvertedSegment])
var beams: [[Path]] = Array(repeating: [], count: n + 1)
beams[0] = [(0, [])]
for j in 1...n {
var cand: [Path] = []
for len in 1...min(maxLen, j) {
let i = j - len
guard !beams[i].isEmpty else { continue }
let range = i..<j
for (text, score) in options(for: range, syllables: syllables, fixed: fixed) {
for path in beams[i] {
cand.append((path.score + score, path.segments + [ConvertedSegment(range: range, text: text)]))
}
}
}
cand.sort { $0.score > $1.score }
beams[j] = Array(cand.prefix(k))
}
return beams[n]
}
private func options(for range: Range<Int>, syllables: [Syllable], fixed: [Range<Int>: String]) -> [(String, Double)] { private func options(for range: Range<Int>, syllables: [Syllable], fixed: [Range<Int>: String]) -> [(String, Double)] {
if let text = fixed[range] { return [(text, 0)] } if let text = fixed[range] { return [(text, 0)] }
// A span may not partially overlap or swallow a fixed span. // A span may not partially overlap or swallow a fixed span.
if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] } if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] }
let slice = Array(syllables[range]) let slice = Array(syllables[range])
var result = lexicon.entries(for: slice).map { entry -> (String, Double) in var result = entries(slice).map { entry -> (String, Double) in
let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0 let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0
return (entry.phrase, entry.score + penalty) return (entry.phrase, entry.score + penalty)
} }
@@ -74,7 +110,7 @@ public struct Converter {
var seen = Set<String>() var seen = Set<String>()
for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) { for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) {
let range = (end - len)..<end let range = (end - len)..<end
for entry in lexicon.entries(for: Array(syllables[range])) where !seen.contains(entry.phrase) { for entry in entries(Array(syllables[range])) where !seen.contains(entry.phrase) {
seen.insert(entry.phrase) seen.insert(entry.phrase)
out.append(Candidate(range: range, text: entry.phrase)) out.append(Candidate(range: range, text: entry.phrase))
} }
+52 -17
View File
@@ -8,33 +8,63 @@ import Foundation
/// - a letter-bigram model trained on those words, for words not in either list. /// - a letter-bigram model trained on those words, for words not in either list.
public final class EnglishModel { public final class EnglishModel {
private var common = Set<String>() private var common = Set<String>()
private var dictionary = Set<String>()
private var sortedWords: [String] = []
private var sortedCommon: [String] = [] private var sortedCommon: [String] = []
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
/// binary searches, so no 230k-entry Set has to be built at startup.
private var sortedDict: [String] = []
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
private var unseenBigram = -12.0 private var unseenBigram = -12.0
public init(commonWords: [String], dictionaryWords: [String] = []) { /// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
public init(commonWords: [String], dictionaryWords: [String] = [],
bigram: (table: [String: Double], unseen: Double)? = nil) {
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty }) common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
sortedCommon = common.sorted() sortedCommon = common.sorted()
sortedWords = common.union(dictionary).sorted() var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
trainBigram() if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
dict = Array(Set(dict)).sorted()
}
sortedDict = dict
if let bigram {
self.bigram = bigram.table
unseenBigram = bigram.unseen
} else {
trainBigram()
}
} }
/// Loads common words from a bundle resource file and the system dictionary if present. /// Loads the bundled data files. Falls back to the system word list and to training
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) { /// the bigram table when the precomputed files are missing.
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
func lines(_ url: URL?) -> [String] { func lines(_ url: URL?) -> [String] {
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] } guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
return text.split(whereSeparator: \.isNewline) return text.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) } .map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.hasPrefix("#") } .filter { !$0.isEmpty && !$0.hasPrefix("#") }
} }
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary)) var bigram: (table: [String: Double], unseen: Double)?
if let url = bigramURL, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
bigram = (table, unseen)
}
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
?? URL(fileURLWithPath: "/usr/share/dict/words")
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
} }
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) } public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) } public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
private func inDictionary(_ w: String) -> Bool {
let i = lowerBound(w, in: sortedDict)
return i < sortedDict.count && sortedDict[i] == w
}
/// Some word strictly longer than `p` starts with it (common words or dictionary).
private func isPrefixOfWord(_ p: String) -> Bool {
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
}
/// - Parameter complete: true when the user ended the word (space / punctuation), /// - Parameter complete: true when the user ended the word (space / punctuation),
/// false while still typing (prefixes of words get partial credit). /// false while still typing (prefixes of words get partial credit).
@@ -43,10 +73,10 @@ public final class EnglishModel {
let n = Double(w.count) let n = Double(w.count)
guard !w.isEmpty else { return 0 } guard !w.isEmpty else { return 0 }
if common.contains(w) { return 3 + n } if common.contains(w) { return 3 + n }
if dictionary.contains(w) { return 1 + 0.6 * n } if inDictionary(w) { return 1 + 0.6 * n }
if !complete { if !complete {
if hasPrefix(w, in: sortedCommon) { return 0.4 * n } if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
if hasPrefix(w, in: sortedWords) { return 0.2 * n } if isPrefixOfWord(w) { return 0.2 * n }
} }
// Unknown word: judge by how English-like its letter sequence is. // Unknown word: judge by how English-like its letter sequence is.
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -63,12 +93,12 @@ public final class EnglishModel {
if common.contains(w) { if common.contains(w) {
f["lit.common"] = 1 f["lit.common"] = 1
f["lit.common_len"] = n f["lit.common_len"] = n
} else if dictionary.contains(w) { } else if inDictionary(w) {
f["lit.dict"] = 1 f["lit.dict"] = 1
f["lit.dict_len"] = n f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) { } else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n f["lit.prefix_common_len"] = n
} else if !complete && hasPrefix(w, in: sortedWords) { } else if !complete && isPrefixOfWord(w) {
f["lit.prefix_dict_len"] = n f["lit.prefix_dict_len"] = n
} else { } else {
let plaus = plausibility(w) let plaus = plausibility(w)
@@ -85,12 +115,17 @@ public final class EnglishModel {
return total / Double(chars.count - 1) return total / Double(chars.count - 1)
} }
private func hasPrefix(_ p: String, in words: [String]) -> Bool { private func lowerBound(_ p: String, in words: [String]) -> Int {
var lo = 0, hi = words.count var lo = 0, hi = words.count
while lo < hi { while lo < hi {
let mid = (lo + hi) / 2 let mid = (lo + hi) / 2
if words[mid] < p { lo = mid + 1 } else { hi = mid } if words[mid] < p { lo = mid + 1 } else { hi = mid }
} }
return lo
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
var lo = lowerBound(p, in: words)
// words[lo] is the first word >= p; skip an exact match, then test the next one. // words[lo] is the first word >= p; skip an exact match, then test the next one.
if lo < words.count && words[lo] == p { lo += 1 } if lo < words.count && words[lo] == p { lo += 1 }
return lo < words.count && words[lo].hasPrefix(p) return lo < words.count && words[lo].hasPrefix(p)
@@ -99,7 +134,7 @@ public final class EnglishModel {
private func trainBigram() { private func trainBigram() {
var pair: [String: Double] = [:] var pair: [String: Double] = [:]
var first: [String: Double] = [:] var first: [String: Double] = [:]
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) { for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
let weight = common.contains(w) ? 5.0 : 1.0 let weight = common.contains(w) ? 5.0 : 1.0
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
for i in 1..<chars.count { for i in 1..<chars.count {
+17 -4
View File
@@ -42,6 +42,9 @@ public enum Features {
"lit.others": -1.5, "lit.others": -1.5,
// Segmentation // Segmentation
"seg.switch": -3.0, "seg.switch": -3.0,
// Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
] ]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid. /// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
@@ -58,7 +61,8 @@ public enum Features {
/// Features of `keys` read as Zhuyin, or nil if impossible. /// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true, public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys) let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone. // A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil } if !isLast && parse.pending != nil { return nil }
@@ -107,18 +111,27 @@ public enum Features {
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) { if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed") add("zh.unconfirmed")
} }
if afterOther { add("seg.switch") } if afterOther {
add("seg.switch")
} else if previous == .zhuyin {
add("ctx.zh")
}
return f return f
} }
/// Features of `keys` output as typed, or nil if impossible. /// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true, public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false) -> FeatureVector? { afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil } if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1] var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v } func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther { add("seg.switch") } if afterOther {
add("seg.switch")
} else if previous == .literal {
add("ctx.lit")
}
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") } if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter }) let letters = String(keys.filter { $0.isLetter })
+242
View File
@@ -0,0 +1,242 @@
import Accelerate
import Foundation
/// Scores Chinese text: natural-log probability of `text` following `context`.
public protocol LanguageModel: AnyObject {
func logProb(_ text: String, context: String) -> Double
}
/// GPT-2 forward pass (CPU, Accelerate) for the character-level CKIP gpt2-tiny-chinese
/// model (GPL-3.0), stored by training/lm/export_lm.py as float16 ("AZLM" format).
/// Weights are expanded to float32 in memory (~46 MB for gpt2-tiny).
public final class GPT2LanguageModel: LanguageModel {
public enum LoadError: Error { case badFormat(String) }
let nLayer: Int, nHead: Int, d: Int, vocabSize: Int
let eps: Float
private var weights: [Float] = []
private var offsets: [String: Int] = [:]
private var vocab: [Character: Int32] = [:]
static let cls: Int32 = 101, unk: Int32 = 100
/// Longest input scored (context + text); longer contexts are cut from the left.
public var maxTokens = 64
public init(modelURL: URL, vocabURL: URL) throws {
let data = try Data(contentsOf: modelURL, options: .mappedIfSafe)
guard data.count > 8, data.prefix(4) == Data("AZLM".utf8) else { throw LoadError.badFormat("magic") }
let headerLength = Int(data[4]) | Int(data[5]) << 8 | Int(data[6]) << 16 | Int(data[7]) << 24
guard let header = try JSONSerialization.jsonObject(with: data.subdata(in: 8..<(8 + headerLength))) as? [String: Any],
let nLayer = header["n_layer"] as? Int, let nHead = header["n_head"] as? Int,
let d = header["n_embd"] as? Int, let vocabSize = header["vocab"] as? Int,
let eps = header["eps"] as? Double, let tensors = header["tensors"] as? [[Any]]
else { throw LoadError.badFormat("header") }
self.nLayer = nLayer; self.nHead = nHead; self.d = d; self.vocabSize = vocabSize; self.eps = Float(eps)
for t in tensors {
guard let name = t[0] as? String, let offset = t[2] as? Int else { throw LoadError.badFormat("tensor") }
offsets[name] = offset
}
let start = 8 + headerLength
let count = (data.count - start) / 2
weights = [Float](repeating: 0, count: count)
try data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
guard let base = raw.baseAddress?.advanced(by: start) else { throw LoadError.badFormat("data") }
try weights.withUnsafeMutableBufferPointer { out in
var src = vImage_Buffer(data: UnsafeMutableRawPointer(mutating: base), height: 1,
width: vImagePixelCount(count), rowBytes: count * 2)
var dst = vImage_Buffer(data: out.baseAddress, height: 1,
width: vImagePixelCount(count), rowBytes: count * 4)
guard vImageConvert_Planar16FtoPlanarF(&src, &dst, 0) == kvImageNoError else {
throw LoadError.badFormat("float16")
}
}
}
var index: Int32 = 0
for line in try String(contentsOf: vocabURL, encoding: .utf8).split(separator: "\n", omittingEmptySubsequences: false) {
if line.count == 1, let c = line.first, vocab[c] == nil { vocab[c] = index }
index += 1
}
for name in ["wte.weight", "wpe.weight", "ln_f.weight"] where offsets[name] == nil {
throw LoadError.badFormat("missing \(name)")
}
}
func tokens(_ text: String) -> [Int32] { text.map { vocab[$0] ?? GPT2LanguageModel.unk } }
public func logProb(_ text: String, context: String) -> Double {
let target = tokens(text)
guard !target.isEmpty else { return 0 }
var ctx = tokens(context)
if ctx.count + target.count + 1 > maxTokens { ctx = Array(ctx.suffix(max(0, maxTokens - target.count - 1))) }
let input = [GPT2LanguageModel.cls] + ctx + target.dropLast()
let h = hidden(input) // [T, d]
return weights.withUnsafeBufferPointer { w in
logSoftmaxScore(h, rows: input.count, from: ctx.count, targets: target, w: w)
}
}
// MARK: - Forward pass
private func w(_ name: String) -> Int { offsets[name]! }
/// Final hidden states [T, d] (row-major).
func hidden(_ ids: [Int32]) -> [Float] {
let T = ids.count, d = self.d
var x = [Float](repeating: 0, count: T * d)
weights.withUnsafeBufferPointer { W in
let wte = w("wte.weight"), wpe = w("wpe.weight")
for t in 0..<T {
vDSP_vadd(W.baseAddress! + wte + Int(ids[t]) * d, 1, W.baseAddress! + wpe + t * d, 1,
&x[t * d], 1, vDSP_Length(d))
}
for l in 0..<nLayer {
let p = "h.\(l)."
var h = layerNorm(x, rows: T, p + "ln_1", W)
let qkv = linear(h, rows: T, inDim: d, outDim: 3 * d, p + "attn.c_attn", W)
let a = attention(qkv, T: T)
var proj = linear(a, rows: T, inDim: d, outDim: d, p + "attn.c_proj", W)
GPT2LanguageModel.add(&x, proj)
h = layerNorm(x, rows: T, p + "ln_2", W)
var f = linear(h, rows: T, inDim: d, outDim: 4 * d, p + "mlp.c_fc", W)
geluNew(&f)
proj = linear(f, rows: T, inDim: 4 * d, outDim: d, p + "mlp.c_proj", W)
GPT2LanguageModel.add(&x, proj)
}
x = layerNorm(x, rows: T, "ln_f", W)
}
return x
}
private func layerNorm(_ x: [Float], rows: Int, _ name: String, _ W: UnsafeBufferPointer<Float>) -> [Float] {
let d = self.d
var out = [Float](repeating: 0, count: x.count)
let g = W.baseAddress! + w(name + ".weight"), b = W.baseAddress! + w(name + ".bias")
for r in 0..<rows {
var mean: Float = 0, sd: Float = 0
x.withUnsafeBufferPointer { xp in
out.withUnsafeMutableBufferPointer { op in
let src = xp.baseAddress! + r * d, dst = op.baseAddress! + r * d
vDSP_normalize(src, 1, dst, 1, &mean, &sd, vDSP_Length(d))
// vDSP_normalize divides by the population std without eps; redo with eps.
let scale = sd / (sd * sd + eps).squareRoot()
var s = scale
vDSP_vsmul(dst, 1, &s, dst, 1, vDSP_Length(d))
vDSP_vma(dst, 1, g, 1, b, 1, dst, 1, vDSP_Length(d))
}
}
}
return out
}
/// x [rows, inDim] · W [inDim, outDim] + bias.
private func linear(_ x: [Float], rows: Int, inDim: Int, outDim: Int, _ name: String,
_ W: UnsafeBufferPointer<Float>) -> [Float] {
var out = [Float](repeating: 0, count: rows * outDim)
vDSP_mmul(x, 1, W.baseAddress! + w(name + ".weight"), 1, &out, 1,
vDSP_Length(rows), vDSP_Length(outDim), vDSP_Length(inDim))
let bias = W.baseAddress! + w(name + ".bias")
out.withUnsafeMutableBufferPointer { op in
for r in 0..<rows {
vDSP_vadd(op.baseAddress! + r * outDim, 1, bias, 1, op.baseAddress! + r * outDim, 1, vDSP_Length(outDim))
}
}
return out
}
/// Causal multi-head self-attention. qkv is [T, 3d] (q | k | v).
private func attention(_ qkv: [Float], T: Int) -> [Float] {
let d = self.d, hd = d / nHead
let scale = 1 / Float(hd).squareRoot()
var out = [Float](repeating: 0, count: T * d)
var scores = [Float](repeating: 0, count: T)
qkv.withUnsafeBufferPointer { q in
out.withUnsafeMutableBufferPointer { o in
for head in 0..<nHead {
for i in 0..<T {
let qi = q.baseAddress! + i * 3 * d + head * hd
var maxScore = -Float.infinity
for j in 0...i {
var s: Float = 0
vDSP_dotpr(qi, 1, q.baseAddress! + j * 3 * d + d + head * hd, 1, &s, vDSP_Length(hd))
scores[j] = s * scale
maxScore = max(maxScore, scores[j])
}
var sum: Float = 0
for j in 0...i { scores[j] = expf(scores[j] - maxScore); sum += scores[j] }
let dst = o.baseAddress! + i * d + head * hd
for j in 0...i {
var a = scores[j] / sum
vDSP_vsma(q.baseAddress! + j * 3 * d + 2 * d + head * hd, 1, &a, dst, 1, dst, 1, vDSP_Length(hd))
}
}
}
}
}
return out
}
/// x += y (same length).
private static func add(_ x: inout [Float], _ y: [Float]) {
x.withUnsafeMutableBufferPointer { xp in
vDSP_vadd(xp.baseAddress!, 1, y, 1, xp.baseAddress!, 1, vDSP_Length(xp.count))
}
}
/// gelu_new: 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))).
private func geluNew(_ x: inout [Float]) {
let n = x.count
var inner = [Float](repeating: 0, count: n)
x.withUnsafeMutableBufferPointer { xb in
inner.withUnsafeMutableBufferPointer { ib in
let xp = xb.baseAddress!, ip = ib.baseAddress!, len = vDSP_Length(n)
var c3: Float = 0.044715, one: Float = 1, k = (2 / Float.pi).squareRoot(), half: Float = 0.5
vDSP_vsq(xp, 1, ip, 1, len) // x²
vDSP_vsmsa(ip, 1, &c3, &one, ip, 1, len) // 1 + 0.044715·x²
vDSP_vmul(ip, 1, xp, 1, ip, 1, len) // x + 0.044715·x³
vDSP_vsmul(ip, 1, &k, ip, 1, len)
var count = Int32(n)
vvtanhf(ip, ip, &count)
vDSP_vsadd(ip, 1, &one, ip, 1, len) // 1 + tanh(…)
vDSP_vmul(ip, 1, xp, 1, xp, 1, len)
vDSP_vsmul(xp, 1, &half, xp, 1, len)
}
}
}
/// Sum over target positions of log softmax(h · wteᵀ)[target]. Row `from + i` predicts target i.
private func logSoftmaxScore(_ h: [Float], rows T: Int, from start: Int, targets: [Int32],
w W: UnsafeBufferPointer<Float>) -> Double {
let d = self.d, V = vocabSize
let rowsUsed = T - start
// hᵀ for the rows we need: [d, rowsUsed]
var hT = [Float](repeating: 0, count: d * rowsUsed)
h.withUnsafeBufferPointer { hp in
vDSP_mtrans(hp.baseAddress! + start * d, 1, &hT, 1, vDSP_Length(d), vDSP_Length(rowsUsed))
}
// logitsᵀ [V, rowsUsed] = wte [V, d] · hᵀ [d, rowsUsed]
var logitsT = [Float](repeating: 0, count: V * rowsUsed)
vDSP_mmul(W.baseAddress! + w("wte.weight"), 1, hT, 1, &logitsT, 1,
vDSP_Length(V), vDSP_Length(rowsUsed), vDSP_Length(d))
var logits = [Float](repeating: 0, count: V * rowsUsed) // [rowsUsed, V]
vDSP_mtrans(logitsT, 1, &logits, 1, vDSP_Length(rowsUsed), vDSP_Length(V))
var total = 0.0
var expBuf = [Float](repeating: 0, count: V)
logits.withUnsafeMutableBufferPointer { lp in
expBuf.withUnsafeMutableBufferPointer { eb in
let e = eb.baseAddress!
for i in 0..<rowsUsed {
let row = lp.baseAddress! + i * V
var m: Float = 0
vDSP_maxv(row, 1, &m, vDSP_Length(V))
var negM = -m
vDSP_vsadd(row, 1, &negM, e, 1, vDSP_Length(V))
var n = Int32(V)
vvexpf(e, e, &n)
var sum: Float = 0
vDSP_sve(e, 1, &sum, vDSP_Length(V))
total += Double(row[Int(targets[i])] - m) - Double(logf(sum))
}
}
}
return total
}
}
+138 -28
View File
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
/// File format, one entry per line (`#` starts a comment): /// File format, one entry per line (`#` starts a comment):
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2 /// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
/// The phrase/reading columns may be in either order; the score is optional. /// The phrase/reading columns may be in either order; the score is optional.
///
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
/// call `prepareTonelessIndex()` from a background queue to have it ready.
public final class Lexicon { public final class Lexicon {
private var exact: [String: [LexiconEntry]] = [:] private var exact: [String: [LexiconEntry]] = [:]
private var toneless: [String: [LexiconEntry]] = [:] private var tonelessIndex: [String: [LexiconEntry]]?
private let lock = NSLock()
public private(set) var maxPhraseLength = 1 public private(set) var maxPhraseLength = 1
public init() {} public init() {}
public convenience init(contentsOf url: URL) throws { public convenience init(contentsOf url: URL) throws {
self.init() self.init()
load(try String(contentsOf: url, encoding: .utf8)) load(data: try Data(contentsOf: url))
} }
public func load(_ text: String) { public func load(_ text: String) {
for line in text.split(whereSeparator: \.isNewline) { load(data: Data(text.utf8))
let trimmed = line.trimmingCharacters(in: .whitespaces)
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
guard cols.count >= 2 else { continue }
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
guard Lexicon.isBopomofo(reading) else { continue }
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
add(reading: reading, phrase: phrase, score: score)
}
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
} }
public func add(reading rawReading: String, phrase: String, score: Double) { public func load(data: Data) {
let entry = LexiconEntry(phrase: phrase, score: score) lock.lock()
defer { lock.unlock() }
exact.reserveCapacity(exact.count + data.count / 30)
data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
let bytes = raw.bindMemory(to: UInt8.self)
let n = bytes.count
var i = 0
while i < n {
var end = i
while end < n && bytes[end] != 0x0A { end += 1 }
parseLine(bytes, i, end)
i = end + 1
}
}
for (key, entries) in exact where entries.count > 1 {
exact[key] = entries.sorted { $0.score > $1.score }
}
tonelessIndex = nil // rebuilt on demand
}
/// Splits one line into up to three space/tab separated fields.
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
var fields: [Range<Int>] = []
var i = start
while i < end && fields.count < 3 {
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
if i >= end { break }
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
let s = i
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
fields.append(s..<i)
}
guard fields.count >= 2 else { return }
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
guard Lexicon.isBopomofo(b, r) else { return }
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
var prefixNeutral = b[r.lowerBound] == 0xCB
var k = r.lowerBound
while !prefixNeutral && k + 2 < r.upperBound {
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
k += 1
}
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
}
/// Parses "-3.9151"-style decimals without creating a String.
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
var i = r.lowerBound
var sign = 1.0
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
var value = 0.0, scale = 1.0, digits = 0, fraction = false
while i < r.upperBound {
let c = b[i]
if c >= 0x30 && c <= 0x39 {
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
digits += 1
} else if c == 0x2E && !fraction {
fraction = true
} else {
return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
}
i += 1
}
return digits > 0 ? sign * value : nil
}
/// Adds one entry (call `load` for bulk data; this does not re-sort).
public func add(reading: String, phrase: String, score: Double) {
lock.lock()
defer { lock.unlock() }
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
exact[key]?.sort { $0.score > $1.score }
tonelessIndex = nil
}
@discardableResult
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
var reading = rawReading
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix. // Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
let syllables = rawReading.split(separator: "-").map { s -> String in if normalise {
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s) reading = reading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
}.joined(separator: "-")
} }
let reading = syllables.joined(separator: "-") maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
maxPhraseLength = max(maxPhraseLength, syllables.count) if exact[reading] == nil {
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) { exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
exact[reading, default: []].append(entry) } else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
} }
let key = Lexicon.stripTones(reading) return reading
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) { }
toneless[key]![i].score = max(toneless[key]![i].score, score)
} else { /// Builds the index that ignores tones (for untoned syllables). Safe to call from a
toneless[key, default: []].append(entry) /// background queue; lookups build it themselves if it is not ready yet.
public func prepareTonelessIndex() {
lock.lock()
defer { lock.unlock() }
_ = buildTonelessIfNeeded()
}
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
if let index = tonelessIndex { return index }
var best: [String: [String: Double]] = [:]
for (reading, entries) in exact {
let key = Lexicon.stripTones(reading)
for e in entries {
if let s = best[key]?[e.phrase], s >= e.score { continue }
best[key, default: [:]][e.phrase] = e.score
}
} }
var index: [String: [LexiconEntry]] = [:]
index.reserveCapacity(best.count)
for (key, phrases) in best {
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
}
tonelessIndex = index
return index
} }
/// Entries for a sequence of readings. Untoned syllables match any tone. /// Entries for a sequence of readings. Untoned syllables match any tone.
public func entries(for syllables: [Syllable]) -> [LexiconEntry] { public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
lock.lock()
defer { lock.unlock() }
if syllables.allSatisfy({ $0.tone != nil }) { if syllables.allSatisfy({ $0.tone != nil }) {
return exact[syllables.map(\.reading).joined(separator: "-")] ?? [] return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
} }
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? [] return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
} }
public func bestScore(for syllable: Syllable) -> Double? { public func bestScore(for syllable: Syllable) -> Double? {
@@ -81,4 +182,13 @@ public final class Lexicon {
guard let first = s.unicodeScalars.first else { return false } guard let first = s.unicodeScalars.first else { return false }
return (0x3105...0x312F).contains(first.value) || first == "˙" return (0x3105...0x312F).contains(first.value) || first == "˙"
} }
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
guard r.count >= 2 else { return false }
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
let c = b[r.lowerBound + 2]
return c >= 0x85 && c <= 0xAF
}
} }
+90
View File
@@ -0,0 +1,90 @@
import Foundation
/// "AI 選字": reranks the converter's best paths with a language model.
///
/// score = lexicon score (log10 → ln) + λ · LM log-probability
///
/// It never blocks typing for long: each rerank has a time budget, and a rerank that runs
/// over it is dropped (the plain lexicon result is used). Three overruns in a row pause
/// AI for a minute. `systemState` lets the app pause or reduce it when the machine is
/// busy, hot or saving power.
public final class Reranker {
public enum Mode: String, CaseIterable {
case off, saver, standard, quality
/// How many paths are reranked, and the time budget for one rerank.
var k: Int { [.saver: 3, .standard: 10, .quality: 30][self] ?? 0 }
var budget: TimeInterval { [.saver: 0.015, .standard: 0.04, .quality: 0.12][self] ?? 0 }
}
/// What the system allows right now (decided by the app).
public enum SystemState { case normal, reduced, paused }
public enum Status: Equatable {
case off, loading, active, pausedBusy, pausedSlow
}
public var mode: Mode { didSet { cache.removeAll() } }
/// nil while loading (or if the model file is missing).
public var model: LanguageModel? { didSet { cache.removeAll() } }
public var lambda = 0.6
public var systemState: () -> SystemState = { .normal }
public var now: () -> Date = Date.init
public private(set) var status: Status = .off
private var overruns = 0
private var pausedUntil = Date.distantPast
private var cache: [String: [ConvertedSegment]] = [:]
static let cacheLimit = 512
static let pauseAfterOverruns = 3
static let slowPause: TimeInterval = 60
public init(mode: Mode = .standard, model: LanguageModel? = nil) {
self.mode = mode
self.model = model
}
/// The mode actually used now (after the system state), or nil when AI is not used.
var effectiveMode: Mode? {
guard mode != .off else { status = .off; return nil }
guard model != nil else { status = .loading; return nil }
if now() < pausedUntil { status = .pausedSlow; return nil }
switch systemState() {
case .paused: status = .pausedBusy; return nil
case .reduced: status = .active; return .saver
case .normal: status = .active; return mode
}
}
/// The reranked conversion, or nil to use the plain lexicon result.
func choose(_ converter: Converter, _ syllables: [Syllable], fixed: [Range<Int>: String],
context: String) -> [ConvertedSegment]? {
guard let mode = effectiveMode, let model, !syllables.isEmpty else { return nil }
let key = "\(mode.rawValue)|\(context)|\(syllables.map(\.reading).joined(separator: "-"))|"
+ fixed.sorted { $0.key.lowerBound < $1.key.lowerBound }.map { "\($0.key)=\($0.value)" }.joined(separator: ",")
if let hit = cache[key] { return hit }
let paths = converter.kBest(syllables, fixed: fixed, k: mode.k)
guard paths.count > 1 else { return paths.first?.segments }
let start = now()
var best: (score: Double, segments: [ConvertedSegment])?
for path in paths {
let text = path.segments.map(\.text).joined()
let score = path.score * log(10) + lambda * model.logProb(text, context: context)
if best == nil || score > best!.score { best = (score, path.segments) }
if now().timeIntervalSince(start) > mode.budget {
overruns += 1
if overruns >= Reranker.pauseAfterOverruns {
pausedUntil = now().addingTimeInterval(Reranker.slowPause)
overruns = 0
status = .pausedSlow
}
return nil
}
}
overruns = 0
if cache.count >= Reranker.cacheLimit { cache.removeAll() }
cache[key] = best?.segments
return best?.segments
}
}
+79
View File
@@ -0,0 +1,79 @@
import Foundation
/// Remembers the candidates the user picked, so they come first next time.
///
/// Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
/// where the lexicon has it for the reading being converted, so a toneless key is safe.
/// Stored as JSON: {reading: {phrase: [count, last use]}} (same format on every platform).
public final class UserHistory {
public typealias Data = [String: [String: [Int]]]
/// (reading, phrase) pairs kept; the least recently used go first.
public var maxEntries = 2000
static let maxCount = 10
public private(set) var data: Data = [:]
private var clock = 0
private let onChange: (Data) -> Void
public init(data: Data = [:], onChange: @escaping (Data) -> Void = { _ in }) {
self.onChange = onChange
for (reading, phrases) in data {
for (phrase, value) in phrases where value.count >= 2 {
self.data[reading, default: [:]][phrase] = [value[0], value[1]]
clock = max(clock, value[1])
}
}
}
/// Loads the JSON file written through `onChange`; empty if missing or unreadable.
public static func load(from url: URL) -> Data {
guard let raw = try? Foundation.Data(contentsOf: url),
let data = try? JSONDecoder().decode(Data.self, from: raw) else { return [:] }
return data
}
static func key(_ syllables: [Syllable]) -> String {
syllables.map(\.toneless).joined(separator: "-")
}
public func record(_ syllables: [Syllable], phrase: String) {
guard !syllables.isEmpty, !phrase.isEmpty else { return }
clock += 1
let key = UserHistory.key(syllables)
let count = data[key]?[phrase]?.first ?? 0
data[key, default: [:]][phrase] = [min(UserHistory.maxCount, count + 1), clock]
evict()
onChange(data)
}
private func evict() {
var pairs: [(used: Int, reading: String, phrase: String)] = []
for (r, phrases) in data { for (p, v) in phrases { pairs.append((v[1], r, p)) } }
guard pairs.count > maxEntries else { return }
pairs.sort { $0.used < $1.used }
for pair in pairs.prefix(pairs.count - maxEntries) {
data[pair.reading]?[pair.phrase] = nil
if data[pair.reading]?.isEmpty == true { data[pair.reading] = nil }
}
}
public func clear() {
data = [:]
onChange(data)
}
/// Lexicon entries with picked phrases moved to the top: just above the best entry,
/// more so the more often they were picked.
func adjust(_ syllables: [Syllable], _ entries: [LexiconEntry]) -> [LexiconEntry] {
guard let picked = data[UserHistory.key(syllables)], let top = entries.first?.score else { return entries }
var out = entries.map { e -> LexiconEntry in
guard let count = picked[e.phrase]?.first else { return e }
return LexiconEntry(phrase: e.phrase, score: max(e.score, top + 0.01 + 0.1 * Double(count)))
}
// Stable sort (keep lexicon order among equal scores), like Python's sort.
out = out.enumerated().sorted { $0.element.score != $1.element.score
? $0.element.score > $1.element.score : $0.offset < $1.offset }.map(\.element)
return out
}
}
+64
View File
@@ -0,0 +1,64 @@
自動注音 AutoZhuyin — 第三方授權聲明 / Third-party notices
==========================================================
本軟體包含或使用下列第三方資料。
This software includes or uses the following third-party material.
1. 小麥注音 McBopomofo — 詞庫資料 (lexicon data)
------------------------------------------------
https://github.com/openvanilla/McBopomofo
lexicon.txt is converted from McBopomofo's data files (BPMFBase.txt,
BPMFMappings.txt, phrase.occ) by scripts/convert-mcbopomofo.py. No McBopomofo
program code is included.
MIT License
Copyright (c) 2011-2026 Mengjuei Hsieh et al.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
2. Webster's Second International Dictionary — 英文字典 (English word list)
--------------------------------------------------------------------------
english-dict.txt is filtered from /usr/share/dict/words (Webster's Second
International Dictionary, 1934), which is in the public domain.
3. PIME — Windows 輸入法框架 (input method framework)
----------------------------------------------------
https://github.com/EasyIME/PIME — GNU LGPL 2.1
The Windows version runs as a module inside PIME, which users install
separately. No PIME code is included in this software.
4. CKIP GPT2 Tiny Chinese — AI 選字語言模型 (language model, macOS)
-------------------------------------------------------------------
https://huggingface.co/ckiplab/gpt2-tiny-chinese
https://github.com/ckiplab/ckip-transformers
Author: Mu Yang at CKIP, Academia Sinica (中央研究院 詞庫小組)
License: GNU General Public License v3.0 (full text: GPL-3.0.txt)
lm.bin is the model's weights converted to float16 by training/lm/export_lm.py;
lm-vocab.txt is its vocabulary (from bert-base-chinese). They are used unmodified
otherwise, to rerank conversion candidates ("AI 選字"). The source code of this input
method, including the conversion script and the code that runs the model, is
available from the project repository.
@@ -6,7 +6,10 @@ private let resources = URL(fileURLWithPath: #filePath)
.appendingPathComponent("Resources") .appendingPathComponent("Resources")
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")) private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase { final class ParserTests: XCTestCase {
@@ -265,10 +268,13 @@ final class ParityTests: XCTestCase {
for fx in fixtures { for fx in fixtures {
let keys = Array(fx["keys"] as! String) let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" } let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let got = c.segment(keys, finishing: fx["finishing"] as! Bool).map { let previous = (fx["previous"] as? String).flatMap(InputMode.init(rawValue:))
let got = c.segment(keys, finishing: fx["finishing"] as! Bool, previous: previous).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)" "\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
} }
if got != expected { mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!): \(got) vs \(expected)") } if got != expected {
mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!) previous=\(String(describing: previous)): \(got) vs \(expected)")
}
} }
XCTAssertGreaterThan(fixtures.count, 300) XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n")) XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
@@ -297,13 +303,16 @@ final class ShiftTapTests: XCTestCase {
return out return out
} }
func testShiftTapForcesNextWordEnglishLowercase() { func testShiftTapTogglesEnglishMode() {
let e = engine() let e = engine()
// "ru" alone would be ㄐㄧ (機); after a Shift tap it stays English, lowercase. // "ru" alone would be ㄐㄧ (機); in English mode keys pass straight through, lowercase.
XCTAssertEqual(type("⇧ru ", into: e), "ru ") XCTAssertEqual(type("⇧ru su ", into: e), "ru su ")
// Automatic detection resumes after the word. XCTAssertTrue(e.isEnglishMode) // still English after spaces
XCTAssertFalse(e.isEnglishArmed) XCTAssertEqual(e.markedText, "") // never composed: no underline
XCTAssertEqual(type("su3cl3\n", into: e), "你好") XCTAssertEqual(type("\n", into: e), "\n") // Enter goes to the app, mode stays
XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(type("⇧su3cl3\n", into: e), "你好") // tap again: automatic again
XCTAssertFalse(e.isEnglishMode)
} }
func testShiftTapTwiceCancels() { func testShiftTapTwiceCancels() {
@@ -316,22 +325,22 @@ final class ShiftTapTests: XCTestCase {
XCTAssertEqual(e.mode, .literal) XCTAssertEqual(e.mode, .literal)
_ = e.handle(.shiftTap) _ = e.handle(.shiftTap)
XCTAssertEqual(e.mode, .zhuyin) XCTAssertEqual(e.mode, .zhuyin)
XCTAssertFalse(e.isEnglishMode) // a flip, not English mode
_ = type("\n", into: e) _ = type("\n", into: e)
XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned
} }
func testForcingBeforeTypingIsNotLearned() { func testEnglishModeIsNotLearned() {
let e = engine() let e = engine()
_ = type("⇧hello ", into: e) _ = type("⇧hello ", into: e)
XCTAssertNil(e.classifier.learned.weights["hello"]) XCTAssertTrue(e.classifier.learned.weights.isEmpty)
} }
func testArmingSurvivesBackspace() { func testBackspaceInEnglishModeGoesToApp() {
let e = engine() let e = engine()
_ = type("⇧r", into: e) _ = type("⇧", into: e)
_ = e.handle(.backspace) XCTAssertFalse(e.handle(.backspace).handled)
XCTAssertTrue(e.isEnglishArmed) XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(type("ru ", into: e), "ru ")
} }
func testDetector() { func testDetector() {
@@ -360,7 +369,8 @@ final class CursorTests: XCTestCase {
func type(_ s: String, into e: CompositionEngine) -> String { func type(_ s: String, into e: CompositionEngine) -> String {
var out = "" var out = ""
for c in s { for c in s {
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter : .character(c) let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter
: c == "\u{8}" ? .backspace : .character(c)
let r = e.handle(key) let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) } if let t = r.commit { out += t } else if !r.handled { out.append(c) }
} }
@@ -390,12 +400,50 @@ final class CursorTests: XCTestCase {
XCTAssertEqual(type("\n", into: e), "今添天氣") XCTAssertEqual(type("\n", into: e), "今添天氣")
} }
func testTypingReturnsCursorToEnd() { func testPunctuationCommitsFromTheMiddle() {
let e = engine() let e = engine()
_ = type("rup wu0 ←", into: e) _ = type("rup wu0 ←", into: e)
XCTAssertEqual(e.cursorOffset, 1) XCTAssertEqual(e.cursorOffset, 1)
_ = type("wu0 ", into: e) _ = type("?", into: e)
XCTAssertEqual(e.cursorOffset, e.markedText.count) XCTAssertEqual(e.markedText, "")
}
func testInsertSyllable() {
let e = engine()
_ = type("rup fu4←wu", into: e)
XCTAssertEqual(e.markedText, "今ㄊㄧ氣")
XCTAssertEqual(e.cursorOffset, 3)
_ = type("0 ", into: e)
XCTAssertEqual(e.markedText, "今天氣")
XCTAssertEqual(e.cursorOffset, 2)
XCTAssertEqual(type("\n", into: e), "今天氣")
}
func testCursorStopsAfterFirstCharacter() {
let e = engine()
_ = type("su3cl3←←←", into: e)
XCTAssertEqual(e.cursorOffset, 1)
}
func testBackspaceDeletesCharacterBeforeCursor() {
let e = engine()
_ = type("rup wu0 wu0 fu4←←\u{8}", into: e)
XCTAssertEqual(e.markedText, "今天氣")
XCTAssertEqual(e.cursorOffset, 1)
}
func testBackspaceInsideInsertedSyllable() {
let e = engine()
_ = type("rup fu4←wu\u{8}", into: e)
XCTAssertEqual(e.markedText, "今ㄊ氣")
}
func testKeepsPickedPhraseBeforeCursor() {
let e = engine()
_ = type("rup wu0 ↓", into: e)
e.selectCandidate(e.candidates.firstIndex { $0.text == "添" }!)
_ = type("fu4←wu0 ", into: e) // insert 天 between 添 and 氣
XCTAssertEqual(String(e.markedText.prefix(3)), "今添天")
} }
func testCandidatesForChineseBeforeEnglish() { func testCandidatesForChineseBeforeEnglish() {
@@ -406,26 +454,321 @@ final class CursorTests: XCTestCase {
} }
} }
final class DirectEnglishTests: XCTestCase { final class FirstToneAfterEnglishTests: XCTestCase {
func testForcedEnglishHasNoComposition() { /// After an English word, a first-tone syllable + space must stay Chinese (機, 吃, 書…).
func testFirstToneAfterEnglish() {
for keys in ["ru", "t", "gj", "vup", "tj", "el"] {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
out += e.handle(.enter).commit ?? ""
XCTAssertEqual(String(out.prefix(6)), "hello ", keys)
XCTAssertEqual(out.count, 7, keys) // one Chinese character after it
XCTAssertFalse(out.last!.isASCII, keys)
}
}
}
final class HeldWordTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// "up " is also ㄧㄣ (因): the next word decides.
func testNextWordDecides() {
let cases = ["up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testHeldWordStaysComposing() {
let e = makeEngine()
XCTAssertEqual(type("up ", into: e), "")
XCTAssertEqual(e.markedText, "up ")
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "up")
XCTAssertEqual(type(" the ", into: e), "up the ")
}
}
final class BacktickSymbolTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testForcedSymbols() {
let cases = ["`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testPlainBacktick() {
let e = makeEngine()
XCTAssertEqual(type("`", into: e), "")
XCTAssertEqual(e.markedText, "`")
XCTAssertEqual(type("ls` ", into: e), "`ls` ")
}
func testBackspaceCancels() {
let e = makeEngine()
_ = type("su3`", into: e)
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "你")
}
}
final class FirstWordHoldTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// The first word is not committed as English on space if it also reads as Zhuyin.
func testNextWordDecides() {
let cases = ["el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
"i love you ": "i love you ", "up the ": "up the "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testNotHeldAfterEnglish() {
let e = makeEngine()
XCTAssertEqual(type("hello world ", into: e), "hello world ")
XCTAssertEqual(type("so ", into: e), "so ") // right after English: commits at once
}
}
final class UserHistoryTests: XCTestCase {
var saved: [UserHistory.Data] = []
var classifier: Classifier!
override func setUp() {
saved = []
classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights,
history: UserHistory { [unowned self] in self.saved.append($0) })
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "↓" ? .down : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
@discardableResult
func pick(_ keys: String, _ phrase: String) -> String {
let e = CompositionEngine(classifier: classifier)
_ = type(keys + "↓", into: e)
e.selectCandidate(e.candidates.firstIndex { $0.text == phrase }!)
return type("\n", into: e)
}
func testPickedCandidateComesFirst() {
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "天")
XCTAssertEqual(pick("wu0 ", "添"), "添")
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
let e = CompositionEngine(classifier: classifier)
_ = type("wu0 ↓", into: e)
XCTAssertEqual(e.candidates.first?.text, "添")
XCTAssertEqual(saved.last, ["ㄊㄧㄢ": ["添": [1, 1]]])
}
func testPhrasesStillWin() {
pick("wu0 ", "添")
XCTAssertEqual(type("rup wu0 \n", into: CompositionEngine(classifier: classifier)), "今天")
}
func testMostPickedWins() {
pick("wu0 ", "添")
pick("wu0 ", "添")
pick("wu0 ", "天")
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
}
func testRoundTripAndClear() throws {
pick("wu0 ", "添")
let json = try JSONEncoder().encode(saved.last!)
let h = UserHistory(data: try JSONDecoder().decode(UserHistory.Data.self, from: json))
XCTAssertEqual(h.data, ["ㄊㄧㄢ": ["添": [1, 1]]])
h.record(ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable), phrase: "天")
XCTAssertEqual(h.data["ㄊㄧㄢ"]?["天"], [1, 2])
h.clear()
XCTAssertTrue(h.data.isEmpty)
}
func testEvictsLeastRecent() {
let h = UserHistory()
h.maxEntries = 2
let syl = ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable)
for p in ["天", "添", "田"] { h.record(syl, phrase: p) }
XCTAssertEqual(Set(h.data["ㄊㄧㄢ"]!.keys), ["添", "田"])
}
}
/// Needs build/lm/lm.bin (scripts/fetch-lm.sh); skipped otherwise.
final class GPT2LanguageModelTests: XCTestCase {
static let dir = resources.deletingLastPathComponent().appendingPathComponent("build/lm")
func model() throws -> GPT2LanguageModel {
let bin = GPT2LanguageModelTests.dir.appendingPathComponent("lm.bin")
try XCTSkipUnless(FileManager.default.fileExists(atPath: bin.path), "run scripts/fetch-lm.sh")
return try GPT2LanguageModel(modelURL: bin, vocabURL: GPT2LanguageModelTests.dir.appendingPathComponent("lm-vocab.txt"))
}
/// Same numbers as the numpy reference (training/lm/gpt2.py, float32 weights).
func testMatchesReference() throws {
let m = try model()
let cases: [(String, String, Double)] = [("他在家", "", -14.329), ("今天天氣很好", "", -30.08),
("今添天氣很好", "", -44.392), ("家", "他在", -5.514),
("天氣很好", "今天", -23.883)]
for (text, context, want) in cases {
XCTAssertEqual(m.logProb(text, context: context), want, accuracy: 0.05, text)
}
}
func testSpeed() throws {
let m = try model()
let start = Date()
for _ in 0..<20 { _ = m.logProb("我們明天早上八點在車站見面", context: "好啊") }
let ms = Date().timeIntervalSince(start) / 20 * 1000
print("GPT2 logProb: \(String(format: "%.2f", ms)) ms")
XCTAssertLessThan(ms, 20)
}
}
/// A language model that likes one string and can pretend to be slow.
private final class FakeLanguageModel: LanguageModel {
var liked: String
var onCall: () -> Void = {}
init(liked: String) { self.liked = liked }
func logProb(_ text: String, context: String) -> Double {
onCall()
return text.contains(liked) ? 0 : -50
}
}
final class RerankerTests: XCTestCase {
func engine(_ reranker: Reranker?) -> CompositionEngine {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
_ = e.handle(.shiftTap) e.reranker = reranker
let r = e.handle(.character("r")) return e
XCTAssertEqual(r.commit, "r") // inserted right away }
XCTAssertEqual(e.markedText, "") // nothing underlined
XCTAssertEqual(e.handle(.character("u")).commit, "u") func type(_ s: String, into e: CompositionEngine) -> String {
XCTAssertEqual(e.handle(.character("'")).commit, "'") var out = ""
XCTAssertFalse(e.handle(.backspace).handled) // the app deletes it for c in s {
XCTAssertTrue(e.isEnglishArmed) let r = e.handle(c == "\n" ? .enter : .character(c))
XCTAssertEqual(e.handle(.character(" ")).commit, " ") if let t = r.commit { out += t } else if !r.handled { out.append(c) }
XCTAssertFalse(e.isEnglishArmed) // back to automatic }
} return out
}
func testEnterEndsForcedEnglish() {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)) func testRerankPicksWhatTheModelLikes() {
_ = e.handle(.shiftTap) XCTAssertEqual(type("wu0 \n", into: engine(nil)), "天")
_ = e.handle(.character("a")) let r = Reranker(mode: .standard, model: FakeLanguageModel(liked: "添"))
XCTAssertFalse(e.handle(.enter).handled) XCTAssertEqual(type("wu0 \n", into: engine(r)), "添")
XCTAssertFalse(e.isEnglishArmed) XCTAssertEqual(r.status, .active)
}
func testOffAndLoadingUseTheLexicon() {
let off = Reranker(mode: .off, model: FakeLanguageModel(liked: "添"))
XCTAssertEqual(type("wu0 \n", into: engine(off)), "天")
XCTAssertEqual(off.status, .off)
let loading = Reranker(mode: .standard, model: nil)
XCTAssertEqual(type("wu0 \n", into: engine(loading)), "天")
XCTAssertEqual(loading.status, .loading)
}
func testPausedWhenSystemBusy() {
let r = Reranker(mode: .standard, model: FakeLanguageModel(liked: "添"))
r.systemState = { .paused }
XCTAssertEqual(type("wu0 \n", into: engine(r)), "天")
XCTAssertEqual(r.status, .pausedBusy)
}
func testReducedUsesFewerPaths() {
let lm = FakeLanguageModel(liked: "添")
var calls = 0
lm.onCall = { calls += 1 }
let r = Reranker(mode: .quality, model: lm)
r.systemState = { .reduced }
_ = type("wu0 \n", into: engine(r))
XCTAssertLessThanOrEqual(calls, 3)
}
func testSlowModelIsDroppedThenPaused() {
var clock = Date(timeIntervalSince1970: 0)
let lm = FakeLanguageModel(liked: "添")
lm.onCall = { clock.addTimeInterval(1) } // every call takes a "second"
let r = Reranker(mode: .standard, model: lm)
r.now = { clock }
for _ in 0..<3 { XCTAssertEqual(type("wu0 \n", into: engine(r)), "天") }
XCTAssertEqual(r.status, .pausedSlow)
lm.onCall = {}
XCTAssertEqual(type("wu0 \n", into: engine(r)), "天") // still paused
clock.addTimeInterval(61)
XCTAssertEqual(type("wu0 \n", into: engine(r)), "添") // back after a minute
}
/// With the real model (build/lm, skipped if missing): sentences the lexicon alone gets wrong.
func testRealModelFixesHomophones() throws {
let dir = GPT2LanguageModelTests.dir
let bin = dir.appendingPathComponent("lm.bin")
try XCTSkipUnless(FileManager.default.fileExists(atPath: bin.path), "run scripts/fetch-lm.sh")
let lm = try GPT2LanguageModel(modelURL: bin, vocabURL: dir.appendingPathComponent("lm-vocab.txt"))
let cases = ["w8 ql32k71u3ji3dj94cp32ji ": "他跑得比我快很多",
"fu/3su3y94gji u h4cl3a87": "請你再說一次好嗎"]
for (keys, want) in cases {
XCTAssertNotEqual(type(keys + "\n", into: engine(nil)), want)
XCTAssertEqual(type(keys + "\n", into: engine(Reranker(mode: .standard, model: lm))), want)
}
}
}
final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws {
var t = Date()
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
} }
} }
+674
View File
@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.
+328
View File
@@ -0,0 +1,328 @@
#!/usr/bin/env python3
"""IBus engine for AutoZhuyin (自動注音) on Linux.
All the logic lives in autozhuyin.core (shared with Windows and the tests); this file
only translates IBus key events and draws the preedit / candidate table.
Run modes:
autozhuyin_ibus.py --ibus started by ibus-daemon (from the component XML)
autozhuyin_ibus.py [--activate] started by the user/autostart: registers the engine
itself if ibus-daemon does not know it, and with
--activate makes it the current input method
"""
import json
import os
import sys
import time
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import gi # noqa: E402
gi.require_version("IBus", "1.0")
from gi.repository import GLib, GObject, IBus # noqa: E402
import autozhuyin # noqa: E402
from autozhuyin.core.history import UserHistory # noqa: E402
from autozhuyin.core.classifier import LITERAL, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import ( # noqa: E402
BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine,
)
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
ENGINE_NAME = "autozhuyin"
BUS_NAME = "org.freedesktop.IBus.AutoZhuyin"
VERSION = "0.0.0"
_DATA = os.path.join(os.path.dirname(os.path.abspath(autozhuyin.__file__)), "data")
_USER_DIR = os.path.join(os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config"), "autozhuyin")
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
try:
with open(os.path.join(HERE, "version.txt"), encoding="utf-8") as _f:
VERSION = _f.read().strip() or VERSION
except OSError:
pass
_classifier = None
def _save_json(path, data):
try:
os.makedirs(_USER_DIR, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False)
except OSError:
pass
def _load_json(path):
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError):
return {}
def _save_learned(weights):
_save_json(_LEARNED_FILE, weights)
def _save_history(data):
_save_json(_HISTORY_FILE, data)
def shared_classifier():
"""Loads the lexicon and English model once per process."""
global _classifier
if _classifier is None:
lexicon = Lexicon.from_file(os.path.join(_DATA, "lexicon.txt"))
if os.path.exists(_USER_PHRASES):
with open(_USER_PHRASES, encoding="utf-8") as f:
lexicon.load(f)
english = EnglishModel.from_files(
os.path.join(_DATA, "english-common.txt"),
os.path.join(_DATA, "english-dict.txt"),
os.path.join(_DATA, "english-bigram.json"),
)
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
return _classifier
_SPECIAL = {
IBus.KEY_BackSpace: BACKSPACE, IBus.KEY_Return: ENTER, IBus.KEY_KP_Enter: ENTER,
IBus.KEY_Escape: ESCAPE, IBus.KEY_Tab: TAB,
IBus.KEY_Up: UP, IBus.KEY_Down: DOWN, IBus.KEY_Left: LEFT, IBus.KEY_Right: RIGHT,
}
_SHIFT_KEYS = (IBus.KEY_Shift_L, IBus.KEY_Shift_R)
_SHORTCUT_MASK = (IBus.ModifierType.CONTROL_MASK | IBus.ModifierType.MOD1_MASK
| IBus.ModifierType.SUPER_MASK | IBus.ModifierType.MOD4_MASK)
_LITERAL_COLOR = 0x1E64DC # blue hint for English segments
class AutoZhuyinEngine(IBus.Engine):
__gtype_name__ = "AutoZhuyinEngine"
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._setup()
def _setup(self):
if getattr(self, "engine", None) is not None:
return
self.engine = CompositionEngine(shared_classifier())
self.shift_tap = ShiftTapDetector()
self.table = IBus.LookupTable.new(self.engine.page_size, 0, True, False)
self.table.set_orientation(IBus.Orientation.VERTICAL)
self.mode_prop = IBus.Property(
key="mode", prop_type=IBus.PropType.NORMAL,
label=IBus.Text.new_from_string("中英自動"),
symbol=IBus.Text.new_from_string("自"),
tooltip=IBus.Text.new_from_string("單按 Shift 切換英文模式"),
sensitive=True, visible=True, state=IBus.PropState.UNCHECKED)
self.props = IBus.PropList()
self.props.append(self.mode_prop)
# --- keys ---
def do_process_key_event(self, keyval, keycode, state):
self._setup()
released = bool(state & IBus.ModifierType.RELEASE_MASK)
# Shift pressed and released alone: English mode (or flip the current word).
if keyval in _SHIFT_KEYS:
now = time.monotonic()
if not released:
if state & _SHORTCUT_MASK:
self.shift_tap.key_pressed()
else:
self.shift_tap.shift_down(now)
return False
if self.shift_tap.shift_up(now):
self.engine.handle(SHIFT_TAP)
self._update(None)
self._update_mode()
return False
if released:
return False
self.shift_tap.key_pressed()
# Shortcuts (Ctrl/Alt/Super) go to the app; commit any composition first.
if state & _SHORTCUT_MASK:
if self.engine.keys:
self._update(self.engine.commit_all())
return False
key = self._key(keyval, state)
if key is None:
return self._busy() # swallow other keys (Delete, Home…) while composing
if not self.engine.wants(key):
return False
handled, commit = self.engine.handle(key)
self._update(commit)
return handled
def _key(self, keyval, state):
if keyval in _SPECIAL:
return _SPECIAL[keyval]
if keyval == IBus.KEY_space:
return " "
code = IBus.keyval_to_unicode(keyval)
if not code or not code.isprintable():
return None
# Caps Lock without Shift: keep Zhuyin working by using lowercase.
if state & IBus.ModifierType.LOCK_MASK and not state & IBus.ModifierType.SHIFT_MASK:
code = code.lower()
return code
def _busy(self):
return bool(self.engine.keys or self.engine.candidates)
# --- candidate panel (mouse) ---
def do_candidate_clicked(self, index, button, state):
self.engine.select_candidate(self.engine.page_start + index)
self._update(None)
def do_page_up(self):
return self._move_candidate(-self.engine.page_size)
def do_page_down(self):
return self._move_candidate(self.engine.page_size)
def do_cursor_up(self):
return self._move_candidate(-1)
def do_cursor_down(self):
return self._move_candidate(1)
def _move_candidate(self, delta):
if not self.engine.candidates:
return False
e = self.engine
e.candidate_index = max(0, min(len(e.candidates) - 1, e.candidate_index + delta))
self._update(None)
return True
# --- lifecycle ---
def do_focus_in(self):
self._setup()
self.register_properties(self.props)
self._update_mode()
def do_focus_out(self):
self._setup()
# PreeditFocusMode.COMMIT: IBus commits the shown text itself on focus out.
self.engine.reset()
self.hide_lookup_table()
def do_reset(self):
self._setup()
self.engine.reset()
self.hide_preedit_text()
self.hide_lookup_table()
def do_disable(self):
self.do_reset()
# --- drawing ---
def _update(self, commit):
if commit:
self.commit_text(IBus.Text.new_from_string(commit))
pieces = self.engine.marked_pieces
text = "".join(t for t, _ in pieces)
if text:
ibus_text = IBus.Text.new_from_string(text)
attrs = IBus.AttrList()
attrs.append(IBus.attr_underline_new(IBus.AttrUnderline.SINGLE, 0, len(text)))
pos = 0
for piece, mode in pieces:
if mode == LITERAL and piece:
attrs.append(IBus.attr_foreground_new(_LITERAL_COLOR, pos, pos + len(piece)))
pos += len(piece)
ibus_text.set_attributes(attrs)
self.update_preedit_text_with_mode(ibus_text, self.engine.cursor_offset, True,
IBus.PreeditFocusMode.COMMIT)
else:
self.hide_preedit_text()
if self.engine.candidates:
self.table.clear()
for _, candidate in self.engine.candidates:
self.table.append_candidate(IBus.Text.new_from_string(candidate))
self.table.set_cursor_pos(self.engine.candidate_index)
self.update_lookup_table(self.table, True)
else:
self.hide_lookup_table()
def _update_mode(self):
english = self.engine.english_mode
self.mode_prop.set_label(IBus.Text.new_from_string("英文" if english else "中英自動"))
self.mode_prop.set_symbol(IBus.Text.new_from_string("英" if english else "自"))
self.update_property(self.mode_prop)
# --- process setup ---
def make_component(exec_path):
component = IBus.Component(
name=BUS_NAME, description="自動注音 AutoZhuyin", version=VERSION,
license="Private", author="AutoZhuyin", homepage="",
command_line=f"{sys.executable} {exec_path} --ibus", textdomain="")
icon = os.path.join(HERE, "icon.png")
component.add_engine(IBus.EngineDesc(
name=ENGINE_NAME, longname="自動注音", description="大千注音+英文,自動判斷中英",
language="zh_TW", license="Private", author="AutoZhuyin",
icon=icon if os.path.exists(icon) else "", layout="us", symbol="自", rank=0))
return component
def main(argv):
IBus.init()
bus = IBus.Bus()
loop = GLib.MainLoop()
started_by_ibus = "--ibus" in argv
activate = "--activate" in argv
factory = None
def on_connected(*_):
nonlocal factory
factory = IBus.Factory.new(bus.get_connection())
factory.add_engine(ENGINE_NAME, GObject.type_from_name("AutoZhuyinEngine"))
if started_by_ibus:
bus.request_name(BUS_NAME, 0)
else:
known = any(e.get_name() == ENGINE_NAME for e in bus.list_engines())
if not known:
bus.register_component(make_component(os.path.abspath(__file__)))
if activate:
bus.set_global_engine(ENGINE_NAME)
def on_disconnected(*_):
if started_by_ibus:
loop.quit()
# Otherwise wait: IBus.Bus reconnects (and emits "connected") when ibus-daemon restarts.
bus.connect("connected", on_connected)
bus.connect("disconnected", on_disconnected)
shared_classifier() # load data before the first key
if bus.is_connected():
on_connected()
elif started_by_ibus:
print("autozhuyin: cannot connect to ibus-daemon", file=sys.stderr)
return 1
loop.run()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+59
View File
@@ -0,0 +1,59 @@
自動注音 AutoZhuyin(Linux 測試版,IBus)
=========================================
用大千注音鍵盤直接打字,不用切換中英文,輸入法會自動判斷你打的是注音還是英文。
按鍵: ji3 hello world su3cl3 (最後按空白)
輸出: 我hello world 你好
適用:Bazzite、SteamOS 桌面模式、Fedora 等有 IBus 的系統(KDE 或 GNOME)。
只在桌面模式可用;Steam 遊戲模式不支援系統輸入法。
安裝
----
1. 解壓縮,在終端機(Konsole)進到這個資料夾,執行:
./install.sh
不需要 sudo,檔案只會裝在你的家目錄(~/.local/share/autozhuyin)。
2. KDE(Bazzite / SteamOS 預設):
系統設定 → 鍵盤 → 虛擬鍵盤,選「IBus Wayland」→ 套用,然後登出再登入。
GNOME:IBus 是內建的,不用另外設定。
3. 之後每次登入都會自動切換到自動注音。
使用方式
--------
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
Enter 送出目前組字
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
← → 在組字區移動游標;游標在中間時直接打注音會插入新字,
Backspace 刪掉游標前的字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
選過的字會被記住,下次優先出現
Esc 取消組字
單按 Shift 切換英文模式(按鍵直接送出),再單按一次回到中英自動判斷
Shift+字母 打大寫字母(也會強制英文)
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
切換與移除
----------
暫時改回一般鍵盤: ibus engine xkb:us::eng
切回自動注音: ibus engine autozhuyin
移除: ./uninstall.sh
個人資料(學習紀錄、自訂詞)在 ~/.config/autozhuyin/:
learned.json 按 Tab 修正的學習紀錄
user-history.json 選字紀錄(選過的候選字)
user-phrases.txt 自訂詞,一行一個,例如:ㄗˋ-ㄉㄨㄥˋ 自動 -1
有問題時
--------
在終端機手動執行看看錯誤訊息:
/usr/bin/python3 ~/.local/share/autozhuyin/autozhuyin_ibus.py --activate
把畫面上的訊息回報給我。
+63
View File
@@ -0,0 +1,63 @@
#!/bin/bash
# Installs AutoZhuyin (自動注音) for the current user as an IBus input method.
# Nothing outside your home folder is touched, so it works on read-only systems
# (Bazzite, SteamOS, Fedora Atomic). No sudo needed.
set -euo pipefail
cd "$(dirname "$0")"
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/autozhuyin"
AUTOSTART="${XDG_CONFIG_HOME:-$HOME/.config}/autostart"
PY=/usr/bin/python3
echo "== 自動注音 AutoZhuyin 安裝 =="
# 1. IBus and its Python bindings must come with the system.
if ! command -v ibus >/dev/null 2>&1; then
echo "找不到 IBus(ibus 指令)。這個系統沒有內建 IBus,自動注音目前只支援 IBus。" >&2
exit 1
fi
if ! "$PY" -c 'import gi; gi.require_version("IBus", "1.0"); from gi.repository import IBus' 2>/dev/null; then
echo "系統的 Python 缺少 IBus 模組(python3-gobject / IBus-1.0 typelib)。" >&2
exit 1
fi
# 2. Copy the program.
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null || true
rm -rf "$DEST"
mkdir -p "$DEST"
cp -R autozhuyin autozhuyin_ibus.py version.txt "$DEST/"
[ -f icon.png ] && cp icon.png "$DEST/"
[ -f THIRD_PARTY_LICENSES.txt ] && cp THIRD_PARTY_LICENSES.txt "$DEST/"
chmod +x "$DEST/autozhuyin_ibus.py"
# 3. Start with the desktop session: registers the engine with IBus and selects it.
mkdir -p "$AUTOSTART"
cat > "$AUTOSTART/autozhuyin.desktop" <<DESKTOP
[Desktop Entry]
Type=Application
Name=自動注音 AutoZhuyin
Comment=Registers the AutoZhuyin input method with IBus
Exec=$PY $DEST/autozhuyin_ibus.py --activate
Icon=$DEST/icon.png
X-GNOME-Autostart-enabled=true
X-KDE-autostart-phase=2
NoDisplay=true
DESKTOP
# 4. Start it now as well.
if pgrep -x ibus-daemon >/dev/null 2>&1; then
nohup "$PY" "$DEST/autozhuyin_ibus.py" --activate >/dev/null 2>&1 &
sleep 1
echo "已啟動並切換到自動注音。"
else
echo "IBus 目前沒有在執行,請照下面的步驟啟用 IBus 後登出再登入。"
fi
cat <<'MSG'
安裝完成。
- KDE(Bazzite / SteamOS 桌面模式):系統設定 → 鍵盤 → 虛擬鍵盤,選「IBus Wayland」,
按套用後登出再登入。之後每次登入都會自動切換到自動注音。
- 想暫時改回一般鍵盤:ibus engine xkb:us::eng ;切回自動注音:ibus engine autozhuyin
- 移除:執行 uninstall.sh
MSG
+14
View File
@@ -0,0 +1,14 @@
#!/bin/bash
# Removes AutoZhuyin (自動注音) for the current user. Your learned corrections and
# user phrases in ~/.config/autozhuyin are kept (delete that folder to remove them).
set -uo pipefail
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/autozhuyin"
AUTOSTART="${XDG_CONFIG_HOME:-$HOME/.config}/autostart/autozhuyin.desktop"
if command -v ibus >/dev/null 2>&1 && [ "$(ibus engine 2>/dev/null)" = "autozhuyin" ]; then
ibus engine xkb:us::eng 2>/dev/null
fi
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null
rm -rf "$DEST" "$AUTOSTART"
echo "已移除自動注音。學習紀錄留在 ~/.config/autozhuyin,不需要可以自行刪除。"
+183
View File
@@ -0,0 +1,183 @@
"""Drives autozhuyin_ibus.AutoZhuyinEngine through a minimal stand-in for
gi.repository.IBus, to catch wrapper bugs without Linux. Run from the repo root:
python3 -m unittest discover -s linux/tests
"""
import os
import sys
import types
import unittest
LINUX = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
import tempfile # noqa: E402
os.environ["XDG_CONFIG_HOME"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
sys.path.insert(0, os.path.join(os.path.dirname(LINUX), "windows")) # autozhuyin package
sys.path.insert(0, LINUX)
# --- stand-in for gi / IBus (only what the wrapper uses) ---
class Flag(int):
def __or__(self, o): return Flag(int(self) | int(o))
def __and__(self, o): return Flag(int(self) & int(o))
IBus = types.SimpleNamespace()
for i, name in enumerate(["BackSpace", "Return", "KP_Enter", "Escape", "Tab", "Up", "Down", "Left",
"Right", "space", "Shift_L", "Shift_R", "Delete"]):
setattr(IBus, "KEY_" + name, 0xff00 + i)
IBus.KEY_space = 0x20
IBus.ModifierType = types.SimpleNamespace(
SHIFT_MASK=Flag(1), LOCK_MASK=Flag(2), CONTROL_MASK=Flag(4), MOD1_MASK=Flag(8),
SUPER_MASK=Flag(1 << 26), MOD4_MASK=Flag(64), RELEASE_MASK=Flag(1 << 30))
IBus.keyval_to_unicode = lambda k: chr(k) if 0x20 <= k < 0x7f else "\x00"
class Text:
def __init__(self, s): self.text, self.attrs = s, None
@staticmethod
def new_from_string(s): return Text(s)
def set_attributes(self, a): self.attrs = a
class AttrList(list):
pass
class LookupTable:
def __init__(self): self.items, self.cursor = [], 0
@staticmethod
def new(*a): return LookupTable()
def set_orientation(self, o): pass
def clear(self): self.items = []
def append_candidate(self, t): self.items.append(t.text)
def set_cursor_pos(self, p): self.cursor = p
class Property:
def __init__(self, **kw): self.label = kw["label"].text
def set_label(self, t): self.label = t.text
def set_symbol(self, t): pass
class Engine:
def __init__(self, *a, **kw):
self.committed, self.preedit, self.cursor, self.shown = "", "", 0, None
self.preedit_attrs = None
def commit_text(self, t): self.committed += t.text
def update_preedit_text_with_mode(self, t, cursor, visible, mode):
self.preedit, self.cursor, self.preedit_attrs = t.text, cursor, t.attrs
def hide_preedit_text(self): self.preedit = ""
def update_lookup_table(self, table, visible): self.shown = list(table.items)
def hide_lookup_table(self): self.shown = None
def register_properties(self, p): pass
def update_property(self, p): self.mode_label = p.label
IBus.Text, IBus.AttrList, IBus.LookupTable, IBus.Property, IBus.Engine = Text, AttrList, LookupTable, Property, Engine
IBus.PropList = lambda: types.SimpleNamespace(append=lambda p: None)
IBus.attr_underline_new = lambda kind, a, b: ("underline", a, b)
IBus.attr_foreground_new = lambda color, a, b: ("color", a, b)
IBus.AttrUnderline = types.SimpleNamespace(SINGLE=1)
IBus.Orientation = types.SimpleNamespace(VERTICAL=1)
IBus.PropType = types.SimpleNamespace(NORMAL=0)
IBus.PropState = types.SimpleNamespace(UNCHECKED=0)
IBus.PreeditFocusMode = types.SimpleNamespace(COMMIT=1)
gi = types.ModuleType("gi")
gi.require_version = lambda *a: None
repo = types.ModuleType("gi.repository")
repo.IBus, repo.GLib, repo.GObject = IBus, types.SimpleNamespace(), types.SimpleNamespace()
gi.repository = repo
sys.modules["gi"], sys.modules["gi.repository"] = gi, repo
import autozhuyin_ibus # noqa: E402
M = IBus.ModifierType
def press(e, keys):
"""Types keys; returns the list of 'handled' results."""
names = {"\n": IBus.KEY_Return, "\b": IBus.KEY_BackSpace, "\t": IBus.KEY_Tab, "←": IBus.KEY_Left,
"→": IBus.KEY_Right, "↓": IBus.KEY_Down, " ": IBus.KEY_space}
out = []
for c in keys:
kv = names.get(c, ord(c))
out.append(e.do_process_key_event(kv, 0, Flag(0)))
e.do_process_key_event(kv, 0, M.RELEASE_MASK)
return out
class IBusEngineTests(unittest.TestCase):
def setUp(self):
autozhuyin_ibus.shared_classifier().history.clear()
autozhuyin_ibus.shared_classifier().learned.clear()
self.e = autozhuyin_ibus.AutoZhuyinEngine()
def test_mixed_sentence(self):
press(self.e, "ji3 hello world su3cl3 ")
self.assertEqual(self.e.committed, "我hello world 你好")
self.assertEqual(self.e.preedit, "")
def test_preedit_marks_english_blue(self):
press(self.e, "ji3hello")
self.assertEqual(self.e.preedit, "我hello")
self.assertIn(("color", 1, 6), self.e.preedit_attrs)
self.assertEqual(self.e.cursor, 6)
def test_space_passes_through_when_idle(self):
self.assertEqual(press(self.e, " "), [False])
def test_candidates_and_click(self):
press(self.e, "su3↓")
self.assertTrue(self.e.shown)
second = self.e.shown[1]
self.e.do_candidate_clicked(1, 1, 0)
self.assertIsNone(self.e.shown)
press(self.e, "\n")
self.assertEqual(self.e.committed, second)
def test_shift_tap_toggles_english(self):
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, Flag(0))
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, M.SHIFT_MASK | M.RELEASE_MASK)
self.assertEqual(self.e.mode_label, "英文")
self.assertEqual(press(self.e, "a"), [False]) # goes to the app as typed
def test_shift_as_modifier_is_not_a_tap(self):
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, Flag(0))
self.e.do_process_key_event(ord("A"), 0, M.SHIFT_MASK)
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, M.SHIFT_MASK | M.RELEASE_MASK)
self.assertFalse(self.e.engine.english_mode)
def test_ctrl_shortcut_commits_and_passes(self):
press(self.e, "su3")
self.assertFalse(self.e.do_process_key_event(ord("c"), 0, M.CONTROL_MASK))
self.assertEqual(self.e.committed, "你")
def test_backtick_symbol(self):
press(self.e, "`<")
self.assertEqual(self.e.committed, ",")
def test_insert_in_middle(self):
press(self.e, "rup fu4←wu0 \n")
self.assertEqual(self.e.committed, "今天氣")
def test_picked_candidate_saved(self):
press(self.e, "su3↓")
second = self.e.shown[1]
self.e.do_candidate_clicked(1, 1, 0)
press(self.e, "\n")
path = os.path.join(os.environ["XDG_CONFIG_HOME"], "autozhuyin", "user-history.json")
with open(path, encoding="utf-8") as f:
self.assertIn(second, f.read())
press(self.e, "su3 \n")
self.assertTrue(self.e.committed.endswith(second))
def test_focus_out_resets(self):
press(self.e, "su3")
self.e.do_focus_out()
self.assertEqual(self.e.engine.keys, [])
if __name__ == "__main__":
unittest.main()
+7
View File
@@ -19,6 +19,13 @@ mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
# "AI 選字" language model (CKIP gpt2-tiny-chinese, GPL-3.0), fetched and converted once.
[ -s build/lm/lm.bin ] || scripts/fetch-lm.sh >/dev/null
cp build/lm/lm.bin build/lm/lm-vocab.txt licenses/GPL-3.0.txt "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
+18
View File
@@ -0,0 +1,18 @@
#!/bin/bash
# Downloads the CKIP gpt2-tiny-chinese language model (GPL-3.0,
# https://huggingface.co/ckiplab/gpt2-tiny-chinese) and converts it to build/lm/:
# lm.bin float16 weights (custom format, see training/lm/export_lm.py)
# lm-vocab.txt
# build.sh bundles them into the app when present ("AI 選字"). Safe to re-run.
set -euo pipefail
cd "$(dirname "$0")/.."
SRC=build/models/ckip-gpt2-tiny
BASE="https://huggingface.co/ckiplab/gpt2-tiny-chinese/resolve/refs%2Fpr%2F1"
mkdir -p "$SRC" build/lm
for f in model.safetensors vocab.txt config.json; do
[ -s "$SRC/$f" ] || curl -sfL -o "$SRC/$f" "$BASE/$f"
done
python3 -I training/lm/export_lm.py "$SRC" build/lm/lm.bin
cp "$SRC/vocab.txt" build/lm/lm-vocab.txt
ls -lh build/lm
+31
View File
@@ -0,0 +1,31 @@
#!/bin/bash
# Builds build/AutoZhuyin-Linux-<version>.tar.gz: the IBus engine, the shared Python
# core and data, and install/uninstall scripts (per-user install, no root needed).
set -euo pipefail
cd "$(dirname "$0")/.."
VERSION="$(python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["version"])' windows/autozhuyin/ime.json)"
NAME="AutoZhuyin-Linux-$VERSION"
OUT="build/$NAME"
python3 -I windows/tools/build_data.py
(cd windows && python3 -m unittest discover -q tests)
python3 -m unittest discover -q -s linux/tests
if [ ! -f build/icon.tiff ]; then
DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon.tiff
fi
rm -rf "$OUT" "$OUT.tar.gz"
mkdir -p "$OUT/autozhuyin"
rsync -a --exclude __pycache__ --exclude '*.pyc' \
windows/autozhuyin/__init__.py windows/autozhuyin/core windows/autozhuyin/data "$OUT/autozhuyin/"
cp linux/autozhuyin_ibus.py linux/installer/install.sh linux/installer/uninstall.sh linux/installer/README.txt \
THIRD_PARTY_LICENSES.txt "$OUT/"
echo "$VERSION" > "$OUT/version.txt"
sips -s format png -z 64 64 build/icon.tiff --out "$OUT/icon.png" >/dev/null
chmod +x "$OUT/install.sh" "$OUT/uninstall.sh" "$OUT/autozhuyin_ibus.py"
# Plain ustar without macOS extended attributes, so GNU tar doesn't warn.
(cd build && COPYFILE_DISABLE=1 tar --no-xattrs --format ustar -czf "$NAME.tar.gz" "$NAME")
echo "Built build/$NAME.tar.gz"
+3
View File
@@ -26,6 +26,9 @@ rsync -a --exclude __pycache__ --exclude '*.pyc' windows/autozhuyin "$OUT/"
for f in install.bat uninstall.bat README.txt; do for f in install.bat uninstall.bat README.txt; do
sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f" sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f"
done done
# Third-party notices: next to the installer and inside the installed folder.
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
(cd build && zip -qr -X "$NAME.zip" "$NAME") (cd build && zip -qr -X "$NAME.zip" "$NAME")
echo "Built build/$NAME.zip" echo "Built build/$NAME.zip"
+5 -3
View File
@@ -1,5 +1,5 @@
#!/bin/bash #!/bin/bash
# One-step release: bump the version, run all tests, build both installers, then # One-step release: bump the version, run all tests, build all installers, then
# commit the version change and tag it. # commit the version change and tag it.
# #
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch) # Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
@@ -38,14 +38,16 @@ xcrun swift test -q 2>&1 | tail -3
echo "==> Packaging (runs the Python tests too)" echo "==> Packaging (runs the Python tests too)"
scripts/package.sh >/dev/null scripts/package.sh >/dev/null
scripts/package-windows.sh >/dev/null scripts/package-windows.sh >/dev/null
scripts/package-linux.sh >/dev/null
trap - ERR trap - ERR
git add Resources/Info.plist windows/autozhuyin/ime.json git add Resources/Info.plist windows/autozhuyin/ime.json
git commit -q -m "Release v$VERSION" git commit -q -m "Release v$VERSION"
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION" git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
echo "==> Tagged v$VERSION" echo "==> Tagged v$VERSION"
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" FILES=("build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" "build/AutoZhuyin-Linux-$VERSION.tar.gz")
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" ls -lh "${FILES[@]}"
shasum -a 256 "${FILES[@]}"
if [ "$PUSH" = 1 ]; then if [ "$PUSH" = 1 ]; then
git push origin HEAD "v$VERSION" git push origin HEAD "v$VERSION"
+147
View File
@@ -0,0 +1,147 @@
"""Measures how much language-model reranking improves Zhuyin → Chinese conversion.
For each test sentence: turn the text into Dachen keystrokes (via the lexicon's
readings), convert with the current Viterbi converter (baseline), take the K best
paths, and rerank them with lexicon score + λ · LM log-probability.
Usage (needs numpy and the model in build/models/ckip-gpt2-tiny):
python training/lm/eval_rerank.py build/lm-eval/chat.txt build/lm-eval/wiki.txt
"""
import heapq
import math
import os
import sys
import time
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.insert(0, os.path.join(ROOT, "windows"))
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from autozhuyin.core import bopomofo # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
from gpt2 import GPT2 # noqa: E402
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
LN10 = math.log(10)
def reading_keys(reading):
out = []
for syl in reading.split("-"):
tone = " "
for ch in syl:
if ch in TONE_KEY:
tone = TONE_KEY[ch]
else:
out.append(KEY_OF[ch])
out.append(tone)
return out
def build_reverse(lex):
best = {}
for reading, entries in lex.exact.items():
for phrase, score in entries:
if phrase not in best or score > best[phrase][1]:
best[phrase] = (reading, score)
return best
def text_to_keys(text, reverse, max_len):
"""Most likely segmentation of the text into lexicon phrases → keystrokes."""
n = len(text)
best = [(-math.inf, None)] * (n + 1)
best[0] = (0.0, None)
for j in range(1, n + 1):
for i in range(max(0, j - max_len), j):
if best[i][0] == -math.inf:
continue
hit = reverse.get(text[i:j])
if hit:
s = best[i][0] + hit[1]
if s > best[j][0]:
best[j] = (s, (i, hit[0]))
if best[n][1] is None:
return None
readings, j = [], n
while j > 0:
i, r = best[j][1]
readings.append(r)
j = i
keys = []
for r in reversed(readings):
keys += reading_keys(r)
return keys
def kbest(conv, syllables, k):
"""K best (score, text) paths through the converter's lattice (scores in log10)."""
n = len(syllables)
max_len = max(1, conv.lexicon.max_phrase_length)
beams = [[] for _ in range(n + 1)]
beams[0] = [(0.0, "")]
for j in range(1, n + 1):
cand = []
for length in range(1, min(max_len, j) + 1):
i = j - length
if not beams[i]:
continue
for text, score in conv._options(syllables, i, j, {}):
for s, t in beams[i]:
cand.append((s + score, t + text))
beams[j] = heapq.nlargest(k, cand, key=lambda e: e[0])
return beams[n]
def char_errors(a, b):
return sum(x != y for x, y in zip(a, b)) + abs(len(a) - len(b))
def main(paths, k=10):
lex = Lexicon.from_file(os.path.join(ROOT, "windows/autozhuyin/data/lexicon.txt"))
conv = Converter(lex)
lm = GPT2(os.path.join(ROOT, "build/models/ckip-gpt2-tiny"))
reverse = build_reverse(lex)
cases = []
for path in paths:
for line in open(path, encoding="utf-8"):
text = line.strip().replace(",", "").replace("、", "")
if not text:
continue
keys = text_to_keys(text, reverse, lex.max_phrase_length)
if keys is None:
continue
syl = [s.syllable for s in bopomofo.parse(keys).syllables]
paths_k = kbest(conv, syl, k)
t0 = time.time()
lm_scores = [lm.logprob(t) for _, t in paths_k]
cases.append((os.path.basename(path), text, paths_k, lm_scores, time.time() - t0))
def evaluate(lam, subset):
chars = errs = sent_ok = 0
for _, text, paths_k, lm_scores, _ in subset:
best = max(range(len(paths_k)), key=lambda i: paths_k[i][0] * LN10 + lam * lm_scores[i])
out = paths_k[best][1]
e = char_errors(out, text)
chars += len(text)
errs += e
sent_ok += e == 0
return 1 - errs / chars, sent_ok / len(subset)
lams = [0, 0.1, 0.2, 0.3, 0.5, 0.7, 1.0, 1.5, 2.0]
for name in sorted({c[0] for c in cases}):
sub = [c for c in cases if c[0] == name]
oracle = sum(any(t == c[1] for _, t in c[2]) for c in sub) / len(sub)
print(f"\n== {name}: {len(sub)} sentences, top-{k} contains the answer: {oracle:.1%}")
for lam in lams:
ca, sa = evaluate(lam, sub)
print(f" λ={lam:<4} char acc {ca:.2%} sentence acc {sa:.1%}" + (" (baseline)" if lam == 0 else ""))
ms = sum(c[4] for c in cases) / len(cases) * 1000
print(f"\nLM time per sentence (top-{k}): {ms:.1f} ms")
return cases
if __name__ == "__main__":
main(sys.argv[1:])
+48
View File
@@ -0,0 +1,48 @@
"""Converts the gpt2-tiny-chinese safetensors file to the input method's format.
Format (little-endian):
"AZLM" magic, uint32 header length, JSON header, then float16 tensors back to back.
Header: {"n_layer", "n_head", "n_embd", "vocab", "eps", "tensors": [[name, [shape], offset]]}
(offset in float16 elements from the start of the data). Linear weights keep the
GPT-2 Conv1D layout [in, out]. The attention mask buffers are dropped.
Usage: export_lm.py <model dir> <out.bin> (standard library only)
"""
import array
import json
import struct
import sys
def main(src, out):
cfg = json.load(open(f"{src}/config.json"))
with open(f"{src}/model.safetensors", "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(n))
base = 8 + n
names = sorted(k for k, v in header.items() if k != "__metadata__" and v["dtype"] == "F32" and v["shape"])
tensors, blobs, offset = [], [], 0
for name in names:
info = header[name]
a, b = info["data_offsets"]
f.seek(base + a)
floats = array.array("f")
floats.frombytes(f.read(b - a))
if sys.byteorder != "little":
floats.byteswap()
half = struct.pack(f"<{len(floats)}e", *floats)
tensors.append([name.replace("transformer.", ""), info["shape"], offset])
blobs.append(half)
offset += len(floats)
meta = json.dumps({"n_layer": cfg["n_layer"], "n_head": cfg["n_head"], "n_embd": cfg["n_embd"],
"vocab": cfg["vocab_size"], "eps": cfg["layer_norm_epsilon"],
"tensors": tensors}).encode()
with open(out, "wb") as f:
f.write(b"AZLM" + struct.pack("<I", len(meta)) + meta)
for blob in blobs:
f.write(blob)
print(f"wrote {out}: {offset} weights")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
+83
View File
@@ -0,0 +1,83 @@
"""Minimal GPT-2 forward pass in numpy, for the CKIP gpt2-tiny-chinese model
(https://huggingface.co/ckiplab/gpt2-tiny-chinese, GPL-3.0). Character-level:
each Chinese character is one token of the bert-base-chinese vocabulary.
Used to evaluate language-model reranking before porting it to the input methods.
"""
import json
import math
import struct
import numpy as np
CLS, UNK = 101, 100
def load_safetensors(path):
with open(path, "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(n))
base = 8 + n
out = {}
for name, info in header.items():
if name == "__metadata__" or info["dtype"] != "F32":
continue
a, b = info["data_offsets"]
f.seek(base + a)
out[name] = np.frombuffer(f.read(b - a), dtype=np.float32).reshape(info["shape"])
return out
class GPT2:
def __init__(self, model_dir):
self.w = load_safetensors(f"{model_dir}/model.safetensors")
cfg = json.load(open(f"{model_dir}/config.json"))
self.n_layer, self.n_head = cfg["n_layer"], cfg["n_head"]
self.eps = cfg["layer_norm_epsilon"]
self.vocab = {}
for i, line in enumerate(open(f"{model_dir}/vocab.txt", encoding="utf-8")):
self.vocab.setdefault(line.rstrip("\n"), i)
def ids(self, text):
return [self.vocab.get(c, UNK) for c in text]
def _ln(self, x, name):
mu = x.mean(-1, keepdims=True)
var = ((x - mu) ** 2).mean(-1, keepdims=True)
return (x - mu) / np.sqrt(var + self.eps) * self.w[name + ".weight"] + self.w[name + ".bias"]
def hidden(self, ids):
"""Final hidden states [T, d] for token ids."""
w, T = self.w, len(ids)
x = w["transformer.wte.weight"][ids] + w["transformer.wpe.weight"][:T]
mask = np.triu(np.full((T, T), -1e9, dtype=np.float32), 1)
for l in range(self.n_layer):
p = f"transformer.h.{l}."
h = self._ln(x, p + "ln_1")
qkv = h @ w[p + "attn.c_attn.weight"] + w[p + "attn.c_attn.bias"]
q, k, v = np.split(qkv, 3, axis=-1)
d = q.shape[-1] // self.n_head
q = q.reshape(T, self.n_head, d).transpose(1, 0, 2)
k = k.reshape(T, self.n_head, d).transpose(1, 0, 2)
v = v.reshape(T, self.n_head, d).transpose(1, 0, 2)
att = q @ k.transpose(0, 2, 1) / math.sqrt(d) + mask
att = np.exp(att - att.max(-1, keepdims=True))
att /= att.sum(-1, keepdims=True)
a = (att @ v).transpose(1, 0, 2).reshape(T, -1)
x = x + a @ w[p + "attn.c_proj.weight"] + w[p + "attn.c_proj.bias"]
h = self._ln(x, p + "ln_2")
h = h @ w[p + "mlp.c_fc.weight"] + w[p + "mlp.c_fc.bias"]
h = 0.5 * h * (1 + np.tanh(math.sqrt(2 / math.pi) * (h + 0.044715 * h ** 3))) # gelu_new
x = x + h @ w[p + "mlp.c_proj.weight"] + w[p + "mlp.c_proj.bias"]
return self._ln(x, "transformer.ln_f")
def logprob(self, text, context=""):
"""Natural-log probability of `text` given `context` (both strings)."""
ctx, tgt = self.ids(context), self.ids(text)
ids = [CLS] + ctx + tgt
h = self.hidden(ids[:-1])
logits = h @ self.w["transformer.wte.weight"].T
logits -= logits.max(-1, keepdims=True)
lse = np.log(np.exp(logits).sum(-1))
start = len(ctx)
return float(sum(logits[start + i, t] - lse[start + i] for i, t in enumerate(tgt)))
+6 -6
View File
@@ -23,14 +23,14 @@ def main():
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH)) clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"), gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7) os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f) for k in HAND_PICKED for f in (False, True)] cases = [(k, f, p) for k in HAND_PICKED for f in (False, True) for p in (None, "literal")]
while len(cases) < 400: while len(cases) < 500:
ex = gen.example() ex = gen.example()
if len(ex["keys"]) <= 28: if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"])) cases.append((ex["keys"], ex["finishing"], ex["previous"]))
cases.append((gen.prefix_of(ex)["keys"], False)) cases.append((gen.prefix_of(ex)["keys"], False, ex["previous"]))
fixtures = [{"keys": k, "finishing": f, fixtures = [{"keys": k, "finishing": f, "previous": p,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f)]} for k, f in cases] "segments": [[s, e, m] for s, e, m in clf.segment(list(k), f, p)]} for k, f, p in cases]
with open(OUT, "w", encoding="utf-8") as fh: with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0) json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT)) print("wrote %d fixtures to %s" % (len(fixtures), OUT))
+5114 -3374
View File
File diff suppressed because it is too large Load Diff
+17 -2
View File
@@ -102,16 +102,31 @@ class Generator:
labels = "".join(m * len(p) for m, p in parts) labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l": if parts[-1][0] == "l":
finishing = True # English / numbers end with a space finishing = True # English / numbers end with a space
elif keys[-1] == " " and self.rng.random() < 0.6:
# The moment the space (first tone) is pressed: the buffer has no space yet and
# the input method decides "first tone" vs "English word + space".
keys, labels, finishing = keys[:-1], labels[:-1], True
kind += "/tone1"
else: else:
# Chinese ends with Enter, or space after a finished (toned) syllable. # Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5 finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind} return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind,
"previous": self.previous_mode(labels[0])}
def previous_mode(self, first_label):
"""Mode of the text committed just before: none, mostly the same, sometimes the other."""
r = self.rng.random()
if r < 0.35:
return None
same = core.ZHUYIN if first_label == "z" else core.LITERAL
other = core.LITERAL if first_label == "z" else core.ZHUYIN
return same if r < 0.80 else other
def prefix_of(self, ex): def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing.""" """The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1 cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False, return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix"} "kind": ex["kind"] + "/prefix", "previous": ex["previous"]}
def _cumulative(ws): def _cumulative(ws):
+19 -10
View File
@@ -45,12 +45,12 @@ def labels_of(segs, n):
return "".join(lab) return "".join(lab)
def seg_features(clf, keys, segs, finishing): def seg_features(clf, keys, segs, finishing, previous):
"""Summed features of a segmentation, or None if any segment is impossible.""" """Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float) total = defaultdict(float)
n = len(keys) n = len(keys)
for idx, (s, e, m) in enumerate(segs): for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing) fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing, previous if idx == 0 else None)
f = fz if m == core.ZHUYIN else fl f = fz if m == core.ZHUYIN else fl
if f is None: if f is None:
return None return None
@@ -62,12 +62,13 @@ def seg_features(clf, keys, segs, finishing):
def evaluate(clf, weights, data): def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0]) by_kind = defaultdict(lambda: [0, 0])
for ex in data: for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], None, weights) segs = clf.segment(list(ex["keys"]), ex["finishing"], ex["previous"], weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"] ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
bucket = by_kind["prefix" if ex["kind"].endswith("/prefix") else ex["kind"]] kind = "prefix" if ex["kind"].endswith("/prefix") else ex["kind"]
bucket[0] += ok for k in (kind, "after-other") if _after_other(ex) else (kind,):
bucket[1] += 1 by_kind[k][0] += ok
commit = [v for k, v in by_kind.items() if k != "prefix"] by_kind[k][1] += 1
commit = [v for k, v in by_kind.items() if k not in ("prefix", "after-other")]
summary = { summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)), "commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]), "prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
@@ -76,6 +77,14 @@ def evaluate(clf, weights, data):
return summary return summary
def _after_other(ex):
"""Committed buffer whose first segment differs from the text typed just before."""
if ex["kind"].endswith("/prefix") or ex["previous"] is None:
return False
first = core.ZHUYIN if ex["labels"][0] == "z" else core.LITERAL
return ex["previous"] != first
def show(name, s): def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items()) kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds)) print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
@@ -92,11 +101,11 @@ def train(clf, data, epochs, lr, seed):
t0 = time.time() t0 = time.time()
for ex in data: for ex in data:
keys = list(ex["keys"]) keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], None, w) pred = clf.segment(keys, ex["finishing"], ex["previous"], w)
if labels_of(pred, len(keys)) != ex["labels"]: if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"]) gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"]) fg = seg_features(clf, keys, gold, ex["finishing"], ex["previous"])
fp = seg_features(clf, keys, pred, ex["finishing"]) fp = seg_features(clf, keys, pred, ex["finishing"], ex["previous"])
if fg is not None and fp is not None: if fg is not None and fp is not None:
mistakes += 1 mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0) scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
@@ -0,0 +1 @@
{"ㄋㄧ": {"妳": [1, 1]}}
+25 -12
View File
@@ -11,6 +11,7 @@ import time
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME) from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
from textService import TextService from textService import TextService
from .core.history import UserHistory
from .core.classifier import Classifier, LearnedPreferences from .core.classifier import Classifier, LearnedPreferences
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
from .core.english import EnglishModel from .core.english import EnglishModel
@@ -22,19 +23,36 @@ _DATA = os.path.join(_HERE, "data")
_USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin") _USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin")
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json") _LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt") _USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
_classifier = None _classifier = None
def _save_learned(weights): def _save_json(path, data):
try: try:
os.makedirs(_USER_DIR, exist_ok=True) os.makedirs(_USER_DIR, exist_ok=True)
with open(_LEARNED_FILE, "w", encoding="utf-8") as f: with open(path, "w", encoding="utf-8") as f:
json.dump(weights, f, ensure_ascii=False) json.dump(data, f, ensure_ascii=False)
except OSError: except OSError:
pass pass
def _load_json(path):
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError):
return {}
def _save_learned(weights):
_save_json(_LEARNED_FILE, weights)
def _save_history(data):
_save_json(_HISTORY_FILE, data)
def _shared_classifier(): def _shared_classifier():
"""Loads the lexicon and English model once per PIME server process.""" """Loads the lexicon and English model once per PIME server process."""
global _classifier global _classifier
@@ -48,14 +66,9 @@ def _shared_classifier():
os.path.join(_DATA, "english-dict.txt"), os.path.join(_DATA, "english-dict.txt"),
os.path.join(_DATA, "english-bigram.json"), os.path.join(_DATA, "english-bigram.json"),
) )
weights = {}
try:
with open(_LEARNED_FILE, encoding="utf-8") as f:
weights = json.load(f)
except (OSError, ValueError):
pass
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json")) trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned), trained) _classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
return _classifier return _classifier
@@ -142,8 +155,8 @@ class AutoZhuyinTextService(TextService):
return False return False
self.engine.handle(SHIFT_TAP) self.engine.handle(SHIFT_TAP)
self._update(None) self._update(None)
if self.engine.english_armed: if not self.engine.keys:
self.showMessage("英文", 1) self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
return True return True
# --- replies --- # --- replies ---
+9 -11
View File
@@ -1,6 +1,7 @@
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift.""" """Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
import json import json
from .history import UserHistory
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
ZHUYIN = "zhuyin" ZHUYIN = "zhuyin"
@@ -30,12 +31,12 @@ class LearnedPreferences:
class Classifier: class Classifier:
context_weight = 1.5
def __init__(self, lexicon, english, learned=None, weights=None): def __init__(self, lexicon, english, learned=None, weights=None, history=None):
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned or LearnedPreferences() self.learned = learned or LearnedPreferences()
self.history = history or UserHistory() # picked candidates (shared by all engines)
self.weights = dict(DEFAULT_WEIGHTS) self.weights = dict(DEFAULT_WEIGHTS)
if weights: if weights:
self.weights.update(weights) self.weights.update(weights)
@@ -49,20 +50,17 @@ class Classifier:
except (OSError, ValueError): except (OSError, ValueError):
return {} return {}
def segment_features(self, seg, is_last, after_other, finishing): def segment_features(self, seg, is_last, after_other, finishing, previous=None):
"""(zhuyin features or None, literal features or None) for one segment.""" """(zhuyin features or None, literal features or None) for one segment.
`previous` (mode of the text committed just before) only affects the first segment."""
seg = list(seg) seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing), return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing, previous),
literal_features(seg, self.english, is_last, after_other, finishing)) literal_features(seg, self.english, is_last, after_other, finishing, previous))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights): def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing) fz, fl = self.segment_features(seg, is_last, after_other, finishing, previous)
zh = dot(weights, fz) if fz is not None else NEG_INF zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF lit = dot(weights, fl) if fl is not None else NEG_INF
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
if whole: if whole:
b = self.learned.bias("".join(seg)) b = self.learned.bias("".join(seg))
if b > 0: if b > 0:
+313 -73
View File
@@ -8,6 +8,7 @@ import time
from . import bopomofo from . import bopomofo
from .classifier import LITERAL, ZHUYIN from .classifier import LITERAL, ZHUYIN
from .features import dot
from .lexicon import Converter from .lexicon import Converter
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>" BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
@@ -20,6 +21,21 @@ FULLWIDTH = {
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
} }
# The first word of a composition that also reads as Zhuyin is not committed as English
# on space; the next word decides ("up jo4" = 因為, "up the" = up the, "no su3" = no 你).
# Right after English text only common one-syllable readings are held (up = 因, url = 教),
# so normal English typing is not delayed.
HOLD_MIN_SCORE = -3.5
# "`" then one of these keys always gives the full-width symbol, whatever the context.
# "`" followed by anything else is just a backtick (so ``` still works).
BACKTICK = "`"
SYMBOLS = {
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", '"': "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
}
PASS = (False, None) PASS = (False, None)
CONSUMED = (True, None) CONSUMED = (True, None)
@@ -30,19 +46,25 @@ class CompositionEngine:
def __init__(self, classifier, now=time.monotonic): def __init__(self, classifier, now=time.monotonic):
self.classifier = classifier self.classifier = classifier
self.converter = Converter(classifier.lexicon) self.converter = Converter(classifier.lexicon, classifier.history)
self.now = now self.now = now
self.keys = [] self.keys = []
self.forced_mode = None self.forced_mode = None
# True when forced_mode corrects the classifier (learned on commit); False when the # English mode, toggled by a Shift tap when nothing is being typed: keys go straight
# user forced English before typing (Shift tap on an empty buffer). # to the app (no composition, no underline) until Shift is tapped again.
self._forced_is_correction = False self.english_mode = False
self.backtick = False # "`" typed; the next key may be a forced symbol
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text} self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = [] self.candidates = []
self.candidate_index = 0 self.candidate_index = 0
self._candidate_segment_start = 0 self._candidate_segment_start = 0
# Composition cursor for choosing candidates: index into _cursor_stops(), None = end. # Composition cursor: a key index into self.keys (None = at the end). ← → move it
# between syllables; ↓ opens candidates for the phrase before it; typing there
# inserts a syllable (kept in self.inserting until its tone key, then spliced in).
self.cursor = None self.cursor = None
self.inserting = []
# Index of a held space (see HOLD_MIN_SCORE), or None.
self.held = None
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None self._cache_key = None
@@ -60,12 +82,55 @@ class CompositionEngine:
if self.forced_mode: if self.forced_mode:
return [(0, len(keys), self.forced_mode)] return [(0, len(keys), self.forced_mode)]
ctx = self.context ctx = self.context
cache_key = (tuple(keys), finishing, ctx) held = self.held if self.held is not None and self.held < len(keys) and keys[self.held] == " " else None
cache_key = (tuple(keys), finishing, ctx, held)
if cache_key != self._cache_key: if cache_key != self._cache_key:
self._cache_key = cache_key self._cache_key = cache_key
self._cache_value = self.classifier.segment(keys, finishing, ctx) if held is None:
self._cache_value = self.classifier.segment(keys, finishing, ctx)
else:
self._cache_value = self._held_segments(keys, held, finishing)
return self._cache_value return self._cache_value
def _held_segments(self, keys, held, finishing):
"""The held word is Chinese if the words typed after it start with Chinese."""
rest = keys[held + 1:]
base = held + 1
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
if rest_segs and rest_segs[0][2] == ZHUYIN and self._held_reads_as_chinese(keys, held, rest_segs[0][1], finishing):
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
return [(0, base, LITERAL)] + rest_segs
def _held_reads_as_chinese(self, keys, held, end, finishing):
"""Chinese all the way to `end` vs. the held English word followed by Chinese."""
c, w, ctx = self.classifier, self.classifier.weights, self.context
is_last = end == len(keys)
zh_whole = c.segment_features(keys[:end], is_last, False, finishing, ctx)[0]
word = c.segment_features(keys[:held], False, False, True, ctx)[1]
zh_rest = c.segment_features(keys[held + 1:end], is_last, True, finishing, None)[0]
if zh_whole is None:
return False
if word is None or zh_rest is None:
return True
return dot(w, zh_whole) >= dot(w, word) + dot(w, zh_rest)
def _is_ambiguous_word(self, keys):
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
if not all(k.isascii() and k.isalpha() for k in keys):
return False
p = bopomofo.parse(list(keys) + [" "])
if p.pending is not None or not p.syllables or p.stray_tones or p.foreign_keys:
return False
best = []
for ps in p.syllables:
entries = self.classifier.lexicon.entries([ps.syllable])
if not entries:
return False
best.append(entries[0][1])
if self.context != LITERAL:
return True # first word: hold anything that reads as Zhuyin
return len(best) == 1 and best[0] >= HOLD_MIN_SCORE
@property @property
def segments(self): def segments(self):
return self._segments(self.keys) return self._segments(self.keys)
@@ -105,42 +170,180 @@ class CompositionEngine:
# --- cursor --- # --- cursor ---
def _cursor_stops(self): def _syllable_ends(self, seg):
"""[(segment index, syllables before the stop)] for every Zhuyin syllable.""" """Absolute key index after each syllable of a Zhuyin segment."""
segs = self.segments return [seg[0] + ps.end for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables]
out = []
for i, seg in enumerate(segs): def _cursor_positions(self):
"""Key indices the cursor can stop at: after every Zhuyin syllable and at every
segment boundary, but never before the first Chinese character (there would be
nothing to pick candidates for)."""
out, ends = {len(self.keys)}, []
for seg in self.segments:
out.add(seg[1])
if seg[2] == ZHUYIN: if seg[2] == ZHUYIN:
ends += self._syllable_ends(seg)
out.update(ends)
first = min(ends) if ends else len(self.keys)
return sorted(p for p in out if p >= first)
def _stop_at(self, pos):
"""(segment index, syllables before pos) for the Zhuyin segment pos is in."""
segs = self.segments
for i, seg in enumerate(segs):
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
n = len(self._convertible(seg, i < len(segs) - 1)[0]) n = len(self._convertible(seg, i < len(segs) - 1)[0])
out.extend((i, k) for k in range(1, n + 1)) local = sum(1 for e in self._syllable_ends(seg) if e <= pos)
return out return (i, min(local, n)) if local else None
return None
def _candidate_stop(self):
"""Where ↓ opens candidates: before the cursor, or the last syllable."""
if self.cursor is not None:
return self._stop_at(self.cursor)
segs = self.segments
for i in range(len(segs) - 1, -1, -1):
if segs[i][2] == ZHUYIN:
n = len(self._convertible(segs[i], i < len(segs) - 1)[0])
if n:
return i, n
return None
def _base_cursor_offset(self):
"""Cursor position in the rendered segments (without the syllable being inserted)."""
pieces = self._segment_pieces()
if self.cursor is None:
return sum(len(t) for t, _ in pieces)
segs = self.segments
for i, seg in enumerate(segs):
if not seg[0] < self.cursor <= seg[1]:
continue
offset = sum(len(t) for t, _ in pieces[:i])
if seg[2] == LITERAL:
return offset + self.cursor - seg[0]
stop = self._stop_at(self.cursor)
local = stop[1] if stop else 0
for (a, b), text in self._conversion(seg, i < len(segs) - 1)[0]:
if b <= local:
offset += len(text)
elif a < local:
offset += min(local - a, len(text))
return offset
return 0
@property @property
def cursor_offset(self): def cursor_offset(self):
"""Cursor position in marked_text (characters).""" """Cursor position in marked_text (characters)."""
pieces = self.marked_pieces return self._base_cursor_offset() + len(self._inserting_text())
stops = self._cursor_stops()
if self.cursor is None or not 0 <= self.cursor < len(stops):
return sum(len(t) for t, _ in pieces)
seg_index, local = stops[self.cursor]
offset = sum(len(t) for t, _ in pieces[:seg_index])
segs = self.segments
for (a, b), text in self._conversion(segs[seg_index], seg_index < len(segs) - 1)[0]:
if b <= local:
offset += len(text)
elif a < local:
offset += min(local - a, len(text))
return offset
def _move_cursor(self, delta): def _move_cursor(self, delta):
stops = self._cursor_stops() self.inserting = []
if not stops: stops = self._cursor_positions()
return here = len(self.keys) if self.cursor is None else self.cursor
nxt = (len(stops) - 1 if self.cursor is None else self.cursor) + delta if delta < 0:
self.cursor = None if nxt >= len(stops) - 1 else max(0, nxt) before = [p for p in stops if p < here]
nxt = before[-1] if before else here
else:
after = [p for p in stops if p > here]
nxt = after[0] if after else here
self.cursor = None if nxt >= len(self.keys) else nxt
def _inserting_text(self):
if not self.inserting:
return ""
return "".join(ps.syllable.reading for ps in bopomofo.parse(self.inserting).syllables)
# --- editing in the middle ---
def _trim_fixed(self, pos):
"""Keeps picked phrases that end at or before key index pos (they don't move)."""
kept = {}
for seg in self.segments:
picks = self.fixed.get(seg[0])
if not picks or seg[2] != ZHUYIN:
continue
if seg[1] <= pos:
kept[seg[0]] = picks
elif seg[0] < pos:
ends = self._syllable_ends(seg)
here = {r: t for r, t in picks.items() if r[1] <= len(ends) and ends[r[1] - 1] <= pos}
if here:
kept[seg[0]] = here
self.fixed = kept
def _splice(self, new_keys):
"""Inserts keys at the cursor and moves the cursor after them."""
pos = self.cursor
self._trim_fixed(pos)
self.keys[pos:pos] = new_keys
if self.held is not None and self.held >= pos:
self.held += len(new_keys)
self.cursor = pos + len(new_keys)
if self.cursor >= len(self.keys):
self.cursor = None
def _delete_before_cursor(self):
"""Backspace in the middle: removes the syllable (or literal key) before the cursor."""
pos = self.cursor
start = pos - 1
for seg in self.segments:
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables:
if seg[0] + ps.end == pos:
start = seg[0] + ps.start
self._trim_fixed(start)
del self.keys[start:pos]
if self.held is not None:
if start <= self.held < pos:
self.held = None
elif self.held >= pos:
self.held -= pos - start
self.cursor = start
if not self.keys:
self.reset()
def _insert_key(self, c):
"""Typing with the cursor in the middle: compose one syllable, splice it in on its tone."""
if c == " ":
if not self.inserting:
return True, self._commit()
self._splice(self.inserting + [" "])
self.inserting = []
return CONSUMED
self.inserting.append(c)
p = bopomofo.parse(self.inserting)
if p.stray_tones or p.foreign_keys:
self.inserting.pop() # a tone with nothing to attach to
return CONSUMED
if p.syllables and p.syllables[-1].toned and len(p.syllables) == 1:
self._splice(self.inserting)
self.inserting = []
elif len(p.syllables) > 1:
# A new syllable started without a tone: splice the first one as it is.
first = p.syllables[0].end
self._splice(self.inserting[:first])
self.inserting = self.inserting[first:]
return CONSUMED
@property @property
def marked_pieces(self): def marked_pieces(self):
pieces = self._segment_pieces()
if self.inserting:
# Show the syllable being inserted at the cursor.
at, out, done = self._base_cursor_offset(), [], False
for text, mode in pieces:
if not done and at <= len(text):
out += [(text[:at], mode), (self._inserting_text(), ZHUYIN), (text[at:], mode)]
done = True
else:
out.append((text, mode))
at -= len(text)
pieces = [(t, m) for t, m in out if t]
if self.backtick:
pieces.append((BACKTICK, LITERAL))
return pieces
def _segment_pieces(self):
segs = self.segments segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)] return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
@@ -150,28 +353,20 @@ class CompositionEngine:
# --- key handling --- # --- key handling ---
@property
def english_armed(self):
"""True while the user has forced English for the next word (Shift tap). In this
mode keys go straight to the app with no composition (no underline)."""
return not self.keys and self.forced_mode == LITERAL and not self._forced_is_correction
def _end_direct_english(self):
if self.english_armed:
self.forced_mode = None
self._previous_mode = LITERAL
self._previous_time = self.now()
def _flip(self): def _flip(self):
"""Forces the whole buffer into the other mode (a correction that is learned)."""
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
self._forced_is_correction = True
def wants(self, key): def wants(self, key):
"""Whether handle(key) would consume the key, without changing state.""" """Whether handle(key) would consume the key, without changing state."""
if self.candidates or key == SHIFT_TAP or self.english_armed: if self.candidates or key == SHIFT_TAP or self.backtick or self.inserting:
return True return True
if self.english_mode and not self.keys:
return False # English mode: everything goes to the app as typed
if key in SPECIAL_KEYS: if key in SPECIAL_KEYS:
return bool(self.keys) return bool(self.keys)
if key == BACKTICK:
return True
if self.keys: if self.keys:
return True return True
if key == " ": if key == " ":
@@ -183,48 +378,60 @@ class CompositionEngine:
def handle(self, key): def handle(self, key):
if self.candidates: if self.candidates:
return self._handle_candidate_key(key) return self._handle_candidate_key(key)
if self.backtick:
return self._after_backtick(key)
if self.inserting and key in (BACKSPACE, ESCAPE):
if key == BACKSPACE:
self.inserting.pop()
else:
self.inserting = []
return CONSUMED
if key == BACKSPACE: if key == BACKSPACE:
if not self.keys: if not self.keys:
return PASS return PASS
self.cursor = None if self.cursor is not None:
if self.cursor > 0:
self._delete_before_cursor()
return CONSUMED
self.keys.pop() self.keys.pop()
if self.held is not None and len(self.keys) <= self.held:
self.held = None
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)} self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys: if not self.keys:
# A Shift-tap "next word is English" stays armed; a correction is dropped. self.forced_mode = None
if self._forced_is_correction:
self.forced_mode = None
self._forced_is_correction = False
self.fixed = {} self.fixed = {}
return CONSUMED return CONSUMED
if key == ENTER: if key == ENTER:
if not self.keys: if not self.keys:
self._end_direct_english()
return PASS return PASS
if self.inserting:
self._splice(self.inserting)
self.inserting = []
return True, self._commit() return True, self._commit()
if key == ESCAPE: if key == ESCAPE:
if not self.keys: if not self.keys:
self._end_direct_english()
return PASS return PASS
self.reset() self.reset()
return CONSUMED return CONSUMED
if key == TAB: if key == TAB:
if not self.keys: if not self.keys:
self._end_direct_english()
return PASS return PASS
self._flip() self._flip()
return CONSUMED return CONSUMED
if key == SHIFT_TAP: if key == SHIFT_TAP:
if not self.keys: if not self.keys:
# Next word is English, typed as-is. Tap again to cancel. # Toggle English mode; it stays on until the next Shift tap.
self.forced_mode = None if self.forced_mode == LITERAL else LITERAL self.english_mode = not self.english_mode
self._forced_is_correction = False self._previous_mode = LITERAL if self.english_mode else None
self._previous_time = self.now()
else: else:
self._flip() self._flip()
return CONSUMED return CONSUMED
if key == DOWN: if key == DOWN:
if not self.keys: if not self.keys:
return PASS return PASS
self._open_candidates() if not self.inserting:
self._open_candidates()
return CONSUMED return CONSUMED
if key in (LEFT, RIGHT): if key in (LEFT, RIGHT):
if not self.keys: if not self.keys:
@@ -235,6 +442,19 @@ class CompositionEngine:
return CONSUMED if self.keys else PASS return CONSUMED if self.keys else PASS
return self._handle_character(key) return self._handle_character(key)
def _after_backtick(self, key):
self.backtick = False
if key in (ESCAPE, BACKSPACE):
return CONSUMED
committed = self._commit() if self.keys else ""
if key in SYMBOLS:
return True, committed + SYMBOLS[key]
# Not a symbol: the backtick was meant literally; then handle the key as usual.
handled, more = self.handle(key)
if not handled and len(key) == 1:
handled, more = True, key
return handled, committed + BACKTICK + (more or "")
def _last_parse(self, segs): def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN: if not segs or segs[-1][2] != ZHUYIN:
return None return None
@@ -242,21 +462,27 @@ class CompositionEngine:
return bopomofo.parse(self.keys[start:end]) return bopomofo.parse(self.keys[start:end])
def _handle_character(self, c): def _handle_character(self, c):
self.cursor = None # typing always continues at the end if self.cursor is not None and self.keys and (c == " " or bopomofo.is_zhuyin_key(c)):
if self.english_armed: return self._insert_key(c)
# Forced English: insert directly. Space ends the word; other symbols end it too. if self.inserting:
if c == " ": self._splice(self.inserting)
self._end_direct_english() self.inserting = []
return True, " " self.cursor = None # anything else continues at the end
if c.isascii() and (c.isalnum() or c in "'.,-/;"): if self.english_mode and not self.keys:
return True, c return PASS # English mode: every key goes to the app as typed
self._end_direct_english() if c == BACKTICK:
return PASS self.backtick = True
return CONSUMED
if c == " ": if c == " ":
if not self.keys: if not self.keys:
return PASS return PASS
segs = self._segments(self.keys, finishing=True) segs = self._segments(self.keys, finishing=True)
if segs[-1][2] == LITERAL: if segs[-1][2] == LITERAL:
if (self.held is None and len(segs) == 1 and not self.forced_mode
and self._is_ambiguous_word(self.keys)):
self.held = len(self.keys)
self.keys.append(" ")
return CONSUMED
return True, self._commit(segs) + " " return True, self._commit(segs) + " "
p = self._last_parse(segs) p = self._last_parse(segs)
if p is not None and p.pending is not None: if p is not None and p.pending is not None:
@@ -285,10 +511,10 @@ class CompositionEngine:
def _open_candidates(self): def _open_candidates(self):
"""Candidates for the phrase ending at the cursor (default: the last syllable).""" """Candidates for the phrase ending at the cursor (default: the last syllable)."""
stops = self._cursor_stops() stop = self._candidate_stop()
if not stops: if stop is None:
return return
seg_index, local = stops[min(len(stops) - 1 if self.cursor is None else self.cursor, len(stops) - 1)] seg_index, local = stop
segs = self.segments segs = self.segments
seg = segs[seg_index] seg = segs[seg_index]
syl = self._convertible(seg, seg_index < len(segs) - 1)[0] syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
@@ -329,20 +555,34 @@ class CompositionEngine:
def _commit(self, segs=None): def _commit(self, segs=None):
segs = segs if segs is not None else self.segments segs = segs if segs is not None else self.segments
text = "".join(self._render(s, True) for s in segs) text = "".join(self._render(s, True) for s in segs)
if self.forced_mode and self._forced_is_correction: self._remember_picks(segs)
if self.forced_mode:
self.classifier.learned.record("".join(self.keys), self.forced_mode) self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = segs[-1][2] if segs else None self._previous_mode = segs[-1][2] if segs else None
self._previous_time = self.now() self._previous_time = self.now()
self.reset() self.reset()
return text return text
def _remember_picks(self, segs):
"""Candidates the user picked are remembered and come first next time."""
for i, seg in enumerate(segs):
picks = self.fixed.get(seg[0])
if seg[2] != ZHUYIN or not picks:
continue
syl = self._convertible(seg, True)[0]
for (a, b), text in picks.items():
if b <= len(syl):
self.classifier.history.record(syl[a:b], text)
def commit_all(self): def commit_all(self):
return self._commit() if self.keys else "" return self._commit() if self.keys else ""
def reset(self): def reset(self):
self.keys = [] self.keys = []
self.inserting = []
self.held = None
self.backtick = False
self.cursor = None self.cursor = None
self.forced_mode = None self.forced_mode = None
self._forced_is_correction = False
self.fixed = {} self.fixed = {}
self.candidates = [] self.candidates = []
+9 -2
View File
@@ -41,6 +41,9 @@ DEFAULT_WEIGHTS = {
"lit.others": -1.5, "lit.others": -1.5,
# Segmentation # Segmentation
"seg.switch": -3.0, "seg.switch": -3.0,
# Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
} }
@@ -52,7 +55,7 @@ def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False): def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` read as Zhuyin, or None if impossible.""" """Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys) parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None: if not is_last and parse.pending is not None:
@@ -99,10 +102,12 @@ def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=Fa
_add(f, "zh.unconfirmed") _add(f, "zh.unconfirmed")
if after_other: if after_other:
_add(f, "seg.switch") _add(f, "seg.switch")
elif previous == "zhuyin":
_add(f, "ctx.zh")
return f return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False): def literal_features(keys, english, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` output as typed, or None if impossible.""" """Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys: if not keys or " " in keys:
return None return None
@@ -110,6 +115,8 @@ def literal_features(keys, english, is_last=True, after_other=False, finishing=F
f = {"lit.bias": 1.0} f = {"lit.bias": 1.0}
if after_other: if after_other:
_add(f, "seg.switch") _add(f, "seg.switch")
elif previous == "literal":
_add(f, "ctx.lit")
if any(c.isupper() for c in keys): if any(c.isupper() for c in keys):
_add(f, "lit.upper") _add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha()) letters = "".join(c for c in keys if c.isalpha())
+65
View File
@@ -0,0 +1,65 @@
"""Remembers the candidates the user picked. Port of UserHistory.swift.
Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
where the lexicon has it for the reading being converted, so a toneless key is safe.
"""
class UserHistory:
max_entries = 2000 # (reading, phrase) pairs kept; the least recently used go first
max_count = 10
def __init__(self, data=None, on_change=None):
# reading -> {phrase: [count, last use]}
self.data = {}
self.on_change = on_change or (lambda data: None)
self._clock = 0
for reading, phrases in (data or {}).items():
if not isinstance(phrases, dict):
continue
for phrase, value in phrases.items():
try:
count, used = int(value[0]), int(value[1])
except (TypeError, ValueError, IndexError):
continue
self.data.setdefault(reading, {})[phrase] = [count, used]
self._clock = max(self._clock, used)
@staticmethod
def key(syllables):
return "-".join(s.toneless for s in syllables)
def record(self, syllables, phrase):
if not syllables or not phrase:
return
self._clock += 1
entry = self.data.setdefault(self.key(syllables), {}).setdefault(phrase, [0, 0])
entry[0] = min(self.max_count, entry[0] + 1)
entry[1] = self._clock
self._evict()
self.on_change(self.data)
def _evict(self):
pairs = [(v[1], r, p) for r, ps in self.data.items() for p, v in ps.items()]
if len(pairs) <= self.max_entries:
return
pairs.sort()
for _, r, p in pairs[:len(pairs) - self.max_entries]:
del self.data[r][p]
if not self.data[r]:
del self.data[r]
def clear(self):
self.data = {}
self.on_change(self.data)
def adjust(self, syllables, entries):
"""Lexicon entries [(phrase, score)] with picked phrases moved to the top: just above
the best entry, more so the more often they were picked. Returns a new list."""
picked = self.data.get(self.key(syllables))
if not picked or not entries:
return entries
top = entries[0][1]
out = [(p, max(s, top + 0.01 + 0.1 * picked[p][0]) if p in picked else s) for p, s in entries]
out.sort(key=lambda e: -e[1])
return out
+26 -15
View File
@@ -16,7 +16,7 @@ class Lexicon:
def __init__(self): def __init__(self):
self.exact = {} self.exact = {}
self.toneless = {} self._toneless = None # built on first use (only untoned syllables need it)
self.max_phrase_length = 1 self.max_phrase_length = 1
@classmethod @classmethod
@@ -42,9 +42,10 @@ class Lexicon:
except ValueError: except ValueError:
score = -6.0 score = -6.0
self.add(reading, phrase, score) self.add(reading, phrase, score)
for table in (self.exact, self.toneless): for v in self.exact.values():
for v in table.values(): if len(v) > 1:
v.sort(key=lambda e: -e[1]) v.sort(key=lambda e: -e[1])
self._toneless = None
def add(self, reading, phrase, score): def add(self, reading, phrase, score):
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")] syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
@@ -53,15 +54,20 @@ class Lexicon:
entries = self.exact.setdefault(reading, []) entries = self.exact.setdefault(reading, [])
if not any(p == phrase for p, _ in entries): if not any(p == phrase for p, _ in entries):
entries.append((phrase, score)) entries.append((phrase, score))
key = _strip_tones(reading) self._toneless = None
tl = self.toneless.setdefault(key, [])
for i, (p, s) in enumerate(tl): @property
if p == phrase: def toneless(self):
if score > s: """Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
tl[i] = (p, score) if self._toneless is None:
break best = {}
else: for reading, entries in self.exact.items():
tl.append((phrase, score)) bucket = best.setdefault(_strip_tones(reading), {})
for phrase, score in entries:
if score > bucket.get(phrase, float("-inf")):
bucket[phrase] = score
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
return self._toneless
def entries(self, syllables): def entries(self, syllables):
if all(s.tone is not None for s in syllables): if all(s.tone is not None for s in syllables):
@@ -78,8 +84,13 @@ TONELESS_PENALTY = -1.5
class Converter: class Converter:
def __init__(self, lexicon): def __init__(self, lexicon, history=None):
self.lexicon = lexicon self.lexicon = lexicon
self.history = history # UserHistory: picked candidates come first
def _entries(self, syllables):
entries = self.lexicon.entries(syllables)
return self.history.adjust(syllables, entries) if self.history else entries
def _options(self, syllables, i, j, fixed): def _options(self, syllables, i, j, fixed):
if (i, j) in fixed: if (i, j) in fixed:
@@ -88,7 +99,7 @@ class Converter:
return [] return []
sl = syllables[i:j] sl = syllables[i:j]
penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0 penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0
out = [(p, s + penalty) for p, s in self.lexicon.entries(sl)] out = [(p, s + penalty) for p, s in self._entries(sl)]
if not out and j - i == 1: if not out and j - i == 1:
out = [(sl[0].reading, UNKNOWN_SCORE)] out = [(sl[0].reading, UNKNOWN_SCORE)]
return out return out
@@ -126,7 +137,7 @@ class Converter:
out, seen = [], set() out, seen = [], set()
for length in range(min(self.lexicon.max_phrase_length, end), 0, -1): for length in range(min(self.lexicon.max_phrase_length, end), 0, -1):
i = end - length i = end - length
for phrase, _ in self.lexicon.entries(syllables[i:end]): for phrase, _ in self._entries(syllables[i:end]):
if phrase not in seen: if phrase not in seen:
seen.add(phrase) seen.add(phrase)
out.append(((i, end), phrase)) out.append(((i, end), phrase))
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.5.1", "version": "0.11.0",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+10 -5
View File
@@ -26,14 +26,19 @@
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白 空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
Enter 送出目前組字 Enter 送出目前組字
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正 Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
← → 在組字區移動游標,選要改哪個字 ← → 在組字區移動游標,選要改哪個字;游標在中間時直接打注音
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字 會插入新字,Backspace 刪掉游標前的字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
選過的字會被記住,下次優先出現
Esc 取消組字 Esc 取消組字
單按 Shift 下一個字強制英文(保持小寫、直接送出、沒有底線), 單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
按空白或 Enter 後恢復自動判斷; 再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
打到一半時按,同 Tab 整段翻轉。會出現「英文」小提示 打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文) Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!: < > ? ! : 中文語境下自動變成全形 ,。?!:
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
`' = 、 `[ `] = 「」 `( `) = () `! = !
` 後面接其他鍵就是普通的反引號
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
+168 -32
View File
@@ -15,6 +15,7 @@ from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BAC
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402 from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402 from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402 from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
from autozhuyin.core.history import UserHistory # noqa: E402
DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data") DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data")
_start = time.time() _start = time.time()
@@ -42,7 +43,7 @@ def make_engine():
def type_keys(s, e): def type_keys(s, e):
out = "" out = ""
for c in s: for c in s:
key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN}.get(c, c) key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN, "\b": BACKSPACE}.get(c, c)
handled, commit = e.handle(key) handled, commit = e.handle(key)
if commit is not None: if commit is not None:
out += commit out += commit
@@ -194,11 +195,15 @@ class MixedSegmentTests(unittest.TestCase):
class ShiftTapTests(unittest.TestCase): class ShiftTapTests(unittest.TestCase):
def test_forces_next_word_english_lowercase(self): def test_toggles_english_mode(self):
e = make_engine() e = make_engine()
self.assertEqual(type_keys("⇧ru ", e), "ru ") self.assertEqual(type_keys("⇧ru su ", e), "ru su ")
self.assertFalse(e.english_armed) self.assertTrue(e.english_mode)
self.assertEqual(type_keys("su3cl3\n", e), "你好") self.assertEqual(e.marked_text, "")
self.assertEqual(e.handle(ENTER), (False, None))
self.assertTrue(e.english_mode)
self.assertEqual(type_keys("⇧su3cl3\n", e), "你好")
self.assertFalse(e.english_mode)
def test_tap_twice_cancels(self): def test_tap_twice_cancels(self):
self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好") self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好")
@@ -208,20 +213,20 @@ class ShiftTapTests(unittest.TestCase):
type_keys("so", e) type_keys("so", e)
e.handle(SHIFT_TAP) e.handle(SHIFT_TAP)
self.assertEqual(e.mode, ZHUYIN) self.assertEqual(e.mode, ZHUYIN)
self.assertFalse(e.english_mode)
type_keys("\n", e) type_keys("\n", e)
self.assertEqual(e.classifier.learned.weights["so"], 1) self.assertEqual(e.classifier.learned.weights["so"], 1)
def test_forcing_before_typing_not_learned(self): def test_english_mode_not_learned(self):
e = make_engine() e = make_engine()
type_keys("⇧hello ", e) type_keys("⇧hello ", e)
self.assertNotIn("hello", e.classifier.learned.weights) self.assertEqual(e.classifier.learned.weights, {})
def test_arming_survives_backspace(self): def test_backspace_goes_to_app(self):
e = make_engine() e = make_engine()
type_keys("⇧r", e) e.handle(SHIFT_TAP)
e.handle(BACKSPACE) self.assertEqual(e.handle(BACKSPACE), (False, None))
self.assertTrue(e.english_armed) self.assertTrue(e.english_mode)
self.assertEqual(type_keys("ru ", e), "ru ")
def test_detector(self): def test_detector(self):
d = ShiftTapDetector() d = ShiftTapDetector()
@@ -242,7 +247,7 @@ class CursorTests(unittest.TestCase):
type_keys("←", e) type_keys("←", e)
self.assertEqual(e.cursor_offset, 3) self.assertEqual(e.cursor_offset, 3)
type_keys("←←←←", e) type_keys("←←←←", e)
self.assertEqual(e.cursor_offset, 1) self.assertEqual(e.cursor_offset, 1) # stops after the first character
type_keys("→→→→", e) type_keys("→→→→", e)
self.assertEqual(e.cursor_offset, 4) self.assertEqual(e.cursor_offset, 4)
@@ -254,12 +259,55 @@ class CursorTests(unittest.TestCase):
e.select_candidate([t for _, t in e.candidates].index("添")) e.select_candidate([t for _, t in e.candidates].index("添"))
self.assertEqual(type_keys("\n", e), "今添天氣") self.assertEqual(type_keys("\n", e), "今添天氣")
def test_typing_returns_cursor_to_end(self): def test_english_returns_cursor_to_end(self):
e = make_engine() e = make_engine()
type_keys("rup wu0 ←", e) type_keys("rup wu0 ←", e)
self.assertEqual(e.cursor_offset, 1) self.assertEqual(e.cursor_offset, 1)
type_keys("wu0 ", e) type_keys("?", e)
self.assertEqual(e.cursor_offset, len(e.marked_text)) self.assertEqual(e.marked_text, "")
class InsertInMiddleTests(unittest.TestCase):
def test_insert_syllable(self):
e = make_engine()
type_keys("rup fu4←wu", e)
self.assertEqual(e.marked_text, "今ㄊㄧ氣")
self.assertEqual(e.cursor_offset, 3)
type_keys("0 ", e)
self.assertEqual(e.marked_text, "今天氣")
self.assertEqual(e.cursor_offset, 2)
self.assertEqual(type_keys("\n", e), "今天氣")
def test_cursor_stops_after_first_character(self):
e = make_engine()
type_keys("su3cl3←←←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("ji3\n", e)
self.assertEqual(e.marked_text, "")
def test_no_stop_before_first_chinese(self):
e = make_engine()
type_keys("hello ji3←←", e) # "hello " held, then 我: nothing to pick before 我
type_keys("↓", e)
self.assertTrue(e.showing_candidates)
def test_backspace_deletes_character_before_cursor(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4←←\b", e)
self.assertEqual(e.marked_text, "今天氣")
self.assertEqual(e.cursor_offset, 1)
def test_backspace_inside_inserted_syllable(self):
e = make_engine()
type_keys("rup fu4←wu\b", e)
self.assertEqual(e.marked_text, "今ㄊ氣")
def test_keeps_picked_phrase_before_cursor(self):
e = make_engine()
type_keys("rup wu0 ↓", e)
e.select_candidate([t for _, t in e.candidates].index("添"))
type_keys("fu4←wu0 ", e) # insert 天 between 添 and 氣
self.assertEqual(e.marked_text[:3], "今添天")
def test_candidates_for_chinese_before_english(self): def test_candidates_for_chinese_before_english(self):
e = make_engine() e = make_engine()
@@ -267,24 +315,112 @@ class CursorTests(unittest.TestCase):
self.assertTrue(e.showing_candidates) self.assertTrue(e.showing_candidates)
class DirectEnglishTests(unittest.TestCase): class FirstToneAfterEnglishTests(unittest.TestCase):
def test_forced_english_has_no_composition(self): def test_first_tone_after_english(self):
e = make_engine() for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
e.handle(SHIFT_TAP) e = make_engine()
self.assertEqual(e.handle("r"), (True, "r")) out = type_keys("hello " + keys + " \n", e)
self.assertEqual(e.marked_text, "") self.assertEqual(out[:6], "hello ", keys)
self.assertEqual(e.handle("'"), (True, "'")) self.assertEqual(len(out), 7, keys) # one Chinese character after it
self.assertEqual(e.handle(BACKSPACE), (False, None)) self.assertFalse(out[6].isascii(), keys)
self.assertTrue(e.english_armed)
self.assertEqual(e.handle(" "), (True, " "))
self.assertFalse(e.english_armed)
def test_enter_ends_forced_english(self):
class HeldWordTests(unittest.TestCase):
def test_next_word_decides(self):
cases = {"up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_held_word_stays_composing(self):
e = make_engine() e = make_engine()
e.handle(SHIFT_TAP) self.assertEqual(type_keys("up ", e), "")
e.handle("a") self.assertEqual(e.marked_text, "up ")
self.assertEqual(e.handle(ENTER), (False, None)) e.handle(BACKSPACE)
self.assertFalse(e.english_armed) self.assertEqual(e.marked_text, "up")
self.assertEqual(type_keys(" the ", e), "up the ")
class BacktickSymbolTests(unittest.TestCase):
def test_forced_symbols(self):
cases = {"`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_plain_backtick(self):
e = make_engine()
self.assertEqual(type_keys("`", e), "")
self.assertEqual(e.marked_text, "`")
self.assertEqual(type_keys("ls` ", e), "`ls` ")
def test_escape_cancels(self):
e = make_engine()
type_keys("su3`", e)
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "你")
class FirstWordHoldTests(unittest.TestCase):
"""The first word is not committed as English on space if it also reads as Zhuyin."""
def test_next_word_decides(self):
cases = {"el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
"i love you ": "i love you ", "up the ": "up the "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_not_held_after_english(self):
e = make_engine()
self.assertEqual(type_keys("hello world ", e), "hello world ")
self.assertEqual(type_keys("so ", e), "so ") # right after English: commits at once
class UserHistoryTests(unittest.TestCase):
def setUp(self):
self.saved = []
self.c = Classifier(LEX, ENG, weights=WEIGHTS, history=UserHistory(on_change=self.saved.append))
def pick(self, keys, phrase):
e = CompositionEngine(self.c)
type_keys(keys + "↓", e)
e.select_candidate([t for _, t in e.candidates].index(phrase))
return type_keys("\n", e)
def test_picked_candidate_comes_first(self):
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "天")
self.assertEqual(self.pick("wu0 ", "添"), "添")
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
e = CompositionEngine(self.c)
type_keys("wu0 ↓", e)
self.assertEqual(e.candidates[0][1], "添")
self.assertEqual(self.saved[-1], {"ㄊㄧㄢ": {"添": [1, 1]}})
def test_phrases_still_win(self):
self.pick("wu0 ", "添")
self.assertEqual(type_keys("rup wu0 \n", CompositionEngine(self.c)), "今天")
def test_most_picked_wins(self):
self.pick("wu0 ", "添")
self.pick("wu0 ", "添")
self.pick("wu0 ", "天")
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
def test_round_trip_and_clear(self):
self.pick("wu0 ", "添")
h = UserHistory(self.saved[-1])
self.assertEqual(h.data, {"ㄊㄧㄢ": {"添": [1, 1]}})
h.record([bopomofo.parse(list("wu0 ")).syllables[0].syllable], "天")
self.assertEqual(h.data["ㄊㄧㄢ"]["天"], [1, 2])
h.clear()
self.assertEqual(h.data, {})
def test_evicts_least_recent(self):
h = UserHistory()
h.max_entries = 2
syl = [bopomofo.parse(list("wu0 ")).syllables[0].syllable]
for p in ["天", "添", "田"]:
h.record(syl, p)
self.assertEqual(set(h.data["ㄊㄧㄢ"]), {"添", "田"})
if __name__ == "__main__": if __name__ == "__main__":
+15 -3
View File
@@ -6,6 +6,9 @@ import types
import unittest import unittest
HERE = os.path.dirname(os.path.abspath(__file__)) HERE = os.path.dirname(os.path.abspath(__file__))
import tempfile # noqa: E402
os.environ["APPDATA"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
sys.path.insert(0, os.path.dirname(HERE)) sys.path.insert(0, os.path.dirname(HERE))
# --- stand-ins for PIME modules (only what the wrapper uses) --- # --- stand-ins for PIME modules (only what the wrapper uses) ---
@@ -46,7 +49,7 @@ class FakeTextService:
textService.TextService = FakeTextService textService.TextService = FakeTextService
sys.modules["textService"] = textService sys.modules["textService"] = textService
from autozhuyin.autozhuyin_ime import AutoZhuyinTextService # noqa: E402 from autozhuyin.autozhuyin_ime import AutoZhuyinTextService, _shared_classifier # noqa: E402
VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28} VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
@@ -99,12 +102,18 @@ def type_text(svc, text):
class WrapperTests(unittest.TestCase): class WrapperTests(unittest.TestCase):
def setUp(self): def setUp(self):
_shared_classifier().history.clear()
_shared_classifier().learned.clear()
self.svc = AutoZhuyinTextService(client=None) self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate() self.svc.onActivate()
def test_mixed_sentence(self): def test_mixed_sentence(self):
self.assertEqual(type_text(self.svc, "ji3 hello world su3cl3 "), "我hello world 你好") self.assertEqual(type_text(self.svc, "ji3 hello world su3cl3 "), "我hello world 你好")
def test_backtick_symbol(self):
self.assertEqual(type_text(self.svc, "`<"), ",")
self.assertEqual(type_text(self.svc, "ji3`."), "我。")
def test_composition_shown(self): def test_composition_shown(self):
type_text(self.svc, "su3c") type_text(self.svc, "su3c")
self.assertEqual(self.svc.reply["compositionString"], "你ㄏ") self.assertEqual(self.svc.reply["compositionString"], "你ㄏ")
@@ -138,20 +147,23 @@ class WrapperTests(unittest.TestCase):
class ShiftTapWrapperTests(unittest.TestCase): class ShiftTapWrapperTests(unittest.TestCase):
def setUp(self): def setUp(self):
_shared_classifier().history.clear()
_shared_classifier().learned.clear()
self.svc = AutoZhuyinTextService(client=None) self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate() self.svc.onActivate()
def test_tap_forces_english_lowercase(self): def test_tap_forces_english_lowercase(self):
reply = tap_shift(self.svc) reply = tap_shift(self.svc)
self.assertEqual(reply.get("showMessage"), "英文") self.assertEqual(reply.get("showMessage"), "英文")
self.assertEqual(type_text(self.svc, "ru "), "ru ") self.assertEqual(type_text(self.svc, "ru su "), "ru su ")
self.assertEqual(tap_shift(self.svc).get("showMessage"), "中英自動")
def test_shift_with_letter_is_not_a_tap(self): def test_shift_with_letter_is_not_a_tap(self):
self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True)) self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True))
press(self.svc, "H", shift=True) press(self.svc, "H", shift=True)
up = FakeKeyEvent("⇧") up = FakeKeyEvent("⇧")
self.assertFalse(self.svc.filterKeyUp(up)) self.assertFalse(self.svc.filterKeyUp(up))
self.assertFalse(self.svc.engine.english_armed) self.assertFalse(self.svc.engine.english_mode)
if __name__ == "__main__": if __name__ == "__main__":