Compare commits

..
33 Commits
Author SHA1 Message Date
ays953 acafd8f96e Release v0.10.1 2026-11-13 15:30:18 +08:00
ays953andClaude Opus 5.5 fa5b266ae4 Stop the cursor after the first Chinese character
Moving left no longer reaches the start of the buffer (or any position before the
first Zhuyin syllable), where there is nothing to pick candidates for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:30:03 +08:00
ays953 6448bf4ca2 Release v0.10.0 2026-11-13 14:31:10 +08:00
ays953andClaude Opus 5.5 2cebf0e700 Remember picked candidates (user history)
Candidates chosen with the arrow-down list are recorded on commit as
toneless reading -> phrase with a count. The converter lifts a picked phrase just
above the best lexicon entry for that reading (more for more picks), so it
comes first next time without overriding longer phrases. Up to 2000 pairs,
least recently used evicted. Stored as user-history.json on every platform; the
Mac menu's clear item clears it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:30:55 +08:00
ays953 815dcf979d Release v0.9.1 2026-11-13 12:06:24 +08:00
ays953andClaude Opus 5.5 8e82105af9 Hold the first word on space when it also reads as Zhuyin
At the start of a composition (not right after English), a word that reads as
valid Zhuyin is no longer committed as English on space. The next word decides,
by comparing "Chinese all the way" with "English word + Chinese" using the
classifier's features: el ji3 = 高我, no su3cl3 = no 你好, up the = up the.
After English text only common one-syllable readings (up, url) are held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:10 +08:00
ays953 9cac3467a0 Release v0.9.0 2026-11-13 11:59:53 +08:00
ays953andClaude Opus 5.5 49e0b52fd5 Add a Linux version (IBus) for Bazzite / SteamOS desktop mode
linux/autozhuyin_ibus.py is an IBus engine on top of the shared Python core:
preedit with the cursor and blue English segments, lookup table, Shift tap for
English mode, candidate clicks. install.sh puts everything in the home folder
(works on read-only systems) and an autostart entry registers the engine with
ibus-daemon and selects it. Tested with a stand-in IBus module.

Also fixes Windows: wants() did not claim the backtick, so backtick + punctuation
never reached the engine under PIME.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:59:40 +08:00
ays953 fcc0f9d2db Release v0.8.0 2026-11-13 09:56:18 +08:00
ays953andClaude Opus 5.5 31d9991e16 Insert and delete syllables at the composition cursor
The ← → cursor is now an editing position (key index), not just a candidate
stop. It can move to the start; typing Zhuyin there composes a syllable at the
cursor and splices it in on its tone key; Backspace removes the character before
the cursor. Picked phrases before the edit point are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:56:06 +08:00
ays953 6b4ec39dc1 Release v0.7.1 2026-11-11 17:01:37 +08:00
ays953andClaude Opus 5.5 955b0ca5c3 Ship third-party notices with both installers
THIRD_PARTY_LICENSES.txt carries McBopomofo's full MIT notice (required when the
converted lexicon is distributed) plus notes on the public-domain word list and
PIME. It goes into AutoZhuyin.app/Contents/Resources and the Windows zip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:01:26 +08:00
ays953 251a16fa08 Release v0.7.0 2026-11-11 16:29:06 +08:00
ays953andClaude Opus 5.5 7cb1f62dfa Backtick + punctuation key forces a full-width symbol
` then , . / ; : ' [ ] ( ) ! ~ - = and their shifted keys always give the
full-width symbol (,。?;:、「」()!~——=), whatever the context. Backtick
followed by any other key is a plain backtick, so ``` still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:28:54 +08:00
ays953 3670305f0f Release v0.6.3 2026-11-11 16:10:59 +08:00
ays953andClaude Opus 5.5 ef9f91a6b7 Hold ambiguous English words (up = 因) until the next word decides
"up " also reads as ㄧㄣ (因). Instead of committing English on space, keep the
word composing: "up jo4" becomes 因為, "up the" stays English. Only applies when
the reading is a common single syllable (lexicon score >= -3.5): up, url.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:10:47 +08:00
ays953 1dacb8f164 Release v0.6.2 2026-11-11 12:13:15 +08:00
ays953andClaude Opus 5.5 bfeed22916 Cut input method startup from ~0.95 s to ~0.15 s
After login macOS starts the input method on first use, and keys typed
while it loads its data were delayed or lost. Startup now:
- parses lexicon.txt as raw bytes (no per-line String splitting or Double
  parsing) and builds the tone-ignoring index lazily on a background queue;
- loads the precomputed English dictionary and bigram table (shared with the
  Windows build) instead of retraining the bigram model on every launch, and
  uses binary search instead of building a 230k-entry Set.
Measured (release, warm cache): lexicon 0.65 -> 0.10 s, English 0.30 -> 0.04 s.
The Windows (Python) core also builds its toneless index lazily: 0.56 -> 0.37 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:13:03 +08:00
ays953 bb9cb0c4e0 Release v0.6.1 2026-11-11 12:00:10 +08:00
ays953andClaude Opus 5.5 2c5c70c066 Learn the previous-word context; fix first-tone characters after English
After an English word, 23 of 50 common first-tone characters (機, 吃, 書…)
were read as English when space was pressed: the fixed +1.5 context bonus
was far too strong for the trained weight scale, and the training data never
contained the "space just pressed on a first-tone syllable" decision.

Context is now the ctx.zh / ctx.lit features, trained with the rest; the
synthetic data includes the previous mode and the first-tone space decision.
Test accuracy: commit 98.22% -> 99.52%, typing 88.73% -> 91.49%, and when the
language differs from the previous word 92.0% -> 99.3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:59:57 +08:00
ays953 17f7d6f50d Release v0.6.0 2026-11-11 11:44:38 +08:00
ays953andClaude Opus 5.5 7e173a1da8 Make the Shift tap a persistent English-mode toggle
A Shift tap with nothing composed now switches English mode on until the
next tap, instead of forcing only the next word: space, Enter and other keys
no longer drop back to automatic detection. In English mode every key goes
straight to the app, so there is no composition or underline. A Shift tap
while typing still flips the buffer like Tab. Windows shows "英文" /
"中英自動" when toggling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:44:25 +08:00
ays953 fa0b78e5a4 Release v0.5.1 2026-11-11 11:36:12 +08:00
ays953andClaude Opus 5.5 e928d08fdc Type forced English directly, without an underlined composition
After a Shift tap, letters, digits and word punctuation go straight to the
app instead of into marked text, so nothing is underlined. Space, Enter, Tab,
Escape or another symbol ends the forced word and automatic detection
resumes; backspace is handled by the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:35:59 +08:00
ays953 82f20a6551 Release v0.5.0 2026-11-11 10:37:37 +08:00
ays953andClaude Opus 5.5 e6e5004630 Add a composition cursor so candidates can be chosen for any character
Left/Right move a cursor between Zhuyin syllables in the composition; Down
opens candidates for the phrase ending at the cursor instead of always the
last syllable. Typing or backspace returns the cursor to the end. Candidates
also work for Chinese that is followed by English in the same buffer. The Mac
caret and PIME composition cursor follow it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:37:24 +08:00
ays953 988ed471be Release v0.4.1 2026-11-11 10:18:14 +08:00
ays953andClaude Opus 5.5 6c36c01f0e Stop declaring Latin repertoire so password fields bypass the IME
With "Latn" in tsInputMethodCharacterRepertoireKey macOS treated AutoZhuyin
as ASCII-capable and kept it active in secure text fields, so typed passwords
showed up as underlined composition text. Apple's Zhuyin and vChewing declare
only Hant (not ASCII-capable); macOS then switches to an ASCII keyboard layout
in password fields and the input method never sees the keystrokes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:18:01 +08:00
ays953andClaude Opus 5.5 0fc42a4b8c install.sh: warn when a system-wide .pkg copy shadows the per-user install
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:05:24 +08:00
ays953 42a2d3a2a3 Release v0.4.0 2026-11-11 09:57:34 +08:00
ays953andClaude Opus 5.5 20291bf515 Add Shift tap to force English for the next word
Pressing and releasing Shift on its own arms literal mode for the next word,
so lowercase English can be typed without it being read as Zhuyin; it resets
after the word is committed and a second tap cancels it. While typing, a
Shift tap flips the buffer like Tab. Forcing before typing is not recorded
as a learned correction; flipping is.

Mac: listens for flagsChanged events. Windows: Shift key-up via PIME's
filterKeyUp/onKeyUp, with a short "英文" tooltip. The tap detection lives in
the core (ShiftTapDetector / shift_tap.py) with tests on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:57:18 +08:00
ays953 21142c2d98 Release v0.3.0 2026-11-11 08:58:43 +08:00
ays953andClaude Opus 5.5 476858d841 Learn classifier weights from synthetic typing data
Scoring is now features x weights (Features.swift / features.py), with the
old hand-tuned values as defaults. training/ generates synthetic mixed
Chinese/English buffers from the lexicon and word lists and trains the
weights with an averaged structured perceptron over the same segmentation
DP the input method runs. On 10,000 test buffers commit-time accuracy goes
from 97.58% to 99.62%; typing-time (prefix) accuracy from 86.15% to 87.81%.

Both apps load Resources/classifier-weights.json. A parity test checks the
Swift segmentation matches the Python one on 400 fixtures.

Also fixes space after a toned syllable being scored as an extra stray tone,
which could turn short Chinese like "u4 " into English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:58:08 +08:00
43 changed files with 10190 additions and 365 deletions
+91 -21
View File
@@ -1,6 +1,6 @@
# 自動注音 AutoZhuyin # 自動注音 AutoZhuyin
給繁體中文使用者的輸入法(macOS、Windows):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。 給繁體中文使用者的輸入法(macOS、Windows、Linux):**不用按切換鍵**,直接用大千注音鍵盤打字,輸入法會自動判斷你打的是注音還是英文。
``` ```
按鍵: ji3 hello world su3cl3␣ 按鍵: ji3 hello world su3cl3␣
@@ -12,10 +12,13 @@
- **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示) - **中英自動判斷**:每按一個鍵就重新判斷,組字區即時顯示結果(Mac 版英文以藍字顯示)
- **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分 - **中英混打不用空白**:`ji3hello` → 我hello、`hellosu3cl3` → hello你好,同一段內會自動切分
- **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對 - **判斷錯了按 Tab**:在中/英之間翻轉,輸入法會記住這組按鍵,下次自動判斷對
- **單按 Shift 切換英文模式**:英文模式下按鍵直接送出、保持小寫、不顯示底線,再單按一次 Shift 回到中英自動判斷
- **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小) - **注音順序打錯也能修正**:`ㄧㄒㄠˇ`、`ㄒㄠㄧˇ` 都會變成 `ㄒㄧㄠˇ`(小)
- **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」 - **可省略聲調**:`jiap7`(ㄨㄛ ㄇㄣ˙)一樣能打出「我們」
- **候選字**:按 ↓ 選最後一個字或詞,選過的會固定下來 - **候選字**:用 ← → 把游標移到要改的字後面,按 ↓ 選字;選過的會固定下來
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,` - **記住選字**:從候選字選過的字詞會被記住,下次打同樣的讀音會優先出現(選越多次越優先)
- **中間補字、刪字**:游標移到前面後直接打注音,會插在游標處;Backspace 刪掉游標前的那個字
- **全形標點**:中文語境下 `?` `!` `<` 自動變成 `?` `!` `,`;任何時候按 `` ` `` 再按標點鍵,一定得到全形標點(例如 `` `< `` = `,`)
- **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權) - **完整詞庫**:約 17 萬筆,來自小麥注音 McBopomofo(MIT 授權)
## 下載與安裝 ## 下載與安裝
@@ -43,6 +46,16 @@ PKG 會裝到 `/Library/Input Methods/`,系統上所有使用者都能用。
SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。 SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安裝:以系統管理員身分執行 `uninstall.bat`。
### Linux(Bazzite、SteamOS 桌面模式、Fedora…)
需求:系統內建 IBus 輸入法框架(Bazzite、Fedora 預設就有)。只在桌面模式可用,Steam 遊戲模式不支援系統輸入法。
1. 解壓縮 `AutoZhuyin-Linux-<版本>.tar.gz`,在終端機進到資料夾執行 `./install.sh`。不需要 sudo,只會裝在家目錄(`~/.local/share/autozhuyin`),所以唯讀系統也能用。
2. KDE(Bazzite / SteamOS 預設):系統設定 → 鍵盤 → 虛擬鍵盤,選 **IBus Wayland** → 套用,登出再登入。GNOME 不用設定。
3. 之後每次登入會自動切換到自動注音(`~/.config/autostart/autozhuyin.desktop`)。
暫時改回一般鍵盤:`ibus engine xkb:us::eng`;切回:`ibus engine autozhuyin`;移除:`./uninstall.sh`。
## 按鍵 ## 按鍵
| 按鍵 | 作用 | | 按鍵 | 作用 |
@@ -50,11 +63,14 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 | | 空白 | 注音:一聲;音節已完成時送出中文。英文:送出單字加空白 |
| Enter | 送出目前組字 | | Enter | 送出目前組字 |
| **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 | | **Tab** | 判斷錯了?按 Tab 把整段翻轉成另一種語言,輸入法會**記住**這次修正 |
| ↓ | 開啟候選字(選最後一個字或詞),數字鍵 1–9 選字,Esc 關閉 | | ← → | 在組字區移動游標(最左停在第一個中文字後面)。游標在中間時:直接打注音會**插入**新字,Backspace 刪掉游標前的字,↓ 選游標前的字 |
| ↓ | 開啟游標前那個字或詞的候選字(游標預設在最後),數字鍵 1–9 選字,Esc 關閉 |
| Esc | 取消組字 | | Esc | 取消組字 |
| Shift + 字母 | 強制英文 | | **單按 Shift** | 沒在打字時:切換**英文模式**,按鍵直接送出、保持小寫、沒有底線,一直維持到再單按一次 Shift 才回到中英自動判斷。打到一半時:同 Tab,整段翻轉 |
| Shift + 字母 | 打大寫字母(也會強制英文) |
| `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` | | `< > ? ! : ( ) [ ]` | 中文語境下自動轉成全形 `,。?!:()「」` |
| `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) | | `, . / ; -` | 英文單字後是標點;在注音中是 ㄝ ㄡ ㄥ ㄤ ㄦ(例如 `2;3` = 檔) |
| `` ` `` + 標點鍵 | **強制全形標點**,不管前後是中文還是英文:`` `, `` `` `< `` = `,`、`` `. `` `` `> `` = `。`、`` `/ `` = `?`、`` `; `` = `;`、`` `: `` = `:`、`` `' `` `` `\ `` = `、`、`` `[ `` `` `] `` = `「」`、`` `{ `` `` `} `` = `『』`、`` `( `` `` `) `` = `()`、`` `! `` = `!`、`` `~ `` = `~`、`` `- `` = `——`、`` `= `` = `=`。`` ` `` 後面接其他鍵就是普通的反引號(` ``` ` 照常可用) |
## 運作方式 ## 運作方式
@@ -69,12 +85,12 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
| 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 | | 使用者過去的修正(學習) | 有大寫字母 → 一定是英文 |
| 一開始稍微偏向注音 | 純數字 → 數字 | | 一開始稍微偏向注音 | 純數字 → 數字 |
分數高的一方勝出。按空白時,注音那邊會把空白當成一聲來評分,英文那邊當作完整單字評分。 每一項都是一個**特徵**(`Features.swift` / `features.py`),分數 = 特徵 × **權重**。權重是用資料訓練出來的(見下方「訓練判斷權重」),存在 `Resources/classifier-weights.json`;檔案不存在時用手調的預設值。分數高的一方勝出。按空白時,還沒打聲調的注音音節會當成一聲來評分,英文那邊當作完整單字評分。
中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎: 中英混打時,`Classifier.segment` 會用動態規劃試遍所有切分點,找出總分最高的切法(例如 `ji3hello` → `ji3` 注音+`hello` 英文)。為了避免把正常的單字或句子切碎:
- 每多切一刀扣 3 分(`segmentPenalty`) - 每多切一刀會扣分(`seg.switch` 特徵)
- 後面還接著其他段的注音段,最後一個音節必須打了聲調 - 後面還接著其他段的注音段,最後一個音節必須打了聲調
- 接在英文後面、還沒打聲調的注音段再扣 2 分(`unconfirmedSwitchPenalty`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」 - 接在英文後面、還沒打聲調的注音段會再扣分(`zh.unconfirmed`),所以打到 `ji3hel` 時會先顯示「我hel」,不會變成「我heㄠ」
音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。 音節解析照大千規則:符號依「聲母 → 介音 → 韻母」填入,聲調鍵結束音節。順序打錯時,如果合併後仍可能是合法音節,就併進同一個音節,否則才切成新音節。
@@ -82,13 +98,15 @@ SmartScreen 出現警告時,按「其他資訊 → 仍要執行」。解除安
## 個人資料 ## 個人資料
| | macOS | Windows | | | macOS | Windows | Linux |
|---|---|---| |---|---|---|---|
| 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | | 學習紀錄(Tab 修正) | `UserDefaults`(`com.autozhuyin.inputmethod.AutoZhuyin`) | `%APPDATA%\PIME\autozhuyin\learned.json` | `~/.config/autozhuyin/learned.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | | 選字紀錄 | `~/Library/Application Support/AutoZhuyin/user-history.json` | `%APPDATA%\PIME\autozhuyin\user-history.json` | `~/.config/autozhuyin/user-history.json` |
| 自訂詞 | `~/Library/Application Support/AutoZhuyin/user-phrases.txt` | `%APPDATA%\PIME\autozhuyin\user-phrases.txt` | `~/.config/autozhuyin/user-phrases.txt` |
自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。 自訂詞的格式和 `lexicon.txt` 一樣,每行一個詞:`ㄗˋ-ㄉㄨㄥˋ-ㄓㄨˋ-ㄧㄣ 自動注音 -4`(讀音、詞、分數;分數越大越優先)。
Mac 版可以在輸入法選單選「清除學習紀錄」。 選字紀錄最多保留 2000 筆,太久沒用的會先淘汰。被選過的字只會排到同讀音第一順位,不會蓋過整個詞組(例如選過「添」之後,ㄊㄧㄢ 單打會出「添」,但「今天」還是「今天」)。
Mac 版可以在輸入法選單選「清除學習紀錄」(中英判斷和選字紀錄一起清除);Windows、Linux 刪除上面的檔案即可。
## 開發 ## 開發
@@ -121,8 +139,47 @@ scripts/package-windows.sh # 產生 build/AutoZhuyin-Wi
- `windows/autozhuyin/data/` 是產生出來的,不進版本控制。 - `windows/autozhuyin/data/` 是產生出來的,不進版本控制。
- 版本號在 `windows/autozhuyin/ime.json`。 - 版本號在 `windows/autozhuyin/ime.json`。
### Linux
Linux 版是 IBus 輸入法(`linux/autozhuyin_ibus.py`),直接使用 Windows 版的 Python 核心(`windows/autozhuyin/core/`)和資料。
```bash
python3 -m unittest discover -s linux/tests # 用假的 IBus 模組測試外殼
scripts/package-linux.sh # 產生 build/AutoZhuyin-Linux-<版本>.tar.gz(會先跑測試)
```
- 安裝在使用者目錄,不寫入 `/usr`:開機時由 autostart 啟動 `autozhuyin_ibus.py --activate`,它會把引擎動態註冊給 ibus-daemon 並切換過去。
- 也支援由 ibus-daemon 啟動(`--ibus`,配合 `/usr/share/ibus/component` 的 XML),之後要做 RPM 時可以用。
> 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。 > 修改核心邏輯時,Swift(`Sources/AutoZhuyinCore/`)和 Python(`windows/autozhuyin/core/`)兩邊都要改,並在兩邊的測試加上相同案例。
### 訓練判斷權重
中英判斷的權重是用合成資料訓練的(結構化感知器,averaged structured perceptron),訓練時跑的就是輸入法實際使用的同一套切分演算法:
1. `training/synth.py` 產生打字資料:從詞庫依詞頻抽中文詞,反查成大千按鍵;英文從常用字表和字典抽;組成「純中文、純英文、中+英、英+中、中+英+中、數字」等情況,每個按鍵都標好正確答案。也會產生「打到一半」的版本,用來訓練打字過程中的顯示。
2. `training/train.py` 先用手調權重評估一次,再訓練、評估,寫入 `Resources/classifier-weights.json`。
3. `training/make_parity.py` 產生 `training/parity-fixtures.json`,Swift 測試會檢查它的切分結果和 Python 版完全一樣。
```bash
python3 windows/tools/build_data.py # 準備資料(Python 版和訓練共用)
python3 training/train.py # 快速訓練(預設 3000 筆,約半分鐘)
python3 training/train.py --train 20000 --test 5000 --epochs 10 # 正式版本用的設定,約 3 分鐘
python3 training/train.py --eval-only # 只評估目前的權重
python3 training/make_parity.py # 權重或特徵改了之後都要重跑
```
目前的結果(10,000 筆測試資料):
| | 送出時整段判斷正確 | 打字過程中顯示正確 | 跟前一個字語言不同時 |
|---|---|---|---|
| 手調權重 | 97.84% | 88.19% | 96.2% |
| 訓練後 | **99.52%** | **91.49%** | **99.3%** |
訓練資料也模擬了「前一個字是中文或英文」和「按下空白(一聲)的那一刻」,所以「前一個字」的影響(`ctx.zh` / `ctx.lit`)也是訓練出來的,不是手調的。「打字過程中」本來就有歧義(例如 `ji3h` 的 `h` 可能是英文的開頭,也可能是注音 ㄘ),所以準確率比較低,但最後送出的結果才是重點。
新增特徵時,Swift 和 Python 兩邊都要加,預設權重設成 0,然後重新訓練。
### 詞庫 ### 詞庫
`Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。 `Resources/lexicon.txt` 由小麥注音的原始資料轉換而來,後面再接上手寫的入門詞庫 `Resources/lexicon-seed.txt`。轉換程式 `scripts/convert-mcbopomofo.py` 只讀取資料檔,不執行小麥注音的程式碼:分數依語料詞頻計算,一字多音的字依讀音常見程度分配權重。
@@ -147,12 +204,12 @@ scripts/release.sh minor --push # 同時 push commit 和 tag
`release.sh` 會自動: `release.sh` 會自動:
1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`) 1. 遞增版本號(`Resources/Info.plist` 的版本與 build 號、`windows/autozhuyin/ime.json`)
2. 跑 Swift 和 Python 全部測試 2. 跑 Swift 和 Python 全部測試
3. 產生 `build/AutoZhuyin-<版本>.pkg` 和 `build/AutoZhuyin-Windows-<版本>.zip` 3. 產生 `build/AutoZhuyin-<版本>.pkg`、`build/AutoZhuyin-Windows-<版本>.zip`、`build/AutoZhuyin-Linux-<版本>.tar.gz`
4. commit「Release v<版本>」並打上 `v<版本>` tag 4. commit「Release v<版本>」並打上 `v<版本>` tag
任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。 任何一步失敗都會把版本號改回去。只想改版本號不打包:`scripts/bump-version.sh [patch|minor|major]`。
最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg` 和 `.zip`。 最後在 Gitea 用這個 tag 建立 Release,附上 `build/` 裡的 `.pkg`、`.zip` 和 `.tar.gz`。
## 專案結構 ## 專案結構
@@ -164,11 +221,12 @@ Sources/AutoZhuyinCore/ 核心邏輯(Swift,可單元測試)
Lexicon.swift 詞庫 Lexicon.swift 詞庫
Converter.swift 音節 → 中文(Viterbi) Converter.swift 音節 → 中文(Viterbi)
EnglishModel.swift 英文判斷 EnglishModel.swift 英文判斷
Classifier.swift 中英判斷 + 學習 Features.swift 判斷用的特徵與預設權重
Classifier.swift 中英判斷(特徵 × 權重)、切分、學習
CompositionEngine.swift 輸入法狀態機 CompositionEngine.swift 輸入法狀態機
Sources/AutoZhuyin/ macOS InputMethodKit 外殼 Sources/AutoZhuyin/ macOS InputMethodKit 外殼
Tests/ Swift 單元測試 Tests/ Swift 單元測試
Resources/ Info.plist、詞庫、英文常用字 Resources/ Info.plist、詞庫、英文常用字、訓練好的權重
scripts/ 建置、安裝、打包、詞庫轉換 scripts/ 建置、安裝、打包、詞庫轉換
pkg/postinstall PKG 安裝後腳本 pkg/postinstall PKG 安裝後腳本
windows/ windows/
@@ -179,13 +237,20 @@ windows/
installer/ install.bat、uninstall.bat、給使用者的 README.txt installer/ install.bat、uninstall.bat、給使用者的 README.txt
tests/ Python 測試 tests/ Python 測試
tools/ 產生資料檔、圖示 tools/ 產生資料檔、圖示
linux/
autozhuyin_ibus.py IBus 外殼(共用 windows/autozhuyin/core)
installer/ install.sh、uninstall.sh、給使用者的 README.txt
tests/ 用假 IBus 模組的外殼測試
training/ 合成資料、訓練權重、Swift/Python 一致性測試資料
THIRD_PARTY_LICENSES.txt 第三方授權聲明(會打包進兩個安裝檔)
``` ```
## 已知限制 ## 已知限制
- Windows 版還沒在實機上測試過。 - Windows 版、Linux 版還沒在實機上測試過。
- Linux 版只支援 IBus。KDE 上有些 Flatpak 程式(例如 Flatpak 版 Chrome)在 IBus Wayland 下可能打不出字。
- 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。 - 聲調鍵打在注音符號中間(例如 `uv3l`)無法自動修正。
- 很短的字串(如 `up` 是英文還是 ㄧㄣ)本質上就有歧義,可能需要按 Tab 修正;修正後會記住。 - 很短的字串本質上就有歧義。開頭第一個詞如果也能讀成注音,按空白後會先保留(顯示成英文),看下一個詞再決定:`up jo4` = 因為、`el ji3` = 高我、`no su3cl3` = no 你好、`up the` = up the。接在英文後面時只保留 `up`(因)這種很常見的讀音,不會拖慢打英文。還是判斷錯的話按 Tab 修正,修正後會記住。
- 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。 - 安裝檔沒有簽章(macOS 沒有 Developer ID,Windows 沒有程式碼簽章憑證)。
## 下一步(Roadmap) ## 下一步(Roadmap)
@@ -193,10 +258,13 @@ windows/
- [x] 匯入完整詞庫 - [x] 匯入完整詞庫
- [x] 注音順序錯誤修正 - [x] 注音順序錯誤修正
- [x] Windows 版 - [x] Windows 版
- [ ] 用真實打字資料調整判斷權重 - [x] Linux 版(IBus)
- [ ] Linux Fcitx5 版(KDE 上相容性較好)
- [x] 用資料訓練判斷權重(合成資料)
- [ ] 加入真實打字資料一起訓練
- [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」 - [ ] 已送出的英文單字若判斷錯誤,可以「事後翻轉」
- [ ] 用 Core ML 小模型取代手調分數,參考整句語境 - [ ] 用 Core ML 小模型取代手調分數,參考整句語境
- [ ] 使用者詞頻學習(選過的候選字優先) - [x] 使用者詞頻學習(選過的候選字優先)
- [ ] 偏好設定視窗 - [ ] 偏好設定視窗
## 致謝與授權 ## 致謝與授權
@@ -204,3 +272,5 @@ windows/
- 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License) - 詞庫資料:[小麥注音 McBopomofo](https://github.com/openvanilla/McBopomofo)(MIT License)
- Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼) - Windows 輸入法框架:[PIME](https://github.com/EasyIME/PIME)(LGPL 2.1,使用者自行安裝,本專案不包含其程式碼)
- 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`) - 英文字典:Webster's Second International Dictionary(公有領域,macOS 的 `/usr/share/dict/words`)
完整的第三方授權聲明(含小麥注音 MIT 授權全文)見 [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt),Mac 版放在 `AutoZhuyin.app/Contents/Resources/`,Windows 版放在 zip 根目錄和安裝後的 `autozhuyin` 資料夾。
+2 -3
View File
@@ -15,9 +15,9 @@
<key>CFBundlePackageType</key> <key>CFBundlePackageType</key>
<string>APPL</string> <string>APPL</string>
<key>CFBundleShortVersionString</key> <key>CFBundleShortVersionString</key>
<string>0.2.0</string> <string>0.10.1</string>
<key>CFBundleVersion</key> <key>CFBundleVersion</key>
<string>2</string> <string>18</string>
<key>InputMethodConnectionName</key> <key>InputMethodConnectionName</key>
<string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string> <string>com.autozhuyin.inputmethod.AutoZhuyin_Connection</string>
<key>InputMethodServerControllerClass</key> <key>InputMethodServerControllerClass</key>
@@ -37,7 +37,6 @@
<key>tsInputMethodCharacterRepertoireKey</key> <key>tsInputMethodCharacterRepertoireKey</key>
<array> <array>
<string>Hant</string> <string>Hant</string>
<string>Latn</string>
</array> </array>
<key>tsInputMethodIconFileKey</key> <key>tsInputMethodIconFileKey</key>
<string>icon.tiff</string> <string>icon.tiff</string>
+41
View File
@@ -0,0 +1,41 @@
{
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": "2026-11-11",
"examples": 40000,
"test_accuracy": {
"commit": 0.9952,
"typing": 0.9149
},
"weights": {
"ctx.lit": 0.0916,
"ctx.zh": 0.1349,
"lit.bias": 0.3982,
"lit.common": -0.2906,
"lit.common_len": 0.8046,
"lit.dict": -1.6662,
"lit.dict_len": 0.6441,
"lit.digits": -2.9689,
"lit.num_digits": -0.018,
"lit.num_others": -0.5554,
"lit.others": -2.0002,
"lit.prefix_common_len": 0.2831,
"lit.prefix_dict_len": 0.0299,
"lit.short": -0.2482,
"lit.unknown_score": 0.2479,
"lit.upper": 20.0,
"seg.switch": -1.7596,
"zh.bias": 1.3423,
"zh.lexscore": 0.2917,
"zh.pending_invalid": -5.0,
"zh.pending_prefix": -0.9162,
"zh.phrase": 0.4499,
"zh.reordered": -1.3079,
"zh.stray": -5.0,
"zh.toned_invalid": -5.0,
"zh.toned_valid_lex": -0.9018,
"zh.toned_valid_nolex": -1.6341,
"zh.unconfirmed": -0.7952,
"zh.untoned_mid_invalid": -5.0,
"zh.untoned_mid_valid": -2.7479
}
}
+48 -10
View File
@@ -9,39 +9,74 @@ enum Shared {
static let learnedKey = "LearnedPreferences" static let learnedKey = "LearnedPreferences"
/// ~/Library/Application Support/AutoZhuyin (user phrases, picked-candidate history).
static let supportDir: URL? = FileManager.default
.urls(for: .applicationSupportDirectory, in: .userDomainMask).first?
.appendingPathComponent("AutoZhuyin")
static let classifier: Classifier = { static let classifier: Classifier = {
let bundle = Bundle.main let bundle = Bundle.main
let start = Date()
let lexicon = Lexicon() let lexicon = Lexicon()
if let url = bundle.url(forResource: "lexicon", withExtension: "txt"), if let url = bundle.url(forResource: "lexicon", withExtension: "txt"),
let text = try? String(contentsOf: url, encoding: .utf8) { let data = try? Data(contentsOf: url) {
lexicon.load(text) lexicon.load(data: data)
} }
// Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt // Optional user phrases: ~/Library/Application Support/AutoZhuyin/user-phrases.txt
if let support = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first { if let userFile = supportDir?.appendingPathComponent("user-phrases.txt"),
let userFile = support.appendingPathComponent("AutoZhuyin/user-phrases.txt") let text = try? String(contentsOf: userFile, encoding: .utf8) {
if let text = try? String(contentsOf: userFile, encoding: .utf8) { lexicon.load(text) } lexicon.load(text)
} }
let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt")) let english = EnglishModel(commonWordsURL: bundle.url(forResource: "english-common", withExtension: "txt"),
dictionaryURL: bundle.url(forResource: "english-dict", withExtension: "txt"),
bigramURL: bundle.url(forResource: "english-bigram", withExtension: "json"))
let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:] let saved = UserDefaults.standard.dictionary(forKey: learnedKey) as? [String: Int] ?? [:]
let learned = LearnedPreferences(weights: saved) { weights in let learned = LearnedPreferences(weights: saved) { weights in
UserDefaults.standard.set(weights, forKey: learnedKey) UserDefaults.standard.set(weights, forKey: learnedKey)
} }
return Classifier(lexicon: lexicon, english: english, learned: learned) let historyFile = supportDir?.appendingPathComponent("user-history.json")
let history = UserHistory(data: historyFile.map(UserHistory.load(from:)) ?? [:]) { data in
guard let url = historyFile, let json = try? JSONEncoder().encode(data) else { return }
try? FileManager.default.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
try? json.write(to: url, options: .atomic)
}
let trained = Features.loadWeights(from: bundle.url(forResource: "classifier-weights", withExtension: "json"))
NSLog("AutoZhuyin: data loaded in %.0f ms", Date().timeIntervalSince(start) * 1000)
// Only needed for untoned syllables; build it off the startup path.
DispatchQueue.global(qos: .utility).async { lexicon.prepareTonelessIndex() }
return Classifier(lexicon: lexicon, english: english, learned: learned, weights: trained, history: history)
}() }()
} }
@objc(AutoZhuyinInputController) @objc(AutoZhuyinInputController)
final class InputController: IMKInputController { final class InputController: IMKInputController {
private let engine = CompositionEngine(classifier: Shared.classifier) private let engine = CompositionEngine(classifier: Shared.classifier)
private var shiftTap = ShiftTapDetector()
// MARK: - Key events // MARK: - Key events
override func recognizedEvents(_ sender: Any!) -> Int { override func recognizedEvents(_ sender: Any!) -> Int {
Int(NSEvent.EventTypeMask.keyDown.rawValue) Int(NSEvent.EventTypeMask.keyDown.rawValue | NSEvent.EventTypeMask.flagsChanged.rawValue)
} }
override func handle(_ event: NSEvent!, client sender: Any!) -> Bool { override func handle(_ event: NSEvent!, client sender: Any!) -> Bool {
guard let event, event.type == .keyDown, let client = sender as? IMKTextInput else { return false } guard let event, let client = sender as? IMKTextInput else { return false }
// Shift pressed and released alone: next word is English (or flip the current one).
if event.type == .flagsChanged {
let flags = event.modifierFlags
let tapped = shiftTap.modifiersChanged(
shift: flags.contains(.shift),
otherModifiers: !flags.intersection([.command, .control, .option, .function]).isEmpty,
time: event.timestamp)
if tapped {
_ = engine.handle(.shiftTap)
updateComposition(client)
}
return false
}
guard event.type == .keyDown else { return false }
shiftTap.keyPressed()
// Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first. // Shortcuts (⌘C, ⌃A, ⌥…) go to the app; commit any composition first.
if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty { if !event.modifierFlags.intersection([.command, .control, .option]).isEmpty {
@@ -99,8 +134,10 @@ final class InputController: IMKInputController {
])) ]))
} }
let text = marked.string let text = marked.string
// Caret at the candidate cursor (UTF-16 offset of the first `cursorOffset` characters).
let caret = String(text.prefix(engine.cursorOffset)).utf16.count
client.setMarkedText(marked, client.setMarkedText(marked,
selectionRange: NSRange(location: (text as NSString).length, length: 0), selectionRange: NSRange(location: caret, length: 0),
replacementRange: NSRange(location: NSNotFound, length: 0)) replacementRange: NSRange(location: NSNotFound, length: 0))
if engine.isShowingCandidates { if engine.isShowingCandidates {
@@ -151,5 +188,6 @@ final class InputController: IMKInputController {
@objc private func clearLearned(_ sender: Any?) { @objc private func clearLearned(_ sender: Any?) {
Shared.classifier.learned.clear() Shared.classifier.learned.clear()
Shared.classifier.history.clear()
} }
} }
+44 -104
View File
@@ -50,48 +50,63 @@ public final class LearnedPreferences {
} }
/// Decides whether a key sequence is Zhuyin or literal (English / numbers). /// Decides whether a key sequence is Zhuyin or literal (English / numbers).
/// Every segment is scored as features · weights (see Features.swift).
public final class Classifier { public final class Classifier {
public let lexicon: Lexicon public let lexicon: Lexicon
public let english: EnglishModel public let english: EnglishModel
public let learned: LearnedPreferences public let learned: LearnedPreferences
/// Picked candidates (shared by every engine using this classifier).
public let history: UserHistory
/// Feature weights: the hand-tuned defaults overlaid with trained ones.
public private(set) var weights: FeatureVector
/// Traditional-Chinese users type Zhuyin most of the time: a small prior toward it. public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences(),
public var zhuyinPrior = 0.5 weights: FeatureVector = [:], history: UserHistory = UserHistory()) {
/// How much the previous segment's mode pulls the current one.
public var contextWeight = 1.5
public init(lexicon: Lexicon, english: EnglishModel, learned: LearnedPreferences = LearnedPreferences()) {
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned self.learned = learned
self.history = history
self.weights = Features.defaultWeights.merging(weights) { _, new in new }
} }
/// (Zhuyin features or nil, literal features or nil) for one segment. `previous` (mode of
/// the text committed just before) only affects the first segment.
public func segmentFeatures(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode? = nil) -> (FeatureVector?, FeatureVector?) {
(Features.zhuyin(seg, lexicon: lexicon, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous),
Features.literal(seg, english: english, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous))
}
private func scores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] {
let (fz, fl) = segmentFeatures(seg, isLast: isLast, afterOther: afterOther,
finishing: finishing, previous: previous)
var zh = fz.map { Features.dot(weights, $0) } ?? -.infinity
var lit = fl.map { Features.dot(weights, $0) } ?? -.infinity
if whole {
let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b }
}
return [zh, lit]
}
/// The whole buffer as one segment.
/// - Parameters: /// - Parameters:
/// - finishing: the user pressed space. Zhuyin is scored as if the space were a /// - finishing: the user pressed space. An unfinished Zhuyin syllable is scored as
/// first-tone key; literal is scored as a completed word. /// first tone; literal is scored as a completed word.
/// - previous: mode of the previously committed segment, if recent. /// - previous: mode of the previously committed segment, if recent.
public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification { public func classify(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> Classification {
var zh = zhuyinScore(finishing ? keys + [" "] : keys) + zhuyinPrior let s = scores(keys, isLast: true, afterOther: false, finishing: finishing, previous: previous, whole: true)
var lit = literalScore(keys, complete: finishing) return Classification(mode: s[0] >= s[1] ? .zhuyin : .literal, zhuyinScore: s[0], literalScore: s[1])
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
let learnedBias = learned.bias(for: String(keys))
if learnedBias > 0 { zh += learnedBias } else { lit -= learnedBias }
return Classification(mode: zh >= lit ? .zhuyin : .literal, zhuyinScore: zh, literalScore: lit)
} }
// MARK: - Segmentation // MARK: - Segmentation
/// Cost of each switch between Zhuyin and literal inside one buffer, so that a normal
/// word or sentence is not chopped up.
public var segmentPenalty = 3.0
/// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" → /// Splits a buffer into alternating Zhuyin / literal segments, e.g. "ji3hello" →
/// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point. /// [ji3: zhuyin, hello: literal]. Dynamic programming over every split point; the
/// A Zhuyin segment followed by another segment must end with a tone key. /// cost of each switch is the "seg.switch" feature.
public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] { public func segment(_ keys: [Character], finishing: Bool = false, previous: InputMode? = nil) -> [Segment] {
let n = keys.count let n = keys.count
guard n > 0 else { return [] } guard n > 0 else { return [] }
@@ -102,12 +117,12 @@ public final class Classifier {
for i in 1...n { for i in 1...n {
for j in 0..<i { for j in 0..<i {
let scores = segmentScores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, let s = scores(Array(keys[j..<i]), isLast: i == n, afterOther: j > 0, finishing: finishing,
finishing: finishing, previous: j == 0 ? previous : nil, previous: j == 0 ? previous : nil, whole: j == 0 && i == n)
whole: j == 0 && i == n) for m in 0..<2 where s[m] > -.infinity {
for m in 0..<2 where scores[m] > -.infinity {
// Adjacent segments must alternate modes; same-mode neighbours are one segment. // Adjacent segments must alternate modes; same-mode neighbours are one segment.
let total = j == 0 ? scores[m] : best[j][1 - m] + scores[m] - segmentPenalty if j > 0 && best[j][1 - m] == -.infinity { continue }
let total = j == 0 ? s[m] : best[j][1 - m] + s[m]
if total > best[i][m] { if total > best[i][m] {
best[i][m] = total best[i][m] = total
back[i][m] = j back[i][m] = j
@@ -130,79 +145,4 @@ public final class Classifier {
} }
return out.reversed() return out.reversed()
} }
/// Extra cost for a trailing Zhuyin segment after English that has no tone key yet
/// ("ji3hel" should stay 我hel, not become 我heㄠ, until a tone confirms the switch).
public var unconfirmedSwitchPenalty = 2.0
private func segmentScores(_ seg: [Character], isLast: Bool, afterOther: Bool, finishing: Bool,
previous: InputMode?, whole: Bool) -> [Double] {
var zh: Double
let parse = ZhuyinParser.parse(seg)
if !isLast && parse.pending != nil {
zh = -.infinity
} else {
zh = zhuyinScore(finishing && isLast ? seg + [" "] : seg) + zhuyinPrior
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
zh -= unconfirmedSwitchPenalty
}
}
var lit = literalScore(seg, complete: !isLast || finishing)
switch previous {
case .zhuyin?: zh += contextWeight
case .literal?: lit += contextWeight
case nil: break
}
if whole {
let b = learned.bias(for: String(seg))
if b > 0 { zh += b } else { lit -= b }
}
return [zh, lit]
}
public func zhuyinScore(_ keys: [Character]) -> Double {
let parse = ZhuyinParser.parse(keys)
if parse.foreignKeys > 0 || parse.syllables.isEmpty { return -.infinity }
var s = -5.0 * Double(parse.strayTones)
for (i, p) in parse.syllables.enumerated() {
let isLast = i == parse.syllables.count - 1
// Out-of-order typing is accepted, but is weaker evidence than normal Zhuyin.
if p.reordered { s -= 0.5 }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
s += 2 + (lexicon.bestScore(for: p.syllable) != nil ? 0.8 : -1.5)
} else {
s -= 5
}
} else if isLast {
s += SyllableTable.isValidPrefix(p.syllable) ? 0 : -5
} else {
// Untoned syllable in the middle: allowed (tone omission) but unusual.
s += valid ? -2.5 : -5
}
}
return s
}
public func literalScore(_ keys: [Character], complete: Bool) -> Double {
if keys.isEmpty || keys.contains(" ") { return -.infinity }
var score = 0.0
if keys.contains(where: { $0.isUppercase }) { score += 20 } // Shift = clearly English
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
// Pure number like "2024" (with possible "." or ",").
return score + 0.8 * Double(digits) - 1.0 * Double(others)
}
score += english.score(letters, complete: complete || digits > 0)
score -= 1.5 * Double(digits)
// `'` inside a word (don't) is fine; Zhuyin punctuation keys inside a word are not.
let apostrophes = keys.filter { $0 == "'" }.count
score -= 1.5 * Double(others - apostrophes)
return score
}
} }
+353 -20
View File
@@ -4,6 +4,8 @@ public enum KeyInput: Equatable {
case character(Character) case character(Character)
case backspace, enter, escape, tab case backspace, enter, escape, tab
case up, down, left, right case up, down, left, right
/// Shift pressed and released on its own.
case shiftTap
} }
public struct KeyResult: Equatable { public struct KeyResult: Equatable {
@@ -33,15 +35,34 @@ public final class CompositionEngine {
public let converter: Converter public let converter: Converter
public private(set) var keys: [Character] = [] public private(set) var keys: [Character] = []
/// Set by Tab: overrides the classifier for the whole buffer. /// Set by Tab / a Shift tap while typing: overrides the classifier for the whole
/// buffer. It is a correction, learned on commit.
public private(set) var forcedMode: InputMode? public private(set) var forcedMode: InputMode?
/// English mode, toggled by a Shift tap when nothing is being typed. Keys go straight
/// to the app (no composition, no underline) until Shift is tapped again.
public private(set) var isEnglishMode = false
/// User-picked phrases: Zhuyin segment start (key index) → syllable range → text. /// User-picked phrases: Zhuyin segment start (key index) → syllable range → text.
private var fixed: [Int: [Range<Int>: String]] = [:] private var fixed: [Int: [Range<Int>: String]] = [:]
public private(set) var candidates: [Candidate] = [] public private(set) var candidates: [Candidate] = []
private var candidateSegmentStart = 0 private var candidateSegmentStart = 0
/// Composition cursor: a key index into `keys` (nil = at the end). ← → move it between
/// syllables; ↓ shows candidates for the phrase before it; typing there inserts a
/// syllable (kept in `inserting` until its tone key, then spliced into `keys`).
private var cursor: Int?
private var inserting: [Character] = []
public var candidateIndex = 0 public var candidateIndex = 0
public var isShowingCandidates: Bool { !candidates.isEmpty } public var isShowingCandidates: Bool { !candidates.isEmpty }
/// Index of a held space: the first word of a composition that also reads as Zhuyin is
/// not committed as English on space; the next word decides ("up jo4" = 因為,
/// "up the" = up the, "no su3" = no 你). Right after English text only common
/// one-syllable readings are held (up = 因, url = 教), so English typing isn't delayed.
private var held: Int?
/// "`" was typed: if the next key is in `symbols` it gives that full-width symbol
/// whatever the context; anything else makes it a plain backtick (so ``` still works).
public private(set) var pendingBacktick = false
static let holdMinScore = -3.5
private var previousMode: InputMode? private var previousMode: InputMode?
private var previousCommitTime = Date.distantPast private var previousCommitTime = Date.distantPast
/// How long the previous segment's mode keeps influencing the next one. /// How long the previous segment's mode keeps influencing the next one.
@@ -50,7 +71,7 @@ public final class CompositionEngine {
public init(classifier: Classifier) { public init(classifier: Classifier) {
self.classifier = classifier self.classifier = classifier
self.converter = Converter(lexicon: classifier.lexicon) self.converter = Converter(lexicon: classifier.lexicon, history: classifier.history)
} }
// MARK: - Derived state // MARK: - Derived state
@@ -62,9 +83,52 @@ public final class CompositionEngine {
private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] { private func segments(of keys: [Character], finishing: Bool = false) -> [Segment] {
guard !keys.isEmpty else { return [] } guard !keys.isEmpty else { return [] }
if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] } if let forced = forcedMode { return [Segment(range: 0..<keys.count, mode: forced)] }
if let h = held, h < keys.count, keys[h] == " " {
return heldSegments(keys, held: h, finishing: finishing)
}
return classifier.segment(keys, finishing: finishing, previous: context) return classifier.segment(keys, finishing: finishing, previous: context)
} }
/// The held word is Chinese if the words typed after it start with Chinese.
private func heldSegments(_ keys: [Character], held h: Int, finishing: Bool) -> [Segment] {
let base = h + 1
let rest = classifier.segment(Array(keys[base...]), finishing: finishing, previous: nil).map {
Segment(range: ($0.range.lowerBound + base)..<($0.range.upperBound + base), mode: $0.mode)
}
if let first = rest.first, first.mode == .zhuyin,
heldReadsAsChinese(keys, held: h, end: first.range.upperBound, finishing: finishing) {
return [Segment(range: 0..<first.range.upperBound, mode: .zhuyin)] + rest.dropFirst()
}
return [Segment(range: 0..<base, mode: .literal)] + rest
}
/// Chinese all the way to `end` vs. the held English word followed by Chinese.
private func heldReadsAsChinese(_ keys: [Character], held h: Int, end: Int, finishing: Bool) -> Bool {
let c = classifier, w = classifier.weights, ctx = context
let isLast = end == keys.count
guard let zhWhole = c.segmentFeatures(Array(keys[..<end]), isLast: isLast, afterOther: false,
finishing: finishing, previous: ctx).0 else { return false }
guard let word = c.segmentFeatures(Array(keys[..<h]), isLast: false, afterOther: false,
finishing: true, previous: ctx).1,
let zhRest = c.segmentFeatures(Array(keys[(h + 1)..<end]), isLast: isLast, afterOther: true,
finishing: finishing, previous: nil).0 else { return true }
return Features.dot(w, zhWhole) >= Features.dot(w, word) + Features.dot(w, zhRest)
}
/// Whether keys + space also read as a common Chinese syllable (see `held`).
private func isAmbiguousWord(_ keys: [Character]) -> Bool {
guard keys.allSatisfy({ $0.isASCII && $0.isLetter }) else { return false }
let p = ZhuyinParser.parse(keys + [" "])
guard p.pending == nil, !p.syllables.isEmpty, p.strayTones == 0, p.foreignKeys == 0 else { return false }
var best: [Double] = []
for ps in p.syllables {
guard let e = classifier.lexicon.entries(for: [ps.syllable]).first else { return false }
best.append(e.score)
}
if context != .literal { return true } // first word: hold anything that reads as Zhuyin
return best.count == 1 && best[0] >= CompositionEngine.holdMinScore
}
/// Current split of the buffer into Zhuyin / literal runs. /// Current split of the buffer into Zhuyin / literal runs.
public var segments: [Segment] { segments(of: keys) } public var segments: [Segment] { segments(of: keys) }
@@ -75,6 +139,29 @@ public final class CompositionEngine {
public var markedText: String { markedPieces.map(\.text).joined() } public var markedText: String { markedPieces.map(\.text).joined() }
public var markedPieces: [MarkedPiece] { public var markedPieces: [MarkedPiece] {
var pieces = segmentPieces
if !inserting.isEmpty {
// Show the syllable being inserted at the cursor.
var at = baseCursorOffset, out: [MarkedPiece] = [], done = false
for piece in pieces {
if !done && at <= piece.text.count {
let i = piece.text.index(piece.text.startIndex, offsetBy: at)
out.append(MarkedPiece(text: String(piece.text[..<i]), mode: piece.mode))
out.append(MarkedPiece(text: insertingText, mode: .zhuyin))
out.append(MarkedPiece(text: String(piece.text[i...]), mode: piece.mode))
done = true
} else {
out.append(piece)
at -= piece.text.count
}
}
pieces = out.filter { !$0.text.isEmpty }
}
if pendingBacktick { pieces.append(MarkedPiece(text: "`", mode: .literal)) }
return pieces
}
private var segmentPieces: [MarkedPiece] {
let segs = segments let segs = segments
return segs.enumerated().map { i, seg in return segs.enumerated().map { i, seg in
MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode) MarkedPiece(text: render(seg, final: i < segs.count - 1), mode: seg.mode)
@@ -83,35 +170,205 @@ public final class CompositionEngine {
/// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo. /// Text for one segment. A non-final Zhuyin segment shows its unfinished syllable as Bopomofo.
private func render(_ seg: Segment, final: Bool) -> String { private func render(_ seg: Segment, final: Bool) -> String {
let segKeys = Array(keys[seg.range]) guard seg.mode == .zhuyin else { return String(keys[seg.range]) }
guard seg.mode == .zhuyin else { return String(segKeys) } let (converted, pending) = conversion(of: seg, final: final)
let p = ZhuyinParser.parse(segKeys) return converted.map(\.text).joined() + pending
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]) }
if final {
let all = p.syllables.map(\.syllable) /// Syllables of a Zhuyin segment that are converted to Chinese (all of them when
return converter.convert(all, fixed: fixedHere.filter { $0.key.upperBound <= all.count }).map(\.text).joined() /// final, otherwise the finished ones).
private func convertibleSyllables(_ seg: Segment, final: Bool) -> (syllables: [Syllable], pending: String) {
let p = ZhuyinParser.parse(Array(keys[seg.range]))
if final { return (p.syllables.map(\.syllable), "") }
return (p.completed.map(\.syllable), p.pending?.syllable.reading ?? "")
}
private func conversion(of seg: Segment, final: Bool) -> (segments: [ConvertedSegment], pending: String) {
let (syl, pending) = convertibleSyllables(seg, final: final)
let fixedHere = (fixed[seg.range.lowerBound] ?? [:]).filter { $0.key.upperBound <= syl.count }
return (converter.convert(syl, fixed: fixedHere), pending)
}
// MARK: - Cursor
/// Absolute key index after each syllable of a Zhuyin segment.
private func syllableEnds(_ seg: Segment) -> [Int] {
ZhuyinParser.parse(Array(keys[seg.range])).syllables.map { seg.range.lowerBound + $0.keyRange.upperBound }
}
/// Key indices the cursor can stop at: after every Zhuyin syllable and at every segment
/// boundary, but never before the first Chinese character (there would be nothing to
/// pick candidates for).
private var cursorPositions: [Int] {
var out: Set<Int> = [keys.count]
var ends: [Int] = []
for seg in segments {
out.insert(seg.range.upperBound)
if seg.mode == .zhuyin { ends += syllableEnds(seg) }
} }
let done = p.completed.map(\.syllable) out.formUnion(ends)
let text = converter.convert(done, fixed: fixedHere.filter { $0.key.upperBound <= done.count }).map(\.text).joined() let first = ends.min() ?? keys.count
return text + (p.pending?.syllable.reading ?? "") return out.filter { $0 >= first }.sorted()
}
/// A place to open candidates: after the `local`-th syllable of Zhuyin segment `segIndex`.
private struct CursorStop { let segIndex: Int; let local: Int }
private func stop(at pos: Int) -> CursorStop? {
let segs = segments
for (i, seg) in segs.enumerated() where seg.mode == .zhuyin
&& seg.range.lowerBound < pos && pos <= seg.range.upperBound {
let n = convertibleSyllables(seg, final: i < segs.count - 1).syllables.count
let local = syllableEnds(seg).filter { $0 <= pos }.count
return local > 0 ? CursorStop(segIndex: i, local: min(local, n)) : nil
}
return nil
}
/// Where ↓ opens candidates: before the cursor, or the last syllable.
private var candidateStop: CursorStop? {
if let c = cursor { return stop(at: c) }
let segs = segments
for i in segs.indices.reversed() where segs[i].mode == .zhuyin {
let n = convertibleSyllables(segs[i], final: i < segs.count - 1).syllables.count
if n > 0 { return CursorStop(segIndex: i, local: n) }
}
return nil
}
/// Cursor position in the rendered segments (without the syllable being inserted).
private var baseCursorOffset: Int {
let pieces = segmentPieces
guard let c = cursor else { return pieces.reduce(0) { $0 + $1.text.count } }
let segs = segments
for (i, seg) in segs.enumerated() where seg.range.lowerBound < c && c <= seg.range.upperBound {
var offset = pieces[..<i].reduce(0) { $0 + $1.text.count }
if seg.mode == .literal { return offset + c - seg.range.lowerBound }
let local = stop(at: c)?.local ?? 0
for conv in conversion(of: seg, final: i < segs.count - 1).segments {
if conv.range.upperBound <= local {
offset += conv.text.count
} else if conv.range.lowerBound < local {
offset += min(local - conv.range.lowerBound, conv.text.count)
}
}
return offset
}
return 0
}
/// Cursor position in `markedText`, in Characters.
public var cursorOffset: Int { baseCursorOffset + insertingText.count }
private var insertingText: String {
inserting.isEmpty ? "" : ZhuyinParser.parse(inserting).syllables.map(\.syllable.reading).joined()
}
private func moveCursor(by delta: Int) {
inserting = []
let stops = cursorPositions
let here = cursor ?? keys.count
let next = delta < 0 ? (stops.last { $0 < here } ?? here) : (stops.first { $0 > here } ?? here)
cursor = next >= keys.count ? nil : next
}
// MARK: - Editing in the middle
/// Keeps picked phrases that end at or before key index `pos` (they don't move).
private func trimFixed(at pos: Int) {
var kept: [Int: [Range<Int>: String]] = [:]
for seg in segments where seg.mode == .zhuyin {
guard let picks = fixed[seg.range.lowerBound] else { continue }
if seg.range.upperBound <= pos {
kept[seg.range.lowerBound] = picks
} else if seg.range.lowerBound < pos {
let ends = syllableEnds(seg)
let here = picks.filter { $0.key.upperBound <= ends.count && ends[$0.key.upperBound - 1] <= pos }
if !here.isEmpty { kept[seg.range.lowerBound] = here }
}
}
fixed = kept
}
/// Inserts keys at the cursor and moves the cursor after them.
private func splice(_ newKeys: [Character]) {
guard let pos = cursor else { keys += newKeys; return }
trimFixed(at: pos)
keys.insert(contentsOf: newKeys, at: pos)
if let h = held, h >= pos { held = h + newKeys.count }
let next = pos + newKeys.count
cursor = next >= keys.count ? nil : next
}
/// Backspace in the middle: removes the syllable (or literal key) before the cursor.
private func deleteBeforeCursor(_ pos: Int) {
var start = pos - 1
for seg in segments where seg.mode == .zhuyin && seg.range.lowerBound < pos && pos <= seg.range.upperBound {
for ps in ZhuyinParser.parse(Array(keys[seg.range])).syllables
where seg.range.lowerBound + ps.keyRange.upperBound == pos {
start = seg.range.lowerBound + ps.keyRange.lowerBound
}
}
trimFixed(at: start)
keys.removeSubrange(start..<pos)
if let h = held {
if h >= start && h < pos { held = nil } else if h >= pos { held = h - (pos - start) }
}
cursor = start
if keys.isEmpty { reset() }
}
/// Typing with the cursor in the middle: compose one syllable, splice it in on its tone.
private func insertKey(_ c: Character) -> KeyResult {
if c == " " {
if inserting.isEmpty { return KeyResult(handled: true, commit: commitCurrent()) }
splice(inserting + [" "])
inserting = []
return .consumed
}
inserting.append(c)
let p = ZhuyinParser.parse(inserting)
if p.strayTones > 0 || p.foreignKeys > 0 {
inserting.removeLast() // a tone with nothing to attach to
} else if p.syllables.count == 1, p.syllables[0].toned {
splice(inserting)
inserting = []
} else if p.syllables.count > 1 {
// A new syllable started without a tone: splice the first one as it is.
let first = p.syllables[0].keyRange.upperBound
splice(Array(inserting[..<first]))
inserting = Array(inserting[first...])
}
return .consumed
} }
// MARK: - Key handling // MARK: - Key handling
public func handle(_ key: KeyInput) -> KeyResult { public func handle(_ key: KeyInput) -> KeyResult {
if isShowingCandidates { return handleCandidateKey(key) } if isShowingCandidates { return handleCandidateKey(key) }
if pendingBacktick { return afterBacktick(key) }
if !inserting.isEmpty && (key == .backspace || key == .escape) {
if key == .backspace { inserting.removeLast() } else { inserting = [] }
return .consumed
}
switch key { switch key {
case .character(let c): case .character(let c):
return handleCharacter(c) return handleCharacter(c)
case .backspace: case .backspace:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
if let c = cursor {
if c > 0 { deleteBeforeCursor(c) }
return .consumed
}
keys.removeLast() keys.removeLast()
if let h = held, keys.count <= h { held = nil }
fixed = fixed.filter { $0.key < keys.count } fixed = fixed.filter { $0.key < keys.count }
if keys.isEmpty { forcedMode = nil; fixed = [:] } if keys.isEmpty { forcedMode = nil; fixed = [:] }
return .consumed return .consumed
case .enter: case .enter:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
if !inserting.isEmpty { splice(inserting); inserting = [] }
return KeyResult(handled: true, commit: commitCurrent()) return KeyResult(handled: true, commit: commitCurrent())
case .escape: case .escape:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
@@ -119,22 +376,61 @@ public final class CompositionEngine {
return .consumed return .consumed
case .tab: case .tab:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin flip()
return .consumed
case .shiftTap:
if keys.isEmpty {
// Toggle English mode; it stays on until the next Shift tap.
isEnglishMode.toggle()
previousMode = isEnglishMode ? .literal : nil
previousCommitTime = now()
} else {
flip()
}
return .consumed return .consumed
case .down: case .down:
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
openCandidates() if inserting.isEmpty { openCandidates() }
return .consumed return .consumed
case .up, .left, .right: case .left:
guard !keys.isEmpty else { return .passThrough }
moveCursor(by: -1)
return .consumed
case .right:
guard !keys.isEmpty else { return .passThrough }
moveCursor(by: 1)
return .consumed
case .up:
return keys.isEmpty ? .passThrough : .consumed return keys.isEmpty ? .passThrough : .consumed
} }
} }
/// Forces the whole buffer into the other mode (a correction that is learned).
private func flip() {
forcedMode = (mode == .zhuyin) ? .literal : .zhuyin
}
private func handleCharacter(_ c: Character) -> KeyResult { private func handleCharacter(_ c: Character) -> KeyResult {
if cursor != nil, !keys.isEmpty, c == " " || DachenLayout.isZhuyinKey(c) {
return insertKey(c)
}
if !inserting.isEmpty { splice(inserting); inserting = [] }
cursor = nil // anything else continues at the end
// English mode: every key goes to the app as typed.
if isEnglishMode && keys.isEmpty { return .passThrough }
if c == "`" {
pendingBacktick = true
return .consumed
}
if c == " " { if c == " " {
guard !keys.isEmpty else { return .passThrough } guard !keys.isEmpty else { return .passThrough }
let segs = segments(of: keys, finishing: true) let segs = segments(of: keys, finishing: true)
if segs.last?.mode == .literal { if segs.last?.mode == .literal {
if held == nil, segs.count == 1, forcedMode == nil, isAmbiguousWord(keys) {
held = keys.count
keys.append(" ")
return .consumed
}
return KeyResult(handled: true, commit: commitCurrent(segs) + " ") return KeyResult(handled: true, commit: commitCurrent(segs) + " ")
} }
if parse(ofLast: segs)?.pending != nil { if parse(ofLast: segs)?.pending != nil {
@@ -171,6 +467,26 @@ public final class CompositionEngine {
return ZhuyinParser.parse(Array(keys[last.range])) return ZhuyinParser.parse(Array(keys[last.range]))
} }
private func afterBacktick(_ key: KeyInput) -> KeyResult {
pendingBacktick = false
if key == .escape || key == .backspace { return .consumed }
let committed = keys.isEmpty ? "" : commitCurrent()
if case .character(let c) = key, let symbol = CompositionEngine.symbols[c] {
return KeyResult(handled: true, commit: committed + symbol)
}
// Not a symbol: the backtick was meant literally; then handle the key as usual.
var r = handle(key)
if !r.handled, case .character(let c) = key { r = KeyResult(handled: true, commit: String(c)) }
return KeyResult(handled: r.handled, commit: committed + "`" + (r.commit ?? ""))
}
/// "`" + key → full-width symbol.
static let symbols: [Character: String] = [
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", "\"": "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
]
static let fullwidth: [Character: String] = [ static let fullwidth: [Character: String] = [
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", "\"": "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
@@ -178,13 +494,14 @@ public final class CompositionEngine {
// MARK: - Candidates (for the last Zhuyin segment) // MARK: - Candidates (for the last Zhuyin segment)
/// Opens candidates for the phrase ending at the cursor (default: the last syllable).
private func openCandidates() { private func openCandidates() {
guard let stop = candidateStop else { return }
let segs = segments let segs = segments
guard let last = segs.last, last.mode == .zhuyin else { return } let seg = segs[stop.segIndex]
let done = ZhuyinParser.parse(Array(keys[last.range])).completed.map(\.syllable) let syl = convertibleSyllables(seg, final: stop.segIndex < segs.count - 1).syllables
guard !done.isEmpty else { return } candidates = converter.candidates(syl, endingAt: stop.local)
candidates = converter.candidates(done, endingAt: done.count) candidateSegmentStart = seg.range.lowerBound
candidateSegmentStart = last.range.lowerBound
candidateIndex = 0 candidateIndex = 0
} }
@@ -224,6 +541,7 @@ public final class CompositionEngine {
private func commitCurrent(_ given: [Segment]? = nil) -> String { private func commitCurrent(_ given: [Segment]? = nil) -> String {
let segs = given ?? segments let segs = given ?? segments
let text = segs.map { render($0, final: true) }.joined() let text = segs.map { render($0, final: true) }.joined()
rememberPicks(segs)
if let forced = forcedMode { if let forced = forcedMode {
classifier.learned.record(String(keys), as: forced) classifier.learned.record(String(keys), as: forced)
} }
@@ -233,6 +551,17 @@ public final class CompositionEngine {
return text return text
} }
/// Candidates the user picked are remembered and come first next time.
private func rememberPicks(_ segs: [Segment]) {
for seg in segs where seg.mode == .zhuyin {
guard let picks = fixed[seg.range.lowerBound], !picks.isEmpty else { continue }
let syl = convertibleSyllables(seg, final: true).syllables
for (range, text) in picks where range.upperBound <= syl.count {
classifier.history.record(Array(syl[range]), phrase: text)
}
}
}
/// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty. /// Commits whatever is composed (e.g. the app lost focus). Returns "" when empty.
public func commitAll() -> String { public func commitAll() -> String {
keys.isEmpty ? "" : commitCurrent() keys.isEmpty ? "" : commitCurrent()
@@ -240,6 +569,10 @@ public final class CompositionEngine {
public func reset() { public func reset() {
keys = [] keys = []
inserting = []
held = nil
pendingBacktick = false
cursor = nil
forcedMode = nil forcedMode = nil
fixed = [:] fixed = [:]
candidates = [] candidates = []
+13 -3
View File
@@ -13,12 +13,22 @@ public struct Candidate: Equatable {
/// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon). /// Turns a syllable sequence into the most likely Chinese text (Viterbi over the lexicon).
public struct Converter { public struct Converter {
public let lexicon: Lexicon public let lexicon: Lexicon
/// Picked candidates come first.
public let history: UserHistory?
/// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted. /// Score used when a syllable has no lexicon entry; the Bopomofo itself is emitted.
static let unknownScore = -30.0 static let unknownScore = -30.0
/// Penalty for matching an untoned syllable against all tones. /// Penalty for matching an untoned syllable against all tones.
static let tonelessPenalty = -1.5 static let tonelessPenalty = -1.5
public init(lexicon: Lexicon) { self.lexicon = lexicon } public init(lexicon: Lexicon, history: UserHistory? = nil) {
self.lexicon = lexicon
self.history = history
}
private func entries(_ syllables: [Syllable]) -> [LexiconEntry] {
let e = lexicon.entries(for: syllables)
return history?.adjust(syllables, e) ?? e
}
/// - Parameter fixed: user-chosen phrases by syllable range; they are always kept. /// - Parameter fixed: user-chosen phrases by syllable range; they are always kept.
public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] { public func convert(_ syllables: [Syllable], fixed: [Range<Int>: String] = [:]) -> [ConvertedSegment] {
@@ -57,7 +67,7 @@ public struct Converter {
if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] } if fixed.keys.contains(where: { $0.overlaps(range) }) { return [] }
let slice = Array(syllables[range]) let slice = Array(syllables[range])
var result = lexicon.entries(for: slice).map { entry -> (String, Double) in var result = entries(slice).map { entry -> (String, Double) in
let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0 let penalty = slice.contains { $0.tone == nil } ? Converter.tonelessPenalty : 0
return (entry.phrase, entry.score + penalty) return (entry.phrase, entry.score + penalty)
} }
@@ -74,7 +84,7 @@ public struct Converter {
var seen = Set<String>() var seen = Set<String>()
for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) { for len in stride(from: min(lexicon.maxPhraseLength, end), through: 1, by: -1) {
let range = (end - len)..<end let range = (end - len)..<end
for entry in lexicon.entries(for: Array(syllables[range])) where !seen.contains(entry.phrase) { for entry in entries(Array(syllables[range])) where !seen.contains(entry.phrase) {
seen.insert(entry.phrase) seen.insert(entry.phrase)
out.append(Candidate(range: range, text: entry.phrase)) out.append(Candidate(range: range, text: entry.phrase))
} }
+74 -15
View File
@@ -8,33 +8,63 @@ import Foundation
/// - a letter-bigram model trained on those words, for words not in either list. /// - a letter-bigram model trained on those words, for words not in either list.
public final class EnglishModel { public final class EnglishModel {
private var common = Set<String>() private var common = Set<String>()
private var dictionary = Set<String>()
private var sortedWords: [String] = []
private var sortedCommon: [String] = [] private var sortedCommon: [String] = []
/// Dictionary words (lowercase, > 2 letters), sorted: membership and prefix checks are
/// binary searches, so no 230k-entry Set has to be built at startup.
private var sortedDict: [String] = []
private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries private var bigram: [String: Double] = [:] // "ab" -> log2 P(b|a); "^a", "a$" for boundaries
private var unseenBigram = -12.0 private var unseenBigram = -12.0
public init(commonWords: [String], dictionaryWords: [String] = []) { /// - Parameter bigram: precomputed table (english-bigram.json); trained from the words if nil.
public init(commonWords: [String], dictionaryWords: [String] = [],
bigram: (table: [String: Double], unseen: Double)? = nil) {
common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty }) common = Set(commonWords.map { $0.lowercased() }.filter { !$0.isEmpty })
dictionary = Set(dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 })
sortedCommon = common.sorted() sortedCommon = common.sorted()
sortedWords = common.union(dictionary).sorted() var dict = dictionaryWords.map { $0.lowercased() }.filter { $0.count > 2 }
trainBigram() if !zip(dict, dict.dropFirst()).allSatisfy({ $0 < $1 }) {
dict = Array(Set(dict)).sorted()
}
sortedDict = dict
if let bigram {
self.bigram = bigram.table
unseenBigram = bigram.unseen
} else {
trainBigram()
}
} }
/// Loads common words from a bundle resource file and the system dictionary if present. /// Loads the bundled data files. Falls back to the system word list and to training
public convenience init(commonWordsURL: URL?, systemDictionary: URL? = URL(fileURLWithPath: "/usr/share/dict/words")) { /// the bigram table when the precomputed files are missing.
public convenience init(commonWordsURL: URL?, dictionaryURL: URL? = nil, bigramURL: URL? = nil) {
func lines(_ url: URL?) -> [String] { func lines(_ url: URL?) -> [String] {
guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] } guard let url, let text = try? String(contentsOf: url, encoding: .utf8) else { return [] }
return text.split(whereSeparator: \.isNewline) return text.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) } .map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.hasPrefix("#") } .filter { !$0.isEmpty && !$0.hasPrefix("#") }
} }
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(systemDictionary)) var bigram: (table: [String: Double], unseen: Double)?
if let url = bigramURL, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let table = obj["table"] as? [String: Double], let unseen = obj["unseen"] as? Double {
bigram = (table, unseen)
}
let dictURL = dictionaryURL.flatMap { FileManager.default.fileExists(atPath: $0.path) ? $0 : nil }
?? URL(fileURLWithPath: "/usr/share/dict/words")
self.init(commonWords: lines(commonWordsURL), dictionaryWords: lines(dictURL), bigram: bigram)
} }
public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) } public func isCommon(_ w: String) -> Bool { common.contains(w.lowercased()) }
public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || dictionary.contains(l) } public func isWord(_ w: String) -> Bool { let l = w.lowercased(); return common.contains(l) || inDictionary(l) }
private func inDictionary(_ w: String) -> Bool {
let i = lowerBound(w, in: sortedDict)
return i < sortedDict.count && sortedDict[i] == w
}
/// Some word strictly longer than `p` starts with it (common words or dictionary).
private func isPrefixOfWord(_ p: String) -> Bool {
hasPrefix(p, in: sortedCommon) || hasPrefix(p, in: sortedDict)
}
/// - Parameter complete: true when the user ended the word (space / punctuation), /// - Parameter complete: true when the user ended the word (space / punctuation),
/// false while still typing (prefixes of words get partial credit). /// false while still typing (prefixes of words get partial credit).
@@ -43,16 +73,40 @@ public final class EnglishModel {
let n = Double(w.count) let n = Double(w.count)
guard !w.isEmpty else { return 0 } guard !w.isEmpty else { return 0 }
if common.contains(w) { return 3 + n } if common.contains(w) { return 3 + n }
if dictionary.contains(w) { return 1 + 0.6 * n } if inDictionary(w) { return 1 + 0.6 * n }
if !complete { if !complete {
if hasPrefix(w, in: sortedCommon) { return 0.4 * n } if hasPrefix(w, in: sortedCommon) { return 0.4 * n }
if hasPrefix(w, in: sortedWords) { return 0.2 * n } if isPrefixOfWord(w) { return 0.2 * n }
} }
// Unknown word: judge by how English-like its letter sequence is. // Unknown word: judge by how English-like its letter sequence is.
let plaus = plausibility(w) let plaus = plausibility(w)
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
} }
/// Feature dict for the classifier (see Features.swift); `score` is their dot
/// product with the default weights.
public func wordFeatures(_ word: String, complete: Bool) -> FeatureVector {
let w = word.lowercased()
let n = Double(w.count)
var f: FeatureVector = [:]
if n <= 2 { f["lit.short"] = 1 }
if common.contains(w) {
f["lit.common"] = 1
f["lit.common_len"] = n
} else if inDictionary(w) {
f["lit.dict"] = 1
f["lit.dict_len"] = n
} else if !complete && hasPrefix(w, in: sortedCommon) {
f["lit.prefix_common_len"] = n
} else if !complete && isPrefixOfWord(w) {
f["lit.prefix_dict_len"] = n
} else {
let plaus = plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
}
return f
}
/// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish). /// Mean log2 probability per letter transition (≈ -3.5 for English, lower for gibberish).
public func plausibility(_ w: String) -> Double { public func plausibility(_ w: String) -> Double {
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
@@ -61,12 +115,17 @@ public final class EnglishModel {
return total / Double(chars.count - 1) return total / Double(chars.count - 1)
} }
private func hasPrefix(_ p: String, in words: [String]) -> Bool { private func lowerBound(_ p: String, in words: [String]) -> Int {
var lo = 0, hi = words.count var lo = 0, hi = words.count
while lo < hi { while lo < hi {
let mid = (lo + hi) / 2 let mid = (lo + hi) / 2
if words[mid] < p { lo = mid + 1 } else { hi = mid } if words[mid] < p { lo = mid + 1 } else { hi = mid }
} }
return lo
}
private func hasPrefix(_ p: String, in words: [String]) -> Bool {
var lo = lowerBound(p, in: words)
// words[lo] is the first word >= p; skip an exact match, then test the next one. // words[lo] is the first word >= p; skip an exact match, then test the next one.
if lo < words.count && words[lo] == p { lo += 1 } if lo < words.count && words[lo] == p { lo += 1 }
return lo < words.count && words[lo].hasPrefix(p) return lo < words.count && words[lo].hasPrefix(p)
@@ -75,7 +134,7 @@ public final class EnglishModel {
private func trainBigram() { private func trainBigram() {
var pair: [String: Double] = [:] var pair: [String: Double] = [:]
var first: [String: Double] = [:] var first: [String: Double] = [:]
for w in sortedWords where w.allSatisfy({ $0.isLetter && $0.isASCII }) { for w in Set(sortedCommon + sortedDict).sorted() where w.allSatisfy({ $0.isLetter && $0.isASCII }) {
let weight = common.contains(w) ? 5.0 : 1.0 let weight = common.contains(w) ? 5.0 : 1.0
let chars = ["^"] + w.map(String.init) + ["$"] let chars = ["^"] + w.map(String.init) + ["$"]
for i in 1..<chars.count { for i in 1..<chars.count {
+151
View File
@@ -0,0 +1,151 @@
import Foundation
/// Segment features for the Zhuyin / literal classifier. A segment's score in each mode
/// is the dot product of its features with the weights. `defaultWeights` reproduces the
/// original hand-tuned scores; learned weights come from Resources/classifier-weights.json
/// (training/train.py).
///
/// Python twin: windows/autozhuyin/core/features.py — both must produce identical
/// features (checked by ParityTests against training/parity-fixtures.json).
public typealias FeatureVector = [String: Double]
public enum Features {
public static let defaultWeights: FeatureVector = [
// Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
// Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
// Segmentation
"seg.switch": -3.0,
// Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
]
/// Reads {"weights": {...}} from a JSON file; empty if missing or invalid.
public static func loadWeights(from url: URL?) -> FeatureVector {
guard let url, let data = try? Data(contentsOf: url),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let w = obj["weights"] as? [String: Double] else { return [:] }
return w
}
public static func dot(_ w: FeatureVector, _ f: FeatureVector) -> Double {
f.reduce(0) { $0 + (w[$1.key] ?? 0) * $1.value }
}
/// Features of `keys` read as Zhuyin, or nil if impossible.
public static func zhuyin(_ keys: [Character], lexicon: Lexicon, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
let parse = ZhuyinParser.parse(keys)
// A Zhuyin segment followed by another must end with a tone.
if !isLast && parse.pending != nil { return nil }
// Space finishes an untoned last syllable as first tone; after a toned syllable it
// just commits, so it must not count as an extra (stray) tone.
let scored = finishing && isLast && parse.pending != nil ? ZhuyinParser.parse(keys + [" "]) : parse
if scored.foreignKeys > 0 || scored.syllables.isEmpty { return nil }
var f: FeatureVector = ["zh.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if scored.strayTones > 0 { add("zh.stray", Double(scored.strayTones)) }
let last = scored.syllables.count - 1
var tonedRun: [Syllable] = []
for (i, p) in scored.syllables.enumerated() {
if p.reordered { add("zh.reordered") }
let valid = SyllableTable.isValid(p.syllable)
if p.toned {
if valid {
if let best = lexicon.bestScore(for: p.syllable) {
add("zh.toned_valid_lex")
add("zh.lexscore", best + 4)
} else {
add("zh.toned_valid_nolex")
}
tonedRun.append(p.syllable)
} else {
add("zh.toned_invalid")
}
} else if i == last {
add(SyllableTable.isValidPrefix(p.syllable) ? "zh.pending_prefix" : "zh.pending_invalid")
} else {
add(valid ? "zh.untoned_mid_valid" : "zh.untoned_mid_invalid")
}
}
// Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
var covered = [Bool](repeating: false, count: tonedRun.count)
for i in tonedRun.indices {
for n in 2...4 where i + n <= tonedRun.count {
if !lexicon.entries(for: Array(tonedRun[i..<(i + n)])).isEmpty {
for k in i..<(i + n) { covered[k] = true }
}
}
}
let coveredCount = covered.filter { $0 }.count
if coveredCount > 0 { add("zh.phrase", Double(coveredCount)) }
if isLast && afterOther && !finishing && !parse.syllables.contains(where: \.toned) {
add("zh.unconfirmed")
}
if afterOther {
add("seg.switch")
} else if previous == .zhuyin {
add("ctx.zh")
}
return f
}
/// Features of `keys` output as typed, or nil if impossible.
public static func literal(_ keys: [Character], english: EnglishModel, isLast: Bool = true,
afterOther: Bool = false, finishing: Bool = false,
previous: InputMode? = nil) -> FeatureVector? {
if keys.isEmpty || keys.contains(" ") { return nil }
let complete = !isLast || finishing
var f: FeatureVector = ["lit.bias": 1]
func add(_ k: String, _ v: Double = 1) { f[k, default: 0] += v }
if afterOther {
add("seg.switch")
} else if previous == .literal {
add("ctx.lit")
}
if keys.contains(where: { $0.isUppercase }) { add("lit.upper") }
let letters = String(keys.filter { $0.isLetter })
let digits = keys.filter { $0.isNumber }.count
let others = keys.count - letters.count - digits
if letters.isEmpty {
if digits > 0 { add("lit.num_digits", Double(digits)) }
if others > 0 { add("lit.num_others", Double(others)) }
return f
}
for (k, v) in english.wordFeatures(letters, complete: complete || digits > 0) { add(k, v) }
if digits > 0 { add("lit.digits", Double(digits)) }
let nonApostrophe = others - keys.filter { $0 == "'" }.count
if nonApostrophe > 0 { add("lit.others", Double(nonApostrophe)) }
return f
}
}
+138 -28
View File
@@ -11,58 +11,159 @@ public struct LexiconEntry: Equatable {
/// File format, one entry per line (`#` starts a comment): /// File format, one entry per line (`#` starts a comment):
/// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2 /// ㄋㄧˇ-ㄏㄠˇ 你好 -4.2
/// The phrase/reading columns may be in either order; the score is optional. /// The phrase/reading columns may be in either order; the score is optional.
///
/// Loading is on the input method's startup path, so the file is parsed as raw bytes,
/// and the tone-ignoring index (only needed for untoned syllables) is built lazily —
/// call `prepareTonelessIndex()` from a background queue to have it ready.
public final class Lexicon { public final class Lexicon {
private var exact: [String: [LexiconEntry]] = [:] private var exact: [String: [LexiconEntry]] = [:]
private var toneless: [String: [LexiconEntry]] = [:] private var tonelessIndex: [String: [LexiconEntry]]?
private let lock = NSLock()
public private(set) var maxPhraseLength = 1 public private(set) var maxPhraseLength = 1
public init() {} public init() {}
public convenience init(contentsOf url: URL) throws { public convenience init(contentsOf url: URL) throws {
self.init() self.init()
load(try String(contentsOf: url, encoding: .utf8)) load(data: try Data(contentsOf: url))
} }
public func load(_ text: String) { public func load(_ text: String) {
for line in text.split(whereSeparator: \.isNewline) { load(data: Data(text.utf8))
let trimmed = line.trimmingCharacters(in: .whitespaces)
if trimmed.isEmpty || trimmed.hasPrefix("#") { continue }
let cols = trimmed.split(whereSeparator: { $0 == " " || $0 == "\t" }).map(String.init)
guard cols.count >= 2 else { continue }
let (reading, phrase) = Lexicon.isBopomofo(cols[0]) ? (cols[0], cols[1]) : (cols[1], cols[0])
guard Lexicon.isBopomofo(reading) else { continue }
let score = cols.count >= 3 ? Double(cols[2]) ?? -6 : -6
add(reading: reading, phrase: phrase, score: score)
}
for key in exact.keys { exact[key]?.sort { $0.score > $1.score } }
for key in toneless.keys { toneless[key]?.sort { $0.score > $1.score } }
} }
public func add(reading rawReading: String, phrase: String, score: Double) { public func load(data: Data) {
let entry = LexiconEntry(phrase: phrase, score: score) lock.lock()
defer { lock.unlock() }
exact.reserveCapacity(exact.count + data.count / 30)
data.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in
let bytes = raw.bindMemory(to: UInt8.self)
let n = bytes.count
var i = 0
while i < n {
var end = i
while end < n && bytes[end] != 0x0A { end += 1 }
parseLine(bytes, i, end)
i = end + 1
}
}
for (key, entries) in exact where entries.count > 1 {
exact[key] = entries.sorted { $0.score > $1.score }
}
tonelessIndex = nil // rebuilt on demand
}
/// Splits one line into up to three space/tab separated fields.
private func parseLine(_ b: UnsafeBufferPointer<UInt8>, _ start: Int, _ end: Int) {
var fields: [Range<Int>] = []
var i = start
while i < end && fields.count < 3 {
while i < end && (b[i] == 0x20 || b[i] == 0x09 || b[i] == 0x0D) { i += 1 }
if i >= end { break }
if fields.isEmpty && b[i] == 0x23 { return } // "#" comment
let s = i
while i < end && b[i] != 0x20 && b[i] != 0x09 && b[i] != 0x0D { i += 1 }
fields.append(s..<i)
}
guard fields.count >= 2 else { return }
func str(_ r: Range<Int>) -> String { String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self) }
let (r, p) = Lexicon.isBopomofo(b, fields[0]) ? (fields[0], fields[1]) : (fields[1], fields[0])
guard Lexicon.isBopomofo(b, r) else { return }
let score = fields.count >= 3 ? Lexicon.parseDouble(b, fields[2]) ?? -6 : -6
// Neutral tone written as a prefix ("˙ㄉㄜ") needs normalising; detect it on bytes.
var prefixNeutral = b[r.lowerBound] == 0xCB
var k = r.lowerBound
while !prefixNeutral && k + 2 < r.upperBound {
if b[k] == 0x2D && b[k + 1] == 0xCB && b[k + 2] == 0x99 { prefixNeutral = true }
k += 1
}
addEntry(reading: str(r), phrase: str(p), score: score, normalise: prefixNeutral)
}
/// Parses "-3.9151"-style decimals without creating a String.
private static func parseDouble(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Double? {
var i = r.lowerBound
var sign = 1.0
if i < r.upperBound && (b[i] == 0x2D || b[i] == 0x2B) { sign = b[i] == 0x2D ? -1 : 1; i += 1 }
var value = 0.0, scale = 1.0, digits = 0, fraction = false
while i < r.upperBound {
let c = b[i]
if c >= 0x30 && c <= 0x39 {
if fraction { scale /= 10; value += Double(c - 0x30) * scale } else { value = value * 10 + Double(c - 0x30) }
digits += 1
} else if c == 0x2E && !fraction {
fraction = true
} else {
return Double(String(decoding: UnsafeBufferPointer(rebasing: b[r]), as: UTF8.self))
}
i += 1
}
return digits > 0 ? sign * value : nil
}
/// Adds one entry (call `load` for bulk data; this does not re-sort).
public func add(reading: String, phrase: String, score: Double) {
lock.lock()
defer { lock.unlock() }
let key = addEntry(reading: reading, phrase: phrase, score: score, normalise: reading.contains("˙"))
exact[key]?.sort { $0.score > $1.score }
tonelessIndex = nil
}
@discardableResult
private func addEntry(reading rawReading: String, phrase: String, score: Double, normalise: Bool) -> String {
var reading = rawReading
// Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix. // Some data sets write the neutral tone as a prefix ("˙ㄉㄜ"); we use a suffix.
let syllables = rawReading.split(separator: "-").map { s -> String in if normalise {
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s) reading = reading.split(separator: "-").map { s -> String in
s.hasPrefix("˙") ? String(s.dropFirst()) + "˙" : String(s)
}.joined(separator: "-")
} }
let reading = syllables.joined(separator: "-") maxPhraseLength = max(maxPhraseLength, reading.utf8.reduce(1) { $0 + ($1 == 0x2D ? 1 : 0) })
maxPhraseLength = max(maxPhraseLength, syllables.count) if exact[reading] == nil {
if !(exact[reading]?.contains { $0.phrase == phrase } ?? false) { exact[reading] = [LexiconEntry(phrase: phrase, score: score)]
exact[reading, default: []].append(entry) } else if !exact[reading]!.contains(where: { $0.phrase == phrase }) {
exact[reading]!.append(LexiconEntry(phrase: phrase, score: score))
} }
let key = Lexicon.stripTones(reading) return reading
if let i = toneless[key]?.firstIndex(where: { $0.phrase == phrase }) { }
toneless[key]![i].score = max(toneless[key]![i].score, score)
} else { /// Builds the index that ignores tones (for untoned syllables). Safe to call from a
toneless[key, default: []].append(entry) /// background queue; lookups build it themselves if it is not ready yet.
public func prepareTonelessIndex() {
lock.lock()
defer { lock.unlock() }
_ = buildTonelessIfNeeded()
}
private func buildTonelessIfNeeded() -> [String: [LexiconEntry]] {
if let index = tonelessIndex { return index }
var best: [String: [String: Double]] = [:]
for (reading, entries) in exact {
let key = Lexicon.stripTones(reading)
for e in entries {
if let s = best[key]?[e.phrase], s >= e.score { continue }
best[key, default: [:]][e.phrase] = e.score
}
} }
var index: [String: [LexiconEntry]] = [:]
index.reserveCapacity(best.count)
for (key, phrases) in best {
index[key] = phrases.map { LexiconEntry(phrase: $0.key, score: $0.value) }
.sorted { $0.score != $1.score ? $0.score > $1.score : $0.phrase < $1.phrase }
}
tonelessIndex = index
return index
} }
/// Entries for a sequence of readings. Untoned syllables match any tone. /// Entries for a sequence of readings. Untoned syllables match any tone.
public func entries(for syllables: [Syllable]) -> [LexiconEntry] { public func entries(for syllables: [Syllable]) -> [LexiconEntry] {
lock.lock()
defer { lock.unlock() }
if syllables.allSatisfy({ $0.tone != nil }) { if syllables.allSatisfy({ $0.tone != nil }) {
return exact[syllables.map(\.reading).joined(separator: "-")] ?? [] return exact[syllables.map(\.reading).joined(separator: "-")] ?? []
} }
return toneless[syllables.map(\.toneless).joined(separator: "-")] ?? [] return buildTonelessIfNeeded()[syllables.map(\.toneless).joined(separator: "-")] ?? []
} }
public func bestScore(for syllable: Syllable) -> Double? { public func bestScore(for syllable: Syllable) -> Double? {
@@ -81,4 +182,13 @@ public final class Lexicon {
guard let first = s.unicodeScalars.first else { return false } guard let first = s.unicodeScalars.first else { return false }
return (0x3105...0x312F).contains(first.value) || first == "˙" return (0x3105...0x312F).contains(first.value) || first == "˙"
} }
/// Same test on raw UTF-8: Bopomofo U+3105–U+312F is E3 84 85…E3 84 AF; "˙" is CB 99.
private static func isBopomofo(_ b: UnsafeBufferPointer<UInt8>, _ r: Range<Int>) -> Bool {
guard r.count >= 2 else { return false }
if b[r.lowerBound] == 0xCB && b[r.lowerBound + 1] == 0x99 { return true }
guard r.count >= 3, b[r.lowerBound] == 0xE3, b[r.lowerBound + 1] == 0x84 else { return false }
let c = b[r.lowerBound + 2]
return c >= 0x85 && c <= 0xAF
}
} }
@@ -0,0 +1,32 @@
import Foundation
/// Detects Shift pressed and released on its own (no other key in between, released
/// within `maxDuration`), as opposed to Shift used for capitals or shortcuts.
///
/// Python twin: windows/autozhuyin/core/shift_tap.py
public struct ShiftTapDetector {
public var maxDuration: TimeInterval = 0.5
private var pressedAt: TimeInterval?
private var used = false
public init() {}
/// Call when the modifier state changes. Returns true when a Shift tap completed.
public mutating func modifiersChanged(shift: Bool, otherModifiers: Bool, time: TimeInterval) -> Bool {
if shift && !otherModifiers {
if pressedAt == nil {
pressedAt = time
used = false
}
return false
}
defer { pressedAt = nil }
guard let start = pressedAt, !shift, !otherModifiers, !used else { return false }
return time - start <= maxDuration
}
/// Call for every other key press: Shift was used as a modifier, not tapped.
public mutating func keyPressed() {
if pressedAt != nil { used = true }
}
}
+79
View File
@@ -0,0 +1,79 @@
import Foundation
/// Remembers the candidates the user picked, so they come first next time.
///
/// Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
/// where the lexicon has it for the reading being converted, so a toneless key is safe.
/// Stored as JSON: {reading: {phrase: [count, last use]}} (same format on every platform).
public final class UserHistory {
public typealias Data = [String: [String: [Int]]]
/// (reading, phrase) pairs kept; the least recently used go first.
public var maxEntries = 2000
static let maxCount = 10
public private(set) var data: Data = [:]
private var clock = 0
private let onChange: (Data) -> Void
public init(data: Data = [:], onChange: @escaping (Data) -> Void = { _ in }) {
self.onChange = onChange
for (reading, phrases) in data {
for (phrase, value) in phrases where value.count >= 2 {
self.data[reading, default: [:]][phrase] = [value[0], value[1]]
clock = max(clock, value[1])
}
}
}
/// Loads the JSON file written through `onChange`; empty if missing or unreadable.
public static func load(from url: URL) -> Data {
guard let raw = try? Foundation.Data(contentsOf: url),
let data = try? JSONDecoder().decode(Data.self, from: raw) else { return [:] }
return data
}
static func key(_ syllables: [Syllable]) -> String {
syllables.map(\.toneless).joined(separator: "-")
}
public func record(_ syllables: [Syllable], phrase: String) {
guard !syllables.isEmpty, !phrase.isEmpty else { return }
clock += 1
let key = UserHistory.key(syllables)
let count = data[key]?[phrase]?.first ?? 0
data[key, default: [:]][phrase] = [min(UserHistory.maxCount, count + 1), clock]
evict()
onChange(data)
}
private func evict() {
var pairs: [(used: Int, reading: String, phrase: String)] = []
for (r, phrases) in data { for (p, v) in phrases { pairs.append((v[1], r, p)) } }
guard pairs.count > maxEntries else { return }
pairs.sort { $0.used < $1.used }
for pair in pairs.prefix(pairs.count - maxEntries) {
data[pair.reading]?[pair.phrase] = nil
if data[pair.reading]?.isEmpty == true { data[pair.reading] = nil }
}
}
public func clear() {
data = [:]
onChange(data)
}
/// Lexicon entries with picked phrases moved to the top: just above the best entry,
/// more so the more often they were picked.
func adjust(_ syllables: [Syllable], _ entries: [LexiconEntry]) -> [LexiconEntry] {
guard let picked = data[UserHistory.key(syllables)], let top = entries.first?.score else { return entries }
var out = entries.map { e -> LexiconEntry in
guard let count = picked[e.phrase]?.first else { return e }
return LexiconEntry(phrase: e.phrase, score: max(e.score, top + 0.01 + 0.1 * Double(count)))
}
// Stable sort (keep lexicon order among equal scores), like Python's sort.
out = out.enumerated().sorted { $0.element.score != $1.element.score
? $0.element.score > $1.element.score : $0.offset < $1.offset }.map(\.element)
return out
}
}
+50
View File
@@ -0,0 +1,50 @@
自動注音 AutoZhuyin — 第三方授權聲明 / Third-party notices
==========================================================
本軟體包含或使用下列第三方資料。
This software includes or uses the following third-party material.
1. 小麥注音 McBopomofo — 詞庫資料 (lexicon data)
------------------------------------------------
https://github.com/openvanilla/McBopomofo
lexicon.txt is converted from McBopomofo's data files (BPMFBase.txt,
BPMFMappings.txt, phrase.occ) by scripts/convert-mcbopomofo.py. No McBopomofo
program code is included.
MIT License
Copyright (c) 2011-2026 Mengjuei Hsieh et al.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
2. Webster's Second International Dictionary — 英文字典 (English word list)
--------------------------------------------------------------------------
english-dict.txt is filtered from /usr/share/dict/words (Webster's Second
International Dictionary, 1934), which is in the public domain.
3. PIME — Windows 輸入法框架 (input method framework)
----------------------------------------------------
https://github.com/EasyIME/PIME — GNU LGPL 2.1
The Windows version runs as a module inside PIME, which users install
separately. No PIME code is included in this software.
@@ -6,7 +6,11 @@ private let resources = URL(fileURLWithPath: #filePath)
.appendingPathComponent("Resources") .appendingPathComponent("Resources")
private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")) private let sharedLexicon = try! Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt"))
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt")) private let windowsData = resources.deletingLastPathComponent().appendingPathComponent("windows/autozhuyin/data")
private let sharedEnglish = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json"))
private let sharedWeights = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json"))
final class ParserTests: XCTestCase { final class ParserTests: XCTestCase {
func testParsesNiHao() { func testParsesNiHao() {
@@ -84,7 +88,7 @@ final class ConverterTests: XCTestCase {
} }
final class ClassifierTests: XCTestCase { final class ClassifierTests: XCTestCase {
let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish) let classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
func mode(_ s: String, finishing: Bool = false) -> InputMode { func mode(_ s: String, finishing: Bool = false) -> InputMode {
classifier.classify(Array(s), finishing: finishing).mode classifier.classify(Array(s), finishing: finishing).mode
@@ -106,7 +110,7 @@ final class ClassifierTests: XCTestCase {
func testLearningFlipsDecision() { func testLearningFlipsDecision() {
let learned = LearnedPreferences() let learned = LearnedPreferences()
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned) let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, learned: learned, weights: sharedWeights)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal) XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .literal)
learned.record("so", as: .zhuyin) learned.record("so", as: .zhuyin)
XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin) XCTAssertEqual(c.classify(Array("so"), finishing: true).mode, .zhuyin)
@@ -115,7 +119,7 @@ final class ClassifierTests: XCTestCase {
final class EngineTests: XCTestCase { final class EngineTests: XCTestCase {
func makeEngine() -> CompositionEngine { func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
} }
/// Types a string and returns everything committed. /// Types a string and returns everything committed.
@@ -188,7 +192,7 @@ final class EngineTests: XCTestCase {
final class DigitInitialTests: XCTestCase { final class DigitInitialTests: XCTestCase {
func engine() -> CompositionEngine { func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
} }
func run(_ s: String) -> (out: String, marked: String) { func run(_ s: String) -> (out: String, marked: String) {
@@ -222,7 +226,7 @@ final class DigitInitialTests: XCTestCase {
final class MixedSegmentTests: XCTestCase { final class MixedSegmentTests: XCTestCase {
func run(_ s: String) -> (out: String, marked: String) { func run(_ s: String) -> (out: String, marked: String) {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish)) let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = "" var out = ""
for c in s { for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c)) let r = e.handle(c == "\n" ? .enter : .character(c))
@@ -253,3 +257,398 @@ final class MixedSegmentTests: XCTestCase {
XCTAssertEqual(run("183\n").out.count, 1) XCTAssertEqual(run("183\n").out.count, 1)
} }
} }
/// The Swift and Python cores must segment identically (fixtures from training/make_parity.py).
final class ParityTests: XCTestCase {
func testMatchesPythonSegmentation() throws {
let url = resources.deletingLastPathComponent().appendingPathComponent("training/parity-fixtures.json")
let fixtures = try JSONSerialization.jsonObject(with: Data(contentsOf: url)) as! [[String: Any]]
let c = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights)
var mismatches: [String] = []
for fx in fixtures {
let keys = Array(fx["keys"] as! String)
let expected = (fx["segments"] as! [[Any]]).map { "\($0[0])-\($0[1]):\($0[2])" }
let previous = (fx["previous"] as? String).flatMap(InputMode.init(rawValue:))
let got = c.segment(keys, finishing: fx["finishing"] as! Bool, previous: previous).map {
"\($0.range.lowerBound)-\($0.range.upperBound):\($0.mode.rawValue)"
}
if got != expected {
mismatches.append("\(String(keys)) finishing=\(fx["finishing"]!) previous=\(String(describing: previous)): \(got) vs \(expected)")
}
}
XCTAssertGreaterThan(fixtures.count, 300)
XCTAssertTrue(mismatches.isEmpty, "\(mismatches.count) mismatches:\n" + mismatches.prefix(10).joined(separator: "\n"))
}
func testToneThenSpaceStaysChinese() {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "u4 " { if let t = e.handle(.character(c)).commit { out += t } }
XCTAssertEqual(out.count, 1, out)
}
}
final class ShiftTapTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "⇧" ? .shiftTap : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testShiftTapTogglesEnglishMode() {
let e = engine()
// "ru" alone would be ㄐㄧ (機); in English mode keys pass straight through, lowercase.
XCTAssertEqual(type("⇧ru su ", into: e), "ru su ")
XCTAssertTrue(e.isEnglishMode) // still English after spaces
XCTAssertEqual(e.markedText, "") // never composed: no underline
XCTAssertEqual(type("\n", into: e), "\n") // Enter goes to the app, mode stays
XCTAssertTrue(e.isEnglishMode)
XCTAssertEqual(type("⇧su3cl3\n", into: e), "你好") // tap again: automatic again
XCTAssertFalse(e.isEnglishMode)
}
func testShiftTapTwiceCancels() {
XCTAssertEqual(type("⇧⇧su3cl3\n", into: engine()), "你好")
}
func testShiftTapMidWordFlips() {
let e = engine()
_ = type("so", into: e)
XCTAssertEqual(e.mode, .literal)
_ = e.handle(.shiftTap)
XCTAssertEqual(e.mode, .zhuyin)
XCTAssertFalse(e.isEnglishMode) // a flip, not English mode
_ = type("\n", into: e)
XCTAssertEqual(e.classifier.learned.weights["so"], 1) // a correction is learned
}
func testEnglishModeIsNotLearned() {
let e = engine()
_ = type("⇧hello ", into: e)
XCTAssertTrue(e.classifier.learned.weights.isEmpty)
}
func testBackspaceInEnglishModeGoesToApp() {
let e = engine()
_ = type("⇧", into: e)
XCTAssertFalse(e.handle(.backspace).handled)
XCTAssertTrue(e.isEnglishMode)
}
func testDetector() {
var d = ShiftTapDetector()
XCTAssertFalse(d.modifiersChanged(shift: true, otherModifiers: false, time: 0))
XCTAssertTrue(d.modifiersChanged(shift: false, otherModifiers: false, time: 0.1))
// Shift used for a capital letter: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 1)
d.keyPressed()
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 1.1))
// Held too long: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 2)
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 3))
// Shift + Command: not a tap.
_ = d.modifiersChanged(shift: true, otherModifiers: false, time: 4)
_ = d.modifiersChanged(shift: true, otherModifiers: true, time: 4.1)
XCTAssertFalse(d.modifiersChanged(shift: false, otherModifiers: false, time: 4.2))
}
}
final class CursorTests: XCTestCase {
func engine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "←" ? .left : c == "→" ? .right : c == "↓" ? .down : c == "\n" ? .enter
: c == "\u{8}" ? .backspace : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testCursorMovesAndClamps() {
let e = engine()
_ = type("rup wu0 wu0 fu4", into: e) // 今天天氣
XCTAssertEqual(e.cursorOffset, 4)
_ = type("←", into: e)
XCTAssertEqual(e.cursorOffset, 3)
_ = type("←←←←", into: e)
XCTAssertEqual(e.cursorOffset, 1) // stops after the first character
_ = type("→→→→", into: e)
XCTAssertEqual(e.cursorOffset, 4)
}
func testChooseCandidateInTheMiddle() {
let e = engine()
_ = type("rup wu0 wu0 fu4", into: e)
_ = type("←←↓", into: e) // cursor after the 2nd syllable (天)
XCTAssertTrue(e.isShowingCandidates)
XCTAssertTrue(e.candidates.allSatisfy { $0.range.upperBound == 2 })
let i = e.candidates.firstIndex { $0.text == "添" }!
e.selectCandidate(i)
XCTAssertEqual(type("\n", into: e), "今添天氣")
}
func testPunctuationCommitsFromTheMiddle() {
let e = engine()
_ = type("rup wu0 ←", into: e)
XCTAssertEqual(e.cursorOffset, 1)
_ = type("?", into: e)
XCTAssertEqual(e.markedText, "")
}
func testInsertSyllable() {
let e = engine()
_ = type("rup fu4←wu", into: e)
XCTAssertEqual(e.markedText, "今ㄊㄧ氣")
XCTAssertEqual(e.cursorOffset, 3)
_ = type("0 ", into: e)
XCTAssertEqual(e.markedText, "今天氣")
XCTAssertEqual(e.cursorOffset, 2)
XCTAssertEqual(type("\n", into: e), "今天氣")
}
func testCursorStopsAfterFirstCharacter() {
let e = engine()
_ = type("su3cl3←←←", into: e)
XCTAssertEqual(e.cursorOffset, 1)
}
func testBackspaceDeletesCharacterBeforeCursor() {
let e = engine()
_ = type("rup wu0 wu0 fu4←←\u{8}", into: e)
XCTAssertEqual(e.markedText, "今天氣")
XCTAssertEqual(e.cursorOffset, 1)
}
func testBackspaceInsideInsertedSyllable() {
let e = engine()
_ = type("rup fu4←wu\u{8}", into: e)
XCTAssertEqual(e.markedText, "今ㄊ氣")
}
func testKeepsPickedPhraseBeforeCursor() {
let e = engine()
_ = type("rup wu0 ↓", into: e)
e.selectCandidate(e.candidates.firstIndex { $0.text == "添" }!)
_ = type("fu4←wu0 ", into: e) // insert 天 between 添 and 氣
XCTAssertEqual(String(e.markedText.prefix(3)), "今添天")
}
func testCandidatesForChineseBeforeEnglish() {
let e = engine()
_ = type("su3cl3hello", into: e) // 你好hello
_ = type("↓", into: e) // last Chinese syllable, not the English word
XCTAssertTrue(e.isShowingCandidates)
}
}
final class FirstToneAfterEnglishTests: XCTestCase {
/// After an English word, a first-tone syllable + space must stay Chinese (機, 吃, 書…).
func testFirstToneAfterEnglish() {
for keys in ["ru", "t", "gj", "vup", "tj", "el"] {
let e = CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
var out = ""
for c in "hello " + keys + " " { if let t = e.handle(.character(c)).commit { out += t } }
out += e.handle(.enter).commit ?? ""
XCTAssertEqual(String(out.prefix(6)), "hello ", keys)
XCTAssertEqual(out.count, 7, keys) // one Chinese character after it
XCTAssertFalse(out.last!.isASCII, keys)
}
}
}
final class HeldWordTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// "up " is also ㄧㄣ (因): the next word decides.
func testNextWordDecides() {
let cases = ["up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testHeldWordStaysComposing() {
let e = makeEngine()
XCTAssertEqual(type("up ", into: e), "")
XCTAssertEqual(e.markedText, "up ")
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "up")
XCTAssertEqual(type(" the ", into: e), "up the ")
}
}
final class BacktickSymbolTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
func testForcedSymbols() {
let cases = ["`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testPlainBacktick() {
let e = makeEngine()
XCTAssertEqual(type("`", into: e), "")
XCTAssertEqual(e.markedText, "`")
XCTAssertEqual(type("ls` ", into: e), "`ls` ")
}
func testBackspaceCancels() {
let e = makeEngine()
_ = type("su3`", into: e)
_ = e.handle(.backspace)
XCTAssertEqual(e.markedText, "你")
}
}
final class FirstWordHoldTests: XCTestCase {
func makeEngine() -> CompositionEngine {
CompositionEngine(classifier: Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights))
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let r = e.handle(c == "\n" ? .enter : .character(c))
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
/// The first word is not committed as English on space if it also reads as Zhuyin.
func testNextWordDecides() {
let cases = ["el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
"i love you ": "i love you ", "up the ": "up the "]
for (keys, want) in cases {
XCTAssertEqual(type(keys, into: makeEngine()), want, keys)
}
}
func testNotHeldAfterEnglish() {
let e = makeEngine()
XCTAssertEqual(type("hello world ", into: e), "hello world ")
XCTAssertEqual(type("so ", into: e), "so ") // right after English: commits at once
}
}
final class UserHistoryTests: XCTestCase {
var saved: [UserHistory.Data] = []
var classifier: Classifier!
override func setUp() {
saved = []
classifier = Classifier(lexicon: sharedLexicon, english: sharedEnglish, weights: sharedWeights,
history: UserHistory { [unowned self] in self.saved.append($0) })
}
func type(_ s: String, into e: CompositionEngine) -> String {
var out = ""
for c in s {
let key: KeyInput = c == "↓" ? .down : c == "\n" ? .enter : .character(c)
let r = e.handle(key)
if let t = r.commit { out += t } else if !r.handled { out.append(c) }
}
return out
}
@discardableResult
func pick(_ keys: String, _ phrase: String) -> String {
let e = CompositionEngine(classifier: classifier)
_ = type(keys + "↓", into: e)
e.selectCandidate(e.candidates.firstIndex { $0.text == phrase }!)
return type("\n", into: e)
}
func testPickedCandidateComesFirst() {
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "天")
XCTAssertEqual(pick("wu0 ", "添"), "添")
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
let e = CompositionEngine(classifier: classifier)
_ = type("wu0 ↓", into: e)
XCTAssertEqual(e.candidates.first?.text, "添")
XCTAssertEqual(saved.last, ["ㄊㄧㄢ": ["添": [1, 1]]])
}
func testPhrasesStillWin() {
pick("wu0 ", "添")
XCTAssertEqual(type("rup wu0 \n", into: CompositionEngine(classifier: classifier)), "今天")
}
func testMostPickedWins() {
pick("wu0 ", "添")
pick("wu0 ", "添")
pick("wu0 ", "天")
XCTAssertEqual(type("wu0 \n", into: CompositionEngine(classifier: classifier)), "添")
}
func testRoundTripAndClear() throws {
pick("wu0 ", "添")
let json = try JSONEncoder().encode(saved.last!)
let h = UserHistory(data: try JSONDecoder().decode(UserHistory.Data.self, from: json))
XCTAssertEqual(h.data, ["ㄊㄧㄢ": ["添": [1, 1]]])
h.record(ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable), phrase: "天")
XCTAssertEqual(h.data["ㄊㄧㄢ"]?["天"], [1, 2])
h.clear()
XCTAssertTrue(h.data.isEmpty)
}
func testEvictsLeastRecent() {
let h = UserHistory()
h.maxEntries = 2
let syl = ZhuyinParser.parse(Array("wu0 ")).syllables.map(\.syllable)
for p in ["天", "添", "田"] { h.record(syl, phrase: p) }
XCTAssertEqual(Set(h.data["ㄊㄧㄢ"]!.keys), ["添", "田"])
}
}
final class StartupTimingTests: XCTestCase {
func testStartupTiming() throws {
var t = Date()
func lap(_ name: String) { print(String(format: "STARTUP %@: %.3fs", name, Date().timeIntervalSince(t))); t = Date() }
_ = try Lexicon(contentsOf: resources.appendingPathComponent("lexicon.txt")); lap("lexicon")
_ = EnglishModel(commonWordsURL: resources.appendingPathComponent("english-common.txt"),
dictionaryURL: windowsData.appendingPathComponent("english-dict.txt"),
bigramURL: windowsData.appendingPathComponent("english-bigram.json")); lap("english")
_ = Features.loadWeights(from: resources.appendingPathComponent("classifier-weights.json")); lap("weights")
}
}
+328
View File
@@ -0,0 +1,328 @@
#!/usr/bin/env python3
"""IBus engine for AutoZhuyin (自動注音) on Linux.
All the logic lives in autozhuyin.core (shared with Windows and the tests); this file
only translates IBus key events and draws the preedit / candidate table.
Run modes:
autozhuyin_ibus.py --ibus started by ibus-daemon (from the component XML)
autozhuyin_ibus.py [--activate] started by the user/autostart: registers the engine
itself if ibus-daemon does not know it, and with
--activate makes it the current input method
"""
import json
import os
import sys
import time
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import gi # noqa: E402
gi.require_version("IBus", "1.0")
from gi.repository import GLib, GObject, IBus # noqa: E402
import autozhuyin # noqa: E402
from autozhuyin.core.history import UserHistory # noqa: E402
from autozhuyin.core.classifier import LITERAL, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import ( # noqa: E402
BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine,
)
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
ENGINE_NAME = "autozhuyin"
BUS_NAME = "org.freedesktop.IBus.AutoZhuyin"
VERSION = "0.0.0"
_DATA = os.path.join(os.path.dirname(os.path.abspath(autozhuyin.__file__)), "data")
_USER_DIR = os.path.join(os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config"), "autozhuyin")
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
try:
with open(os.path.join(HERE, "version.txt"), encoding="utf-8") as _f:
VERSION = _f.read().strip() or VERSION
except OSError:
pass
_classifier = None
def _save_json(path, data):
try:
os.makedirs(_USER_DIR, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False)
except OSError:
pass
def _load_json(path):
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError):
return {}
def _save_learned(weights):
_save_json(_LEARNED_FILE, weights)
def _save_history(data):
_save_json(_HISTORY_FILE, data)
def shared_classifier():
"""Loads the lexicon and English model once per process."""
global _classifier
if _classifier is None:
lexicon = Lexicon.from_file(os.path.join(_DATA, "lexicon.txt"))
if os.path.exists(_USER_PHRASES):
with open(_USER_PHRASES, encoding="utf-8") as f:
lexicon.load(f)
english = EnglishModel.from_files(
os.path.join(_DATA, "english-common.txt"),
os.path.join(_DATA, "english-dict.txt"),
os.path.join(_DATA, "english-bigram.json"),
)
trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
_classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
return _classifier
_SPECIAL = {
IBus.KEY_BackSpace: BACKSPACE, IBus.KEY_Return: ENTER, IBus.KEY_KP_Enter: ENTER,
IBus.KEY_Escape: ESCAPE, IBus.KEY_Tab: TAB,
IBus.KEY_Up: UP, IBus.KEY_Down: DOWN, IBus.KEY_Left: LEFT, IBus.KEY_Right: RIGHT,
}
_SHIFT_KEYS = (IBus.KEY_Shift_L, IBus.KEY_Shift_R)
_SHORTCUT_MASK = (IBus.ModifierType.CONTROL_MASK | IBus.ModifierType.MOD1_MASK
| IBus.ModifierType.SUPER_MASK | IBus.ModifierType.MOD4_MASK)
_LITERAL_COLOR = 0x1E64DC # blue hint for English segments
class AutoZhuyinEngine(IBus.Engine):
__gtype_name__ = "AutoZhuyinEngine"
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._setup()
def _setup(self):
if getattr(self, "engine", None) is not None:
return
self.engine = CompositionEngine(shared_classifier())
self.shift_tap = ShiftTapDetector()
self.table = IBus.LookupTable.new(self.engine.page_size, 0, True, False)
self.table.set_orientation(IBus.Orientation.VERTICAL)
self.mode_prop = IBus.Property(
key="mode", prop_type=IBus.PropType.NORMAL,
label=IBus.Text.new_from_string("中英自動"),
symbol=IBus.Text.new_from_string("自"),
tooltip=IBus.Text.new_from_string("單按 Shift 切換英文模式"),
sensitive=True, visible=True, state=IBus.PropState.UNCHECKED)
self.props = IBus.PropList()
self.props.append(self.mode_prop)
# --- keys ---
def do_process_key_event(self, keyval, keycode, state):
self._setup()
released = bool(state & IBus.ModifierType.RELEASE_MASK)
# Shift pressed and released alone: English mode (or flip the current word).
if keyval in _SHIFT_KEYS:
now = time.monotonic()
if not released:
if state & _SHORTCUT_MASK:
self.shift_tap.key_pressed()
else:
self.shift_tap.shift_down(now)
return False
if self.shift_tap.shift_up(now):
self.engine.handle(SHIFT_TAP)
self._update(None)
self._update_mode()
return False
if released:
return False
self.shift_tap.key_pressed()
# Shortcuts (Ctrl/Alt/Super) go to the app; commit any composition first.
if state & _SHORTCUT_MASK:
if self.engine.keys:
self._update(self.engine.commit_all())
return False
key = self._key(keyval, state)
if key is None:
return self._busy() # swallow other keys (Delete, Home…) while composing
if not self.engine.wants(key):
return False
handled, commit = self.engine.handle(key)
self._update(commit)
return handled
def _key(self, keyval, state):
if keyval in _SPECIAL:
return _SPECIAL[keyval]
if keyval == IBus.KEY_space:
return " "
code = IBus.keyval_to_unicode(keyval)
if not code or not code.isprintable():
return None
# Caps Lock without Shift: keep Zhuyin working by using lowercase.
if state & IBus.ModifierType.LOCK_MASK and not state & IBus.ModifierType.SHIFT_MASK:
code = code.lower()
return code
def _busy(self):
return bool(self.engine.keys or self.engine.candidates)
# --- candidate panel (mouse) ---
def do_candidate_clicked(self, index, button, state):
self.engine.select_candidate(self.engine.page_start + index)
self._update(None)
def do_page_up(self):
return self._move_candidate(-self.engine.page_size)
def do_page_down(self):
return self._move_candidate(self.engine.page_size)
def do_cursor_up(self):
return self._move_candidate(-1)
def do_cursor_down(self):
return self._move_candidate(1)
def _move_candidate(self, delta):
if not self.engine.candidates:
return False
e = self.engine
e.candidate_index = max(0, min(len(e.candidates) - 1, e.candidate_index + delta))
self._update(None)
return True
# --- lifecycle ---
def do_focus_in(self):
self._setup()
self.register_properties(self.props)
self._update_mode()
def do_focus_out(self):
self._setup()
# PreeditFocusMode.COMMIT: IBus commits the shown text itself on focus out.
self.engine.reset()
self.hide_lookup_table()
def do_reset(self):
self._setup()
self.engine.reset()
self.hide_preedit_text()
self.hide_lookup_table()
def do_disable(self):
self.do_reset()
# --- drawing ---
def _update(self, commit):
if commit:
self.commit_text(IBus.Text.new_from_string(commit))
pieces = self.engine.marked_pieces
text = "".join(t for t, _ in pieces)
if text:
ibus_text = IBus.Text.new_from_string(text)
attrs = IBus.AttrList()
attrs.append(IBus.attr_underline_new(IBus.AttrUnderline.SINGLE, 0, len(text)))
pos = 0
for piece, mode in pieces:
if mode == LITERAL and piece:
attrs.append(IBus.attr_foreground_new(_LITERAL_COLOR, pos, pos + len(piece)))
pos += len(piece)
ibus_text.set_attributes(attrs)
self.update_preedit_text_with_mode(ibus_text, self.engine.cursor_offset, True,
IBus.PreeditFocusMode.COMMIT)
else:
self.hide_preedit_text()
if self.engine.candidates:
self.table.clear()
for _, candidate in self.engine.candidates:
self.table.append_candidate(IBus.Text.new_from_string(candidate))
self.table.set_cursor_pos(self.engine.candidate_index)
self.update_lookup_table(self.table, True)
else:
self.hide_lookup_table()
def _update_mode(self):
english = self.engine.english_mode
self.mode_prop.set_label(IBus.Text.new_from_string("英文" if english else "中英自動"))
self.mode_prop.set_symbol(IBus.Text.new_from_string("英" if english else "自"))
self.update_property(self.mode_prop)
# --- process setup ---
def make_component(exec_path):
component = IBus.Component(
name=BUS_NAME, description="自動注音 AutoZhuyin", version=VERSION,
license="Private", author="AutoZhuyin", homepage="",
command_line=f"{sys.executable} {exec_path} --ibus", textdomain="")
icon = os.path.join(HERE, "icon.png")
component.add_engine(IBus.EngineDesc(
name=ENGINE_NAME, longname="自動注音", description="大千注音+英文,自動判斷中英",
language="zh_TW", license="Private", author="AutoZhuyin",
icon=icon if os.path.exists(icon) else "", layout="us", symbol="自", rank=0))
return component
def main(argv):
IBus.init()
bus = IBus.Bus()
loop = GLib.MainLoop()
started_by_ibus = "--ibus" in argv
activate = "--activate" in argv
factory = None
def on_connected(*_):
nonlocal factory
factory = IBus.Factory.new(bus.get_connection())
factory.add_engine(ENGINE_NAME, GObject.type_from_name("AutoZhuyinEngine"))
if started_by_ibus:
bus.request_name(BUS_NAME, 0)
else:
known = any(e.get_name() == ENGINE_NAME for e in bus.list_engines())
if not known:
bus.register_component(make_component(os.path.abspath(__file__)))
if activate:
bus.set_global_engine(ENGINE_NAME)
def on_disconnected(*_):
if started_by_ibus:
loop.quit()
# Otherwise wait: IBus.Bus reconnects (and emits "connected") when ibus-daemon restarts.
bus.connect("connected", on_connected)
bus.connect("disconnected", on_disconnected)
shared_classifier() # load data before the first key
if bus.is_connected():
on_connected()
elif started_by_ibus:
print("autozhuyin: cannot connect to ibus-daemon", file=sys.stderr)
return 1
loop.run()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+59
View File
@@ -0,0 +1,59 @@
自動注音 AutoZhuyin(Linux 測試版,IBus)
=========================================
用大千注音鍵盤直接打字,不用切換中英文,輸入法會自動判斷你打的是注音還是英文。
按鍵: ji3 hello world su3cl3 (最後按空白)
輸出: 我hello world 你好
適用:Bazzite、SteamOS 桌面模式、Fedora 等有 IBus 的系統(KDE 或 GNOME)。
只在桌面模式可用;Steam 遊戲模式不支援系統輸入法。
安裝
----
1. 解壓縮,在終端機(Konsole)進到這個資料夾,執行:
./install.sh
不需要 sudo,檔案只會裝在你的家目錄(~/.local/share/autozhuyin)。
2. KDE(Bazzite / SteamOS 預設):
系統設定 → 鍵盤 → 虛擬鍵盤,選「IBus Wayland」→ 套用,然後登出再登入。
GNOME:IBus 是內建的,不用另外設定。
3. 之後每次登入都會自動切換到自動注音。
使用方式
--------
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
Enter 送出目前組字
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
← → 在組字區移動游標;游標在中間時直接打注音會插入新字,
Backspace 刪掉游標前的字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
選過的字會被記住,下次優先出現
Esc 取消組字
單按 Shift 切換英文模式(按鍵直接送出),再單按一次回到中英自動判斷
Shift+字母 打大寫字母(也會強制英文)
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
切換與移除
----------
暫時改回一般鍵盤: ibus engine xkb:us::eng
切回自動注音: ibus engine autozhuyin
移除: ./uninstall.sh
個人資料(學習紀錄、自訂詞)在 ~/.config/autozhuyin/:
learned.json 按 Tab 修正的學習紀錄
user-history.json 選字紀錄(選過的候選字)
user-phrases.txt 自訂詞,一行一個,例如:ㄗˋ-ㄉㄨㄥˋ 自動 -1
有問題時
--------
在終端機手動執行看看錯誤訊息:
/usr/bin/python3 ~/.local/share/autozhuyin/autozhuyin_ibus.py --activate
把畫面上的訊息回報給我。
+63
View File
@@ -0,0 +1,63 @@
#!/bin/bash
# Installs AutoZhuyin (自動注音) for the current user as an IBus input method.
# Nothing outside your home folder is touched, so it works on read-only systems
# (Bazzite, SteamOS, Fedora Atomic). No sudo needed.
set -euo pipefail
cd "$(dirname "$0")"
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/autozhuyin"
AUTOSTART="${XDG_CONFIG_HOME:-$HOME/.config}/autostart"
PY=/usr/bin/python3
echo "== 自動注音 AutoZhuyin 安裝 =="
# 1. IBus and its Python bindings must come with the system.
if ! command -v ibus >/dev/null 2>&1; then
echo "找不到 IBus(ibus 指令)。這個系統沒有內建 IBus,自動注音目前只支援 IBus。" >&2
exit 1
fi
if ! "$PY" -c 'import gi; gi.require_version("IBus", "1.0"); from gi.repository import IBus' 2>/dev/null; then
echo "系統的 Python 缺少 IBus 模組(python3-gobject / IBus-1.0 typelib)。" >&2
exit 1
fi
# 2. Copy the program.
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null || true
rm -rf "$DEST"
mkdir -p "$DEST"
cp -R autozhuyin autozhuyin_ibus.py version.txt "$DEST/"
[ -f icon.png ] && cp icon.png "$DEST/"
[ -f THIRD_PARTY_LICENSES.txt ] && cp THIRD_PARTY_LICENSES.txt "$DEST/"
chmod +x "$DEST/autozhuyin_ibus.py"
# 3. Start with the desktop session: registers the engine with IBus and selects it.
mkdir -p "$AUTOSTART"
cat > "$AUTOSTART/autozhuyin.desktop" <<DESKTOP
[Desktop Entry]
Type=Application
Name=自動注音 AutoZhuyin
Comment=Registers the AutoZhuyin input method with IBus
Exec=$PY $DEST/autozhuyin_ibus.py --activate
Icon=$DEST/icon.png
X-GNOME-Autostart-enabled=true
X-KDE-autostart-phase=2
NoDisplay=true
DESKTOP
# 4. Start it now as well.
if pgrep -x ibus-daemon >/dev/null 2>&1; then
nohup "$PY" "$DEST/autozhuyin_ibus.py" --activate >/dev/null 2>&1 &
sleep 1
echo "已啟動並切換到自動注音。"
else
echo "IBus 目前沒有在執行,請照下面的步驟啟用 IBus 後登出再登入。"
fi
cat <<'MSG'
安裝完成。
- KDE(Bazzite / SteamOS 桌面模式):系統設定 → 鍵盤 → 虛擬鍵盤,選「IBus Wayland」,
按套用後登出再登入。之後每次登入都會自動切換到自動注音。
- 想暫時改回一般鍵盤:ibus engine xkb:us::eng ;切回自動注音:ibus engine autozhuyin
- 移除:執行 uninstall.sh
MSG
+14
View File
@@ -0,0 +1,14 @@
#!/bin/bash
# Removes AutoZhuyin (自動注音) for the current user. Your learned corrections and
# user phrases in ~/.config/autozhuyin are kept (delete that folder to remove them).
set -uo pipefail
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/autozhuyin"
AUTOSTART="${XDG_CONFIG_HOME:-$HOME/.config}/autostart/autozhuyin.desktop"
if command -v ibus >/dev/null 2>&1 && [ "$(ibus engine 2>/dev/null)" = "autozhuyin" ]; then
ibus engine xkb:us::eng 2>/dev/null
fi
pkill -f "autozhuyin/autozhuyin_ibus.py" 2>/dev/null
rm -rf "$DEST" "$AUTOSTART"
echo "已移除自動注音。學習紀錄留在 ~/.config/autozhuyin,不需要可以自行刪除。"
+183
View File
@@ -0,0 +1,183 @@
"""Drives autozhuyin_ibus.AutoZhuyinEngine through a minimal stand-in for
gi.repository.IBus, to catch wrapper bugs without Linux. Run from the repo root:
python3 -m unittest discover -s linux/tests
"""
import os
import sys
import types
import unittest
LINUX = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
import tempfile # noqa: E402
os.environ["XDG_CONFIG_HOME"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
sys.path.insert(0, os.path.join(os.path.dirname(LINUX), "windows")) # autozhuyin package
sys.path.insert(0, LINUX)
# --- stand-in for gi / IBus (only what the wrapper uses) ---
class Flag(int):
def __or__(self, o): return Flag(int(self) | int(o))
def __and__(self, o): return Flag(int(self) & int(o))
IBus = types.SimpleNamespace()
for i, name in enumerate(["BackSpace", "Return", "KP_Enter", "Escape", "Tab", "Up", "Down", "Left",
"Right", "space", "Shift_L", "Shift_R", "Delete"]):
setattr(IBus, "KEY_" + name, 0xff00 + i)
IBus.KEY_space = 0x20
IBus.ModifierType = types.SimpleNamespace(
SHIFT_MASK=Flag(1), LOCK_MASK=Flag(2), CONTROL_MASK=Flag(4), MOD1_MASK=Flag(8),
SUPER_MASK=Flag(1 << 26), MOD4_MASK=Flag(64), RELEASE_MASK=Flag(1 << 30))
IBus.keyval_to_unicode = lambda k: chr(k) if 0x20 <= k < 0x7f else "\x00"
class Text:
def __init__(self, s): self.text, self.attrs = s, None
@staticmethod
def new_from_string(s): return Text(s)
def set_attributes(self, a): self.attrs = a
class AttrList(list):
pass
class LookupTable:
def __init__(self): self.items, self.cursor = [], 0
@staticmethod
def new(*a): return LookupTable()
def set_orientation(self, o): pass
def clear(self): self.items = []
def append_candidate(self, t): self.items.append(t.text)
def set_cursor_pos(self, p): self.cursor = p
class Property:
def __init__(self, **kw): self.label = kw["label"].text
def set_label(self, t): self.label = t.text
def set_symbol(self, t): pass
class Engine:
def __init__(self, *a, **kw):
self.committed, self.preedit, self.cursor, self.shown = "", "", 0, None
self.preedit_attrs = None
def commit_text(self, t): self.committed += t.text
def update_preedit_text_with_mode(self, t, cursor, visible, mode):
self.preedit, self.cursor, self.preedit_attrs = t.text, cursor, t.attrs
def hide_preedit_text(self): self.preedit = ""
def update_lookup_table(self, table, visible): self.shown = list(table.items)
def hide_lookup_table(self): self.shown = None
def register_properties(self, p): pass
def update_property(self, p): self.mode_label = p.label
IBus.Text, IBus.AttrList, IBus.LookupTable, IBus.Property, IBus.Engine = Text, AttrList, LookupTable, Property, Engine
IBus.PropList = lambda: types.SimpleNamespace(append=lambda p: None)
IBus.attr_underline_new = lambda kind, a, b: ("underline", a, b)
IBus.attr_foreground_new = lambda color, a, b: ("color", a, b)
IBus.AttrUnderline = types.SimpleNamespace(SINGLE=1)
IBus.Orientation = types.SimpleNamespace(VERTICAL=1)
IBus.PropType = types.SimpleNamespace(NORMAL=0)
IBus.PropState = types.SimpleNamespace(UNCHECKED=0)
IBus.PreeditFocusMode = types.SimpleNamespace(COMMIT=1)
gi = types.ModuleType("gi")
gi.require_version = lambda *a: None
repo = types.ModuleType("gi.repository")
repo.IBus, repo.GLib, repo.GObject = IBus, types.SimpleNamespace(), types.SimpleNamespace()
gi.repository = repo
sys.modules["gi"], sys.modules["gi.repository"] = gi, repo
import autozhuyin_ibus # noqa: E402
M = IBus.ModifierType
def press(e, keys):
"""Types keys; returns the list of 'handled' results."""
names = {"\n": IBus.KEY_Return, "\b": IBus.KEY_BackSpace, "\t": IBus.KEY_Tab, "←": IBus.KEY_Left,
"→": IBus.KEY_Right, "↓": IBus.KEY_Down, " ": IBus.KEY_space}
out = []
for c in keys:
kv = names.get(c, ord(c))
out.append(e.do_process_key_event(kv, 0, Flag(0)))
e.do_process_key_event(kv, 0, M.RELEASE_MASK)
return out
class IBusEngineTests(unittest.TestCase):
def setUp(self):
autozhuyin_ibus.shared_classifier().history.clear()
autozhuyin_ibus.shared_classifier().learned.clear()
self.e = autozhuyin_ibus.AutoZhuyinEngine()
def test_mixed_sentence(self):
press(self.e, "ji3 hello world su3cl3 ")
self.assertEqual(self.e.committed, "我hello world 你好")
self.assertEqual(self.e.preedit, "")
def test_preedit_marks_english_blue(self):
press(self.e, "ji3hello")
self.assertEqual(self.e.preedit, "我hello")
self.assertIn(("color", 1, 6), self.e.preedit_attrs)
self.assertEqual(self.e.cursor, 6)
def test_space_passes_through_when_idle(self):
self.assertEqual(press(self.e, " "), [False])
def test_candidates_and_click(self):
press(self.e, "su3↓")
self.assertTrue(self.e.shown)
second = self.e.shown[1]
self.e.do_candidate_clicked(1, 1, 0)
self.assertIsNone(self.e.shown)
press(self.e, "\n")
self.assertEqual(self.e.committed, second)
def test_shift_tap_toggles_english(self):
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, Flag(0))
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, M.SHIFT_MASK | M.RELEASE_MASK)
self.assertEqual(self.e.mode_label, "英文")
self.assertEqual(press(self.e, "a"), [False]) # goes to the app as typed
def test_shift_as_modifier_is_not_a_tap(self):
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, Flag(0))
self.e.do_process_key_event(ord("A"), 0, M.SHIFT_MASK)
self.e.do_process_key_event(IBus.KEY_Shift_L, 0, M.SHIFT_MASK | M.RELEASE_MASK)
self.assertFalse(self.e.engine.english_mode)
def test_ctrl_shortcut_commits_and_passes(self):
press(self.e, "su3")
self.assertFalse(self.e.do_process_key_event(ord("c"), 0, M.CONTROL_MASK))
self.assertEqual(self.e.committed, "你")
def test_backtick_symbol(self):
press(self.e, "`<")
self.assertEqual(self.e.committed, ",")
def test_insert_in_middle(self):
press(self.e, "rup fu4←wu0 \n")
self.assertEqual(self.e.committed, "今天氣")
def test_picked_candidate_saved(self):
press(self.e, "su3↓")
second = self.e.shown[1]
self.e.do_candidate_clicked(1, 1, 0)
press(self.e, "\n")
path = os.path.join(os.environ["XDG_CONFIG_HOME"], "autozhuyin", "user-history.json")
with open(path, encoding="utf-8") as f:
self.assertIn(second, f.read())
press(self.e, "su3 \n")
self.assertTrue(self.e.committed.endswith(second))
def test_focus_out_resets(self):
press(self.e, "su3")
self.e.do_focus_out()
self.assertEqual(self.e.engine.keys, [])
if __name__ == "__main__":
unittest.main()
+5 -1
View File
@@ -18,7 +18,11 @@ rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources" mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin" cp "$BIN" "$APP/Contents/MacOS/AutoZhuyin"
cp Resources/Info.plist "$APP/Contents/Info.plist" cp Resources/Info.plist "$APP/Contents/Info.plist"
cp Resources/lexicon.txt Resources/english-common.txt "$APP/Contents/Resources/" cp Resources/lexicon.txt Resources/english-common.txt Resources/classifier-weights.json "$APP/Contents/Resources/"
cp THIRD_PARTY_LICENSES.txt "$APP/Contents/Resources/"
# Precomputed English dictionary and bigram table (shared with the Windows build).
python3 -I windows/tools/build_data.py >/dev/null
cp windows/autozhuyin/data/english-dict.txt windows/autozhuyin/data/english-bigram.json "$APP/Contents/Resources/"
if [ ! -f build/icon.tiff ]; then if [ ! -f build/icon.tiff ]; then
xcrun swift scripts/make-icon.swift build/icon.tiff xcrun swift scripts/make-icon.swift build/icon.tiff
+11
View File
@@ -13,6 +13,17 @@ rm -rf "$DEST/AutoZhuyin.app"
cp -R build/AutoZhuyin.app "$DEST/" cp -R build/AutoZhuyin.app "$DEST/"
"$DEST/AutoZhuyin.app/Contents/MacOS/AutoZhuyin" --register "$DEST/AutoZhuyin.app/Contents/MacOS/AutoZhuyin" --register
SYSTEM_APP="/Library/Input Methods/AutoZhuyin.app"
if [ -d "$SYSTEM_APP" ]; then
OLD="$(plutil -extract CFBundleShortVersionString raw "$SYSTEM_APP/Contents/Info.plist" 2>/dev/null || echo '?')"
echo
echo "WARNING: a system-wide copy (v$OLD, installed from the .pkg) exists at:"
echo " $SYSTEM_APP"
echo "macOS runs that copy instead of this one. Install the new .pkg"
echo "(scripts/package.sh), or remove it with:"
echo " sudo rm -rf \"$SYSTEM_APP\" && pkill -x AutoZhuyin"
fi
echo echo
echo "Installed to $DEST/AutoZhuyin.app" echo "Installed to $DEST/AutoZhuyin.app"
echo "First install: System Settings → Keyboard → Text Input → Edit… → + → 繁體中文 → 自動注音" echo "First install: System Settings → Keyboard → Text Input → Edit… → + → 繁體中文 → 自動注音"
+31
View File
@@ -0,0 +1,31 @@
#!/bin/bash
# Builds build/AutoZhuyin-Linux-<version>.tar.gz: the IBus engine, the shared Python
# core and data, and install/uninstall scripts (per-user install, no root needed).
set -euo pipefail
cd "$(dirname "$0")/.."
VERSION="$(python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["version"])' windows/autozhuyin/ime.json)"
NAME="AutoZhuyin-Linux-$VERSION"
OUT="build/$NAME"
python3 -I windows/tools/build_data.py
(cd windows && python3 -m unittest discover -q tests)
python3 -m unittest discover -q -s linux/tests
if [ ! -f build/icon.tiff ]; then
DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" xcrun swift scripts/make-icon.swift build/icon.tiff
fi
rm -rf "$OUT" "$OUT.tar.gz"
mkdir -p "$OUT/autozhuyin"
rsync -a --exclude __pycache__ --exclude '*.pyc' \
windows/autozhuyin/__init__.py windows/autozhuyin/core windows/autozhuyin/data "$OUT/autozhuyin/"
cp linux/autozhuyin_ibus.py linux/installer/install.sh linux/installer/uninstall.sh linux/installer/README.txt \
THIRD_PARTY_LICENSES.txt "$OUT/"
echo "$VERSION" > "$OUT/version.txt"
sips -s format png -z 64 64 build/icon.tiff --out "$OUT/icon.png" >/dev/null
chmod +x "$OUT/install.sh" "$OUT/uninstall.sh" "$OUT/autozhuyin_ibus.py"
# Plain ustar without macOS extended attributes, so GNU tar doesn't warn.
(cd build && COPYFILE_DISABLE=1 tar --no-xattrs --format ustar -czf "$NAME.tar.gz" "$NAME")
echo "Built build/$NAME.tar.gz"
+3
View File
@@ -26,6 +26,9 @@ rsync -a --exclude __pycache__ --exclude '*.pyc' windows/autozhuyin "$OUT/"
for f in install.bat uninstall.bat README.txt; do for f in install.bat uninstall.bat README.txt; do
sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f" sed 's/$/\r/' "windows/installer/$f" > "$OUT/$f"
done done
# Third-party notices: next to the installer and inside the installed folder.
sed 's/$/\r/' THIRD_PARTY_LICENSES.txt > "$OUT/THIRD_PARTY_LICENSES.txt"
cp "$OUT/THIRD_PARTY_LICENSES.txt" "$OUT/autozhuyin/"
(cd build && zip -qr -X "$NAME.zip" "$NAME") (cd build && zip -qr -X "$NAME.zip" "$NAME")
echo "Built build/$NAME.zip" echo "Built build/$NAME.zip"
+5 -3
View File
@@ -1,5 +1,5 @@
#!/bin/bash #!/bin/bash
# One-step release: bump the version, run all tests, build both installers, then # One-step release: bump the version, run all tests, build all installers, then
# commit the version change and tag it. # commit the version change and tag it.
# #
# Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch) # Usage: scripts/release.sh [patch|minor|major] [--push] (default: patch)
@@ -38,14 +38,16 @@ xcrun swift test -q 2>&1 | tail -3
echo "==> Packaging (runs the Python tests too)" echo "==> Packaging (runs the Python tests too)"
scripts/package.sh >/dev/null scripts/package.sh >/dev/null
scripts/package-windows.sh >/dev/null scripts/package-windows.sh >/dev/null
scripts/package-linux.sh >/dev/null
trap - ERR trap - ERR
git add Resources/Info.plist windows/autozhuyin/ime.json git add Resources/Info.plist windows/autozhuyin/ime.json
git commit -q -m "Release v$VERSION" git commit -q -m "Release v$VERSION"
git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION" git tag -a "v$VERSION" -m "AutoZhuyin v$VERSION"
echo "==> Tagged v$VERSION" echo "==> Tagged v$VERSION"
ls -lh "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" FILES=("build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" "build/AutoZhuyin-Linux-$VERSION.tar.gz")
shasum -a 256 "build/AutoZhuyin-$VERSION.pkg" "build/AutoZhuyin-Windows-$VERSION.zip" ls -lh "${FILES[@]}"
shasum -a 256 "${FILES[@]}"
if [ "$PUSH" = 1 ]; then if [ "$PUSH" = 1 ]; then
git push origin HEAD "v$VERSION" git push origin HEAD "v$VERSION"
+30
View File
@@ -0,0 +1,30 @@
"""Shared setup: imports the Python core (windows/autozhuyin/core) and its data."""
import os
import sys
import types
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
WIN = os.path.join(ROOT, "windows")
DATA = os.path.join(WIN, "autozhuyin", "data")
RESOURCES = os.path.join(ROOT, "Resources")
WEIGHTS_PATH = os.path.join(RESOURCES, "classifier-weights.json")
sys.path.insert(0, WIN)
from autozhuyin.core import bopomofo, features # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Lexicon # noqa: E402
core = types.SimpleNamespace(bopomofo=bopomofo, features=features, Classifier=Classifier,
ZHUYIN=ZHUYIN, LITERAL=LITERAL)
def load_models():
if not os.path.exists(os.path.join(DATA, "english-dict.txt")):
sys.exit("Run windows/tools/build_data.py first.")
lexicon = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
english = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json"))
return lexicon, english
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Writes training/parity-fixtures.json: segmentations produced by the Python core
with the current weights. The Swift tests (ParityTests) must produce the same ones,
which keeps the two implementations in sync. Re-run after changing features or weights.
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator # noqa: E402
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "parity-fixtures.json")
HAND_PICKED = ["ji3hello", "hellosu3cl3", "su3cl3github", "ji3hel", "2;3", "2.5", "uvl3", "so", "up",
"u4", "183", "2024", "Hello", "don't", "mp3", "ji3emailsu3cl3", "rup wu0 wu0 fu4cp3cl3"]
def main():
lexicon, english = load_models()
clf = core.Classifier(lexicon, english, weights=core.Classifier.load_weights(WEIGHTS_PATH))
gen = Generator(os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), seed=7)
cases = [(k, f, p) for k in HAND_PICKED for f in (False, True) for p in (None, "literal")]
while len(cases) < 500:
ex = gen.example()
if len(ex["keys"]) <= 28:
cases.append((ex["keys"], ex["finishing"], ex["previous"]))
cases.append((gen.prefix_of(ex)["keys"], False, ex["previous"]))
fixtures = [{"keys": k, "finishing": f, "previous": p,
"segments": [[s, e, m] for s, e, m in clf.segment(list(k), f, p)]} for k, f, p in cases]
with open(OUT, "w", encoding="utf-8") as fh:
json.dump(fixtures, fh, ensure_ascii=False, indent=0)
print("wrote %d fixtures to %s" % (len(fixtures), OUT))
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+147
View File
@@ -0,0 +1,147 @@
"""Synthetic typing data for the Zhuyin / literal classifier.
Each example is what sits in the input buffer when the user commits it, with the
intended mode of every key:
{"keys": "ji3hello", "labels": "zzzlllll", "finishing": True, "kind": "zh+en"}
labels: "z" = Zhuyin, "l" = literal. finishing: committed with space (True) or Enter.
Chinese comes from lexicon phrases sampled by frequency and converted to Dachen keys;
English from the common-word list and the dictionary. Real typing commits English at
a space, so a buffer holds at most one English word next to Chinese.
"""
import random
from common import core
bopomofo = core.bopomofo
KEY_OF = {sym: key for key, sym in bopomofo.SYMBOLS.items()}
TONE_KEY = {"ˊ": "6", "ˇ": "3", "ˋ": "4", "˙": "7"}
def reading_to_keys(reading):
"""'ㄋㄧˇ-ㄏㄠˇ' -> 'su3cl3'; first tone -> space. None if not typeable."""
out = []
for syl in reading.split("-"):
tone = " "
if syl and syl[-1] in TONE_KEY:
tone = TONE_KEY[syl[-1]]
syl = syl[:-1]
if not syl or syl not in bopomofo.VALID:
return None
out.append("".join(KEY_OF[c] for c in syl) + tone)
return "".join(out)
def _is_cjk(s):
return all(0x4E00 <= ord(c) <= 0x9FFF for c in s)
class Generator:
def __init__(self, lexicon_path, common_path, dict_path, seed=0):
self.rng = random.Random(seed)
phrases, weights = [], []
with open(lexicon_path, encoding="utf-8") as f:
for line in f:
cols = line.split()
if len(cols) != 3 or line.startswith("#") or not _is_cjk(cols[1]):
continue
n = cols[0].count("-") + 1
if n > 4:
continue
keys = reading_to_keys(cols[0])
if keys is None:
continue
phrases.append((cols[1], keys))
weights.append(10 ** (float(cols[2]) * 0.6)) # tempered frequency
self.phrases = phrases
self.phrase_cum = _cumulative(weights)
self.common = [w.strip() for w in open(common_path, encoding="utf-8")
if w.strip() and not w.startswith("#")]
self.dict_words = [w.strip() for w in open(dict_path, encoding="utf-8")
if 3 <= len(w.strip()) <= 9]
# --- pieces ---
def chinese(self):
n = self.rng.choice([1, 1, 2, 2, 3])
picked = self.rng.choices(self.phrases, cum_weights=self.phrase_cum, k=n)
return "".join(keys for _, keys in picked)
def english(self):
w = self.rng.choice(self.common) if self.rng.random() < 0.75 else self.rng.choice(self.dict_words)
if self.rng.random() < 0.1:
w = w[:1].upper() + w[1:]
return w
def number(self):
r = self.rng.random()
if r < 0.4:
return str(self.rng.randint(0, 9999))
if r < 0.7:
return str(self.rng.choice([2023, 2024, 2025, 2026, 100, 50, 10, 30]))
return "%d.%d" % (self.rng.randint(0, 99), self.rng.randint(0, 9))
# --- examples ---
def example(self):
r = self.rng.random()
if r < 0.40:
parts, kind = [("z", self.chinese())], "zh"
elif r < 0.65:
parts, kind = [("l", self.english())], "en"
elif r < 0.80:
parts, kind = [("z", self.chinese()), ("l", self.english())], "zh+en"
elif r < 0.93:
parts, kind = [("l", self.english()), ("z", self.chinese())], "en+zh"
elif r < 0.96:
parts, kind = [("z", self.chinese()), ("l", self.english()), ("z", self.chinese())], "zh+en+zh"
else:
parts, kind = [("l", self.number())], "num"
keys = "".join(p for _, p in parts)
labels = "".join(m * len(p) for m, p in parts)
if parts[-1][0] == "l":
finishing = True # English / numbers end with a space
elif keys[-1] == " " and self.rng.random() < 0.6:
# The moment the space (first tone) is pressed: the buffer has no space yet and
# the input method decides "first tone" vs "English word + space".
keys, labels, finishing = keys[:-1], labels[:-1], True
kind += "/tone1"
else:
# Chinese ends with Enter, or space after a finished (toned) syllable.
finishing = keys[-1] != " " and self.rng.random() < 0.5
return {"keys": keys, "labels": labels, "finishing": finishing, "kind": kind,
"previous": self.previous_mode(labels[0])}
def previous_mode(self, first_label):
"""Mode of the text committed just before: none, mostly the same, sometimes the other."""
r = self.rng.random()
if r < 0.35:
return None
same = core.ZHUYIN if first_label == "z" else core.LITERAL
other = core.LITERAL if first_label == "z" else core.ZHUYIN
return same if r < 0.80 else other
def prefix_of(self, ex):
"""The buffer while still typing: a random cut, not finishing."""
cut = self.rng.randint(1, len(ex["keys"]) - 1) if len(ex["keys"]) > 1 else 1
return {"keys": ex["keys"][:cut], "labels": ex["labels"][:cut], "finishing": False,
"kind": ex["kind"] + "/prefix", "previous": ex["previous"]}
def _cumulative(ws):
out, t = [], 0.0
for w in ws:
t += w
out.append(t)
return out
def gold_segments(labels):
"""'zzzlll' -> [(0, 3, 'zhuyin'), (3, 6, 'literal')]"""
segs, start = [], 0
for i in range(1, len(labels) + 1):
if i == len(labels) or labels[i] != labels[start]:
segs.append((start, i, core.ZHUYIN if labels[start] == "z" else core.LITERAL))
start = i
return segs
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""Learns the classifier weights from synthetic typing data (averaged structured
perceptron over the same segmentation DP the input method runs) and reports
accuracy against the hand-tuned defaults.
Usage (from the repo root, after windows/tools/build_data.py):
python3 training/train.py # train, evaluate, write Resources/classifier-weights.json
python3 training/train.py --eval-only # just evaluate the current weights file
"""
import argparse
import datetime
import json
import os
import random
import sys
import time
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import DATA, WEIGHTS_PATH, core, load_models # noqa: E402
from synth import Generator, gold_segments # noqa: E402
MAX_KEYS = 28
FIXED = {"lit.upper"} # hard rule (Shift = English), not learned
def make_set(gen, n):
out = []
while len(out) < 2 * n:
ex = gen.example()
if len(ex["keys"]) > MAX_KEYS:
continue
out.append(ex)
if len(ex["keys"]) > 1:
out.append(gen.prefix_of(ex))
return out
def labels_of(segs, n):
lab = [""] * n
for s, e, m in segs:
for i in range(s, e):
lab[i] = "z" if m == core.ZHUYIN else "l"
return "".join(lab)
def seg_features(clf, keys, segs, finishing, previous):
"""Summed features of a segmentation, or None if any segment is impossible."""
total = defaultdict(float)
n = len(keys)
for idx, (s, e, m) in enumerate(segs):
fz, fl = clf.segment_features(keys[s:e], e == n, idx > 0, finishing, previous if idx == 0 else None)
f = fz if m == core.ZHUYIN else fl
if f is None:
return None
for k, v in f.items():
total[k] += v
return total
def evaluate(clf, weights, data):
by_kind = defaultdict(lambda: [0, 0])
for ex in data:
segs = clf.segment(list(ex["keys"]), ex["finishing"], ex["previous"], weights)
ok = labels_of(segs, len(ex["keys"])) == ex["labels"]
kind = "prefix" if ex["kind"].endswith("/prefix") else ex["kind"]
for k in (kind, "after-other") if _after_other(ex) else (kind,):
by_kind[k][0] += ok
by_kind[k][1] += 1
commit = [v for k, v in by_kind.items() if k not in ("prefix", "after-other")]
summary = {
"commit": sum(c for c, _ in commit) / max(1, sum(t for _, t in commit)),
"prefix": by_kind["prefix"][0] / max(1, by_kind["prefix"][1]),
"by_kind": {k: round(c / t, 4) for k, (c, t) in sorted(by_kind.items())},
}
return summary
def _after_other(ex):
"""Committed buffer whose first segment differs from the text typed just before."""
if ex["kind"].endswith("/prefix") or ex["previous"] is None:
return False
first = core.ZHUYIN if ex["labels"][0] == "z" else core.LITERAL
return ex["previous"] != first
def show(name, s):
kinds = " ".join("%s %.1f%%" % (k, 100 * v) for k, v in s["by_kind"].items())
print("%-9s commit %.2f%% typing %.2f%% [%s]" % (name, 100 * s["commit"], 100 * s["prefix"], kinds))
def train(clf, data, epochs, lr, seed):
rng = random.Random(seed)
w = dict(clf.weights)
total = defaultdict(float)
steps = 0
for epoch in range(epochs):
rng.shuffle(data)
mistakes = 0
t0 = time.time()
for ex in data:
keys = list(ex["keys"])
pred = clf.segment(keys, ex["finishing"], ex["previous"], w)
if labels_of(pred, len(keys)) != ex["labels"]:
gold = gold_segments(ex["labels"])
fg = seg_features(clf, keys, gold, ex["finishing"], ex["previous"])
fp = seg_features(clf, keys, pred, ex["finishing"], ex["previous"])
if fg is not None and fp is not None:
mistakes += 1
scale = lr * (0.5 if ex["kind"].endswith("/prefix") else 1.0)
for k in set(fg) | set(fp):
if k not in FIXED:
w[k] = w.get(k, 0.0) + scale * (fg.get(k, 0.0) - fp.get(k, 0.0))
steps += 1
for k, v in w.items():
total[k] += v
print("epoch %d: %d updates (%.0fs)" % (epoch + 1, mistakes, time.time() - t0), flush=True)
return {k: round(v / steps, 4) for k, v in total.items()}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--train", type=int, default=3000, help="full examples (plus as many prefixes)")
ap.add_argument("--test", type=int, default=1500)
ap.add_argument("--epochs", type=int, default=6)
ap.add_argument("--lr", type=float, default=0.1)
ap.add_argument("--eval-only", action="store_true")
args = ap.parse_args()
lexicon, english = load_models()
paths = (os.path.join(DATA, "lexicon.txt"), os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"))
test = make_set(Generator(*paths, seed=2), args.test)
clf = core.Classifier(lexicon, english) # default (hand-tuned) weights
show("default", evaluate(clf, clf.weights, test))
if os.path.exists(WEIGHTS_PATH):
current = dict(clf.weights)
current.update(core.Classifier.load_weights(WEIGHTS_PATH))
show("current", evaluate(clf, current, test))
if args.eval_only:
return
data = make_set(Generator(*paths, seed=1), args.train)
print("training on %d buffers (%d test)" % (len(data), len(test)), flush=True)
learned = train(clf, data, args.epochs, args.lr, seed=3)
result = evaluate(clf, learned, test)
show("learned", result)
with open(WEIGHTS_PATH, "w", encoding="utf-8") as f:
json.dump({
"about": "AutoZhuyin classifier weights, learned by training/train.py",
"trained": datetime.date.today().isoformat(),
"examples": len(data),
"test_accuracy": {"commit": round(result["commit"], 4), "typing": round(result["prefix"], 4)},
"weights": dict(sorted(learned.items())),
}, f, ensure_ascii=False, indent=2)
f.write("\n")
print("wrote", WEIGHTS_PATH)
if __name__ == "__main__":
main()
@@ -0,0 +1 @@
{"ㄋㄧ": {"妳": [1, 1]}}
+51 -12
View File
@@ -6,33 +6,53 @@ tests); this file only translates PIME key events and replies.
""" """
import json import json
import os import os
import time
from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME) from keycodes import * # noqa: F401,F403 (VK_* constants, provided by PIME)
from textService import TextService from textService import TextService
from .core.history import UserHistory
from .core.classifier import Classifier, LearnedPreferences from .core.classifier import Classifier, LearnedPreferences
from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, TAB, UP, CompositionEngine from .core.engine import BACKSPACE, DOWN, ENTER, ESCAPE, LEFT, RIGHT, SHIFT_TAP, TAB, UP, CompositionEngine
from .core.english import EnglishModel from .core.english import EnglishModel
from .core.lexicon import Lexicon from .core.lexicon import Lexicon
from .core.shift_tap import ShiftTapDetector
_HERE = os.path.dirname(os.path.abspath(__file__)) _HERE = os.path.dirname(os.path.abspath(__file__))
_DATA = os.path.join(_HERE, "data") _DATA = os.path.join(_HERE, "data")
_USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin") _USER_DIR = os.path.join(os.environ.get("APPDATA", _HERE), "PIME", "autozhuyin")
_LEARNED_FILE = os.path.join(_USER_DIR, "learned.json") _LEARNED_FILE = os.path.join(_USER_DIR, "learned.json")
_USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt") _USER_PHRASES = os.path.join(_USER_DIR, "user-phrases.txt")
_HISTORY_FILE = os.path.join(_USER_DIR, "user-history.json")
_classifier = None _classifier = None
def _save_learned(weights): def _save_json(path, data):
try: try:
os.makedirs(_USER_DIR, exist_ok=True) os.makedirs(_USER_DIR, exist_ok=True)
with open(_LEARNED_FILE, "w", encoding="utf-8") as f: with open(path, "w", encoding="utf-8") as f:
json.dump(weights, f, ensure_ascii=False) json.dump(data, f, ensure_ascii=False)
except OSError: except OSError:
pass pass
def _load_json(path):
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except (OSError, ValueError):
return {}
def _save_learned(weights):
_save_json(_LEARNED_FILE, weights)
def _save_history(data):
_save_json(_HISTORY_FILE, data)
def _shared_classifier(): def _shared_classifier():
"""Loads the lexicon and English model once per PIME server process.""" """Loads the lexicon and English model once per PIME server process."""
global _classifier global _classifier
@@ -46,13 +66,9 @@ def _shared_classifier():
os.path.join(_DATA, "english-dict.txt"), os.path.join(_DATA, "english-dict.txt"),
os.path.join(_DATA, "english-bigram.json"), os.path.join(_DATA, "english-bigram.json"),
) )
weights = {} trained = Classifier.load_weights(os.path.join(_DATA, "classifier-weights.json"))
try: _classifier = Classifier(lexicon, english, LearnedPreferences(_load_json(_LEARNED_FILE), _save_learned),
with open(_LEARNED_FILE, encoding="utf-8") as f: trained, UserHistory(_load_json(_HISTORY_FILE), _save_history))
weights = json.load(f)
except (OSError, ValueError):
pass
_classifier = Classifier(lexicon, english, LearnedPreferences(weights, _save_learned))
return _classifier return _classifier
@@ -66,6 +82,7 @@ class AutoZhuyinTextService(TextService):
def __init__(self, client): def __init__(self, client):
TextService.__init__(self, client) TextService.__init__(self, client)
self.engine = CompositionEngine(_shared_classifier()) self.engine = CompositionEngine(_shared_classifier())
self.shift_tap = ShiftTapDetector()
# --- lifecycle --- # --- lifecycle ---
@@ -104,6 +121,13 @@ class AutoZhuyinTextService(TextService):
return bool(self.engine.keys or self.engine.candidates) return bool(self.engine.keys or self.engine.candidates)
def filterKeyDown(self, ev): def filterKeyDown(self, ev):
if ev.keyCode == VK_SHIFT: # noqa: F405
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
self.shift_tap.key_pressed()
else:
self.shift_tap.shift_down(time.monotonic())
return False
self.shift_tap.key_pressed()
# Shortcuts (Ctrl/Alt) go to the app. # Shortcuts (Ctrl/Alt) go to the app.
if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405 if ev.isKeyDown(VK_CONTROL) or ev.isKeyDown(VK_MENU): # noqa: F405
return False return False
@@ -120,6 +144,21 @@ class AutoZhuyinTextService(TextService):
self._update(commit) self._update(commit)
return handled return handled
def filterKeyUp(self, ev):
# Shift pressed and released alone: next word is English (or flip the current one).
if ev.keyCode == VK_SHIFT: # noqa: F405
return self.shift_tap.shift_up(time.monotonic())
return False
def onKeyUp(self, ev):
if ev.keyCode != VK_SHIFT: # noqa: F405
return False
self.engine.handle(SHIFT_TAP)
self._update(None)
if not self.engine.keys:
self.showMessage("英文" if self.engine.english_mode else "中英自動", 1)
return True
# --- replies --- # --- replies ---
def _update(self, commit): def _update(self, commit):
@@ -127,7 +166,7 @@ class AutoZhuyinTextService(TextService):
self.setCommitString(commit) self.setCommitString(commit)
marked = self.engine.marked_text marked = self.engine.marked_text
self.setCompositionString(marked) self.setCompositionString(marked)
self.setCompositionCursor(len(marked)) self.setCompositionCursor(self.engine.cursor_offset)
if self.engine.showing_candidates: if self.engine.showing_candidates:
page, cursor = self.engine.page() page, cursor = self.engine.page()
self.setCandidateList(page) self.setCandidateList(page)
+45 -84
View File
@@ -1,5 +1,8 @@
"""Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift.""" """Zhuyin-vs-literal classifier with learned corrections. Port of Classifier.swift."""
from . import bopomofo import json
from .history import UserHistory
from .features import DEFAULT_WEIGHTS, dot, literal_features, zhuyin_features
ZHUYIN = "zhuyin" ZHUYIN = "zhuyin"
LITERAL = "literal" LITERAL = "literal"
@@ -28,28 +31,47 @@ class LearnedPreferences:
class Classifier: class Classifier:
zhuyin_prior = 0.5
context_weight = 1.5
def __init__(self, lexicon, english, learned=None): def __init__(self, lexicon, english, learned=None, weights=None, history=None):
self.lexicon = lexicon self.lexicon = lexicon
self.english = english self.english = english
self.learned = learned or LearnedPreferences() self.learned = learned or LearnedPreferences()
self.history = history or UserHistory() # picked candidates (shared by all engines)
self.weights = dict(DEFAULT_WEIGHTS)
if weights:
self.weights.update(weights)
@classmethod
def load_weights(cls, path):
"""Reads a weights JSON file ({"weights": {...}}); returns {} if missing."""
try:
with open(path, encoding="utf-8") as f:
return json.load(f).get("weights", {})
except (OSError, ValueError):
return {}
def segment_features(self, seg, is_last, after_other, finishing, previous=None):
"""(zhuyin features or None, literal features or None) for one segment.
`previous` (mode of the text committed just before) only affects the first segment."""
seg = list(seg)
return (zhuyin_features(seg, self.lexicon, is_last, after_other, finishing, previous),
literal_features(seg, self.english, is_last, after_other, finishing, previous))
def _scores(self, seg, is_last, after_other, finishing, previous, whole, weights):
fz, fl = self.segment_features(seg, is_last, after_other, finishing, previous)
zh = dot(weights, fz) if fz is not None else NEG_INF
lit = dot(weights, fl) if fl is not None else NEG_INF
if whole:
b = self.learned.bias("".join(seg))
if b > 0:
zh += b
else:
lit -= b
return zh, lit
def classify(self, keys, finishing=False, previous=None): def classify(self, keys, finishing=False, previous=None):
"""Returns (mode, zhuyin_score, literal_score).""" """Whole buffer as one segment. Returns (mode, zhuyin_score, literal_score)."""
keys = list(keys) zh, lit = self._scores(list(keys), True, False, finishing, previous, True, self.weights)
zh = self.zhuyin_score(keys + [" "] if finishing else keys) + self.zhuyin_prior
lit = self.literal_score(keys, complete=finishing)
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
b = self.learned.bias("".join(keys))
if b > 0:
zh += b
else:
lit -= b
return (ZHUYIN if zh >= lit else LITERAL), zh, lit return (ZHUYIN if zh >= lit else LITERAL), zh, lit
def mode(self, keys, finishing=False, previous=None): def mode(self, keys, finishing=False, previous=None):
@@ -57,13 +79,11 @@ class Classifier:
# --- segmentation --- # --- segmentation ---
segment_penalty = 3.0 def segment(self, keys, finishing=False, previous=None, weights=None):
unconfirmed_switch_penalty = 2.0
def segment(self, keys, finishing=False, previous=None):
"""Splits keys into alternating segments: [(start, end, mode)], e.g. """Splits keys into alternating segments: [(start, end, mode)], e.g.
"ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over "ji3hello" -> [(0, 3, zhuyin), (3, 8, literal)]. Dynamic programming over
split points; a Zhuyin segment followed by another must end with a tone.""" split points; each segment is scored by features x weights."""
weights = weights or self.weights
keys = list(keys) keys = list(keys)
n = len(keys) n = len(keys)
if n == 0: if n == 0:
@@ -73,8 +93,8 @@ class Classifier:
back = [[-1, -1] for _ in range(n + 1)] back = [[-1, -1] for _ in range(n + 1)]
for i in range(1, n + 1): for i in range(1, n + 1):
for j in range(i): for j in range(i):
scores = self._segment_scores(keys[j:i], i == n, j > 0, finishing, scores = self._scores(keys[j:i], i == n, j > 0, finishing,
previous if j == 0 else None, j == 0 and i == n) previous if j == 0 else None, j == 0 and i == n, weights)
for m in (0, 1): for m in (0, 1):
if scores[m] == NEG_INF: if scores[m] == NEG_INF:
continue continue
@@ -83,7 +103,7 @@ class Classifier:
else: else:
if best[j][1 - m] == NEG_INF: if best[j][1 - m] == NEG_INF:
continue continue
total = best[j][1 - m] + scores[m] - self.segment_penalty total = best[j][1 - m] + scores[m]
if total > best[i][m]: if total > best[i][m]:
best[i][m] = total best[i][m] = total
back[i][m] = j back[i][m] = j
@@ -96,62 +116,3 @@ class Classifier:
out.append((j, i, modes[m])) out.append((j, i, modes[m]))
i, m = j, 1 - m i, m = j, 1 - m
return out[::-1] return out[::-1]
def _segment_scores(self, seg, is_last, after_other, finishing, previous, whole):
parse = bopomofo.parse(seg)
if not is_last and parse.pending is not None:
zh = NEG_INF
else:
zh = self.zhuyin_score(seg + [" "] if finishing and is_last else seg) + self.zhuyin_prior
if is_last and after_other and not finishing and not any(s.toned for s in parse.syllables):
zh -= self.unconfirmed_switch_penalty
lit = self.literal_score(seg, complete=(not is_last) or finishing)
if previous == ZHUYIN:
zh += self.context_weight
elif previous == LITERAL:
lit += self.context_weight
if whole:
b = self.learned.bias("".join(seg))
if b > 0:
zh += b
else:
lit -= b
return zh, lit
def zhuyin_score(self, keys):
p = bopomofo.parse(keys)
if p.foreign_keys > 0 or not p.syllables:
return NEG_INF
s = -5.0 * p.stray_tones
last = len(p.syllables) - 1
for i, ps in enumerate(p.syllables):
if ps.reordered:
s -= 0.5
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
s += 2 + (0.8 if self.lexicon.best_score(ps.syllable) is not None else -1.5)
else:
s -= 5
elif i == last:
s += 0 if bopomofo.is_valid_prefix(ps.syllable) else -5
else:
s += -2.5 if valid else -5
return s
def literal_score(self, keys, complete):
if not keys or " " in keys:
return NEG_INF
score = 0.0
if any(c.isupper() for c in keys):
score += 20
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
return score + 0.8 * digits - 1.0 * others
score += self.english.score(letters, complete or digits > 0)
score -= 1.5 * digits
apostrophes = keys.count("'")
score -= 1.5 * (others - apostrophes)
return score
+366 -27
View File
@@ -8,17 +8,34 @@ import time
from . import bopomofo from . import bopomofo
from .classifier import LITERAL, ZHUYIN from .classifier import LITERAL, ZHUYIN
from .features import dot
from .lexicon import Converter from .lexicon import Converter
BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>" BACKSPACE, ENTER, ESCAPE, TAB = "<backspace>", "<enter>", "<escape>", "<tab>"
UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>" UP, DOWN, LEFT, RIGHT = "<up>", "<down>", "<left>", "<right>"
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT} SHIFT_TAP = "<shift-tap>" # Shift pressed and released on its own
SPECIAL_KEYS = {BACKSPACE, ENTER, ESCAPE, TAB, UP, DOWN, LEFT, RIGHT, SHIFT_TAP}
FULLWIDTH = { FULLWIDTH = {
"<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")", "<": ",", ">": "。", "?": "?", "!": "!", ":": ":", '"': "「", "(": "(", ")": ")",
"{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——", "{": "『", "}": "』", "[": "「", "]": "」", "~": "~", "_": "——",
} }
# The first word of a composition that also reads as Zhuyin is not committed as English
# on space; the next word decides ("up jo4" = 因為, "up the" = up the, "no su3" = no 你).
# Right after English text only common one-syllable readings are held (up = 因, url = 教),
# so normal English typing is not delayed.
HOLD_MIN_SCORE = -3.5
# "`" then one of these keys always gives the full-width symbol, whatever the context.
# "`" followed by anything else is just a backtick (so ``` still works).
BACKTICK = "`"
SYMBOLS = {
",": ",", "<": ",", ".": "。", ">": "。", "/": "?", "?": "?", ";": ";", ":": ":",
"'": "、", "\\": "、", '"': "「", "[": "「", "]": "」", "{": "『", "}": "』",
"(": "(", ")": ")", "!": "!", "~": "~", "-": "——", "_": "——", "=": "=",
}
PASS = (False, None) PASS = (False, None)
CONSUMED = (True, None) CONSUMED = (True, None)
@@ -29,14 +46,25 @@ class CompositionEngine:
def __init__(self, classifier, now=time.monotonic): def __init__(self, classifier, now=time.monotonic):
self.classifier = classifier self.classifier = classifier
self.converter = Converter(classifier.lexicon) self.converter = Converter(classifier.lexicon, classifier.history)
self.now = now self.now = now
self.keys = [] self.keys = []
self.forced_mode = None self.forced_mode = None
# English mode, toggled by a Shift tap when nothing is being typed: keys go straight
# to the app (no composition, no underline) until Shift is tapped again.
self.english_mode = False
self.backtick = False # "`" typed; the next key may be a forced symbol
self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text} self.fixed = {} # zhuyin segment start -> {(syl_start, syl_end): text}
self.candidates = [] self.candidates = []
self.candidate_index = 0 self.candidate_index = 0
self._candidate_segment_start = 0 self._candidate_segment_start = 0
# Composition cursor: a key index into self.keys (None = at the end). ← → move it
# between syllables; ↓ opens candidates for the phrase before it; typing there
# inserts a syllable (kept in self.inserting until its tone key, then spliced in).
self.cursor = None
self.inserting = []
# Index of a held space (see HOLD_MIN_SCORE), or None.
self.held = None
self._previous_mode = None self._previous_mode = None
self._previous_time = float("-inf") self._previous_time = float("-inf")
self._cache_key = None self._cache_key = None
@@ -54,12 +82,55 @@ class CompositionEngine:
if self.forced_mode: if self.forced_mode:
return [(0, len(keys), self.forced_mode)] return [(0, len(keys), self.forced_mode)]
ctx = self.context ctx = self.context
cache_key = (tuple(keys), finishing, ctx) held = self.held if self.held is not None and self.held < len(keys) and keys[self.held] == " " else None
cache_key = (tuple(keys), finishing, ctx, held)
if cache_key != self._cache_key: if cache_key != self._cache_key:
self._cache_key = cache_key self._cache_key = cache_key
self._cache_value = self.classifier.segment(keys, finishing, ctx) if held is None:
self._cache_value = self.classifier.segment(keys, finishing, ctx)
else:
self._cache_value = self._held_segments(keys, held, finishing)
return self._cache_value return self._cache_value
def _held_segments(self, keys, held, finishing):
"""The held word is Chinese if the words typed after it start with Chinese."""
rest = keys[held + 1:]
base = held + 1
rest_segs = [(a + base, b + base, m) for a, b, m in self.classifier.segment(rest, finishing, None)]
if rest_segs and rest_segs[0][2] == ZHUYIN and self._held_reads_as_chinese(keys, held, rest_segs[0][1], finishing):
return [(0, rest_segs[0][1], ZHUYIN)] + rest_segs[1:]
return [(0, base, LITERAL)] + rest_segs
def _held_reads_as_chinese(self, keys, held, end, finishing):
"""Chinese all the way to `end` vs. the held English word followed by Chinese."""
c, w, ctx = self.classifier, self.classifier.weights, self.context
is_last = end == len(keys)
zh_whole = c.segment_features(keys[:end], is_last, False, finishing, ctx)[0]
word = c.segment_features(keys[:held], False, False, True, ctx)[1]
zh_rest = c.segment_features(keys[held + 1:end], is_last, True, finishing, None)[0]
if zh_whole is None:
return False
if word is None or zh_rest is None:
return True
return dot(w, zh_whole) >= dot(w, word) + dot(w, zh_rest)
def _is_ambiguous_word(self, keys):
"""Whether keys + space also read as a common Chinese syllable (see HOLD_MIN_SCORE)."""
if not all(k.isascii() and k.isalpha() for k in keys):
return False
p = bopomofo.parse(list(keys) + [" "])
if p.pending is not None or not p.syllables or p.stray_tones or p.foreign_keys:
return False
best = []
for ps in p.syllables:
entries = self.classifier.lexicon.entries([ps.syllable])
if not entries:
return False
best.append(entries[0][1])
if self.context != LITERAL:
return True # first word: hold anything that reads as Zhuyin
return len(best) == 1 and best[0] >= HOLD_MIN_SCORE
@property @property
def segments(self): def segments(self):
return self._segments(self.keys) return self._segments(self.keys)
@@ -77,22 +148,202 @@ class CompositionEngine:
def page_start(self): def page_start(self):
return (self.candidate_index // self.page_size) * self.page_size return (self.candidate_index // self.page_size) * self.page_size
def _convertible(self, seg, final):
"""(syllables converted to Chinese, pending Bopomofo) for a Zhuyin segment."""
start, end, _ = seg
p = bopomofo.parse(self.keys[start:end])
if final:
return [s.syllable for s in p.syllables], ""
return [s.syllable for s in p.completed], (p.pending.syllable.reading if p.pending else "")
def _conversion(self, seg, final):
syl, pending = self._convertible(seg, final)
fx = {k: v for k, v in self.fixed.get(seg[0], {}).items() if k[1] <= len(syl)}
return self.converter.convert(syl, fx), pending
def _render(self, seg, final): def _render(self, seg, final):
start, end, mode = seg start, end, mode = seg
seg_keys = self.keys[start:end]
if mode == LITERAL: if mode == LITERAL:
return "".join(seg_keys) return "".join(self.keys[start:end])
p = bopomofo.parse(seg_keys) converted, pending = self._conversion(seg, final)
fixed_here = self.fixed.get(start, {}) return "".join(t for _, t in converted) + pending
syl = [s.syllable for s in (p.syllables if final else p.completed)]
fx = {k: v for k, v in fixed_here.items() if k[1] <= len(syl)} # --- cursor ---
text = "".join(t for _, t in self.converter.convert(syl, fx))
if not final and p.pending: def _syllable_ends(self, seg):
text += p.pending.syllable.reading """Absolute key index after each syllable of a Zhuyin segment."""
return text return [seg[0] + ps.end for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables]
def _cursor_positions(self):
"""Key indices the cursor can stop at: after every Zhuyin syllable and at every
segment boundary, but never before the first Chinese character (there would be
nothing to pick candidates for)."""
out, ends = {len(self.keys)}, []
for seg in self.segments:
out.add(seg[1])
if seg[2] == ZHUYIN:
ends += self._syllable_ends(seg)
out.update(ends)
first = min(ends) if ends else len(self.keys)
return sorted(p for p in out if p >= first)
def _stop_at(self, pos):
"""(segment index, syllables before pos) for the Zhuyin segment pos is in."""
segs = self.segments
for i, seg in enumerate(segs):
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
n = len(self._convertible(seg, i < len(segs) - 1)[0])
local = sum(1 for e in self._syllable_ends(seg) if e <= pos)
return (i, min(local, n)) if local else None
return None
def _candidate_stop(self):
"""Where ↓ opens candidates: before the cursor, or the last syllable."""
if self.cursor is not None:
return self._stop_at(self.cursor)
segs = self.segments
for i in range(len(segs) - 1, -1, -1):
if segs[i][2] == ZHUYIN:
n = len(self._convertible(segs[i], i < len(segs) - 1)[0])
if n:
return i, n
return None
def _base_cursor_offset(self):
"""Cursor position in the rendered segments (without the syllable being inserted)."""
pieces = self._segment_pieces()
if self.cursor is None:
return sum(len(t) for t, _ in pieces)
segs = self.segments
for i, seg in enumerate(segs):
if not seg[0] < self.cursor <= seg[1]:
continue
offset = sum(len(t) for t, _ in pieces[:i])
if seg[2] == LITERAL:
return offset + self.cursor - seg[0]
stop = self._stop_at(self.cursor)
local = stop[1] if stop else 0
for (a, b), text in self._conversion(seg, i < len(segs) - 1)[0]:
if b <= local:
offset += len(text)
elif a < local:
offset += min(local - a, len(text))
return offset
return 0
@property
def cursor_offset(self):
"""Cursor position in marked_text (characters)."""
return self._base_cursor_offset() + len(self._inserting_text())
def _move_cursor(self, delta):
self.inserting = []
stops = self._cursor_positions()
here = len(self.keys) if self.cursor is None else self.cursor
if delta < 0:
before = [p for p in stops if p < here]
nxt = before[-1] if before else here
else:
after = [p for p in stops if p > here]
nxt = after[0] if after else here
self.cursor = None if nxt >= len(self.keys) else nxt
def _inserting_text(self):
if not self.inserting:
return ""
return "".join(ps.syllable.reading for ps in bopomofo.parse(self.inserting).syllables)
# --- editing in the middle ---
def _trim_fixed(self, pos):
"""Keeps picked phrases that end at or before key index pos (they don't move)."""
kept = {}
for seg in self.segments:
picks = self.fixed.get(seg[0])
if not picks or seg[2] != ZHUYIN:
continue
if seg[1] <= pos:
kept[seg[0]] = picks
elif seg[0] < pos:
ends = self._syllable_ends(seg)
here = {r: t for r, t in picks.items() if r[1] <= len(ends) and ends[r[1] - 1] <= pos}
if here:
kept[seg[0]] = here
self.fixed = kept
def _splice(self, new_keys):
"""Inserts keys at the cursor and moves the cursor after them."""
pos = self.cursor
self._trim_fixed(pos)
self.keys[pos:pos] = new_keys
if self.held is not None and self.held >= pos:
self.held += len(new_keys)
self.cursor = pos + len(new_keys)
if self.cursor >= len(self.keys):
self.cursor = None
def _delete_before_cursor(self):
"""Backspace in the middle: removes the syllable (or literal key) before the cursor."""
pos = self.cursor
start = pos - 1
for seg in self.segments:
if seg[2] == ZHUYIN and seg[0] < pos <= seg[1]:
for ps in bopomofo.parse(self.keys[seg[0]:seg[1]]).syllables:
if seg[0] + ps.end == pos:
start = seg[0] + ps.start
self._trim_fixed(start)
del self.keys[start:pos]
if self.held is not None:
if start <= self.held < pos:
self.held = None
elif self.held >= pos:
self.held -= pos - start
self.cursor = start
if not self.keys:
self.reset()
def _insert_key(self, c):
"""Typing with the cursor in the middle: compose one syllable, splice it in on its tone."""
if c == " ":
if not self.inserting:
return True, self._commit()
self._splice(self.inserting + [" "])
self.inserting = []
return CONSUMED
self.inserting.append(c)
p = bopomofo.parse(self.inserting)
if p.stray_tones or p.foreign_keys:
self.inserting.pop() # a tone with nothing to attach to
return CONSUMED
if p.syllables and p.syllables[-1].toned and len(p.syllables) == 1:
self._splice(self.inserting)
self.inserting = []
elif len(p.syllables) > 1:
# A new syllable started without a tone: splice the first one as it is.
first = p.syllables[0].end
self._splice(self.inserting[:first])
self.inserting = self.inserting[first:]
return CONSUMED
@property @property
def marked_pieces(self): def marked_pieces(self):
pieces = self._segment_pieces()
if self.inserting:
# Show the syllable being inserted at the cursor.
at, out, done = self._base_cursor_offset(), [], False
for text, mode in pieces:
if not done and at <= len(text):
out += [(text[:at], mode), (self._inserting_text(), ZHUYIN), (text[at:], mode)]
done = True
else:
out.append((text, mode))
at -= len(text)
pieces = [(t, m) for t, m in out if t]
if self.backtick:
pieces.append((BACKTICK, LITERAL))
return pieces
def _segment_pieces(self):
segs = self.segments segs = self.segments
return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)] return [(self._render(s, i < len(segs) - 1), s[2]) for i, s in enumerate(segs)]
@@ -102,12 +353,20 @@ class CompositionEngine:
# --- key handling --- # --- key handling ---
def _flip(self):
"""Forces the whole buffer into the other mode (a correction that is learned)."""
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN
def wants(self, key): def wants(self, key):
"""Whether handle(key) would consume the key, without changing state.""" """Whether handle(key) would consume the key, without changing state."""
if self.candidates: if self.candidates or key == SHIFT_TAP or self.backtick or self.inserting:
return True return True
if self.english_mode and not self.keys:
return False # English mode: everything goes to the app as typed
if key in SPECIAL_KEYS: if key in SPECIAL_KEYS:
return bool(self.keys) return bool(self.keys)
if key == BACKTICK:
return True
if self.keys: if self.keys:
return True return True
if key == " ": if key == " ":
@@ -119,17 +378,36 @@ class CompositionEngine:
def handle(self, key): def handle(self, key):
if self.candidates: if self.candidates:
return self._handle_candidate_key(key) return self._handle_candidate_key(key)
if self.backtick:
return self._after_backtick(key)
if self.inserting and key in (BACKSPACE, ESCAPE):
if key == BACKSPACE:
self.inserting.pop()
else:
self.inserting = []
return CONSUMED
if key == BACKSPACE: if key == BACKSPACE:
if not self.keys: if not self.keys:
return PASS return PASS
if self.cursor is not None:
if self.cursor > 0:
self._delete_before_cursor()
return CONSUMED
self.keys.pop() self.keys.pop()
if self.held is not None and len(self.keys) <= self.held:
self.held = None
self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)} self.fixed = {k: v for k, v in self.fixed.items() if k < len(self.keys)}
if not self.keys: if not self.keys:
self.forced_mode = None self.forced_mode = None
self.fixed = {} self.fixed = {}
return CONSUMED return CONSUMED
if key == ENTER: if key == ENTER:
return (True, self._commit()) if self.keys else PASS if not self.keys:
return PASS
if self.inserting:
self._splice(self.inserting)
self.inserting = []
return True, self._commit()
if key == ESCAPE: if key == ESCAPE:
if not self.keys: if not self.keys:
return PASS return PASS
@@ -138,17 +416,45 @@ class CompositionEngine:
if key == TAB: if key == TAB:
if not self.keys: if not self.keys:
return PASS return PASS
self.forced_mode = LITERAL if self.mode == ZHUYIN else ZHUYIN self._flip()
return CONSUMED
if key == SHIFT_TAP:
if not self.keys:
# Toggle English mode; it stays on until the next Shift tap.
self.english_mode = not self.english_mode
self._previous_mode = LITERAL if self.english_mode else None
self._previous_time = self.now()
else:
self._flip()
return CONSUMED return CONSUMED
if key == DOWN: if key == DOWN:
if not self.keys: if not self.keys:
return PASS return PASS
self._open_candidates() if not self.inserting:
self._open_candidates()
return CONSUMED return CONSUMED
if key in (UP, LEFT, RIGHT): if key in (LEFT, RIGHT):
if not self.keys:
return PASS
self._move_cursor(-1 if key == LEFT else 1)
return CONSUMED
if key == UP:
return CONSUMED if self.keys else PASS return CONSUMED if self.keys else PASS
return self._handle_character(key) return self._handle_character(key)
def _after_backtick(self, key):
self.backtick = False
if key in (ESCAPE, BACKSPACE):
return CONSUMED
committed = self._commit() if self.keys else ""
if key in SYMBOLS:
return True, committed + SYMBOLS[key]
# Not a symbol: the backtick was meant literally; then handle the key as usual.
handled, more = self.handle(key)
if not handled and len(key) == 1:
handled, more = True, key
return handled, committed + BACKTICK + (more or "")
def _last_parse(self, segs): def _last_parse(self, segs):
if not segs or segs[-1][2] != ZHUYIN: if not segs or segs[-1][2] != ZHUYIN:
return None return None
@@ -156,11 +462,27 @@ class CompositionEngine:
return bopomofo.parse(self.keys[start:end]) return bopomofo.parse(self.keys[start:end])
def _handle_character(self, c): def _handle_character(self, c):
if self.cursor is not None and self.keys and (c == " " or bopomofo.is_zhuyin_key(c)):
return self._insert_key(c)
if self.inserting:
self._splice(self.inserting)
self.inserting = []
self.cursor = None # anything else continues at the end
if self.english_mode and not self.keys:
return PASS # English mode: every key goes to the app as typed
if c == BACKTICK:
self.backtick = True
return CONSUMED
if c == " ": if c == " ":
if not self.keys: if not self.keys:
return PASS return PASS
segs = self._segments(self.keys, finishing=True) segs = self._segments(self.keys, finishing=True)
if segs[-1][2] == LITERAL: if segs[-1][2] == LITERAL:
if (self.held is None and len(segs) == 1 and not self.forced_mode
and self._is_ambiguous_word(self.keys)):
self.held = len(self.keys)
self.keys.append(" ")
return CONSUMED
return True, self._commit(segs) + " " return True, self._commit(segs) + " "
p = self._last_parse(segs) p = self._last_parse(segs)
if p is not None and p.pending is not None: if p is not None and p.pending is not None:
@@ -188,15 +510,16 @@ class CompositionEngine:
# --- candidates (for the last Zhuyin segment) --- # --- candidates (for the last Zhuyin segment) ---
def _open_candidates(self): def _open_candidates(self):
"""Candidates for the phrase ending at the cursor (default: the last syllable)."""
stop = self._candidate_stop()
if stop is None:
return
seg_index, local = stop
segs = self.segments segs = self.segments
if not segs or segs[-1][2] != ZHUYIN: seg = segs[seg_index]
return syl = self._convertible(seg, seg_index < len(segs) - 1)[0]
start, end, _ = segs[-1] self.candidates = self.converter.candidates(syl, local)
syl = [s.syllable for s in bopomofo.parse(self.keys[start:end]).completed] self._candidate_segment_start = seg[0]
if not syl:
return
self.candidates = self.converter.candidates(syl, len(syl))
self._candidate_segment_start = start
self.candidate_index = 0 self.candidate_index = 0
def page(self): def page(self):
@@ -232,6 +555,7 @@ class CompositionEngine:
def _commit(self, segs=None): def _commit(self, segs=None):
segs = segs if segs is not None else self.segments segs = segs if segs is not None else self.segments
text = "".join(self._render(s, True) for s in segs) text = "".join(self._render(s, True) for s in segs)
self._remember_picks(segs)
if self.forced_mode: if self.forced_mode:
self.classifier.learned.record("".join(self.keys), self.forced_mode) self.classifier.learned.record("".join(self.keys), self.forced_mode)
self._previous_mode = segs[-1][2] if segs else None self._previous_mode = segs[-1][2] if segs else None
@@ -239,11 +563,26 @@ class CompositionEngine:
self.reset() self.reset()
return text return text
def _remember_picks(self, segs):
"""Candidates the user picked are remembered and come first next time."""
for i, seg in enumerate(segs):
picks = self.fixed.get(seg[0])
if seg[2] != ZHUYIN or not picks:
continue
syl = self._convertible(seg, True)[0]
for (a, b), text in picks.items():
if b <= len(syl):
self.classifier.history.record(syl[a:b], text)
def commit_all(self): def commit_all(self):
return self._commit() if self.keys else "" return self._commit() if self.keys else ""
def reset(self): def reset(self):
self.keys = [] self.keys = []
self.inserting = []
self.held = None
self.backtick = False
self.cursor = None
self.forced_mode = None self.forced_mode = None
self.fixed = {} self.fixed = {}
self.candidates = [] self.candidates = []
+23
View File
@@ -56,6 +56,29 @@ class EnglishModel:
plaus = self.plausibility(w) plaus = self.plausibility(w)
return min(0.5 * n, max(-2 * n, n * (plaus + 4.2))) return min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
def word_features(self, word, complete):
"""Feature dict for the classifier (see features.py); score() is their
dot product with the default weights."""
w = word.lower()
n = float(len(w))
f = {}
if n <= 2:
f["lit.short"] = 1.0
if w in self.common:
f["lit.common"] = 1.0
f["lit.common_len"] = n
elif w in self.dictionary:
f["lit.dict"] = 1.0
f["lit.dict_len"] = n
elif not complete and _has_prefix(w, self.sorted_common):
f["lit.prefix_common_len"] = n
elif not complete and _has_prefix(w, self.sorted_words):
f["lit.prefix_dict_len"] = n
else:
plaus = self.plausibility(w)
f["lit.unknown_score"] = min(0.5 * n, max(-2 * n, n * (plaus + 4.2)))
return f
def plausibility(self, w): def plausibility(self, w):
chars = ["^"] + list(w) + ["$"] chars = ["^"] + list(w) + ["$"]
total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars))) total = sum(self.bigram.get(chars[i - 1] + chars[i], self.unseen) for i in range(1, len(chars)))
+138
View File
@@ -0,0 +1,138 @@
"""Segment features for the Zhuyin / literal classifier.
A segment's score in each mode is the dot product of its feature dict with a weight
dict. DEFAULT_WEIGHTS reproduces the original hand-tuned scores; trained weights
(training/train.py → Resources/classifier-weights.json) replace them.
Swift port: Sources/AutoZhuyinCore/Features.swift. Both must produce identical
features; Tests/.../ParityTests checks this against training/parity-fixtures.json.
"""
from . import bopomofo
DEFAULT_WEIGHTS = {
# Zhuyin
"zh.bias": 0.5,
"zh.stray": -5.0,
"zh.reordered": -0.5,
"zh.toned_valid_lex": 2.8,
"zh.toned_valid_nolex": 0.5,
"zh.toned_invalid": -5.0,
"zh.pending_prefix": 0.0,
"zh.pending_invalid": -5.0,
"zh.untoned_mid_valid": -2.5,
"zh.untoned_mid_invalid": -5.0,
"zh.unconfirmed": -2.0,
"zh.lexscore": 0.0,
"zh.phrase": 0.0,
# Literal
"lit.bias": 0.0,
"lit.upper": 20.0,
"lit.num_digits": 0.8,
"lit.num_others": -1.0,
"lit.common": 3.0,
"lit.common_len": 1.0,
"lit.dict": 1.0,
"lit.dict_len": 0.6,
"lit.prefix_common_len": 0.4,
"lit.prefix_dict_len": 0.2,
"lit.unknown_score": 1.0,
"lit.short": 0.0,
"lit.digits": -1.5,
"lit.others": -1.5,
# Segmentation
"seg.switch": -3.0,
# Context: the first segment has the same mode as the previously committed text
"ctx.zh": 1.5,
"ctx.lit": 1.5,
}
def dot(weights, feats):
return sum(weights.get(k, 0.0) * v for k, v in feats.items())
def _add(f, k, v=1.0):
f[k] = f.get(k, 0.0) + v
def zhuyin_features(keys, lexicon, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` read as Zhuyin, or None if impossible."""
parse = bopomofo.parse(keys)
if not is_last and parse.pending is not None:
return None # a Zhuyin segment followed by another must end with a tone
# Pressing space finishes an untoned last syllable as first tone; after a toned
# syllable it just commits, so it must not count as an extra (stray) tone.
scored = bopomofo.parse(keys + [" "]) if finishing and is_last and parse.pending is not None else parse
if scored.foreign_keys > 0 or not scored.syllables:
return None
f = {"zh.bias": 1.0}
if scored.stray_tones:
_add(f, "zh.stray", scored.stray_tones)
last = len(scored.syllables) - 1
toned_run = []
for i, ps in enumerate(scored.syllables):
if ps.reordered:
_add(f, "zh.reordered")
valid = bopomofo.is_valid(ps.syllable)
if ps.toned:
if valid:
best = lexicon.best_score(ps.syllable)
if best is not None:
_add(f, "zh.toned_valid_lex")
_add(f, "zh.lexscore", best + 4.0)
else:
_add(f, "zh.toned_valid_nolex")
toned_run.append(ps.syllable)
else:
_add(f, "zh.toned_invalid")
elif i == last:
_add(f, "zh.pending_prefix" if bopomofo.is_valid_prefix(ps.syllable) else "zh.pending_invalid")
else:
_add(f, "zh.untoned_mid_valid" if valid else "zh.untoned_mid_invalid")
# Syllables covered by a multi-syllable lexicon phrase (real Chinese forms words).
covered = [False] * len(toned_run)
for i in range(len(toned_run)):
for n in (2, 3, 4):
if i + n <= len(toned_run) and lexicon.entries(toned_run[i:i + n]):
for k in range(i, i + n):
covered[k] = True
if any(covered):
_add(f, "zh.phrase", sum(covered))
if is_last and after_other and not finishing and not any(ps.toned for ps in parse.syllables):
_add(f, "zh.unconfirmed")
if after_other:
_add(f, "seg.switch")
elif previous == "zhuyin":
_add(f, "ctx.zh")
return f
def literal_features(keys, english, is_last=True, after_other=False, finishing=False, previous=None):
"""Features of `keys` output as typed, or None if impossible."""
if not keys or " " in keys:
return None
complete = (not is_last) or finishing
f = {"lit.bias": 1.0}
if after_other:
_add(f, "seg.switch")
elif previous == "literal":
_add(f, "ctx.lit")
if any(c.isupper() for c in keys):
_add(f, "lit.upper")
letters = "".join(c for c in keys if c.isalpha())
digits = sum(1 for c in keys if c.isdigit())
others = len(keys) - len(letters) - digits
if not letters:
if digits:
_add(f, "lit.num_digits", digits)
if others:
_add(f, "lit.num_others", others)
return f
for k, v in english.word_features(letters, complete or digits > 0).items():
_add(f, k, v)
if digits:
_add(f, "lit.digits", digits)
non_apostrophe = others - keys.count("'")
if non_apostrophe:
_add(f, "lit.others", non_apostrophe)
return f
+65
View File
@@ -0,0 +1,65 @@
"""Remembers the candidates the user picked. Port of UserHistory.swift.
Keys are toneless readings ("ㄊㄧㄢ", "ㄐㄧㄣ-ㄊㄧㄢ"); a picked phrase is boosted only
where the lexicon has it for the reading being converted, so a toneless key is safe.
"""
class UserHistory:
max_entries = 2000 # (reading, phrase) pairs kept; the least recently used go first
max_count = 10
def __init__(self, data=None, on_change=None):
# reading -> {phrase: [count, last use]}
self.data = {}
self.on_change = on_change or (lambda data: None)
self._clock = 0
for reading, phrases in (data or {}).items():
if not isinstance(phrases, dict):
continue
for phrase, value in phrases.items():
try:
count, used = int(value[0]), int(value[1])
except (TypeError, ValueError, IndexError):
continue
self.data.setdefault(reading, {})[phrase] = [count, used]
self._clock = max(self._clock, used)
@staticmethod
def key(syllables):
return "-".join(s.toneless for s in syllables)
def record(self, syllables, phrase):
if not syllables or not phrase:
return
self._clock += 1
entry = self.data.setdefault(self.key(syllables), {}).setdefault(phrase, [0, 0])
entry[0] = min(self.max_count, entry[0] + 1)
entry[1] = self._clock
self._evict()
self.on_change(self.data)
def _evict(self):
pairs = [(v[1], r, p) for r, ps in self.data.items() for p, v in ps.items()]
if len(pairs) <= self.max_entries:
return
pairs.sort()
for _, r, p in pairs[:len(pairs) - self.max_entries]:
del self.data[r][p]
if not self.data[r]:
del self.data[r]
def clear(self):
self.data = {}
self.on_change(self.data)
def adjust(self, syllables, entries):
"""Lexicon entries [(phrase, score)] with picked phrases moved to the top: just above
the best entry, more so the more often they were picked. Returns a new list."""
picked = self.data.get(self.key(syllables))
if not picked or not entries:
return entries
top = entries[0][1]
out = [(p, max(s, top + 0.01 + 0.1 * picked[p][0]) if p in picked else s) for p, s in entries]
out.sort(key=lambda e: -e[1])
return out
+26 -15
View File
@@ -16,7 +16,7 @@ class Lexicon:
def __init__(self): def __init__(self):
self.exact = {} self.exact = {}
self.toneless = {} self._toneless = None # built on first use (only untoned syllables need it)
self.max_phrase_length = 1 self.max_phrase_length = 1
@classmethod @classmethod
@@ -42,9 +42,10 @@ class Lexicon:
except ValueError: except ValueError:
score = -6.0 score = -6.0
self.add(reading, phrase, score) self.add(reading, phrase, score)
for table in (self.exact, self.toneless): for v in self.exact.values():
for v in table.values(): if len(v) > 1:
v.sort(key=lambda e: -e[1]) v.sort(key=lambda e: -e[1])
self._toneless = None
def add(self, reading, phrase, score): def add(self, reading, phrase, score):
syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")] syllables = [s[1:] + "˙" if s.startswith("˙") else s for s in reading.split("-")]
@@ -53,15 +54,20 @@ class Lexicon:
entries = self.exact.setdefault(reading, []) entries = self.exact.setdefault(reading, [])
if not any(p == phrase for p, _ in entries): if not any(p == phrase for p, _ in entries):
entries.append((phrase, score)) entries.append((phrase, score))
key = _strip_tones(reading) self._toneless = None
tl = self.toneless.setdefault(key, [])
for i, (p, s) in enumerate(tl): @property
if p == phrase: def toneless(self):
if score > s: """Index ignoring tones: toneless reading -> [(phrase, best score)], best first."""
tl[i] = (p, score) if self._toneless is None:
break best = {}
else: for reading, entries in self.exact.items():
tl.append((phrase, score)) bucket = best.setdefault(_strip_tones(reading), {})
for phrase, score in entries:
if score > bucket.get(phrase, float("-inf")):
bucket[phrase] = score
self._toneless = {k: sorted(v.items(), key=lambda e: -e[1]) for k, v in best.items()}
return self._toneless
def entries(self, syllables): def entries(self, syllables):
if all(s.tone is not None for s in syllables): if all(s.tone is not None for s in syllables):
@@ -78,8 +84,13 @@ TONELESS_PENALTY = -1.5
class Converter: class Converter:
def __init__(self, lexicon): def __init__(self, lexicon, history=None):
self.lexicon = lexicon self.lexicon = lexicon
self.history = history # UserHistory: picked candidates come first
def _entries(self, syllables):
entries = self.lexicon.entries(syllables)
return self.history.adjust(syllables, entries) if self.history else entries
def _options(self, syllables, i, j, fixed): def _options(self, syllables, i, j, fixed):
if (i, j) in fixed: if (i, j) in fixed:
@@ -88,7 +99,7 @@ class Converter:
return [] return []
sl = syllables[i:j] sl = syllables[i:j]
penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0 penalty = TONELESS_PENALTY if any(s.tone is None for s in sl) else 0.0
out = [(p, s + penalty) for p, s in self.lexicon.entries(sl)] out = [(p, s + penalty) for p, s in self._entries(sl)]
if not out and j - i == 1: if not out and j - i == 1:
out = [(sl[0].reading, UNKNOWN_SCORE)] out = [(sl[0].reading, UNKNOWN_SCORE)]
return out return out
@@ -126,7 +137,7 @@ class Converter:
out, seen = [], set() out, seen = [], set()
for length in range(min(self.lexicon.max_phrase_length, end), 0, -1): for length in range(min(self.lexicon.max_phrase_length, end), 0, -1):
i = end - length i = end - length
for phrase, _ in self.lexicon.entries(syllables[i:end]): for phrase, _ in self._entries(syllables[i:end]):
if phrase not in seen: if phrase not in seen:
seen.add(phrase) seen.add(phrase)
out.append(((i, end), phrase)) out.append(((i, end), phrase))
+24
View File
@@ -0,0 +1,24 @@
"""Detects Shift pressed and released on its own. Port of ShiftTapDetector.swift."""
class ShiftTapDetector:
max_duration = 0.5
def __init__(self):
self._pressed_at = None
self._used = False
def shift_down(self, time):
if self._pressed_at is None: # ignore auto-repeat
self._pressed_at = time
self._used = False
def key_pressed(self):
"""Any other key while Shift is held: Shift was used as a modifier."""
if self._pressed_at is not None:
self._used = True
def shift_up(self, time):
"""Returns True when this release completes a Shift tap."""
start, self._pressed_at = self._pressed_at, None
return start is not None and not self._used and time - start <= self.max_duration
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "自動注音 AutoZhuyin", "name": "自動注音 AutoZhuyin",
"version": "0.2.0", "version": "0.10.1",
"guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}", "guid": "{3895C6E3-6D40-4B4B-B4EA-CEEC75382D23}",
"locale": "zh-Hant-TW", "locale": "zh-Hant-TW",
"fallbackLocale": "zh-TW", "fallbackLocale": "zh-TW",
+11 -2
View File
@@ -26,10 +26,19 @@
空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白 空白 注音:一聲;音節打完時送出中文。英文:送出單字加空白
Enter 送出目前組字 Enter 送出目前組字
Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正 Tab 判斷錯了?按 Tab 在中/英之間翻轉,輸入法會記住這次修正
↓ 開啟候選字,按數字鍵 1–9 選字 ← → 在組字區移動游標,選要改哪個字;游標在中間時直接打注音
會插入新字,Backspace 刪掉游標前的字
↓ 開啟游標前那個字或詞的候選字,按數字鍵 1–9 選字;
選過的字會被記住,下次優先出現
Esc 取消組字 Esc 取消組字
Shift+字母 強制英文 單按 Shift 切換英文模式(按鍵直接送出、保持小寫、沒有底線),
再單按一次回到中英自動判斷,會出現「英文」/「中英自動」小提示;
打到一半時按,同 Tab 整段翻轉
Shift+字母 打大寫字母(也會強制英文)
< > ? ! : 中文語境下自動變成全形 ,。?!: < > ? ! : 中文語境下自動變成全形 ,。?!:
` + 標點 強制全形標點,例如 `< 或 `, = , `. = 。 `/ = ? `; = ;
`' = 、 `[ `] = 「」 `( `) = () `! = !
` 後面接其他鍵就是普通的反引號
注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。 注音符號的順序打錯也沒關係,例如 ㄧㄒㄠˇ 會自動變成 ㄒㄧㄠˇ(小)。
+240 -5
View File
@@ -11,9 +11,11 @@ sys.path.insert(0, os.path.dirname(HERE))
from autozhuyin.core import bopomofo # noqa: E402 from autozhuyin.core import bopomofo # noqa: E402
from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402 from autozhuyin.core.classifier import LITERAL, ZHUYIN, Classifier, LearnedPreferences # noqa: E402
from autozhuyin.core.engine import DOWN, ENTER, TAB, CompositionEngine # noqa: E402 from autozhuyin.core.engine import DOWN, ENTER, LEFT, RIGHT, SHIFT_TAP, TAB, BACKSPACE, CompositionEngine # noqa: E402
from autozhuyin.core.shift_tap import ShiftTapDetector # noqa: E402
from autozhuyin.core.english import EnglishModel # noqa: E402 from autozhuyin.core.english import EnglishModel # noqa: E402
from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402 from autozhuyin.core.lexicon import Converter, Lexicon # noqa: E402
from autozhuyin.core.history import UserHistory # noqa: E402
DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data") DATA = os.path.join(os.path.dirname(HERE), "autozhuyin", "data")
_start = time.time() _start = time.time()
@@ -21,6 +23,7 @@ LEX = Lexicon.from_file(os.path.join(DATA, "lexicon.txt"))
ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"), ENG = EnglishModel.from_files(os.path.join(DATA, "english-common.txt"),
os.path.join(DATA, "english-dict.txt"), os.path.join(DATA, "english-dict.txt"),
os.path.join(DATA, "english-bigram.json")) os.path.join(DATA, "english-bigram.json"))
WEIGHTS = Classifier.load_weights(os.path.join(DATA, "classifier-weights.json"))
LOAD_TIME = time.time() - _start LOAD_TIME = time.time() - _start
@@ -34,13 +37,13 @@ def convert(keys):
def make_engine(): def make_engine():
return CompositionEngine(Classifier(LEX, ENG)) return CompositionEngine(Classifier(LEX, ENG, weights=WEIGHTS))
def type_keys(s, e): def type_keys(s, e):
out = "" out = ""
for c in s: for c in s:
key = ENTER if c == "\n" else TAB if c == "\t" else c key = {"\n": ENTER, "\t": TAB, "⇧": SHIFT_TAP, "←": LEFT, "→": RIGHT, "↓": DOWN, "\b": BACKSPACE}.get(c, c)
handled, commit = e.handle(key) handled, commit = e.handle(key)
if commit is not None: if commit is not None:
out += commit out += commit
@@ -86,7 +89,7 @@ class ConverterTests(unittest.TestCase):
class ClassifierTests(unittest.TestCase): class ClassifierTests(unittest.TestCase):
c = Classifier(LEX, ENG) c = Classifier(LEX, ENG, weights=WEIGHTS)
def test_zhuyin(self): def test_zhuyin(self):
for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]: for s in ["su3cl3", "ji3", "cl3", "ru ", "u ", "fu ", "183", "2k7", "5j/ "]:
@@ -101,7 +104,7 @@ class ClassifierTests(unittest.TestCase):
def test_learning(self): def test_learning(self):
learned = LearnedPreferences() learned = LearnedPreferences()
c = Classifier(LEX, ENG, learned) c = Classifier(LEX, ENG, learned, WEIGHTS)
self.assertEqual(c.mode(list("so"), finishing=True), LITERAL) self.assertEqual(c.mode(list("so"), finishing=True), LITERAL)
learned.record("so", ZHUYIN) learned.record("so", ZHUYIN)
self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN) self.assertEqual(c.mode(list("so"), finishing=True), ZHUYIN)
@@ -150,6 +153,9 @@ class EngineTests(unittest.TestCase):
self.assertEqual(type_keys("2j/ \n", make_engine()), "東") self.assertEqual(type_keys("2j/ \n", make_engine()), "東")
self.assertEqual(type_keys("2.5\n", make_engine()), "2.5") self.assertEqual(type_keys("2.5\n", make_engine()), "2.5")
def test_tone_then_space_stays_chinese(self):
self.assertEqual(len(type_keys("u4 ", make_engine())), 1)
def test_out_of_order_converts(self): def test_out_of_order_converts(self):
self.assertEqual(type_keys("uvl3\n", make_engine()), "小") self.assertEqual(type_keys("uvl3\n", make_engine()), "小")
@@ -188,5 +194,234 @@ class MixedSegmentTests(unittest.TestCase):
self.assertLess(per_key, 0.05) self.assertLess(per_key, 0.05)
class ShiftTapTests(unittest.TestCase):
def test_toggles_english_mode(self):
e = make_engine()
self.assertEqual(type_keys("⇧ru su ", e), "ru su ")
self.assertTrue(e.english_mode)
self.assertEqual(e.marked_text, "")
self.assertEqual(e.handle(ENTER), (False, None))
self.assertTrue(e.english_mode)
self.assertEqual(type_keys("⇧su3cl3\n", e), "你好")
self.assertFalse(e.english_mode)
def test_tap_twice_cancels(self):
self.assertEqual(type_keys("⇧⇧su3cl3\n", make_engine()), "你好")
def test_mid_word_flips_and_learns(self):
e = make_engine()
type_keys("so", e)
e.handle(SHIFT_TAP)
self.assertEqual(e.mode, ZHUYIN)
self.assertFalse(e.english_mode)
type_keys("\n", e)
self.assertEqual(e.classifier.learned.weights["so"], 1)
def test_english_mode_not_learned(self):
e = make_engine()
type_keys("⇧hello ", e)
self.assertEqual(e.classifier.learned.weights, {})
def test_backspace_goes_to_app(self):
e = make_engine()
e.handle(SHIFT_TAP)
self.assertEqual(e.handle(BACKSPACE), (False, None))
self.assertTrue(e.english_mode)
def test_detector(self):
d = ShiftTapDetector()
d.shift_down(0)
self.assertTrue(d.shift_up(0.1))
d.shift_down(1)
d.key_pressed()
self.assertFalse(d.shift_up(1.1))
d.shift_down(2)
self.assertFalse(d.shift_up(3))
class CursorTests(unittest.TestCase):
def test_moves_and_clamps(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4", e)
self.assertEqual(e.cursor_offset, 4)
type_keys("←", e)
self.assertEqual(e.cursor_offset, 3)
type_keys("←←←←", e)
self.assertEqual(e.cursor_offset, 1) # stops after the first character
type_keys("→→→→", e)
self.assertEqual(e.cursor_offset, 4)
def test_choose_candidate_in_the_middle(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4←←↓", e)
self.assertTrue(e.showing_candidates)
self.assertTrue(all(r[1] == 2 for r, _ in e.candidates))
e.select_candidate([t for _, t in e.candidates].index("添"))
self.assertEqual(type_keys("\n", e), "今添天氣")
def test_english_returns_cursor_to_end(self):
e = make_engine()
type_keys("rup wu0 ←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("?", e)
self.assertEqual(e.marked_text, "")
class InsertInMiddleTests(unittest.TestCase):
def test_insert_syllable(self):
e = make_engine()
type_keys("rup fu4←wu", e)
self.assertEqual(e.marked_text, "今ㄊㄧ氣")
self.assertEqual(e.cursor_offset, 3)
type_keys("0 ", e)
self.assertEqual(e.marked_text, "今天氣")
self.assertEqual(e.cursor_offset, 2)
self.assertEqual(type_keys("\n", e), "今天氣")
def test_cursor_stops_after_first_character(self):
e = make_engine()
type_keys("su3cl3←←←", e)
self.assertEqual(e.cursor_offset, 1)
type_keys("ji3\n", e)
self.assertEqual(e.marked_text, "")
def test_no_stop_before_first_chinese(self):
e = make_engine()
type_keys("hello ji3←←", e) # "hello " held, then 我: nothing to pick before 我
type_keys("↓", e)
self.assertTrue(e.showing_candidates)
def test_backspace_deletes_character_before_cursor(self):
e = make_engine()
type_keys("rup wu0 wu0 fu4←←\b", e)
self.assertEqual(e.marked_text, "今天氣")
self.assertEqual(e.cursor_offset, 1)
def test_backspace_inside_inserted_syllable(self):
e = make_engine()
type_keys("rup fu4←wu\b", e)
self.assertEqual(e.marked_text, "今ㄊ氣")
def test_keeps_picked_phrase_before_cursor(self):
e = make_engine()
type_keys("rup wu0 ↓", e)
e.select_candidate([t for _, t in e.candidates].index("添"))
type_keys("fu4←wu0 ", e) # insert 天 between 添 and 氣
self.assertEqual(e.marked_text[:3], "今添天")
def test_candidates_for_chinese_before_english(self):
e = make_engine()
type_keys("su3cl3hello↓", e)
self.assertTrue(e.showing_candidates)
class FirstToneAfterEnglishTests(unittest.TestCase):
def test_first_tone_after_english(self):
for keys in ["ru", "t", "gj", "vup", "tj", "el"]:
e = make_engine()
out = type_keys("hello " + keys + " \n", e)
self.assertEqual(out[:6], "hello ", keys)
self.assertEqual(len(out), 7, keys) # one Chinese character after it
self.assertFalse(out[6].isascii(), keys)
class HeldWordTests(unittest.TestCase):
def test_next_word_decides(self):
cases = {"up jo4 ": "因為", "up the ": "up the ", "pick up the ": "pick up the ",
"up\n": "up", "i love ": "i love "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_held_word_stays_composing(self):
e = make_engine()
self.assertEqual(type_keys("up ", e), "")
self.assertEqual(e.marked_text, "up ")
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "up")
self.assertEqual(type_keys(" the ", e), "up the ")
class BacktickSymbolTests(unittest.TestCase):
def test_forced_symbols(self):
cases = {"`<": ",", "`.": "。", "`?": "?", "`;": ";", "`[": "「",
"hello`,": "hello,", "su3cl3`.": "你好。", "```\n": "```"}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_plain_backtick(self):
e = make_engine()
self.assertEqual(type_keys("`", e), "")
self.assertEqual(e.marked_text, "`")
self.assertEqual(type_keys("ls` ", e), "`ls` ")
def test_escape_cancels(self):
e = make_engine()
type_keys("su3`", e)
e.handle(BACKSPACE)
self.assertEqual(e.marked_text, "你")
class FirstWordHoldTests(unittest.TestCase):
"""The first word is not committed as English on space if it also reads as Zhuyin."""
def test_next_word_decides(self):
cases = {"el ji3 ": "高我", "no su3cl3 ": "no 你好", "ok ji3 ": "ok 我",
"i love you ": "i love you ", "up the ": "up the "}
for keys, want in cases.items():
self.assertEqual(type_keys(keys, make_engine()), want, keys)
def test_not_held_after_english(self):
e = make_engine()
self.assertEqual(type_keys("hello world ", e), "hello world ")
self.assertEqual(type_keys("so ", e), "so ") # right after English: commits at once
class UserHistoryTests(unittest.TestCase):
def setUp(self):
self.saved = []
self.c = Classifier(LEX, ENG, weights=WEIGHTS, history=UserHistory(on_change=self.saved.append))
def pick(self, keys, phrase):
e = CompositionEngine(self.c)
type_keys(keys + "↓", e)
e.select_candidate([t for _, t in e.candidates].index(phrase))
return type_keys("\n", e)
def test_picked_candidate_comes_first(self):
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "天")
self.assertEqual(self.pick("wu0 ", "添"), "添")
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
e = CompositionEngine(self.c)
type_keys("wu0 ↓", e)
self.assertEqual(e.candidates[0][1], "添")
self.assertEqual(self.saved[-1], {"ㄊㄧㄢ": {"添": [1, 1]}})
def test_phrases_still_win(self):
self.pick("wu0 ", "添")
self.assertEqual(type_keys("rup wu0 \n", CompositionEngine(self.c)), "今天")
def test_most_picked_wins(self):
self.pick("wu0 ", "添")
self.pick("wu0 ", "添")
self.pick("wu0 ", "天")
self.assertEqual(type_keys("wu0 \n", CompositionEngine(self.c)), "添")
def test_round_trip_and_clear(self):
self.pick("wu0 ", "添")
h = UserHistory(self.saved[-1])
self.assertEqual(h.data, {"ㄊㄧㄢ": {"添": [1, 1]}})
h.record([bopomofo.parse(list("wu0 ")).syllables[0].syllable], "天")
self.assertEqual(h.data["ㄊㄧㄢ"]["天"], [1, 2])
h.clear()
self.assertEqual(h.data, {})
def test_evicts_least_recent(self):
h = UserHistory()
h.max_entries = 2
syl = [bopomofo.parse(list("wu0 ")).syllables[0].syllable]
for p in ["天", "添", "田"]:
h.record(syl, p)
self.assertEqual(set(h.data["ㄊㄧㄢ"]), {"添", "田"})
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+46 -3
View File
@@ -6,6 +6,9 @@ import types
import unittest import unittest
HERE = os.path.dirname(os.path.abspath(__file__)) HERE = os.path.dirname(os.path.abspath(__file__))
import tempfile # noqa: E402
os.environ["APPDATA"] = tempfile.mkdtemp(prefix="autozhuyin-test-") # learned / history files
sys.path.insert(0, os.path.dirname(HERE)) sys.path.insert(0, os.path.dirname(HERE))
# --- stand-ins for PIME modules (only what the wrapper uses) --- # --- stand-ins for PIME modules (only what the wrapper uses) ---
@@ -36,6 +39,8 @@ class FakeTextService:
def setCandidateList(self, c): self.reply["candidateList"] = c def setCandidateList(self, c): self.reply["candidateList"] = c
def setCandidateCursor(self, p): self.reply["candidateCursor"] = p def setCandidateCursor(self, p): self.reply["candidateCursor"] = p
def showMessage(self, message, duration=3): self.reply["showMessage"] = message
def setShowCandidates(self, show): def setShowCandidates(self, show):
self.showCandidates = show self.showCandidates = show
self.reply["showCandidates"] = show self.reply["showCandidates"] = show
@@ -44,15 +49,15 @@ class FakeTextService:
textService.TextService = FakeTextService textService.TextService = FakeTextService
sys.modules["textService"] = textService sys.modules["textService"] = textService
from autozhuyin.autozhuyin_ime import AutoZhuyinTextService # noqa: E402 from autozhuyin.autozhuyin_ime import AutoZhuyinTextService, _shared_classifier # noqa: E402
VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28} VK = {"\n": 0x0D, "\t": 0x09, " ": 0x20, "\b": 0x08, "↓": 0x28}
class FakeKeyEvent: class FakeKeyEvent:
def __init__(self, ch, shift=False, ctrl=False): def __init__(self, ch, shift=False, ctrl=False):
self.keyCode = VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA) self.keyCode = 0x10 if ch == "⇧" else VK.get(ch, ord(ch.upper()) if ch.isalnum() else 0xBA)
self.charCode = 0 if ch in ("\n", "\t", "\b", "↓") else ord(ch) self.charCode = 0 if ch in ("\n", "\t", "\b", "↓", "⇧") else ord(ch)
self.keyStates = [0] * 256 self.keyStates = [0] * 256
if shift: if shift:
self.keyStates[0x10] = 0x80 self.keyStates[0x10] = 0x80
@@ -74,6 +79,17 @@ def press(svc, ch, **kw):
return True, svc.reply return True, svc.reply
def tap_shift(svc):
"""Shift down then up with nothing in between, as PIME delivers it."""
svc.reply = {}
down = FakeKeyEvent("⇧", shift=True)
assert not svc.filterKeyDown(down)
up = FakeKeyEvent("⇧")
if svc.filterKeyUp(up):
svc.onKeyUp(up)
return svc.reply
def type_text(svc, text): def type_text(svc, text):
out = "" out = ""
for ch in text: for ch in text:
@@ -86,12 +102,18 @@ def type_text(svc, text):
class WrapperTests(unittest.TestCase): class WrapperTests(unittest.TestCase):
def setUp(self): def setUp(self):
_shared_classifier().history.clear()
_shared_classifier().learned.clear()
self.svc = AutoZhuyinTextService(client=None) self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate() self.svc.onActivate()
def test_mixed_sentence(self): def test_mixed_sentence(self):
self.assertEqual(type_text(self.svc, "ji3 hello world su3cl3 "), "我hello world 你好") self.assertEqual(type_text(self.svc, "ji3 hello world su3cl3 "), "我hello world 你好")
def test_backtick_symbol(self):
self.assertEqual(type_text(self.svc, "`<"), ",")
self.assertEqual(type_text(self.svc, "ji3`."), "我。")
def test_composition_shown(self): def test_composition_shown(self):
type_text(self.svc, "su3c") type_text(self.svc, "su3c")
self.assertEqual(self.svc.reply["compositionString"], "你ㄏ") self.assertEqual(self.svc.reply["compositionString"], "你ㄏ")
@@ -123,5 +145,26 @@ class WrapperTests(unittest.TestCase):
self.assertEqual(out + reply.get("commitString", ""), "Hi ") self.assertEqual(out + reply.get("commitString", ""), "Hi ")
class ShiftTapWrapperTests(unittest.TestCase):
def setUp(self):
_shared_classifier().history.clear()
_shared_classifier().learned.clear()
self.svc = AutoZhuyinTextService(client=None)
self.svc.onActivate()
def test_tap_forces_english_lowercase(self):
reply = tap_shift(self.svc)
self.assertEqual(reply.get("showMessage"), "英文")
self.assertEqual(type_text(self.svc, "ru su "), "ru su ")
self.assertEqual(tap_shift(self.svc).get("showMessage"), "中英自動")
def test_shift_with_letter_is_not_a_tap(self):
self.svc.filterKeyDown(FakeKeyEvent("⇧", shift=True))
press(self.svc, "H", shift=True)
up = FakeKeyEvent("⇧")
self.assertFalse(self.svc.filterKeyUp(up))
self.assertFalse(self.svc.engine.english_mode)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Fills windows/autozhuyin/data/ from the shared Mac resources. """Fills windows/autozhuyin/data/ from the shared Mac resources.
- lexicon.txt, english-common.txt: copied from Resources/ - lexicon.txt, english-common.txt, classifier-weights.json: copied from Resources/
- english-dict.txt: filtered from a system word list (default /usr/share/dict/words, - english-dict.txt: filtered from a system word list (default /usr/share/dict/words,
Webster's 2nd edition, public domain), since Windows has no such file Webster's 2nd edition, public domain), since Windows has no such file
- english-bigram.json: precomputed letter-bigram table - english-bigram.json: precomputed letter-bigram table
@@ -21,7 +21,7 @@ from autozhuyin.core.english import train_bigram # noqa: E402
data = WIN / "autozhuyin" / "data" data = WIN / "autozhuyin" / "data"
data.mkdir(parents=True, exist_ok=True) data.mkdir(parents=True, exist_ok=True)
for name in ("lexicon.txt", "english-common.txt"): for name in ("lexicon.txt", "english-common.txt", "classifier-weights.json"):
shutil.copy(ROOT / "Resources" / name, data / name) shutil.copy(ROOT / "Resources" / name, data / name)
word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words") word_list = Path(sys.argv[1] if len(sys.argv) > 1 else "/usr/share/dict/words")