Engineering case study · Open-source Khmer language tools

pheasaKhmer

Open-source tools I created for the Khmer language: a Unicode normalizer for Khmer text and language AI, an engine that turns chat-style romanized Khmer into Khmer script and back, an on-device Android keyboard with a Rust core, and a web converter and Telegram bot.

Python / Rust / Kotlin / UniFFI / FastAPI / Docker

sok sabay teសុខសប្បាយទេ“te”ទេតែ

The problem

Many Khmer speakers type in Latin letters because the Khmer keyboard is slow, and chat romanization has no standard: “sous dey”, “suosdey” and “sursdey” all mean សួស្តី. Khmer script has a problem of its own for software. Unicode allows several code point orders for the same visible syllable, so text that looks identical fails search, deduplication, tokenizers, and evaluation metrics. pheasaKhmer is a set of open tools for both problems, built to work offline where they can, document where their data comes from, and be checked by native speakers.

What’s in it

pheasa
A normalizer for Khmer text, so text that looks the same is the same, and a benchmark for Khmer language AI in progress.
khmer-engine
Converts chat-style romanized Khmer to Khmer script (sok sabay te → សុខសប្បាយទេ) and Khmer script back to romanized text, in chat style or the UNGEGN standard.
khmer-keyboard
An Android keyboard on a Rust core: type Khmer in Latin letters or on a Khmer layout. Nothing typed leaves the phone. iOS is next.
khmer-converter
A web page and Telegram bot for the same conversions, with a button to report a wrong conversion that feeds the engine’s test set.

Engineering decisions

  • Wrote a normalizer that puts Khmer code points in the order described by Unicode Technical Note #61, reconciling spellings that Unicode NFC leaves different. It can report every change with input offsets and the rule responsible, and options that remove information are off by default.
  • Ported the normalizer to Rust for places without Python, such as phone keyboards, and tested it against cases generated from the Python implementation under the same normalization version.
  • Indexed each word of a 61,980-word pronunciation lexicon under several romanizations: by sound, by spelling rules, by the UNGEGN standard, and by the consonant abbreviations people use in chat. Matching keys fold spelling variants together, so the many ways one word is typed all find it.
  • Chose words with a Viterbi search over word-pair probabilities from 6.5 million words of Khmer web text, so context decides between words that sound alike, such as បង and បង់.
  • Ported the engine to a Rust core with Kotlin and Swift bindings generated by UniFFI. Golden tests hold it to the Python engine on 18,254 matching keys, 294 conversions, and 2,611 keystroke suggestion lists.
  • Kept learning on the device: a picked word ranks higher next time, most after the same previous word, so one choice does not push a common word down everywhere.

The result

The normalizer makes Khmer text that looks the same byte-identical, in Python and in a Rust port that gives the same output for every input. The conversion engine turns romanized Khmer into Khmer script and back using a 61,980-word lexicon. Its Rust port powers an Android keyboard that suggests words in about 0.36 ms per keystroke, needs no permissions, and keeps everything typed on the phone. A web page and Telegram bot offer the same conversions without storing what people type.