把单词拆成“配方”，词汇表缩水两成
现在的 AI 分词器把 hello、Hello、HELLO 当成三个互不相干的词，白白占掉词汇表的位置；要么就粗暴地全转成小写，丢掉信息。这篇论文换了个思路：把“词根”和“怎么变”拆开存——hello 是原料，大写、加音调符号、重复字母都是可逆的“操作码”，像配方一样拼回去。结果词汇表最多省了 19.7%，训练出的模型写 Python 代码语法正确率还更高。它不是你明天能直接用的东西，但指向一个更省、更懂结构的 AI 分词方向。
📄 原文摘要(英文)
Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization. We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area. We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible. Across natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained exhaustion conditions, reducing actual vocabulary slot requirements by up to 19.7%. Downstream evaluations on 98M-parameter GPT-2 models show that the Functionalizer improves Python code syntax validity (9.12% vs. 7.70%) while reducing duplicate n-gram repetition in natural language prose. These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.