把单词拆成“零件+操作码”，词表缩水两成
现在的 AI 分词器把 hello、Hello、HELLO、Héllo 当成四个互不相干的词，白白占掉词表位置；这篇论文反着来：先拆出“基础词”和“怎么改它”的指令(比如大写、加变音符号、重复字母)，存进一个保留区，用完还能原样拼回去。在自然语言和代码语料上，词表实际占用最多省了 19.7%，训练出的 GPT-2 写 Python 语法正确率还更高。它不是你明天能直接用的东西，但指向一个趋势：与其让模型死记每个变体，不如教它“变形规则”。
📄 原文摘要(英文)
Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization. We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area. We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible. Across natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained exhaustion conditions, reducing actual vocabulary slot requirements by up to 19.7%. Downstream evaluations on 98M-parameter GPT-2 models show that the Functionalizer improves Python code syntax validity (9.12% vs. 7.70%) while reducing duplicate n-gram repetition in natural language prose. These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.