删掉0.05%的权重，大模型就学会了新本事
想让大模型学会新技能，通常得动它几亿个参数。这篇论文发现：只把不到0.05%的权重直接清零，效果竟然和完整微调一样好，而且忘得更少——学新任务时，旧本事几乎不掉分，而常规方法最多掉5.4分。做法很简单：在Transformer的特定位置学一个二进制开关，决定哪些权重删掉。额外好处是，存下来的开关文件比完整模型小几百到几千倍。它不是你明天能直接用的工具，但它暗示了一件事：大模型的能力可能集中在极少数关键连接上，这给以后做更轻量、更不健忘的模型更新指了条路。
📄 原文摘要(英文)
How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.