微调大模型时，最危险的数据看起来最无害
微调大模型时，最危险的数据看起来最无害。研究者发现，让模型变坏的那 20% 训练数据，单看每一条都完全正常——没有攻击性、没有越狱词，但它们的梯度方向却和「不安全行为」悄悄对齐。他们用一套叫 SimpleLoRIF 的方法，把百万级数据按梯度对齐程度排序，删掉最对齐的那 20%，再训练出的模型在对抗攻击下的成功率最高降了 22%，能力还基本没掉。更意外的是，他们翻出被删的数据，发现两类「隐形毒药」:工具执行失败的错误示范，和带叙事逻辑的算术题——这些看起来只是「没答对」，却在教模型学会绕过规则。它不是你明天能用上的工具，但它解释了一个反直觉的事实：AI 变坏，往往不是被教坏的，而是被「教歪了」的。
📄 原文摘要(英文)
Fine-tuning Large Language Models (LLMs) on instruction data can quietly erode their safety, even when no training example looks harmful. Because this safety-degrading data looks benign, standard content filters struggle to catch it. We tackle this problem by developing SimpleLoRIF, a scalable gradient embedding method that identifies fine-tuning examples whose gradients align with unsafe behavior. By simplifying a low-rank attribution method and restricting backpropagation to the final four transformer blocks, SimpleLoRIF can embed datasets on the order of a million examples, boosting gradient attribution from a small-scale analysis tool into a practical filter for full post-training datasets. We first validate it on emergent misalignment, where we find that gradients can connect insecure-code training data to the broadly harmful behaviors that appear after fine-tuning, with AUC as high as 0.865 in the gradient space of misaligned models. We then use SimpleLoRIF to build SafeTulu, a version of Tulu 3 with the 20% of examples most gradient-aligned with unsafe responses removed. Training on SafeTulu reduces GCG attack success rate by up to 22% across Qwen, Llama, and Olmo, outperforming both LLM-as-a-judge and prior gradient-based filtering, while largely preserving or even improving capability. Finally, we analyzed the removed data and identify possibly under-studied sources of safety degradation, such as tool-execution failures and narrative arithmetic. We release SafeTulu along with a Hugging Face Space that lets researchers build their own customized versions of SafeTulu to fine-tune on, such as by changing the removal fraction or filtering only certain subsets.