RL 让大模型多任务不打架，SFT 却会
同一个大模型，先学 A 任务再学 B 任务，顺序不同结果天差地别。研究者发现：监督微调(SFT)在多任务连续训练时，任务之间会互相干扰——学完一个再学另一个，前面的能力会被冲掉；而强化学习(RL)却能让多个任务和平共处，甚至越学越稳。他们把原因挖到了参数层面：RL 每次更新只动少数参数，而且不同任务的更新方向几乎垂直，互不踩踏；SFT 的更新则又大又重叠，后学的任务直接覆盖前面的。基于这个发现，他们提出一种叫 Parallel-RL 的训练方式，把多任务拆开并行跑，效率和灵活性都大幅提升。这不是你明天能直接用的技巧，但它解释了为什么 RL 在复杂推理上往往比 SFT 更可靠——也提醒你，如果还在用 SFT 微调多任务模型，顺序和任务组合可能正在悄悄拖后腿。
📄 原文摘要(英文)
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.