同一批任务，微调互相打架，强化学习却能和平共处
同一批任务，让大模型用两种主流方式学，结果完全相反：监督微调(SFT)在多任务下会互相打架——学完数学再学代码，前面学的就忘了；而强化学习(RL)却能让不同任务稳定共存，甚至越学越顺。研究者从参数层面找到了原因：微调的更新方向又大又乱，任务之间互相踩踏；强化学习因为自带奖励归一化，每次更新方向又小又正交，任务之间几乎不干扰。基于这个发现，他们提出了一种把多任务训练拆开并行的方法，效率和灵活性都大幅提升。
📄 原文摘要(英文)
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.