强化学习白训了?一个固定偏移就够了
强化学习把扩散模型调得更会画字、更会摆物体,大家默认这得靠模型学会「看人下菜」的复杂本事。这篇反过来试:在冻结的基座模型每个模块输出上加一组固定向量,对任何提示词、任何位置、任何随机种子都一模一样,参数量比 RL 训出来的 LoRA 少 65 倍(0.29M 对 18.8M),训练目标里根本没有奖励,只去模仿公开 RL 模型的输出。结果这组死板的向量拿回了六个图像模型 90.0–96.3% 的奖励提升,两个视频模型上是 89.8% 和 111.9%;从零开始用在线 RL 训这组向量,在 SD3.5-M 上画字(OCR)和人类偏好(PickScore)甚至超过改权重的 RL。也就是说,RL 后训练的大部分收益可能只是把预训练模型整体往它本来就会的方向推了一把,而不是教会了它新东西。它不是你明天能用上的工具,但它改的是你对「微调」这件事的理解:那些昂贵的、按提示词定制的改造,可能有一大块只是换了个姿势的全局偏移。
📄 原文摘要(英文)
Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.