给扩散模型装个“随机方向盘”：训练快4倍，生成质量还更好
想让AI画得更好，通常得在生成过程中加随机扰动，代价是训练又慢又贵。这篇论文换了个思路：不动生成器本身，只在它内部的“注意力路由”上做随机探索——就像开车时偶尔换条车道试试，但最终路线还是原来的。训练时，模型在少数几个去噪步骤里随机选择“看哪些关键区域”，用奖励信号来学习哪条路更好；推理时，它又恢复成原来的确定性采样，完全不影响使用。结果是：生成质量提升，训练效率比现有方法快近4倍。这不是你明天就能用的工具，但它指向一个趋势：以后微调AI生成模型，可能不再需要动大手术，而是像调方向盘一样精准、省力。
📄 原文摘要(英文)
Reinforcement learning can align visual generators with reward signals beyond their pretraining objectives, but existing score-function methods typically introduce stochasticity by perturbing the reverse denoising trajectory, leading to costly rollout-based optimization. We instead ask whether stochastic exploration can be introduced inside the generator while preserving its original sampler. Routing Policy Optimization (RPO) answers this question by turning attention routing into a stochastic exploration action. At a few selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention. The sampled route perturbs the velocity of that transition, and a Gaussian likelihood around the dense velocity scores the perturbation with a score-function objective. RPO further uses a controlled two-rollout estimator: two rollouts share the same prompt, initial latent, routed denoising levels, and dense anchor layers, differing only in their sampled attention routes. Their reward difference provides an advantage signal for the routed transitions. Because the likelihood differentiates the dense velocity itself, the learned updates remain effective after routing is removed. At inference, RPO restores dense attention and retains the original deterministic sampler. Experiments on text-to-image and text-to-video generation show that RPO improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods. Please see our project page at https://velox-world.com/rpo for visual results.