一个模型，随时切换它更听话还是更聪明
现在的大模型训练完就定型了：要么更安全，要么更聪明，想要不同侧重得重新训练。这篇让一个模型同时学会多种偏好，用的时候拨一个旋钮就行——想要它更无害还是更能解题，实时切换，不用重训。做法是把不同目标(比如有用性和安全性)分别训练成独立的小模块，推理时按你的偏好比例混合它们。在四个模型上，它比所有对比方法覆盖的权衡范围更广，而且训练成本更低。它不是你明天能用上的产品功能，但它指向一个方向：AI 不再是一个固定性格，而是可以按需调节的。
📄 原文摘要(英文)
Language model deployment often requires different trade-offs between competing objectives as user preferences change, yet standard post-training methods typically produce a single policy for a fixed objective weighting. We introduce LambdaMix, a new parameter efficient method for learning a preference conditioned family of policies with group-relative online reinforcement learning. LambdaMix jointly trains one low-rank adapter per objective over a frozen base model. One preference vector selects the adapter mixture that generates responses and weights the separately normalized rewards used to update the adapters. A single training run supports changing preferences at inference without further optimization. We experiment with four models on helpfulness and harmlessness and on solving math problems while following instructions. We also evaluate the two largest models on rewriting comments to preserve wording while reducing clues about personal attributes. On the first two tasks, LambdaMix achieves higher hypervolume than all evaluated conditioned and policy merging baselines across all four models. LambdaMix uses fewer training completions and stores fewer adapter parameters than a set of GRPO specialist policies. With architecture and initialization fixed, varying adapter mixtures and objective weights together gives wider score ranges and higher coverage than varying either alone.