让AI少说废话的新招：不背答案，只比好坏
现在训练AI听话，主流办法是直接教它“这个回答好、那个回答差”，但有个副作用：AI会为了讨好训练数据，把话说得又长又绕，反而丢了自然感。这篇论文换了个思路——不直接告诉AI哪个答案好，而是给它一对回答，只问“哪个更好”，像用裁判打分而不是背标准答案。研究者把这套方法在Mistral、Llama、Gemma等5个主流模型上试了一遍，结果比现有方法更会赢，而且AI说话更简洁、不注水。它不是你明天就能装进App里的东西，但指向一个更省力、更接近人类学习方式的AI调教方向。
📄 原文摘要(英文)
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.