AI 推荐训练：别再奖励单个答案
训练 AI 推荐时，一条帖子底下往往有多个好回复。现有方法随机挑一个当标准，导致 AI 学到的 85% 是「猜中挑了哪个」，而不是「推荐得好不好」。这篇提出 ReMaRk:把整条帖子的所有回复合并成一个标准，一次打分。结果训练成本大幅下降——达到同样效果只需原来 25% 到 53% 的步数，效果不输甚至更好。这不是你明天能用的产品功能，但它指向一个更本质的转变：当人类答案本来就不唯一时，AI 该学的是「满足所有人」，而不是「押中一个」。
📄 原文摘要(英文)
A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.