游戏AI当老师：教大模型学会“走一步看三步”
大模型在复杂游戏里常犯“短视”错误——赢了最后一步，却忘了前面哪步走错了。研究者发现，游戏AI（解算器）能算出每一步的“价值变化”：比如推箱子时，这一步让箱子离目标更近，价值就涨。他们用这个变化值当“老师信号”，教大模型在每一步都做对选择，而不是只看最终输赢。在推箱子、扫雷、华容道等游戏中，这种方法让大模型表现全面超越传统训练方式，甚至能零样本迁移到购物、网页操作等现实任务。它不是你明天能用上的，但揭示了AI学会“长远规划”的一条新路。
📄 原文摘要(英文)
Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.