大模型解题像人一样：先想清楚再动手
我们总以为大模型解题是黑箱里疯狂计算，这篇论文发现它其实很像人：先想清楚问题属于哪一类(代数、几何)，再决定用什么招(化简、倒推)。研究者用数学教育里的两个概念——语义空间和启发式——拆解模型的解题过程，发现决定答案对错的不是步骤多长，而是用了哪些数学招数；而且模型和人类一样，集中精力在少数几个思路上更容易做对，东一榔头西一棒子反而容易错。更意外的是，强化学习训练会让模型变得「一根筋」，只爱用少数几种招数，这反而可能限制它。研究者试着在训练时鼓励模型多用不同招数，准确率确实提升了。它不是你明天能用上的工具，但给了我们一个看大模型「怎么想」的新视角：原来它的思考方式，比我们以为的更接近人。
📄 原文摘要(英文)
Large language models (LLMs) achieve strong performance on mathematical reasoning benchmarks, yet the mathematically meaningful skills underlying their reasoning remain underexplored. We introduce SHAPE, a framework that analyzes Chain-of-Thought (CoT) trajectories through two lenses developed in mathematics education: (1) semantic spaces: the model's evolving mathematical interpretations of a problem (e.g., algebraic, geometric), and (2) heuristics: the specific mathematical actions taken within those spaces (e.g., simplifying the problem, working backward). We first use SHAPE to analyze the reasoning patterns of various models. Our findings reveal that the mathematical heuristics employed by a model better explain final answer correctness than traditional CoT features. Furthermore, models are likely to reach correct solutions by concentrating their reasoning effort within a few semantic spaces rather than exploring many disparate ones -- a pattern consistent with human behavior. Next, we utilize the SHAPE lens to evaluate whether post-training truly enhances mathematical proficiency. We find that reinforcement learning induces mode-seeking in heuristic usage. Lastly, we post-train LLMs by promoting diverse heuristics and demonstrate its effectiveness in improving accuracy. Overall, SHAPE provides a theoretically-grounded diagnostic framework for decoding LLM reasoning and offers a new path toward post-training LLMs for math reasoning. The code for our model is available at https://github.com/holi-lab/SHAPE-of-CoT