AI 模拟人类？它连扔骰子都学不像
我们总想让 AI 模拟真实世界——比如在经济学里代替人类做决策。但这篇论文发现，大模型连最基本的随机性都抓不住：让它从某个概率分布里抽 100 个样本，最好的模型也只能通过 20% 的统计检验，多数模型接近 0%。研究者造了 448 道题，从抛硬币到随机程序再到自然语言描述的随机过程，用 KS 检验看模型生成的样本和真实分布有没有区别。结果很诚实：加推理链能好一点，但远不够用。它不是你明天能用上的，但提醒你：别信 AI 能当好“替身”，它连随机都学不会。
📄 原文摘要(英文)
We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.