金融AI的考试全是开卷，这次终于闭卷了
金融AI的考试一直有个漏洞：考题、答案、甚至时间线都是提前定好的，AI只要背套路就能拿高分。这篇论文把考试改成闭卷：一个模拟用户心里藏着一套投资策略，但AI不知道，它得自己上网找数据、自己设定时间截点做回测，最后交出的报告还要被程序逐项验证——有没有用到未来数据、算的指标对不对、策略是不是用户真正要的那个。结果四个主流AI在150次考试里全军覆没，没有一次通过；用户签了字的报告，程序一查就露馅。更扎心的是，AI连用户没明说的细节只能自己想起来54%到64%，人类专家把88%的失败归咎于AI本身。这不是你明天能用的工具，但它说明一件事：金融AI在真实世界里远没有演示里那么可靠，因为真实世界不会把答案提前给你。
📄 原文摘要(英文)
Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64\% of the details they were not told. Human experts attribute 88.33\% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.