AI 说自己不知道，其实知道
金融回测的前提是模型没提前知道答案，业界靠三种办法防：信厂商给的「知识截止日」、直接问模型「你知道吗」、或让它「假装回到过去某天作答」。这篇把 2,226 条 1999–2026 年的美国金融事实(月度宏观数据、月末股价、美联储决议)做成基准，每条事实配四种问法，专门测这三种办法灵不灵。结果是：模型普遍低报自己的知识。GPT-5 在受控事实上答对 76%，却只承认知道 38%;在它自称「不知道」的事实里，仍有 61% 答对。它声称的知识边界还能往后拖最多七个季度；而给它一个历史日期，它大多不再声称「我本来就知道」，但答对率没变。也就是说，回测落在训练窗口内时，靠自述、靠指令、靠厂商截止日，都无法证明没有泄漏。它不是你明天能用的工具，但如果你或你所在的公司用 AI 做历史行情回测，这条结论直接推翻了你正在依赖的那道防线。
📄 原文摘要(英文)
Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.