大模型读长文档，中间部分最易出错
现在的视觉大模型读长文档时，中间部分最容易出错——这不是偶然，而是系统性缺陷。研究者用全合成文档做了可控测试，发现模型在文档长度增加时性能急剧下降，且对文档中间三分之一的内容特别不敏感，五个模型中有四个在中间区域表现最差。此外，图表理解在长文档中几乎崩溃。这些失败模式在现有基准测试中无法暴露，说明模型可能只是在刷题，而非真正理解长文档。它不是你明天能用上的，但提醒你：别轻信模型处理长文档的能力。
📄 原文摘要(英文)
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.