大模型读长文档，中间部分最易出错
大模型读长文档的能力评测一直有，但现有测试把长度、布局、问题难度混在一起，模型到底哪里不行说不清。这篇论文造了一套纯合成的长文档测试集，把文档长度、排版样式、内容模态、问题类型拆开单独控制，然后测了7个顶级视觉语言模型。结果发现三个现有测试暴露不了的问题：文档一长，模型表现直线下滑；所有模型对文档中间三分之一的部分最不敏感，最差的一个从开头到结尾掉了8.3个百分点；图表理解在长文档里直接崩掉。这些模型可能只是在刷现有测试的套路，并没有真正理解长文档。它不是你明天能用上的，但如果你在选模型做文档处理，这个测试能帮你看出哪个模型在长文档中间部分更稳。
📄 原文摘要(英文)
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.