AI跑长任务：平均花85分钟，成功率不到5%
现在的AI评测大多只考几分钟的短任务，看最终结果给分。这篇论文反其道而行，设计了一套需要几十分钟到几小时、包含46个长任务的测试集，比如复现实验、写软件、分析多模态数据。每个任务被拆成细粒度子任务，AI做对一步就给一分，而不是只看最后成不成。结果：最强模型在95%得分率下通过率仅15.2%，完美通过率10.9%，所有模型平均通过率只有4.3%和1.7%。每个任务平均消耗990万token、231轮交互、85分钟。这不是你明天能用上的工具，但它揭示了当前AI在需要长期规划和迭代调试的任务上离可靠还很远。
📄 原文摘要(英文)
AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.