让AI照着成品抄作业：新基准专测“看会”而非“听会”
现在的编程AI评测，都是给它一个需求描述，让它写代码。但真实开发里，程序员经常是看着一个能跑的成品，去另一个残缺的代码库里把它复刻出来——这靠的是“看会”，不是“听会”。新基准ProgramDistill把26个真实Web应用拆成1975个可回放的行为，生成4063道“照着成品补代码”的任务。结果：最强的GPT-6 Astra在完整重建上只成功49.2%，而残缺程度越深，成功率从100%暴跌到64%再到32%。也就是说，AI“看懂别人代码再复刻”的能力，远没有“听指令写代码”成熟。它不是你明天能用的工具，但它是下一代编程AI训练方向的信号：光喂需求不够，得让AI学会从成品里反推意图。
📄 原文摘要(英文)
Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.