AI Pulse
📄 论文解读

AI边看边挑错,还来得及纠正

监控一个人做事的 AI,最有价值的时刻不是事后复盘,而是错误刚发生、还来得及补救的那几秒。这篇让模型一边看第一人称视频、一边对照任务说明,实时判断你正在做哪一步、做得对不对——把动作分成做对、走样、手滑、补救、多加,漏掉的步骤则在视频结束后再算。研究者比较了两种做法:拆成「认步骤、定位时间、验执行」三段的分步流水线,和一个模型一次吐出全部结果的联合模型;后者在「抓错」上更准(84.1% 对 80.7%),尤其在「手滑」这类小失误上差距最大(56.4% 对 37.5%),分步流水线则被上一环的错误拖累。它离你家的厨房还远,但方向是:未来的 AI 助手不只是等你做完再打分,而是在你拧错螺丝的那一刻就开口。

📄 原文摘要(英文)

An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation [email protected] (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection [email protected] (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新