2000小时人做家务视频开源,机器人学动作不再缺数据
机器人学做家务一直缺数据:自己录太慢,网上视频又没标注。Open-AoE 直接拉来500多人用手机拍了2000小时日常操作视频——开冰箱、拧瓶盖、叠衣服——还自动标好了手部姿势、动作起止和文字说明。更关键的是,它附赠一套工具链,能把手机视频直接转成机器人能用的训练格式,连不同机械臂的映射都帮你算好。这不是你明天能用的App,但做机器人研发的团队可以省下一年数据采集时间。
📄 原文摘要(英文)
Egocentric videos of human manipulation provide scalable supervision for embodied intelligence, yet existing resources rarely combine low-cost continuous capture, manipulation-level structured annotations, and reusable tools for robot learning. We present Open-AoE, an open, community-oriented egocentric manipulation dataset and toolchain spanning the full pipeline from smartphone capture to model training. Its first release contains approximately 2,000 hours of manipulation video collected in natural environments by 500+ contributors using 400+ smartphones. The dataset provides text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions. Open-AoE further includes a data processing pipeline that transforms raw recordings into structured samples through temporal action segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction. Meanwhile, we provide a separate downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models. By integrating scalable capture, structured processing, and downstream adaptation, Open-AoE reduces the barriers to both data contribution and reuse, providing practical open infrastructure for embodied model training, human-to-robot transfer, and world modeling.