3D场景像搭积木:物体是原生单元
现在的AI看3D场景,看到的是一堆点或小方块,要找到“哪个点属于哪个物体”得事后猜。这篇让AI直接输出“物体组”——每个物体是一个独立令牌组,包含身份令牌和外观令牌,像给每个物体发一张身份证和一张照片。从多张照片直接学,不需要人工标3D数据。效果上,分割比传统方法准,还能直接拖拽、删除、替换物体,检索时按物体搜而不是按点搜。它不是你明天能用上的,但方向很明确:让AI理解场景像人一样,先认物体,再认细节。
📄 原文摘要(英文)
A 3D scene is understood through its objects, not the primitives that compose them. Yet feed-forward reconstruction methods output dense, unstructured sets of points or Gaussians, leaving object-level structure to be recovered after the fact. We propose a feed-forward framework that decomposes a scene into instance-structured 3D token groups directly from unposed multi-view images -- compact object-centric units from which reconstruction, segmentation, and manipulation all follow. Each token group pairs an instance token capturing entity-level identity with anchor tokens that encode local geometry and appearance, which are decoded into a set of 3D Gaussians. This two-level factorization decouples object identity from local appearance, making object instances a native interface of the representation rather than a derived product. The token groups are learned through differentiable rendering with joint reconstruction and segmentation supervision, requiring no 3D annotations. Our feed-forward model surpasses per-scene optimization baselines in class-agnostic instance segmentation while remaining competitive in novel view synthesis. Beyond these metrics, the same token groups directly unlock instance-level scene editing -- removing, translating, or inserting objects by operating on their groups -- as well as efficient open-vocabulary 3D instance retrieval, where retrieval complexity scales with the number of instances rather than primitives.