LEMuR: Language Embedded 3D Segmentation and Object Tracking for Mixed Reality

Abstract

Mixed reality enables distributed collaboration across distinct physical spaces. However, collaborator interactions are restricted from shared manipulation of physical objects in their surroundings to aid communication. Existing approaches either substitute low-fidelity virtual proxies for physical objects, stripping away the communicative context essential for interpreting actions, or rely on external cameras that constrains interaction to fixed capture regions and impede practical deployment. We introduce LEMuR, a language embedded 3D segmentation and tracking pipeline that, following an offline static scene reconstruction phase, enables real-time sharing and synchronization of everyday ph..