Geometric and semantic foundation models are increasingly used to build spatial maps for robots, but most work has focused on the mapping itself rather than on using those maps in a complete autonomy pipeline. In a new paper, researchers from MIT and the University of Illinois bridge that gap by integrating FOUND-IT—a system that constructs open-set 3D scene graphs from a single uncalibrated camera—into a mobile manipulator platform.
The team demonstrated that the resulting system can handle several practical objectives: detecting changes in objects, supporting multiple mapping sessions, determining navigable paths, and directly guiding object grasping for scene rearrangement. These capabilities were tested in rigorous on-robot experiments, moving beyond the typical simulation-only or mapping-only evaluations common in this area.
The paper also offers a candid assessment of the approach's weaknesses. Because FOUND-IT relies on monocular vision and foundation-model predictions, the authors highlight specific limitations that emerge when the scene graph is used as the robot's spatial memory. The findings suggest that while foundation-model-driven scene graphs are promising for task-driven autonomy, careful attention is still needed to make them reliable in real-world mobile manipulation. The code is publicly available.