SpatialScout: Evaluating VLM Agents for Active 3D Object Mapping
ORIGINAL / SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?
It introduces a benchmark for active object-level 3D mapping that requires vision-language model agents to autonomously explore indoor scenes and estimate object centers under bounded interaction budgets, filling the gap that spatial question answering cannot test metric map construction.
01 ABSTRACT
SpatialScout is a new benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping, comprising 80 indoor scenes and 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and poses, agents control a camera to explore each scene and estimate object centers, with evaluation jointly measuring localization accuracy and target coverage. Experiments reveal substantial limitations: the strongest model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding IoU 0.5 and only 3.25% exceeding 0.9.
02 KEY FINDINGS
- The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets.
- Evaluation uses object-scale overlap scores that also penalize missing predictions, jointly measuring localization accuracy and target coverage.
- The strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with only 3.25% of targets exceeding IoU 0.9.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL