Humanoid robots that interact naturally with people need to understand social context and plan actions, not just map speech to motion. Existing methods typically generate continuous trajectories from pre-made speech, but they lack awareness of complex social environments and explicit high-level semantic planning. To address this, the authors introduce SocialCoAct, a unified agent-centered framework for vision-language-model (VLM) based egocentric social co-action.
The framework includes SocialCoAct-Agent, which processes egocentric observations and dialogue history to produce verbal responses and atomic action sequences, aligning each action with its corresponding utterance span. The authors test the agent in MuJoCo simulations and on a physical humanoid robot. To evaluate the required VLM capabilities, they built SocialCoAct-Bench from continuous egocentric recordings of natural multi-party interactions, containing 423 human-verified multiple-choice questions across three subtasks: Long Episode Understanding, Speaker Intention Understanding, and Coordinated Action Planning.
Across a range of closed-source and open-weight VLMs, including general and egocentric models, the highest overall accuracy reached 77.54%, while accuracy on the hard subset was only 46.51%. Extended reasoning improved Qwen3-VL-2B and Qwen3-VL-4B by 19.62 and 17.02 percentage points, respectively, but reduced inference throughput by 53.6× and 20.2×. Visual-input ablations showed that the benefits of visual context vary across models, indicating that the task remains challenging and that current VLMs have substantial room for improvement in social co-action planning.