Researchers have introduced CAPSafe, a benchmark designed to test how well large language models reason about safety in embodied agents. It covers two levels: whether a single action is safe given its environmental context, and whether an agent can generate safe long-horizon plans for complex hazardous household tasks. The benchmark models safety conditions where hazards are situational, can co-occur, and carry relative severity annotations.
Evaluating nine closed and open-source LLMs, the authors found a clear gap between hazard recognition and safe planning. Several models exceeded 90% accuracy on identifying unsafe actions, yet safe long-horizon planning proved much harder. Gemini-3-pro achieved the highest safe success rate at 78.19%, while all other models remained below 65%.
The results suggest that robust safety reasoning over long action sequences is still an open challenge. Recognizing a single hazard is not enough; agents must also weigh multiple, co-occurring risks across extended plans before acting in the real world.