NVIDIA has introduced Cosmos3-H-Dreams, an interactive generative simulation model for clinical surgery. It builds on earlier tabletop suturing work, but now learns from video and robot kinematics recorded during clinical cholecystectomy procedures on CMR Surgical's Versius platform. According to the authors, it is the first real-time generative surgical simulator trained on such recordings to support live kinematic control.
The model is based on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment, then specialized on curated cholecystectomy video and kinematics. It uses 28 action dimensions covering pose, gripper, energy, thumbstick axes, and thumbstick buttons. A bidirectional teacher is trained on 24 then 72 future frames, then distilled into causal students with four-step and two-step denoising schedules. The two-step model is separately trained, not simply the four-step model run with fewer sampling steps.
To run interactively, the system uses FlashDreams accelerated streaming inference and an adapted FadeMem module that consolidates the growing key/value cache during long rollouts. The authors caution that visual realism alone is not proof of biomechanical accuracy; generated scenes must follow tool commands and preserve anatomy over time. Small errors can accumulate, so temporal consistency is a central challenge. This article is based solely on the NVIDIA/Hugging Face announcement and includes no independent evaluation.