What was released
NVIDIA released Cosmos-H-Dreams on July 27, 2026. The action-conditioned world model takes an initial RGB surgical scene and a live stream of robot kinematics, then generates future video in short blocks. A person can control it with a keyboard or Meta Quest, or a learned policy can run in a closed loop.
How it reaches real time
The team distilled the bidirectional Cosmos-H-Surgical-Simulator into a causal few-step student and used FlashDreams with bounded-memory KV caching, CUDA Graphs, and compilation. NVIDIA reports roughly 160 fps on one RTX PRO 6000 versus about 10 fps for standard simulator inference. The released checkpoint specializes in dVRK tabletop suturing.
Why it matters
Physical surgical-robot experiments are costly and hard to repeat after failures. A responsive generative simulator may accelerate early policy evaluation, skills rehearsal, and synthetic-data generation. The model, code, and adaptation recipe are public, but adopters must review the NVIDIA Open Model License and supported GPU, BF16, and Linux conditions.
Medical-safety limits
Generated video does not guarantee tissue physics or patient outcomes. Out-of-distribution scenes, occlusions, deformation, and collisions may look plausible while being wrong. It must not directly control surgery or clinical decisions and needs independent physics simulation, expert review, physical testing, and applicable regulatory processes.