What do NVIDIA's new Cosmos 3 Nano models actually simulate?

The four checkpoints and what each one does

The four checkpoints share a 16B trainable parameter backbone called Cosmos 3 Nano. All four carry the OpenMDW1.1 license which permits commercial and non-commercial use under its terms. Transfer-Auto generates synchronized video across up to eleven camera views including fisheyes. It also supports joint RGB–LiDAR generation. Per-camera World Scenario Map controls specify scene structure and traffic motion while text prompts guide appearance. Developers can vary environmental conditions to expand training and evaluation coverage. Optional initial RGB images anchor scene appearance. Joint LiDAR generation additionally uses aligned HD-map range-map controls.

Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps. Generated history and updated driving controls support fast streaming and long interactive rollouts. Per-camera captions and synchronized World Scenario Map controls are required. For LiDAR generation HD-Map range controls aligned with camera controls must be included. The distilled model generates successive chunks using its rolling history. Closed-loop interaction requires a simulation runtime and World Scenario Map renderer to update those controls from driving inputs.

Sim-Transfer takes a different approach. Your simulator computes motion contacts and physical state. Sim-Transfer uses its spatial controls and a text prompt to render color observations that a policy can consume. Depth segmentation edges and blur serve as spatial controls. Text guides appearance while physics and ground-truth state stay with the simulator. Action leads to simulator update which produces spatial controls which generate RGB which feeds the next policy action.

Sim-Bimanual simulates how an environment evolves in response to a bimanual robot's actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments controlled by a policy or human-operated joysticks. Start with an image move the robot or camera and generate the next observations. One control sequence can switch between viewpoint changes and manipulation. The mixed-action example uses fifty-nine control values padded to sixty-four during training. Its schema defines coordinate frames rotations gripper calibration validity masks and control domains. Follow those conventions so each value describes the intended motion. Keep timing aligned: T transitions need T plus one observations. Training transforms raw captures into the model's action representation; released inference actions are already normalized.

How streaming generation works under the hood

Cosmos 3's Mixture-of-Transformers backbone connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt available controls and generated history. Controls plus history feed into chunk generation which decodes images and updates history. Denoising refines noisy compressed video representations into a segment. Temporal-causal attention lets it read available history and controls while blocking future target observations. A key/value cache retains attention information for the next prediction.

The model operates on compressed video representations called latents. After the initial image each latent time step represents four RGB frames. Transfer predicts one step per chunk; the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often. Four models. Sixteen billion parameters.

The four-stage training recipe

Stage zero — bidirectional adaptation — learns the domain and its control-to-video relationships from complete recorded clips. Stage one — teacher forcing — predicts each chunk from earlier recorded observations and available controls with the clean answer and future targets masked out. Stage two — consistency distillation — uses a frozen teacher to train a student that generates each chunk in four denoising steps. Stage three — self-forcing — trains on the student's generated history. A frozen teacher and an auxiliary score model supply the distribution-matching signal.

Why the final stage matters: a slightly mispredicted object becomes context for later frames so errors can compound. Self-forcing exposes training to that generated history. Distillation and caching reduce the cost of producing each observation.

Adapting the models and running them

The post-training recipes start from Cosmos3-Nano or compatible base weights and cover all four stages. Keep full training checkpoints to resume; exported inference weights omit optimizer and other training state. Transfer-Auto post-trains on your driving data to adapt to your sensor rig and domain and generates offline synthetic data. Sim-Auto applies causal training and distillation to an adapted Transfer-Auto model for fast streaming simulation. Sim-Transfer post-trains the checkpoint for your environment with local data preparation stage configurations and resume. Sim-Bimanual post-trains for your embodiment with data validation stage configurations checkpoint handoffs and export.

Give your coding agent the skill and dataset location. Follow the environment checks validate a sample and test training and restart before a longer run. Evaluate quality on held-out episodes: a successful short run verifies execution not convergence.

Requirements listed in the announcement: Linux an NVIDIA GPU Docker and NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on one GB200; this is not a minimum-memory specification. The server downloads weights; the clients fetch sample inputs. How much VRAM is actually needed for smaller GPUs the announcement does not say. Whether the models can run quantized the announcement does not say.