What is ToyBlocks and why does it give robotics a shared testbed?

A simulator that writes its own curriculum

ToyBlocks is the first FoundationWorld. Each FoundationWorld pairs a controlled simulated environment with tools that automatically generate tasks, training demonstrations, and metrics for measuring robot behavior over time. The scope is deliberately narrow: a single robot arm and blocks on a table. That narrow scope makes it possible to satisfy four requirements at once — realistic physics and appearance, automatic generation at scale, challenges that combine reasoning with dexterity, and broad coverage of the tasks possible within the domain.

Skills are short-horizon behaviors such as pushing a block across the table or flipping it onto another face without picking it up. They are learned through reinforcement learning and motion planning inside Isaac Lab, Nvidia's open-source framework for robot learning in simulation. Policies learn by trial and error. Skill generation can draw on privileged information — the exact position and orientation of a block that the simulator knows but a real robot cannot see — instead of estimating it from camera images. This enables automatic generation of skills that would otherwise require a person to remotely control the robot.

A task-and-motion planner sequences the skills needed to solve a task and plans the robot's movements between them. Running each plan in simulation produces a dataset of recorded demonstrations with labels describing which task or skill the robot performs. The framework scales data generation by executing hundreds of such plans in parallel on a single graphics processing unit, reaching up to 24 times real-time performance. Domain randomization varies block size, table height, and other physical properties from scene to scene. Pictures and letters on the block faces allow searching, sorting, and arranging tasks. A robot may need to turn a block to see a hidden face before deciding where it belongs.

What the controlled setting reveals about model behavior

Low-level action models map language instructions and visual observations to robot actions. The premise is that large-scale pretraining lets them generalize rapidly. Evidence has been conflicting. Some studies report brittleness to changes in object layout and lighting; others show good generalization. ToyBlocks lets researchers test spatial generalization in a simple, reproducible setting with meaningful task-space diversity.

Four tasks are used: Stack, Unstack, Flip, and Rotate. Studies distinguish interpolation — new configurations within the training range — from extrapolation — configurations beyond that range. For example, models train on scenarios where objects spawn only in the left region and test on the right region, or train on stack heights of 2 and 5 blocks and test on heights 3 and 4. Despite large-scale pretraining, popular action models including ManiFlow, π0.5, and MolmoAct 2 generalize poorly in initial results. LoRA — low-rank adaptation, a method for fine-tuning with a small set of trainable weight updates — improves generalization. The testbed makes it possible to analyze the contribution of each technique systematically.

High-level reasoning models decide what to do; low-level models control the robot's movements. Long-horizon tasks require many steps. An example is spelling "Ai2" with a block stack. Because letters may not be immediately visible, the agent must first choose actions that reveal information. Then it plans and executes a sequence to build the stack. Molmo 2, a vision-language model, serves as the high-level planner. ManiFlow acts as the low-level policy. The low-level model pauses after completing a skill and queries the high-level model for the next instruction. Three difficulty scenarios are evaluated: easiest with characters upright and facing the camera, medium with characters that may not be upright, hardest with characters not visible at the start.

Agentic systems test how frontier models plan and act through tool calls. Claude Opus 5.5 is exposed to ToyBlocks skills through the Model Context Protocol, a standard for connecting AI applications to tools. The agent sees the scene through camera images, calls skills such as pick, flip, or rotate by pointing at a pixel, and decides when to search, build, or declare the task complete or impossible. In the search-and-spell setting the agent had to generate stacks spelling three-letter words among up to eight blocks. Fifty percent of scenes did not have blocks with all required letters, rendering the task infeasible. The agent successfully solved or correctly declared infeasible 63 percent of tasks. Errors were mostly caused by confusing letters with look-alike letters (13 percent) or choosing an incorrect orientation for a letter in a stack (15 percent). With patterned blocks the agent finishes having seen fewer faces because it can infer what hidden faces must depict rather than checking every one.

An agent can also learn how its skills behave. When a skill behaves unexpectedly, the agent forms a hypothesis, designs a small test, and revises its understanding from the result. Learned skill descriptions raise success rates from 24 percent with default descriptions to 63 percent, outperforming the 36 percent achieved with descriptions written by a human expert.

Action-conditioned world models predict how a scene changes in response to a robot's actions. ToyBlocks provides ground truth — the actual simulated scene state — enabling highly controlled tests. Four skills were tested: pick-and-place, flip, rotate, and a mixed task. Nine hundred trajectories were generated for each skill. Fifty percent were successful demonstrations; fifty percent were unsuccessful because of small changes to spatial configurations. With equal numbers of successful and unsuccessful executions, a model that always predicts success would be correct only half the time. Three models were tested: Wan 2.2, a video prediction model with 5 billion parameters; Astra, an agentic model that created its own physics simulator tuned to match the demonstration data; and LeWM, a latent world model. Wan 2.2 predicts block motion and task outcomes most accurately. Astra tracks the gripper well but struggles to simulate contact. LeWM is fast and more accurate over short interaction segments, though it drifts over full trajectories. Adding 100 model-generated trajectories from the best video-generation model raises a policy's success in an unseen region from 19 percent to 78 percent.

From simulation to a physical Franka arm

Sim-to-real evaluations test policies trained in simulation on physical robots performing block manipulation tasks. A Franka FR3 arm with a Robotiq gripper is used across thirty scene configurations. Each scene contains four to ten blocks seen during training. An episode is successful only if the relevant blocks are moved to target poses without collisions. If the operator intervenes to stop the robot due to an unsafe motion, the episode is marked as a failure.

ManiFlow's atomic-skill success rates were similar in simulation and the real world: 80.0 percent and 81.1 percent. The hierarchical Molmo 2 + ManiFlow system succeeded in 55.5 percent of simulated episodes (111 of 200) and 50.0 percent of real-world episodes (15 of 30); further trials are needed to establish statistical confidence across tasks. The Molmo 2–ManiFlow evaluations use the easiest scene configuration defined in the simulation evaluation, with block characters visible and upright.

Transfer happens in three stages. Before data generation, system identification tunes simulated robot parameters — including joint friction and armature (the motor and gearbox inertia reflected at the joint) — to match real-robot trajectories. During data generation, physical properties (object size, friction, table height) and visual properties (lighting, backgrounds, materials, camera parameters) are randomized. At inference time, observations are captured using a front-facing ZED camera and a wrist-mounted ZED Mini. For policies that use depth images, clean depth maps are generated from paired RGB camera images using FastFoundationStereo rather than relying on noisy depth sensor measurements.

What ships today and what comes next

The release includes simulation assets, procedural task and demonstration generators, automatically generated demonstrations, two trained models showcasing what the framework can do — one based on Molmo 2 and one based on ManiFlow — training code, and evaluation code with validation sets. The initial