Ai2 introduces ToyBlocks, a controlled world for robot learning
Ai2 has introduced ToyBlocks, an open simulation framework that lets researchers train and evaluate robot arms in a controlled tabletop world. The system is designed to help researchers vary training data, repeat experiments and investigate why a robot succeeds or fails.
- ToyBlocks is Ai2’s first FoundationWorld.
- The open simulation features a robot arm and blocks on a tabletop.
- It automatically generates skills, tasks and labeled demonstrations.
- Researchers can vary training data and test models on unseen tasks.
- Evaluations include speed, force and task-progress metrics.
A controlled environment for robot research
ToyBlocks is the first example of Ai2’s FoundationWorlds, a framework for creating simulated environments focused on specific robotics questions. Each FoundationWorld combines a controlled world with tools for automatically generating tasks and training demonstrations, along with metrics for measuring robot behavior over time.
Ai2 says the deliberately narrow setting makes it possible to combine realistic physics and appearance, automatic task generation, challenges involving both reasoning and dexterity, and broad coverage of tasks within one domain. The approach is intended to give researchers more control than open-ended robotics environments, where the range of objects and interactions can make experiments difficult to reproduce and interpret.
ToyBlocks centers on a robot arm and blocks on a tabletop. The blocks can carry pictures and letters on their faces, allowing the system to support searching, sorting and arranging tasks as well as picking and stacking. A robot may need to rotate a block to reveal a hidden face before deciding where it belongs.
Automatic demonstrations and controllable training data
ToyBlocks automatically generates demonstrations, which are recordings of a robot completing tasks in simulation. This avoids the time-consuming teleoperation required when a person remotely controls a robot. Researchers can change the balance of experiences in the training data, such as generating more demonstrations of pushing or flipping, and then test performance on tasks that were excluded from training.
The framework uses reinforcement learning and motion planning to generate short-horizon skills, including pushing a block across the table or flipping it onto another face without picking it up. A task-and-motion planner selects and sequences the skills needed for a task, plans the movements between them and produces labeled demonstration datasets by running those plans in simulation.
Ai2 says ToyBlocks can generate and execute hundreds of plans in parallel on a single graphics processing unit, reaching up to 24 times real-time performance. Skill training takes place in Isaac Lab, Nvidia's open-source framework for robot learning in simulation.
The system separates high-level decisions from low-level movement control. In an example described by Ai2, Molmo 2 selects skills such as rotation and flipping, while ManiFlow controls the simulated arm. ToyBlocks also models perception by training systems to map raw sensor observations to actions, requiring them to estimate a block’s position and orientation rather than receiving those values directly from the simulator.
Measuring generalization and robot behavior
ToyBlocks supports domain randomization, which changes properties such as block size and table height between scenes. This allows researchers to test how models respond when the geometry changes. They can also study how much training data a model needs, which combinations of experiences support generalization, and whether a model can perform tasks kept out of training.
Ai2’s examples include testing whether rotation and pick-and-place demonstrations improve performance on flipping, and comparing models trained on different workspace regions. The research setup covers tasks including Stack, Unstack, Flip and Rotate, with tests for interpolation within a training range and extrapolation beyond it. Other evaluations vary skill sequences, letters or colors on blocks, and the number of blocks in a scene.
Evaluation produces more than success rates. According to Ai2, the framework records execution speeds, forces exerted on objects and task progress using the same specifications as the planning module. A webpage interface also allows users to connect Claude or another frontier model and give the robot natural-language instructions.
Ai2 says it plans to extend the FoundationWorlds approach to increasingly complex environments. For researchers, the immediate significance of ToyBlocks is a reproducible setting for linking training choices with robot behavior, while keeping the tasks simple enough to inspect and repeat.
