Penn Engineering Teaches Robots Multi-Step Tasks From Five Minutes of Video

Combining symbolic planning with dynamical systems

For robots to be useful in everyday settings, they must adapt when circumstances change, recover from errors and interact safely with people, according to Penn Engineering. Nadia Figueroa, the Shalini and Rajeev Misra Presidential Assistant Professor in Mechanical Engineering and Applied Mechanics, approaches this by focusing on what the university calls the physics of intelligence: how objects move, how a robot's body can move through space and what constraints govern those movements.

Her group's project SymSkill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulation targets a basic limit in robot learning, the step from single movements to multi-step tasks. Lead author Yifei Simon Shao, a doctoral student in the Figueroa Lab, says a robot may know how to grasp, move and place an object, but a real task may need those skills in a particular sequence, and the sequence can change midway.

SymSkill uses two levels of hierarchy. The first, based on symbolic planning, helps the robot decide what to do. In the example of moving a banana from a covered pan to a plate, it tells the robot to remove the lid, pick up the banana and place it on the plate. The second level uses a dynamical systems approach to carry out each movement physically and to react when a person gets in the way.

Five minutes of video, up to 12 steps

According to the researchers, the system learns these capabilities from just five minutes of video demonstrations. From that small amount of data it learns individual skills that can be recombined into longer tasks. The team demonstrated this in simulation and on a real robot, which completed multi-step tasks involving up to 12 steps without needing additional demonstrations.

The system can also respond when reality does not match the plan. If a person interferes or the robot fails at one step, SymSkill can re-plan and recover. Figueroa says: "We can make robots more capable without simply throwing more data at the problem."

The work received the Best Paper Award at the 2026 IEEE International Conference on Robotics and Automation (ICRA) in Planning and Control, and the Best Paper Award at the 2025 Conference on Robot Learning (CoRL) LEAP Workshop.

Gaussian Graphs: a roadmap built from demonstrations

A second project, Zero-Shot Generalization from Motion Demonstrations to New Tasks, addresses data efficiency from another direction. Figueroa worked with Kilian Freitag and Alvin Combrink, both doctoral students at Chalmers University of Technology in Sweden. They asked whether existing demonstrations in a workspace, such as a kitchen, could be combined into a "roadmap" that lets a robot find a path between points it was never shown.

The researchers introduce a structure called a Gaussian Graph. Learned movements are broken into smaller regions of motion, which become vertices on a graph. The robot can connect these points to reach locations it has not been explicitly taught to reach. As a result, it does not need a new demonstration for every possible route.

"Think of it like creating roads," Figueroa explains. "Once we set up the map and provide the rules of the road, the robot can be given a destination and use what it knows to get there without being shown the exact route."

The robot can use the roadmap in two ways: combining learned movements into a new, direct path, or linking several movements to accomplish a more complex task. In both cases it can adjust if something changes or gets in its way while still reaching its goal.

What comes next

Penn Engineering says both approaches draw on physics, mathematics and computer science rather than simply adding more data. Figueroa's lab is now exploring how to combine them with language models so robots can better understand how humans describe tasks. The team also wants to see how the methods extend to dual-arm and humanoid robots, and to move beyond stationary arms toward robots that navigate spaces while manipulating objects.

The stated goal is to develop underlying systems that let robots continually reuse, adapt and combine what they have learned. If the approach scales, it points toward robots that need far less demonstration data to handle the variable conditions of everyday environments, though the source describes results only in simulation and on a real robot arm in research settings.