Berkeley researchers outline an agentic approach to robot programming

A different route to robot control

Ken Goldberg, a UC Berkeley roboticist and professor of engineering, developed the approach with Spencer Huang, director of robotics engineering at NVIDIA, and a team of researchers. Goldberg is presenting the method and results from months of testing this week at the Conference on Robot Learning.

Traditional model-based robotics relies on extensive engineering to program how robots perceive objects, plan movements and control their actions. Model-free robotics trains robots to learn behaviors from demonstrations, but requires enormous amounts of data and time. The researchers say agentic robotics provides a link between these methods.

"Instead of collecting massive amounts of data and hand-tuning every behavior, teams of agents can quickly generate, test and refine robot programs," Goldberg said. Human engineers can then interpret the programs for fine-tuning and verification.

Agents build and test programs in simulation

Goldberg's team organizes robot tasks into individual skills using a framework called Graph-as-Policy. Examples include finding an object, planning a grasp and moving a gripper without collisions. AI agents connect these skills into a larger robot program, then use NVIDIA's Isaac simulation libraries to rehearse it across different task variations.

The agents can identify failures and revise the program before it is deployed. The researchers are also releasing open-source code built on NVIDIA's Isaac open simulation framework, together with an industry-standard benchmark for evaluating robots across industrial tasks.

In one experiment, postdoctoral students Eric Kaiyuan Chen and Shuangyu Xie gave OpenAI's Astra model access to their simulation interface and a short video of a robot wiping ink from a metal bar with a sponge. In less than an hour, the tool created and fine-tuned a physics simulation that reproduced the video, including deformations of the sponge. The resulting robot-control program was tested in simulation and then worked on a real robot.

One trial exposed a failure when the robot pushed over the metal bar because it did not reliably infer the bar's mass. The system used video of the failed attempt to improve the simulation, revise the control program and test another approach.

Early industrial use points to practical benefits

Berkeley-based Ambi Robotics applied the methods earlier this year to analyze parcel-sorting data and test updates to robot behavior. The company reported an increased capacity of over 15,000 packages sorted annually per robot, with the change deployed across 30% of its U.S. fleet.

Jeff Mahler, Ambi Robotics' co-founder and chief technology officer, said the approach converted production data into robot behavior updates while keeping engineers in control.

Goldberg said the implications are already promising for industrial environments, where tasks are repetitive and performed at scale. He does not expect home assistants to fold laundry yet, but said the ability to self-improve parts of a task marks a major advance. The researchers' stated goal is to make robot programming faster and more adaptable while retaining the structure needed for reliable industrial systems.