AI Agents Generate and Refine Robot Control Programs

Ken Goldberg, a roboticist and professor of engineering at the University of California, Berkeley, stated that recent advancements in artificial intelligence are enabling teams of AI agents to develop robot-control programs, test them in simulated environments, and revise the code when operational failures occur. Goldberg, in collaboration with Spencer Huang, director of robotics engineering at NVIDIA, and a broader team of researchers, has designated this emerging methodology as “agentic robotics.” The researchers assert that this approach provides a potential solution to two significant obstacles in the robotics field: the enormous quantities of data required to train robots and the intensive engineering efforts necessary to program them, both of which traditionally demand vast amounts of physical and technical resources to accomplish effectively.

Goldberg described agentic robotics as a fundamental shift in how robots are controlled, noting that instead of collecting massive amounts of data and hand-tuning every behavior, teams of agents can quickly generate, test, and refine robot programs. This process allows human engineers to interpret how the programs work for fine-tuning and verification. Goldberg, who was recently inducted into the National Academy of Engineering, presented the new approach and results from months of testing at the Conference on Robot Learning. His team is also releasing open-source code for other roboticists to experiment with, built on NVIDIA’s Isaac open simulation framework for developers, as well as an industry-standard benchmark for evaluating the performance of robots across industrial tasks.

The goal of this collaboration between Goldberg and Huang is to make robot programming faster and more adaptable while preserving the structure needed for reliable industrial systems. Huang noted that while robots are advancing quickly, developing them for real-world environments remains slow, data-intensive, and difficult to scale. Working with UC Berkeley, NVIDIA is helping advance a new approach that combines agentic AI, simulation, and open industrial task-focused benchmarks to help developers rapidly test and improve robot behavior before deployment.

Traditionally, engineers have two primary methods for programming robots. In model-based robotics, engineers directly program the software that tells robots how to perceive objects, plan movements, and control their actions. Alternatively, in model-free robotics, engineers train robots to learn behaviors from demonstrations through imitation learning.

In an attempt to reset public expectations last year, Goldberg stated that a “100,000-year data gap” would prevent robots from gaining real-world skills as quickly as AI agents have gained language fluency over the past several years. Those AI agents were trained on an immense amount of internet text and image data for which there is no parallel in robotics. Goldberg expressed that he was discouraged by the daunting amount of data that would need to be collected, but agentic robotics provides a missing link between model-based and model-free methods.

Earlier this year, Goldberg and his team began experimenting with a change to the way robot code is organized. Using a framework they called Graph-as-Policy, robot tasks are segmented into individual skills, such as finding an object, planning a grasp, and moving the gripper without collisions. Goldberg likened these skills to individual building blocks that fit together into something much larger. AI agents then generate and connect these skills into a complete robot program.

OpenAI’s recent release of its Astra model made the process even more promising, according to Goldberg. His postdoctoral students, Eric Kaiyuan Chen and Shuangyu Xie, gave Astra access to their simulation interface and a short video of a real robot whose assignment was using a sponge to wipe ink off a metal bar. In less than an hour, the AI tool created and fine-tuned a physics simulation that accurately reproduced the video, including very realistic deformations of the sponge.

The researchers then gave the AI the simulation and their Graph-as-Policy framework. The AI generated a robot-control program and repeatedly tested and improved it in simulation. When the researchers ran the resulting code on a real robot, it worked. In one trial, Goldberg noted that the robot pushed over the metal bar because it did not reliably infer its mass. This misstep created new evidence for the agent to review and illustrated the edge provided by agentic robotics: the AI could use video of the failed attempt to improve its simulation, revise the robot’s control program, and test a new approach.

Berkeley-based Ambi Robotics, which specializes in logistics and warehouse order fulfillment, applied the new methods earlier this year to analyze parcel sorting data and test updates to robot behavior. Tests showed an increased capacity of over 15,000 packages sorted annually per robot; the change is now deployed across 30% of the company’s U.S. fleet. Jeff Mahler, Ambi Robotics’ co-founder and chief technology officer, stated that AI-powered industrial robots must continuously improve reliability and performance.

Goldberg said the implications are already promising in industrial settings, where tasks are repetitive and at-scale. He cautioned not to expect home assistants to fold laundry yet, but emphasized that the ability to self-improve key elements of tasks is a major leap forward in the field. It is also an exciting time for him and his students. He said he can think of only two moments in his 40-year robotics career that have elicited similar feelings: the public release of the World Wide Web in 1993 and the debut of ChatGPT in 2022.

Goldberg explained that agentic robotics combines classical control theory and physics with rapidly evolving agentic coding. He described the approach as fast and practical for real engineering applications, stating that his sense is that it is a true paradigm shift.

The release of the Graph-as-Policy research paper and the RoboSimStudio video demonstrations website provides additional resources for the community to explore these methods.

Subject of Research: Recent advances in AI are creating a 'paradigm shift' for robotics

Article Title: Recent advances in AI are creating a 'paradigm shift' for robotics

Article References: Recent advances in AI are creating a 'paradigm shift' for robotics. Original publication (Bibliographic details unavailable.)

Image Credits: AI Generated

DOI: Not provided

Keywords: Recent, advances, creating, paradigm, shift', robotics, scientific research