Robots learn their own moves: AI framework maps every motion a machine can make

Robots are getting better at doing remarkable things, but teaching them what to do remains slow, expensive and heavily dependent on specialists. A research team from the German Research Center for Artificial Intelligence and the University of Bremen now reports a different way of working: a development cycle called Q-Rock, in which a robot’s possible behaviors are discovered largely by the machine itself, then labeled, stored and matched to tasks by software. The work, published as an open access research article in the journal AI Perspectives on 5 July 2021, describes both the formal framework and a working demonstration on simulated robotic hardware.

The core idea is deceptively simple. The authors argue that the set of all possible behaviors of a robotic system is inherently defined by its hardware and software, and that this set can be found by letting the system explore itself. Rather than programming a robot to perform a specific task and hoping the skills generalize, Q-Rock first asks what the machine can physically do, before anyone decides what it should do. The exploration is deliberately goal-agnostic, meaning the robot is not steered toward any particular objective during this phase. The point is to generate a maximal variety of capabilities, so that the resulting library may even contain novel, unanticipated movements that no engineer thought to program.

The full behavior space of a robot is, however, astronomically large. Because robots are digital systems with finite sensor resolutions, the number of possible states and transitions grows so quickly with the number of joints that enumerating every movement is not tractable for machines with many degrees of freedom. The team therefore restricts automated exploration to kinematic capabilities, the trajectories a robot can produce, and works with a representative set rather than a complete one. Movements are limited to a fixed length, and the robot is explored in a minimal, deterministic and static environment, an assumption meant to make the discovered capabilities transferable to more complex settings later. Even with these simplifications, the authors acknowledge that the curse of dimensionality cannot be fully negated.

To make exploration practical, the researchers use a parameter-based encoding. A capability function model maps parameters from a defined space to capability functions, which are policies or control laws that select actions based on the robot’s current state and time. In the demonstration, these parameters defined polynomial trajectories for each joint. Repeatedly executing such a function in simulation produces a capability, defined as a sequence of world states over a finite time horizon. Because the mapping from parameters to trajectories is locally smooth, small changes in parameters lead to small changes in motion, a property the team highlights as important for the clustering stage that follows.

Once a library of capabilities exists, the second stage of the cycle begins: classification and annotation. The explored trajectories are transformed into feature spaces, each capturing a specific aspect of a movement, such as its starting state, its end effector end state, or its directness, a measure of how closely a path follows the straight route between start and finish. Standard k-means clustering then groups similar trajectories, and probabilistic generative models, implemented as neural ordinary differential equation based normalizing flows, are trained on the parameter sets of each cluster. These models can later be sampled to generate new parameter sets that likely produce motions belonging to the intended cluster, allowing behaviors to be recreated without exhaustively searching the original library.

Human input remains essential at this stage. The robot, after exploration, has no notion of causality or purpose, the authors stress; only through human semantic descriptions do the raw capabilities become meaningful behaviors. Feature spaces and behavior models are hand-crafted in the current implementation, and a user reviews and confirms the labels assigned to each cognitive core, the framework’s name for an executable grounding of a behavior model on a specific robot. A behavior model specifies constraints in feature spaces, for example requiring that a reach movement have a directness value between 0.8 and 1.0 while leaving start and end states variable. When clusters satisfying those constraints exist for a given robot, a cognitive core can be instantiated and inherits the behavior’s label.

The third stage is reasoning, which links the sub-symbolic world of trajectories and neural networks to symbolic task descriptions. A user can describe a problem using a planning language, and the system decomposes it into primitive tasks, each carrying a semantic annotation. These annotations are matched against the annotations of known cognitive cores in the central graph database, which stores components, assemblies, capabilities, clusters and behaviors. If a single robotic system can be found whose cognitive cores cover all required tasks, the system is suggested as a solution. The reasoning also works in reverse: given a new hardware assembly, the framework can identify which functions that composition is likely to support.

To demonstrate the cycle end to end, the team built three test systems in simulation: a three-degree-of-freedom robotic arm, a wheeled mobile base, and a combination of the two. For the arm, 106 trajectories of four seconds each were generated, with five polynomial parameters per joint, giving fifteen parameters in total, sampled uniformly from a defined range. A validation model, a fully connected neural network with four hidden layers, was trained to predict whether a given parameter set corresponds to a motion the robot can actually execute, reaching an accuracy of 96.6 plus or minus 0.4 percent on unseen data. The mobile base used twelve parameters applied to wheel velocities over twenty-second trajectories, and the combined system had twenty-seven parameters in total.

Clustering results varied across the three systems and feature spaces. For the arm, generative model accuracies were 91 plus or minus 2.1 percent for the start-state feature, 87 plus or minus 3.3 percent for directness, and 59 plus or minus 4.1 percent for the end effector end state. The mobile base yielded 89, 73 and 47 percent respectively, while the combined system proved hardest, with accuracies of 82, 55 and 32 percent. These figures, determined by sampling parameters, simulating the resulting trajectories and checking cluster assignment over ten training runs, show that the approach works but also that performance degrades as system complexity grows, a limitation the authors discuss openly.

With the cognitive cores in place, the team showed that sampling from the constrained reach core produced direct end effector motion toward requested targets for all three systems, including a combined arm-and-base configuration whose target lay beyond the arm’s roughly 0.4-meter reach alone. An unconstrained variant of the same core, lacking the directness requirement, generated more indirect paths, confirming that the constraint mechanism shapes behavior as intended. In a final demonstration, a user composing a simple mission from predefined actions could retrieve the reach core through its semantic label, and the system suggested the custom-designed assembly, named NewShoppingCart, as a suitable robot for the mission. The authors present this as a qualitative proof of concept: a fully integrated workflow in which annotated hardware models generate capabilities, capabilities are clustered and labeled, and semantic annotations close the loop between bottom-up system analysis and top-down task-driven robot design. Scaling the exploration to systems of systems, automating feature learning and behavior model definition, and grounding generic plans on real hardware remain open challenges, and the datasets from the use case are not publicly available. Still, the demonstration marks a concrete step toward robots that can, in a meaningful sense, discover what they are capable of before anyone tells them what to do.

Subject of Research: An automated development cycle for exploring, classifying and reasoning about robot behaviors

Article Title: A development cycle for automated self-exploration of robot behaviors

Article References: Roehr, T. M., Harnack, D., Wöhrle, H., Wiebe, F., Schilling, M., Lima, O., Langosz, M., Kumar, S., Straube, S., & Kirchner, F. (2021). A development cycle for automated self-exploration of robot behaviors. AI Perspectives, 3(1), Article 1. https://doi.org/10.1186/s42467-021-00008-9

Image Credits: AI Generated

DOI: 10.1186/s42467-021-00008-9

Keywords: robotics, Q-Rock, self-exploration, machine learning, kinematic capabilities, cognitive cores, semantic annotation, behavior models, clustering, graph database, robot design, AI framework