Teaching Robots to Do More with Less Data

For robots to become useful for everyday tasks that require physical interaction with people, they will need to do more than follow instructions in carefully controlled environments. They will need to adapt when circumstances change, recover when something goes wrong and interact safely with the people and objects around them.

And they will need to learn efficiently.

That challenge is at the center of research by Nadia Figueroa, Shalini and Rajeev Misra Presidential Assistant Professor in Mechanical Engineering and Applied Mechanics at Penn Engineering.

Rather than relying solely on increasingly large data sets to train robots, Figueroa’s approach leans heavily on the physics of intelligence: how objects move, how a robot’s body can move through space and what constraints govern those movements. 

“We’re asking how we can make robots more useful in the real world while making the systems that teach them scalable, efficient, safe and effective for human interaction,” says Figueroa. “Current research in this realm does not typically include humans, as it is very hard to design for those kinds of interactions.”

From Individual Movements to Complex Tasks

In a recent research project, SymSkill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulation, lead author Yifei Simon Shao, a doctoral student in the Figueroa Lab, tackles a fundamental limitation in robot learning: moving from performing individual movements to accomplishing a multi-step task.

“A robot might already know how to grasp an object, move it and place it somewhere else,” says Simon Shao. “But a real-world task could require those skills to happen in a particular sequence, and they could change midway through the task. We are aiming to build a system that allows a robot to learn how to perform those tasks without having to observe each individual motion and every possible combination of those motions.”

SymSkill combines two levels of hierarchy when teaching robots: one that helps them understand what to do in a sequence of tasks, and another that helps them figure out how to actually move the manipulator. 

Symbolic planning has been used in AI for decades to represent objects, actions and the relationships between them. In a task like moving a banana from a covered pan to a plate, this first level of hierarchy can tell the robot that it first needs to remove the lid, then pick up the banana and finally place it on the plate and adjust when something unexpected happens, such as a person moving the lid. The next level of hierarchy uses a dynamical systems approach which then helps the robot physically carry out each movement and react when a person gets in its way.

Crucially, the system learns these capabilities from just five minutes of video demonstrations. From that small amount of data, it learns individual skills that can be recombined into longer, more complex tasks. The researchers demonstrated this in both simulation and on a real robot, where it was able to complete multi-step tasks involving up to 12 steps without needing additional demonstrations.

A robotic arm using Symskill is tasked with putting a banana on a plate (left), which requires a series of steps to accomplish. Symskill is able to adapt to interruptions by humans, as show in the image on the right, and recover from failed attempts until a task is accomplished.

And the system allowed the robot to respond when reality did not match the plan.

If a person interferes with the robot or the robot fails at one step, SymSkill can re-plan and recover. 

“What makes this exciting is that the robot doesn’t have to experience every possible version of a task before it can perform it,” says Shao. “Once it has learned all basic skills and understands how they relate to the world around it, it can start putting those skills together in ways it hasn’t seen before. That’s an important step toward robots that can actually operate in the unpredictable and highly variable environments of everyday life.”

“We can make robots more capable without simply throwing more data at the problem,” says Figueroa. “We’ve been developing this approach to dynamical systems for years, and now we are showing that there is real potential in combining our understanding of physics with newer AI approaches to build robots that can learn, adapt and operate more effectively in the real world.”

The work received major recognition, including the Best Paper Award at the 2026 IEEE International Conference on Robotics and Automation (ICRA) in Planning and Control and overall Best Conference Paper as well as the Best Paper Award at the 2025 Conference on Robot Learning (CoRL) LEAP Workshop.

Vijay Kumar, Professor and Nemirovsky Family Dean in Mechanical Engineering and Applied Mechanics (left), sits with Nadia Figueroa and the study’s lead author, Yifei Simon Shao at the ICRA award banquet.

Reusing What a Robot Already Knows

Another project approaches the same data-efficiency problem from a different direction.

In Zero-Shot Generalization from Motion Demonstrations to New Tasks, Figueroa and collaborators Kilian Freitag and Alvin Combrink, both doctoral students at Chalmers University of Technology in Sweden, investigate how learned motions in a given workspace can build infrastructure for a robot to learn new motions in that same workspace to accomplish a task it has never seen before. 

Imagine a robot learning several different movements in a kitchen workspace: moving an object from one location to another, or moving between different points on a counter. Traditionally, if researchers wanted the robot to perform a new movement, they would need to demonstrate that movement as well. Figueroa and her collaborators asked whether those existing demonstrations could instead be combined to create a “roadmap” of the robot’s workspace, allowing it to find a path between points it had never been shown before. In other words, could a robot use a handful of demonstrations to figure out how to get somewhere it had never been taught to go?

To achieve their aims, the researchers introduce a structure called a Gaussian Graph. They begin with a handful of demonstrations for different tasks performed in a shared workspace. The researchers turn the robot’s learned movements into a kind of map of possible paths. Each movement is broken down into smaller regions of motion, which become points — or “vertices” — on a graph. The robot can then connect those points to find a path between locations it has never been explicitly taught to reach.

This work showed that the robot does not need a new demonstration for every possible route through the workspace. Instead, the demonstrations establish a map of how movement works within that space.

“Think of it like creating roads,” Figueroa explains. “Once we set up the map and provide the rules of the road, the robot can be given a destination and use what it knows to get there without being shown the exact route.” 

The researchers demonstrate two ways the robot can use this roadmap. It can either combine learned movements into a new, direct path or link several movements together to accomplish a more complex task. In both cases, the robot can adjust its movements if something changes or gets in its way without losing its ability to reach the intended goal.

Building Robots That Can Learn, Adapt and Collaborate

SymSkill shows how dynamical systems can be combined with symbolic AI to move from individual skills to complex, long-horizon tasks. The Gaussian Graph work shows how those learned motion policies can be reused and recombined to create behaviors the robot was never explicitly taught.

Both approaches rely on a combination of methodologies from physics, mathematics and computer science rather than simply adding more data.

That distinction will help advance AI systems that can operate not just in a digital environment, but in the messy, unpredictable physical world.

Figueroa’s lab is now exploring how these approaches can be combined with language models so robots can better understand how humans describe tasks, as well as how the methods might extend to dual-arm and humanoid robots. The researchers also want to move beyond stationary robotic arms toward robots that can navigate through spaces while manipulating objects.

The goal is not simply to teach robots more skills. It is to develop the underlying systems that allow robots to continually reuse, adapt and combine what they have learned.

Learn more about how research in the Figueroa Robotics Lab is building robots that can move beyond controlled demonstrations and become genuinely useful partners in the real world. And check out the accolades for these specific research papers: the Best Paper Award at the 2026 IEEE International Conference on Robotics and Automation (ICRA) in Planning and Control and the overall Best Conference Paper, the Best Paper Award at the 2025 Conference on Robot Learning (CoRL) LEAP Workshop and a Best Paper Award finalist at the 2026 IEEE International Conference on Automation Science and Engineering (CASE).