How Robots Can Learn Complex Tasks from Just 10 Demos: Introducing Oriented Affordance Frames

If you have ever tried to teach a robotic arm to perform a task using machine learning, you know the struggle: data hunger. To teach a robot to simply pick up a mug and place it on a coaster often requires hundreds, if not thousands, of human demonstrations. If you move the mug slightly to the left, or swap it for a taller mug, the robot frequently fails.

This lack of “sample efficiency” (needing too much data) and “generalisation” (failing when things change slightly) is a massive bottleneck in robotics. It is the primary reason we don’t yet have Rosie the Robot tidying our kitchens.

In a fascinating new paper, Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames, researchers from QUT Centre for Robotics and the University of Adelaide propose a clever solution. They demonstrate a method that allows a robot to learn long-horizon, multi-object tasks—like making a cup of tea—from as few as 10 demonstrations.

How? By rethinking the coordinate systems, or “frames,” through which the robot views the world. In this post, we will break down their approach, explain the concept of Oriented Affordance Frames, and see how this leads to robust, generalized robotic behavior.

The Problem: The “Frame” of Reference

To understand why this paper is significant, we first need to understand how robots typically “see” a task during Imitation Learning (Behavior Cloning).

When you guide a robot through a task (a demonstration), the robot records the state of the world and the actions taken. But how does it represent that state?

1. Global Frame: The robot memorizes the exact position of the object in the room (e.g., “Move to coordinate X=10, Y=20”). If you move the table or the object, the robot moves to the empty space where the object used to be.
2. End-Effector Frame: The robot learns movements relative to its own hand. This helps, but it still requires the robot to see the object from every possible angle during training to understand how to interact with it.
3. Standard Affordance Frame: The robot learns movements relative to a specific part of the object (like a handle). This is better, but it still struggles with the approach trajectory if the robot starts in a different location relative to the object.

The researchers illustrate this problem perfectly in the figure below:

As shown in Figure 2, a Global Frame (left) requires demonstrations covering every inch of the workspace. End-Effector/Affordance Frames (middle) reduce this burden but still require extensive data to capture the relationship between the robot and the object.

The researchers propose a third option: the Oriented Affordance Frame (right). This frame aligns the coordinate system not just with the object, but also orients it toward the robot. This subtle shift makes the mathematical representation of the task nearly identical, regardless of where the robot or the object is actually located in the room.

The Solution: Oriented Affordance Frames

The core contribution of this paper is a structured representation for state and action spaces that drastically reduces the data needed for training. Let’s break down the three main pillars of their method.

1. Task Decomposition and Affordances

Long tasks, like making tea, are hard because errors compound over time. The authors tackle this by breaking the long-horizon task into smaller sub-policies.

Instead of one giant neural network trying to “Make Tea,” they train small, independent policies for specific interactions, such as “Grasp Cup,” “Place Cup on Saucer,” or “Pour Tea.”

Crucially, each sub-task is defined by an Affordance. An affordance is a property of an object that defines how it can be used. For a cup, the handle is an affordance for grasping; the opening is an affordance for pouring.

Figure 5 shows this hierarchy. The complex task is partitioned into interactions defined by specific object parts (affordance frames) and the robot’s tool (tool frames).

2. The Oriented Affordance Frame (OAF)

This is the “secret sauce” of the paper.

In standard robotics, a coordinate frame on an object (like a cup handle) is fixed. If the cup rotates, the frame rotates. However, the researchers realized that for the policy (the learned behavior), the relationship between the robot and the object is what matters most.

They define the Oriented Affordance Frame as follows:

1. Origin: Centered on the target affordance (e.g., the cup handle).
2. Orientation: Rotated so that one axis (specifically the x-axis, or “funnel axis”) points directly at the robot’s tool frame at the start of the task.

In Figure 3, notice the brown arrow. In the Oriented Affordance Frame (middle), the frame is rotated so that the axis points toward the robot’s gripper (the tool frame).

Why is this brilliant? Imagine you are training a robot to pick up a cup.

• Scenario A: The cup is to the left of the robot.
• Scenario B: The cup is to the right of the robot.

In a global frame, these look like two completely different movements (move left vs. move right). But in the Oriented Affordance Frame, because the coordinate system rotates to face the robot, both scenarios look mathematically identical: “Move forward along the x-axis and close gripper.”

This creates a “funnel” effect for data. All demonstrations, regardless of where they happen in the room, get aligned into a consistent trajectory in the OAF.

As visualized in Figure 4, this alignment collapses the variability of the task. The policy doesn’t need to learn how to approach from the left and the right. It just learns to “approach forward” in its own relative coordinate system. This is why 10 demonstrations are enough—the robot doesn’t need to see every variation because the coordinate frame normalizes the variations away.

3. Perception via Foundation Models

For this method to work in the real world without cheating (using QR codes or markers), the robot needs to automatically find these affordance frames.

The authors propose a perception pipeline leveraging modern Vision Foundation Models:

• Grounding DINO: Detects the object (e.g., finds the “teapot”).
• SAM (Segment Anything Model): Cuts the object out of the image.
• FoundationPose: Tracks the object’s 6D pose (position and rotation) in real-time.
• DINO-ViT: Matches specific features (like the handle) to define the affordance point.

This pipeline (shown in Figure 9) allows the system to be “zero-shot” capable regarding perception. You don’t need to train a custom detector for every new mug; the foundation models handle the heavy lifting of figuring out where the object and its handle are.

4. Self-Progress Prediction

How does the robot know when to stop “Grasping Cup” and start “Placing Cup”?

Usually, researchers code complex “if-then” rules or train a separate high-level manager policy. Here, the authors used a simpler, elegant approach. During training, they calculate a progress scalar (from 0 to 1) based on how far along the demonstration is.

The policy learns to predict this value. When the robot is running, it simply checks its own predicted progress. If the “Grasp” policy says “I am 100% done,” the system automatically switches to the next sub-policy in the chain.

Figure 15 shows these predictions in action. You can see the confidence (progress) rising steadily as the robot performs each sub-task.

Experimental Results

The researchers tested their method on three complex real-world tasks: Tea Serving, Shoe Racking, and Coffee Making. The primary focus was the Tea Serving task, which requires precise manipulation of liquids and fragile objects.

Sample Efficiency and Success Rates

The results were striking. With only 10 demonstrations, the Oriented Affordance Frame (OAF) approach achieved a 90.9% average success rate across sub-tasks.

In comparison:

• End-Effector Frame: ~48% success.
• Global Frame: ~59% success (failed completely on out-of-distribution tasks).

Look at chart (b) in Figure 6 above. The Blue line (Oriented Affordance Frame) reaches high success with very few demos. The Red line (Non-Oriented) struggles significantly. Chart (d) is even more telling: to achieve the performance OAF gets with 10 demos, a standard image-based policy would need nearly 300 demos.

Generalisation

The true test of a robot is whether it can handle things it hasn’t seen before.

Spatial Generalisation: The researchers trained the robot with objects in one specific arrangement. They then scrambled the objects to new positions (Out-of-Distribution). Because the OAF “funnels” the data relative to the robot, the policy still worked, achieving 83.1% success on average. The Global Frame baseline dropped to 0% success in these scenarios.

Intra-Category Generalisation: Could the robot use a teapot it had never seen before? The authors trained on one tea set and tested on unseen mugs and pots with different shapes and colors.

As shown in Figure 8 (and the object set in Figure 7 below), the robot successfully manipulated novel objects. This is because the perception pipeline (FoundationPose + DINO) correctly identified the new “handle” locations, and the policy simply executed the learned “grasp handle” motion relative to that new frame.

Compositional Generalisation

Finally, the ability to chain these skills is vital. The method allows “compositional generalisation.” This means you can train a “Pour” policy in isolation and a “Grasp” policy in isolation. At runtime, you can chain them together to solve a long-horizon task, even if the robot ends the “Grasp” task in a slightly different position than it started the “Pour” task during training. The OAF absorbs these small inconsistencies.

Figure 1 summarizes this perfectly: learn from 10 demos on the left, and deploy in a messy, complex, novel environment on the right.

Robustness to Base Movement

An unexpected but cool finding was the robustness to mobile manipulation. Because the frame is relative to the tool and object, the robot’s base can actually be moving while the arm operates.

Figure 10 shows the robot successfully picking up a cup even while its base (the cart it is mounted on) is shifting position. A global frame policy would have failed immediately here.

Conclusion and Implications

The paper “Learning from 10 Demos” offers a compelling argument against the brute-force data approach often seen in deep learning. By injecting a specific structural bias—the Oriented Affordance Frame—the researchers turned a hard learning problem into an easy one.

Here are the key takeaways for students and practitioners:

1. Coordinate Frames Matter: How you represent your data (input space) is often more important than the architecture of your neural network. A smart coordinate transformation can do the work of thousands of training examples.
2. Relative over Absolute: For robotics, learning relative motions (Robot-to-Object) is almost always superior to learning absolute coordinates, especially for generalization.
3. Foundation Models Enable Abstraction: We can now rely on powerful vision models to give us high-level semantic information (like “where is the handle?”), allowing control policies to focus purely on movement rather than image processing.
4. Sample Efficiency is Key: For robots to be useful in homes, they must learn quickly. Techniques that work with 10 demos are viable for real-world products; techniques that need 1,000 are not.

This work paves the way for robots that can walk into a new kitchen, look at a never-before-seen kettle, identifying its handle, and pour a cup of tea—all based on a handful of lessons learned in a lab miles away.