Introduction

Imagine trying to teach a robot how to cook a meal. You could grab its arm and guide it through the motions: opening the fridge, grabbing an egg, cracking it, and frying it. This is the essence of Imitation Learning (IL). It works wonderfully for short, specific skills. But what happens when you ask that same robot to prepare a three-course dinner?

Suddenly, the robot isn’t just copying a motion; it needs to plan. It needs to understand that the “egg” must be “cracked” before it can be “fried,” and that the “pan” must be “hot” before the egg goes in. If the robot only knows how to mimic motions, it will inevitably fail over a long sequence. It might try to fry the egg shell, or put the egg in a cold pan, because it doesn’t understand the logic behind the task.

To solve complex, long-horizon tasks, humans rely on symbolic reasoning—we think in terms of objects, states, and rules. Robots, on the other hand, typically excel at continuous control—calculating motor torques and trajectories. Bridging this gap is the domain of Neuro-Symbolic AI.

In a fascinating new paper titled “Few-Shot Neuro-Symbolic Imitation Learning for Long-Horizon Planning and Acting,” researchers from Tufts University and the Austrian Institute of Technology propose a framework that bridges this divide. They have developed a system that can look at a handful of raw demonstrations and simultaneously learn two things:

  1. The High-Level Logic: It discovers the symbolic rules of the game (like PDDL planning domains) without being told what they are.
  2. The Low-Level Skills: It learns the precise motor movements to execute those rules.

The result is a robot that can solve complex puzzles, assemble nuts and bolts, and drive forklifts with as few as five demonstrations, outpacing traditional deep learning methods that require hundreds.

The Core Problem: The Gap Between Acting and Planning

Current approaches to robotic learning generally fall into two camps, neither of which is perfect for long-horizon tasks.

1. End-to-End Imitation Learning: These are neural networks that map camera pixels directly to motor actions. They are great at acquiring intuition-like skills (like grasping a weirdly shaped object). However, they require massive amounts of data. More critically, they struggle with “distribution shift.” Over a long task, small errors accumulate. If a robot is slightly off-position in step 5 of a 50-step process, it encounters a state it has never seen before and fails.

2. Task and Motion Planning (TAMP): This is the classical robotics approach. You give the robot a symbolic map of the world (e.g., Object A is on Table). You define rules using a language like PDDL (Planning Domain Definition Language). A planner calculates the sequence of steps, and a controller executes them. This is robust and interpretable. The catch? You have to hand-code everything. You must manually define what a “state” is, what “predicates” exist, and what every action does. This is brittle and labor-intensive.

The Contribution

The researchers argue that we shouldn’t have to choose. Their framework automates the creation of the symbolic layer. By watching a human perform a task a few times, the system builds its own symbolic representation of the world and links it to learned neural policies.

Figure 1: Our neuro-symbolic framework integrates graph construction, symbolic abstraction, planning, action decomposition, imitation learning and space filtering. Starting with just a few skill demonstrations (left), we construct a transition graph capturing transitions (edges), which are skill transitions between two high-level states (black-box nodes). This graph enables automatic PDDL model extraction via an ASP solver, which powers high-level planning. The oracle decomposes complex tasks into primitive action step policies executed by learned diffusion-based controllers, allowing generalization to novel long-horizon tasks while requiring minimal training data.

As shown in Figure 1, the workflow moves from raw demonstrations to a transition graph, then to abstract symbolic logic, and finally back down to specific motor controllers.

Building the Brain: From Graphs to Symbols

The first major innovation in this paper is how it extracts logic from visual data. The robot isn’t given a manual; it has to write one.

Step 1: The Transition Graph

The process begins with a human demonstrating a task. The system views each demonstration as a transition between “high-level states.”

For example, imagine a Tower of Hanoi puzzle.

  • State A: All disks are on Peg 1.
  • Action: Move small disk to Peg 2.
  • State B: Small disk is on Peg 2, others on Peg 1.

The system captures visual snapshots of the start and end of these skills. The human provides a simple label for the transition (e.g., “Move”). Crucially, the human helps identify when two different visual states are actually the same “abstract” state (e.g., matching two photos where the disks are in the same configuration).

This results in a Transition Graph, where nodes are states and edges are actions.

Figure 8: Task graphs for different environments. Stacking and Forklift Load/Unload Palets graphs consist only of two nodes and a connecting edge, as they consider one-skill tasks and do not require planning.

Figure 8 illustrates these graphs. While simple tasks like stacking (a) might look linear, complex tasks like the Kitchen environment (c, d) create intricate webs of dependencies.

Step 2: Symbolic Abstraction with ASP

Here is where the “neuro-symbolic” magic happens. The system possesses a graph, but it doesn’t yet understand the rules that generated that graph.

The researchers use an Answer Set Programming (ASP) solver. Think of ASP as a logic puzzle solver. The system asks the solver: “Find the simplest set of symbolic rules (predicates and operators) that could mathematically explain this graph structure.”

The solver outputs a PDDL domain. It might discover that:

  • There is a predicate we’ll call p1 (which might correspond to “on top of”).
  • There is an operator Move that requires p1(A, B) to be false before it can happen.

The system has now “learned” the physics and logic of the puzzle without ever being explicitly programmed with concepts like “gravity” or “stacking.”

The Oracle: Learning to Focus

Once the robot understands the plan (symbolic level), it needs to learn the movement (neural level). This is where many imitation learning methods fail: they overwhelm the neural network with too much data.

If a robot is trying to pick up a specific nut to screw onto a bolt, the position of a coffee mug on the other side of the table is irrelevant. If the neural network pays attention to the coffee mug, it might fail when the mug is moved.

The authors introduce an Oracle—a filtering mechanism derived directly from the symbolic abstraction learned in the previous step. Because the ASP solver identified which objects are involved in a specific transition, the system knows exactly which objects matter for each skill.

The paper defines a filtering function \(\gamma\):

() \\gamma ( \\tilde { s } , o _ { i } ) = \\tilde { s } ( \\mathcal { E } _ { o _ { i } } ) . ()

This function takes the full state of the world \(\tilde{s}\) and an operator \(o_i\) (like “Pick Up Nut”), and returns only the subset of objects \(\mathcal{E}\) relevant to that operator.

But they go a step further. Robots need to generalize. Picking up a block at location X should be the same skill as picking it up at location Y. To achieve this, the Oracle applies a transformation \(\alpha\) to convert absolute coordinates into coordinates relative to the robot’s end-effector:

() \\phi ( \\tilde { s } _ { t } ) = \\alpha \\circ \\gamma ( \\tilde { s } _ { t } , o _ { i } ) . ()

This combined function \(\phi\) (Phi) ensures that the policy \(\pi\) receives a “canonical” view of the task. It sees only the necessary objects, positioned relative to the hand. This drastically simplifies the learning problem.

Figure 2: Shown here is an example demonstration of the MOVE operator in the Towers of Hanoi domain, where block 1 is moved off block 2 and placed onto a platform. The agent partitions skill demonstrations into action steps, with an oracle \\(\\phi\\) filtering observations to simplify learning. The demonstration is first collected, then filtered using \\(\\gamma\\) to retain operator-relevant objects (block 1 and platform 3) and \\(\\alpha\\) to express coordinates relative to the end-effector. The trajectory is then decomposed into a sequence of simpler action steps. This enables efficient training of low-level controllers, which are sequenced to execute each symbolic operator (Alg. 1.line 5).

Figure 2 visualizes this pipeline perfectly. Notice how the complex scene is filtered down to just the relevant block and the gripper, making the “Move” skill applicable anywhere in the workspace.

Learning the Moves: Diffusion Policies

With the data filtered and relative-ized, the system trains the actual motor policies. The researchers use Diffusion Policies, a modern class of imitation learning algorithms inspired by image generation models (like Stable Diffusion).

Diffusion policies work by adding noise to the expert’s action and learning to “denoise” it to recover the correct movement. They are particularly good at capturing complex, multi-modal distributions (handling different ways to perform the same move).

() a _ { t } = \\pi _ { e x e c } ( \\phi ( \\tilde { s } _ { t } ) ) . ()

The policy \(\pi_{exec}\) takes the filtered observation from the Oracle and outputs the action \(a_t\).

The training objective is to minimize the difference between the robot’s predicted actions and the expert’s demonstration:

() \\pi ^ { * } = \\arg \\operatorname* { m i n } _ { \\pi \\in \\Pi } \\sum _ { \\tilde { \\tau } _ { i } \\in \\mathcal { D } } \\sum _ { t = 0 } ^ { T _ { i } } \\mathcal { L } ( \\pi ( \\tilde { s } _ { t } ) , a _ { t } ) , ()

Action Decomposition

To further simplify learning, the system automatically breaks down complex skills into smaller “Action Steps.” A “Move” command isn’t just one long blur of motion; it is a sequence: Reach -> Pick -> Carry -> Drop.

Figure 7: Example of “MOVE” skill decomposition into action steps using the Stacking and Towers of Hanoi demonstrations. From left to right: reach-pick, pick, reach-drop, and drop.

By clustering the action data (Figure 7), the system identifies these sub-phases automatically. Learning four simple, short-horizon policies is much easier than learning one long, complex one.

Putting It to the Test

The researchers validated their framework on six diverse domains, ranging from table-top manipulation to mobile robotics.

Figure 3: Ilustrations of some of the simulation domains used for evaluation. Figure 6: Ilustration of the Nut Assembly environment in Robosuite.

The domains included:

  1. Stacking: Building towers of blocks.
  2. Towers of Hanoi: The classic puzzle requiring recursive planning.
  3. Nut Assembly: Precision manipulation (Figure 6).
  4. Kitchen: A complex sequence of turning on stoves, cooking, and serving.
  5. Forklift: A simulated warehouse environment involving vehicle navigation and pallet manipulation.

The Results: Efficiency and Robustness

The primary metric was success rate on long-horizon tasks. The comparison included standard Imitation Learning (IL) and Hierarchical Imitation Learning (H-IL) baselines.

The results were stark.

Figure 4: Performance comparison between our Neuro-Symbolic (N-S) framework and baseline methods. Our approach achieves high success rates on short—Stacking & Forklift Pallt Loading/Unloading—and long-horizon tasks—including Towers of Hanoi, Multiple Pallets Storage, Nut Assemly & Kitchen—even with as few as 5 demonstrations. Our approach is domain agnostic, and works in very different scenarios.

As seen in Figure 4:

  • Data Efficiency: The Neuro-Symbolic (N-S) approach (red bars) achieved high success rates (90%+) with as few as 5 demonstrations.
  • Baseline Failure: The baselines (blue/purple bars) often failed completely (0% success) on complex tasks like Towers of Hanoi, even when given 500 demonstrations. They simply couldn’t learn the long-term dependencies.
  • Consistency: Whether it was a robotic arm or a driving forklift, the N-S approach worked consistently.

The Power of Generalization

Perhaps the most impressive result is the system’s ability to generalize. Because the robot learns the symbolic rules of the Towers of Hanoi, it isn’t just memorizing where the disks are.

The researchers trained the agent on a simple 3-disk configuration. They then asked it to solve 4-disk and 5-disk versions of the puzzle—tasks it had never seen demonstrated.

Figure 5: Zero- and few-shot generalization results on Diffrent Hanoi Towers configurations.

Figure 5 shows these results. The agent showed strong zero-shot generalization. It could solve harder puzzles immediately because the PDDL domain it discovered (the logic of moving disks) holds true regardless of how many disks there are. By adding just a few “expert” corrections (few-shot), the performance on even the hardest tasks skyrocketed.

Conclusion: The Future of Robot Learning

This paper presents a compelling argument for the marriage of Neural Networks and Symbolic AI. By letting the neural network handle the “messy” real-world motor control and the symbolic engine handle the “clean” high-level logic, we get the best of both worlds.

The key takeaways are:

  1. Interpretability: Unlike a “black box” neural network, we can look at the generated PDDL code and see exactly what the robot thinks the rules are.
  2. Data Efficiency: Breaking the problem down allows the robot to learn from 5 examples instead of 5,000.
  3. Generalization: Logic scales. A rule learned on a small puzzle applies to a large one.

This framework paves the way for robots that can be taught new, complex jobs in minutes by a human demonstrator, without a single line of code being written. Whether it’s organizing a warehouse or helping in a kitchen, the future of robotics looks increasingly Neuro-Symbolic.