Papers

Introduction We are living in the golden age of imitation learning. From robots that can cook shrimp to those that can repair themselves, we’ve seen incredible breakthroughs driven by large-scale demonstration data. However, there is a massive bottleneck hiding behind these viral videos: the cost of data. Projects like ALOHA Unleashed or DROID required months of labor, dozens of operators, and tens of thousands of collected trajectories. If we want robots to generalize to every cup, hammer, or door handle in the world, we cannot possibly teleoperate them through every variation. We need a way to multiply our data automatically. ...

Introduction Imagine you are thirsty. You decide to reach for a cup of coffee sitting on your desk. What happens first? Before your arm muscles even engage, your eyes move. You scan the table, saccade toward the cup to lock in its position, and then guide your hand toward it. Once you grasp the cup, your eyes might immediately dart to the coaster where you plan to place it. This sequence feels instantaneous, but it reveals a fundamental truth about biological intelligence: we do not passively absorb the visual world like a video camera; we actively look in order to act. ...

Introduction In the world of robotics, we are constantly chasing the dream of efficient learning. If you have ever trained a neural network for image recognition, you know the power of backpropagation. You calculate the error, compute the gradient (the direction to adjust parameters to reduce that error), and update the network. It’s elegant, mathematical, and efficient. However, when we try to apply this same logic to robots interacting with the physical world—specifically legged robots that need to walk—we hit a massive wall: Contact. ...

Introduction Imagine a future where a surgical robot operates autonomously on a patient. It’s stitching soft tissue with precision, relieving an overworked surgeon who oversees the process. Suddenly, the robot encounters a piece of tissue that is slightly more slippery or deformed than what it was trained on. In a standard automation scenario, the robot might plow ahead, confident in a wrong decision, potentially causing harm. Ideally, however, the robot would “realize” it is confused, pause, and ask the human surgeon to take over for a moment. Once the tricky part is navigated, the robot resumes control. ...

Introduction Imagine unboxing a new robot, turning it on, and telling it to “go to the kitchen.” Thanks to recent advancements in foundation models and Vision-Language Models (VLMs), this is becoming a reality. Robots can now understand high-level instructions and navigate through environments they have never seen before. However, there is a catch. While these modern AI models are excellent at understanding where to go based on visual context, they often lack a precise understanding of physical geometry. They might recognize a path to the kitchen but fail to notice the small cardboard box left in the hallway or a chair leg protruding from under a table. The result? Collisions. ...

Introduction When we imagine a humanoid robot, we typically picture it doing one of two things: walking on two legs or standing still while using its hands to manipulate an object. This mirrors how traditional robotics control has evolved—treating the robot as a bipedal platform for mobile manipulation. But think about how humans actually move. We don’t just walk. We sit in chairs, we crawl under low obstacles, we trip and roll to break a fall, and we push ourselves up from the ground. We use our elbows, knees, backs, and shoulders to interact with the world. In robotics terms, humans embrace “full-body ground contacts.” ...

Introduction The dream of generalist robotics is a machine that can walk into any room, look around, and perform a task simply by being asked. Whether it’s “clean up that spill” or “make me a sandwich,” the robot needs to understand the visual world, parse the language command, and translate that into precise physical movements. To achieve this, the field has coalesced around Vision-Language-Action (VLA) models. Think of these as the robotic equivalent of Large Language Models (LLMs). But instead of outputting text, they output robot actions. ...

Imagine you are driving down a busy street. You see a ball roll out from between two parked cars. You don’t just see a “spherical orange object”; you immediately infer that a child might be chasing it, and you instinctively slow down. This is commonsense reasoning. Now, consider an autonomous driving (AD) system. Most modern end-to-end (E2E) models are excellent at pattern matching—they see the road geometry and other cars, and they mimic the trajectories found in their training data. However, they often lack the “why” behind the driving decisions. They might navigate a standard intersection perfectly but struggle in “long-tail” scenarios (rare, complex events) because they lack the underlying reasoning capabilities that humans possess. ...

Introduction In the rapidly evolving world of robotics, the ability to create high-fidelity 3D maps is crucial. Whether it is a drone inspecting a warehouse or a quadruped exploring a disaster zone, robots rely on these maps to understand their environment. Recently, 3D Gaussian Splatting (GSplat) has emerged as a powerful technique for representing scenes, offering photorealistic quality and real-time rendering speeds that surpass traditional point clouds or voxel maps. ...

Imagine a robot entering a new kitchen for the first time. To a human, the function of the room is obvious: the refrigerator handle pulls open, the drawer slides out, and the cabinet door swings on a hinge. We understand these mechanics intuitively, often predicting how an object moves before we even touch it. For a robot, however, this is a geometric nightmare. A cabinet isn’t just a static box; it is an articulated object—a rigid body capable of specific motions relative to another part. If a robot miscalculates the axis of rotation for a heavy fridge door, it could rip the handle off or damage its own arm. ...

Imagine trying to move a heavy sofa up a winding staircase with two friends. It requires constant communication: “Pivot left,” “lift higher,” “wait, it’s slipping.” Now, imagine doing that while flying, buffeted by wind, connected to the object only by loose cables, and—here is the kicker—you aren’t allowed to speak to each other. This is the challenge of cooperative aerial manipulation. While a single drone (Micro Aerial Vehicle or MAV) is often too weak to carry heavy payloads, a team of drones can lift significantly more. By tethering multiple drones to a single load, we can transport construction materials or emergency supplies to remote areas. ...

Introduction: The High Cost of Robotic Vision In the rapidly evolving landscape of Artificial Intelligence, we have witnessed a massive shift from text-based Large Language Models (LLMs) to multimodal systems. We no longer just want AI to write poetry; we want it to see the world and, more importantly, act upon it. This ambition has given rise to Vision-Language-Action (VLA) models—systems that ingest visual data and language instructions to output robotic control actions. ...

Introduction Large Language Models (LLMs) like GPT-4 have transformed our expectations of artificial intelligence. We have grown accustomed to their ability to write code, summarize history, and even reason through logic puzzles. Recently, roboticists have begun connecting these “brains” to robot “bodies,” allowing LLMs to generate high-level plans or write control code. However, a significant gap remains. While LLMs understand language, they don’t natively “understand” the complex, low-level physics required to slide a puck across a table or swing a rope into a specific shape. ...

Introduction Imagine you are teaching a robot to pick up a coffee mug. You guide the robot’s hand, gripping the mug and placing it on a coaster. The robot records this motion and the video feed from its camera. You run the policy, and it works perfectly. But then, you move the mug three inches to the left, or perhaps you rotate the robot’s base slightly. Suddenly, the robot flails, misses the mug, or crashes into the table. ...

The dream of general-purpose robotics is a machine that can walk into a messy, unseen kitchen and simply “wash the dishes” or “put away the groceries” without needing weeks of specific training for that exact room. However, the reality of robotic learning is often far more brittle. Most robots are trained via Behavior Cloning (BC), where they meticulously mimic collected datasets of robot actions. The problem? Collecting robot data is expensive, slow, and dangerous. ...

Introduction Imagine you are reaching for a coffee mug on a cluttered desk. You don’t consciously hallucinate a photorealistic video of your hand moving, frame by frame, texture by texture, before you act. Instead, your brain operates on an intuitive, implicit level. It predicts the consequences of your movement—the spatial feeling of the grasp, the weight of the cup, the avoidance of the stapler—without needing to render every pixel of the scene in your mind’s eye. ...

Introduction When you look at a coffee mug, you don’t just see a cylindrical object with a curved protrusion; you intuitively see a handle that can be grasped. When you look at a drawer, you see a knob meant to be pulled. In psychology and robotics, this concept is known as affordance—the actionable properties of an object that define how an agent can interact with it. For humans, recognizing affordances is effortless. For robots, it is an immense challenge. While recent advancements in Vision-Language Models (VLMs) have given machines the ability to describe scenes and answer questions, bridging the gap between high-level semantic understanding (“That is a mug”) and low-level robotic control (“Grasp the handle at these coordinates”) remains a bottleneck. ...

Introduction: The Long-Horizon Challenge Imagine asking a robot to “make dinner.” To you, this is a single request. To a robot, it is a staggering sequence of complex, physically grounded actions: open the fridge, identify the ingredients, grasp the onion, place it on the cutting board, pick up the knife, and so on. In the field of robot learning, we call these long-horizon manipulation tasks. They are notoriously difficult because errors compound. If the robot fumbles opening the fridge, the rest of the plan is irrelevant. ...

Imagine trying to plug a USB charger into a wall socket in a pitch-black room. You can’t see the socket, but you can feel around, sensing the resistance when you hit the plastic faceplate and the satisfying “click” when the plug slides in. Now, imagine a robot trying to do the same thing relying only on a camera. If its hand blocks the view, or if the lighting is bad, the robot is effectively blind and numb. It pushes, fails, and doesn’t know why. ...

Introduction In the context of human motor skills, placing an object is deceptively simple. Whether you are hanging a mug on a rack, sliding a book onto a shelf, or inserting a battery into a remote, your brain seamlessly processes the geometry of the objects, identifies valid target locations, and coordinates your hand to execute the move. You don’t need to be retrained from scratch every time you encounter a slightly different mug or a new type of shelf. ...