Papers

Deep learning models are exceptional at learning specific tasks. Train a model to classify dogs, and it will do it perfectly. But ask that same model to learn how to classify cars afterward, and you encounter a notorious problem: Catastrophic Forgetting. In the process of learning about cars, the model completely forgets what a dog looks like. This is the central challenge of Continual Learning (CL)—how can we teach machines to learn sequentially, task after task, just like humans do, without erasing previous knowledge? ...

Can AI Hallucinate Depth and Pose to Catch Crimes? Inside PI-VAD Imagine you are watching a security camera feed in a busy store. You see a customer pick up an item, look at it, and put it in their bag. Is this a normal shopping event, or is it shoplifting? To a human observer, the context matters. Did they look around nervously? Did they scan the item with a personal scanner? To a standard computer vision model relying solely on pixel data (RGB), the visual difference between “buying” and “stealing” is frustratingly subtle. Both involve reaching, grabbing, and bagging. ...

Introduction We are currently living through a golden age of visual synthesis. Text-to-image models like Stable Diffusion, Midjourney, and DALL-E have revolutionized how we create content, allowing us to conjure photorealistic scenes from a single sentence. However, if you have spent any time experimenting with these tools, you have likely encountered the “uncanny valley” of AI generation: the anatomy problem. You ask for a portrait of a guitarist, and the model generates a stunning lighting setup, perfect skin texture, and… three hands. Or perhaps a person standing in a field, missing an ear, or with legs that merge into the grass. These “abnormal human bodies” are not just minor glitches; they are structural failures that shatter the illusion of realism. ...

Introduction Imagine an autonomous vehicle driving through a busy intersection. It suddenly brakes for a pedestrian. As an engineer or a user, you might ask: Did it actually see the pedestrian? Or did it react to a shadow on the pavement that looked like a person? In the era of deep learning, answering this question is notoriously difficult. We have entered the age of Object-level Foundation Models—powerful AI systems like Grounding DINO and Florence-2 that can detect objects and understand complex textual descriptions (e.g., “the guy in white”). While these models achieve incredible accuracy, they operate as “black boxes.” Their internal decision-making processes are vast, complex webs of parameters that are opaque to humans. ...

Imagine you are building a virtual world. You have a 3D model of a chair and a 3D model of a human. Now, you want the human to sit on the chair. In traditional animation, this is a manual, tedious process. You have to drag the character, bend their knees, ensure they don’t clip through the wood, and place their hands naturally on the armrests. Now, imagine asking an AI to “make the person sit on the chair,” and it just happens. ...

Introduction In the world of computer animation and robotics, walking is a solved problem. We can simulate bipedal locomotion with impressive fidelity. However, as soon as you ask a virtual character to interact with the world—pick up a box, sit on a chair, or push a cart—the illusion often breaks. Hands float inches above objects, feet slide through table legs, or the character simply flails and falls over. This is the challenge of Physics-Based Human-Object Interaction (HOI). Unlike standard animation, where characters move along predefined paths (kinematics), physics-based characters must use virtual muscles (actuators) to generate forces. They must balance, account for friction, and manipulate dynamic objects that have mass and inertia. ...

InstructMove: How Watching Videos Teaches AI to Perform Complex Image Edits The field of text-to-image generation has exploded in recent years. We can now conjure hyper-realistic scenes from a simple sentence. However, a significant challenge remains: editing. Once an image is generated (or if you have a real photo), how do you change specific elements—like making a person smile or rotating a car—without destroying the rest of the image’s identity? ...

The dream of “Holographic” communication—where you can view a remote event from any angle in real-time—has long been a staple of science fiction. In computer vision, this is known as Free-Viewpoint Video (FVV). The goal is to reconstruct dynamic 3D scenes from multiple camera feeds instantly. While recent technologies like 3D Gaussian Splatting (3DGS) have revolutionized how we render static scenes, handling dynamic scenes (videos where people and objects move) remains a massive computational bottleneck. Traditional methods require processing the entire video offline, which is useless for live interactions like virtual meetings or sports broadcasting. Even existing “streaming” methods often take over 10 seconds to process a single frame, creating an unacceptable lag. ...

Imagine asking a robot to “pick up the red mug next to the laptop.” To us, this is a trivial request. To an AI, it is a geometric and semantic nightmare. The AI must identify objects in 3D space, understand what “red” and “mug” look like, and figure out the spatial relationship “next to.” While Large Language Models (LLMs) have mastered text, and Vision-Language Models (VLMs) have conquered 2D images, 3D scene understanding remains a frontier filled with challenges. Most current approaches awkwardly stitch together 2D image data and 3D point clouds, often losing the fine-grained details that make a scene coherent. They struggle to understand how objects relate to one another in physical space and are notoriously computationally expensive. ...

Introduction In the rapid evolution of Artificial Intelligence, we have witnessed a shift from models that simply predict the next word to models that can solve complex logic puzzles. With the release of systems like OpenAI’s o1, text-based Large Language Models (LLMs) have demonstrated “System 2” thinking—the ability to deliberate, reason step-by-step, and self-correct before answering. However, there is a glaring gap in this progress: Vision. While Multimodal Large Language Models (MLLMs)—models that can see and talk—have become excellent at describing images (perception), they often struggle when asked to perform complex reasoning about what they see. If you show an AI a chart and ask for a deep economic analysis, or a geometric figure and ask for a multi-step proof, it frequently hallucinates or takes a shortcut to the wrong answer. ...

Imagine a drone flying at high speed through a dimly lit tunnel. A standard camera would likely fail in this scenario; the fast motion causes severe blur, and the low light results in grainy, unusable footage. This is the bottleneck for many robotics applications today. However, there is a different kind of sensor that thrives in exactly these conditions: the Event Camera. Event cameras are bio-inspired sensors that work differently than the cameras in our phones. Instead of capturing full frames at a fixed rate, they operate asynchronously, detecting changes in brightness at the pixel level. This gives them incredible advantages: microsecond latency, high dynamic range, and no motion blur. ...

If you have ever played with Text-to-Image (T2I) models like Stable Diffusion, you know the struggle: you type a prompt, get a weird result, tweak the prompt, get a slightly less weird result, and repeat. While these models are powerful, they aren’t naturally aligned with human aesthetic preferences or detailed instruction following. In the world of Large Language Models (LLMs), we solved this using Reinforcement Learning from Human Feedback (RLHF) and, more recently, Direct Preference Optimization (DPO). These methods take “winning” and “losing” outputs and teach the model to prefer the winner. ...

Imagine you have a digital photo album containing thousands of images. You want to find a picture of your specific pet, “Fido,” catching a frisbee. You type “Fido catching a frisbee” into the search bar. Standard Vision-Language Models (VLMs) like CLIP differ from standard object detectors because they can understand open-ended text. However, they have a major limitation: they know what a dog looks like, but they don’t know what your dog, Fido, looks like. ...

The arrival of 3D Gaussian Splatting (3DGS) marked a paradigm shift in neural rendering. Unlike Neural Radiance Fields (NeRFs), which rely on expensive ray marching through implicit volumes, 3DGS utilizes explicit point clouds—specifically, 3D Gaussians—to render scenes in real-time with photorealistic quality. However, despite its speed and visual fidelity, 3DGS has a “messy room” problem. The quality of the final render is heavily dependent on how the Gaussian points are distributed. If the initialization (usually via Structure from Motion) is poor, or if the optimization process fails to place points where they are needed, the model suffers from artifacts. You might see “floaters” (random blobs floating in space), blurred details in complex geometry, or erroneous depth estimations. ...

Beyond Matching: How Implicit Learning Solves Image-to-Point Cloud Registration Imagine you are a robot navigating a city. You have a pre-built 3D map of the city (a point cloud), and you just took a picture with your onboard camera. To know where you are, you need to figure out exactly where that 2D picture fits inside that massive 3D world. This problem is known as Image-to-Point Cloud Registration. It sounds simple in theory—just line up the picture with the 3D model—but in practice, it is incredibly difficult. Why? Because you are trying to match two completely different types of data: a 2D grid of pixels (the image) and an unordered set of spatial coordinates (the point cloud). ...

Dreaming of Data: How ImagineFSL Revolutionizes Few-Shot Learning with Synthetic Pretraining In the world of deep learning, data is the fuel that powers the engine. But what happens when that fuel runs low? This is the core challenge of Few-Shot Learning (FSL)—teaching a model to recognize new concepts with only one or a handful of examples. Recently, Vision-Language Models (VLMs) like CLIP have shown incredible promise in this area. However, adapting these massive models to specific tasks with tiny datasets remains a hurdle. The community’s recent answer has been Generative AI. If we don’t have enough data, why not just generate it using Text-to-Image (T2I) models like Stable Diffusion? ...

Solving the Ghosting Problem: How to Reconstruct Images from Multiplexed Single-Photon Detectors Imagine trying to take a picture in near-total darkness, where light is so scarce that you are counting individual particles—photons—as they hit the sensor. This is the realm of single-photon detectors. These devices are revolutionizing fields ranging from biological imaging and Lidar to quantum optics. Among these, Superconducting Nanowire Single-Photon Detectors (SNSPDs) are the gold standard. They offer incredible efficiency and precision. However, they have a major scaling problem: they need to be cooled to cryogenic temperatures. If you want a megapixel camera, you can’t easily run a million wires out of a deep-freeze cryostat without introducing too much heat. ...

Have you ever compressed an image so much that it looked blocky and pixelated, yet your phone still recognized the face in it perfectly? Conversely, have you ever taken a photo that looked fine to you, but your smart camera refused to focus or detect the object? For decades, the field of image processing has been obsessed with one question: “Does this look good to a human?” We built compression algorithms (like JPEG), cameras, and restoration filters designed to please the Human Visual System (HVS). But the world has changed. According to recent data, Machine-to-Machine (M2M) connections have surpassed Human-to-Machine connections. Today, the primary consumer of visual data isn’t you or me—it’s Artificial Intelligence. ...

The dream of Virtual Reality (VR) has always been the “Holodeck” concept—the ability to step into a digital recording of the real world and experience it exactly as if you were there. You want to be able to walk around, lean in to see details, look behind you, and hear the soundscape change as you move. While we have made massive strides with technologies like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting, we hit a wall when it comes to dynamic scenes. Most current datasets are either static (frozen in time), object-centric (looking at a single object from the outside), or silent (no audio). ...

Imagine you want to direct a short film. You have a script, and you have a photo of your lead actor. In the traditional world, this requires cameras, lighting crews, and days of shooting. In the world of Generative AI, we have moved closer to doing this with just a text prompt. However, if you have ever tried to generate a video of a specific person using standard text-to-video models, you have likely encountered the “Identity Problem.” You might upload a photo of yourself and ask for a video of you playing basketball. The result? A person playing basketball who looks vaguely like you in one frame, like your cousin in the next, and like a complete stranger in the third. ...