Can Robots Feel What They See? Translating Touch Across Sensors

Introduction

In the world of computer vision, a camera is generally just a camera. Whether you swap a Logitech webcam for a high-end DSLR, the fundamental data structure—an array of pixels representing light—remains consistent. You might need to resize an image, but a neural network trained on JPEGs can usually handle the switch with minimal fuss.

In robotics, however, the sense of touch is far more chaotic. Tactile sensors come in every conceivable shape and form factor. Some are soft, air-filled bubbles; others are rigid, gel-based pads; some use internal cameras to track deformation, while others measure resistance. This hardware diversity creates a massive bottleneck: an algorithm trained to manipulate a cup using a Soft Bubble sensor will likely fail completely if you switch to a GelSlim sensor. The data distributions are simply too different.

This forces roboticists to start from scratch every time they change hardware, collecting expensive new datasets and retraining models. But what if we could translate the “language” of one sensor into another?

In the paper “Cross-Sensor Touch Generation,” researchers from the University of Michigan and Cornell University propose a generative AI framework to solve this problem. They demonstrate that despite mechanical differences, vision-based tactile sensors share a common geometric reality. By translating the signals from a source sensor into the “imagined” signals of a target sensor, they allow robots to execute manipulation skills on hardware they were never trained to use.

The Problem: The Tower of Babel in Tactile Sensing

To understand the difficulty of this task, look at the figure below. It illustrates a robot picking up a cup.

When the robot grasps the object, its sensors generate a feedback signal. If the robot is equipped with a GelSlim sensor, it sees high-resolution texture. If it uses a Soft Bubble, it sees a lower-resolution, depth-based deformation map.

The researchers’ goal is Cross-Modal Tactile Generation. As shown above, the system takes the real signal (Source) and generates a synthetic signal (Target) that mimics what the other sensor would have felt in that exact scenario.

Once the signal is translated, it can be fed into a downstream task—like the pose estimation shown above—allowing the robot to stack the cup successfully using a model that has never seen the physical sensor currently attached to the robot’s hand.

Core Method: Two Paths to Translation

The authors propose two distinct architectures to bridge the gap between sensors. The choice between them depends on data availability:

1. Touch-to-Touch (T2T): A direct translation method requiring paired data.
2. Touch-to-Depth-to-Touch (T2D2): An indirect method using depth as a bridge, requiring no paired data.

1. Touch-to-Touch (T2T): The End-to-End Approach

The first method, T2T, treats the problem as an image-to-image translation task, similar to how one might use AI to turn a sketch into a photorealistic image.

The researchers collected a dataset where a robot physically probed objects with two different sensors at the exact same location. Using this paired data, they trained a Latent Diffusion Model. The model takes the reading from the source sensor (e.g., GelSlim), encodes it, and conditions the diffusion process to “hallucinate” the corresponding reading for the target sensor (e.g., Soft Bubble).

Because this method is trained end-to-end on paired examples, it is highly accurate at capturing fine-grained details. However, collecting perfectly paired tactile data is mechanically difficult and time-consuming, requiring precise alignment of different sensors on the same robot arm.

2. Touch-to-Depth-to-Touch (T2D2): The Geometry Bridge

To overcome the burden of collecting paired data, the authors introduce T2D2. This method relies on a key insight: while the images produced by sensors look different, the physical geometry of the contact is absolute.

T2D2 uses Depth as an intermediate representation (IR). The pipeline consists of three steps:

Step A: Depth Estimation

First, the model predicts a depth map from the source tactile image. The authors adapted the Depth Anything V2 model for this purpose. They trained it using a scale-invariant logarithmic loss to ensure the predicted geometry matches reality:

Here, \(D_S\) is the ground truth depth and \(D'_S\) is the estimated depth. This allows the model to extract the 3D shape of the object pressing into the sensor.

Step B: Cross-Sensor Depth Adaptation

This is the most technically intricate step. A depth map is specific to a sensor’s camera perspective and field of view. To translate it, the system must “move” the depth information from the source sensor’s coordinate frame to the target sensor’s frame.

First, they define a mask of valid pixels (pixels actually touching the object):

Next, they back-project these pixels into a 3D point cloud (\(\mathcal{P}_T\)) using the source sensor’s camera intrinsics (\(K_S^{-1}\)) and transform them into the target sensor’s frame (\(T_{S \to T}\)):

Finally, this 3D point cloud is projected back onto the 2D image plane of the target sensor to create a new, adapted depth map (\(D''_T\)) and a new contact mask (\(M''_T\)):

This process mathematically ensures that the geometry is preserved, even if the target sensor has a different size or camera angle.

As shown in the alignment diagram above, the coordinate transformations must account for the specific physical differences between sensors like the GelSlim and Soft Bubble, including rotation and differing contact areas.

Step C: Depth-to-Touch Generation

Once the depth map is adapted to the target sensor’s specifications, a diffusion model generates the final tactile image. Since this model only learns to turn depth into touch for a specific sensor, it does not require paired data from other sensors.

Experimental Results

The researchers evaluated their models on three distinct sensors: GelSlim (high-res, rigid), Soft Bubble (compliant, depth-based), and DIGIT (low-cost, compact).

Qualitative Performance

The visual results are striking. In the figure below, you can see the model’s ability to translate between GelSlim and Soft Bubble outputs.

• Top (Figure 3): The T2T model (direct translation) generates crisp, accurate images that closely resemble the Ground Truth (GT).
• Bottom (Figure 4): The T2D2 model (depth-based) successfully captures the overall contact shape but loses some high-frequency texture information. This is expected, as the intermediate depth representation acts as a bottleneck that filters out fine surface details.

To test robustness, the team used a diverse set of tools for data collection, ensuring the models could generalize to object shapes they hadn’t seen during training.

Quantitative Analysis

The researchers measured success using both standard image metrics (PSNR, SSIM) and tactile-specific metrics (Pose Estimation Error).

1. Image Quality: T2T consistently outperformed T2D2. Direct translation preserves more visual fidelity.
2. Geometric Accuracy: When translating from GelSlim \(\rightarrow\) Soft Bubble, the error was higher. This is because the Soft Bubble is physically larger. The model has to “outpaint” or hallucinate tactile data for areas that the smaller GelSlim sensor didn’t even touch. Conversely, going from Soft Bubble \(\rightarrow\) GelSlim was easier, as it mostly involved cropping and refining existing data.

Downstream Robotics Tasks

The ultimate test of these generated images is whether a robot can actually use them.

1. Peg-in-Hole & Cup Stacking: The team took a policy trained only on Soft Bubble data and deployed it on a robot equipped with GelSlim. Using T2T to translate the live GelSlim images into “fake” Soft Bubble images, the robot successfully estimated the object poses and completed the tasks. The T2T method achieved success rates nearly identical to using the native sensor.

2. Marble Rolling (Behavior Cloning): This task involved rolling a marble to the center of the sensor. A policy was trained on GelSlim data. The robot was then switched to use a DIGIT sensor.

Using the T2D2 pipeline, the system translated DIGIT readings into GelSlim readings in real-time. As the timeline in the image above shows, the policy (which had never seen DIGIT data) successfully guided the marble to the center. This “zero-shot” transfer highlights the power of the depth-based approach: you can plug in a new sensor (like DIGIT) just by calibrating its depth, without needing to collect paired training data against every other sensor in existence.

Conclusion and Implications

This research presents a significant step toward unifying the fragmented hardware landscape of robotics. By treating tactile sensing as a geometric problem, the authors demonstrated that we don’t always need to retrain our brains (or algorithms) for new bodies.

• T2T proves that with enough paired data, we can achieve high-fidelity translation suitable for precise tasks like pose estimation.
• T2D2 proves that we can achieve modularity through geometry. By using depth as a universal language, we can integrate new sensors into existing ecosystems with minimal effort.

For students and researchers, this work suggests that the future of robotic touch might not lie in a single “perfect” sensor, but in generative models that allow all sensors to understand one another.