If you have ever tried your hand at 3D sculpting—creating digital characters, monsters, or environments—you know the pain of detailing. Sculpting the basic silhouette of a dragon is one thing; sculpting every individual scale, horn, and skin pore is an entirely different battle.

To solve this, professional artists don’t sculpt every detail from scratch. They use “stamps,” known technically as Vector Displacement Maps (VDMs). These are powerful tools that allow an artist to take a complex shape (like a nose, an ear, or a set of scales) and “stamp” it onto a base mesh instantly.

However, there is a catch: creating these stamps is incredibly difficult. It requires technical mastery of topology and baking pipelines. As a result, artists are often limited to buying pre-made “brush packs” from third parties. If you need a specific type of alien ear and it’s not in your library, you are out of luck.

Enter GenVDM, a novel research paper that proposes a generative AI pipeline capable of turning a single RGB image into a functional, high-quality Vector Displacement Map.

Figure 1. We introduce GenVDM, a method that can generate a highly detailed Vector Displacement Map (VDM) from a single input image. The generated VDMs can be directly applied to mesh surfaces to create intricate geometric details.

In this post, we will dive deep into how GenVDM works. We will explore why standard 3D generative models fail at this task, the clever “two-step” reconstruction pipeline the authors designed, and how they solved the massive problem of having zero training data available for this specific task.

Background: The Power of the VDM

Before dissecting the neural network, we need to understand the data format. Why are we generating VDMs and not just standard 3D meshes?

Displacement Maps: Scalar vs. Vector

In computer graphics, a standard Displacement Map is essentially a heightfield—a grayscale image where bright pixels pull the surface “up” and dark pixels push it “down” along the surface normal. While useful for rough textures like concrete or bark, heightfields have a fatal flaw: they can only displace geometry in one direction (vertical). They cannot create overhangs, undercuts, or complex cavities.

A Vector Displacement Map (VDM) solves this. Instead of a single height value, every pixel in a VDM contains a 3D vector \((x, y, z)\). This tells the geometry exactly where to move in 3D space. This allows a flat plane to twist and curl into complex shapes like a human ear, a mushroom, or a hooked claw—shapes that fold back on themselves.

The Generative Gap

The recent explosion in 3D Generative AI (like models that turn text into 3D meshes) has focused on generating complete objects—entire chairs, cars, or avatars. These models are generally not designed to generate parts or surface details. Furthermore, simply predicting a depth map from an image isn’t enough because depth maps, like scalar displacement maps, cannot represent the complex undercut geometry required for high-quality sculpting brushes.

GenVDM fills this gap by focusing specifically on generating these geometric patches that can be seamlessly blended onto existing 3D surfaces.

The GenVDM Pipeline

The authors’ approach is broken down into three logical stages. Because the geometry of a VDM is complex and requires precise topology (the wireframe structure of the 3D model), a simple “image-to-3D” network wasn’t sufficient.

The pipeline operates as follows:

  1. Input Processing: Prepare the image to look like a stamp.
  2. Multi-View Normal Generation: Use a diffusion model to understand the shape from multiple angles.
  3. VDM Reconstruction: Convert those multi-view predictions into the final vector map using a novel neural deformation technique.

Figure 2. Overview of our image-to-VDM pipeline.

Step 1: Multi-View Normal Generation

The process begins with a single RGB image. To help the model understand that this object is meant to be a “patch” on a surface rather than a floating object in a void, the authors place a gray square behind the object (Figure 2a).

The researchers then fine-tune a pre-trained image-to-multiview diffusion model, specifically Zero123++. Standard Zero123++ is designed to generate views surrounding an entire object (front, back, sides). However, for a VDM, the “back” of the object is inside the surface it’s stamped onto—we don’t need to see it.

Therefore, the authors modified the camera poses. Instead of circling the object, the model generates six normal maps from the front hemisphere (various angles of azimuth and elevation). They choose to generate normal maps (which represent surface orientation) rather than RGB images because normals provide pure geometric information without the distraction of lighting or texture.

Step 2: VDM Reconstruction

Once the model has hallucinated these six normal maps, the system needs to combine them into a single, cohesive 3D shape. This is the most technically intricate part of the paper.

Directly training a large model to regress a VDM pixel-by-pixel is difficult due to the lack of massive datasets. Instead, the authors use per-shape optimization. This is a slower but more accurate process where a 3D representation is iteratively tweaked until it matches the six generated normal maps.

Figure 3. Reconstructing VDM from multi-view normal maps. We adopt a two-step approach.

As shown in Figure 3, this reconstruction has two phases:

Phase A: Neural SDF Optimization

First, they reconstruct an implicit 3D mesh using a Neural Signed Distance Function (SDF). This creates a “raw” mesh that looks like the target object (Figure 3b). While accurate, this mesh is just a collection of triangles. It doesn’t have the specific UV mapping or topology required to be a VDM. It effectively “floats” in space and might have noise or holes.

Phase B: Parameterization via Neural Deformation

This is the critical innovation. To turn that raw mesh into a VDM, the system must figure out how to warp a flat, square grid so that it wraps perfectly onto the shape of the raw mesh.

Standard geometry processing techniques (like Tutte embedding) try to flatten a 3D mesh onto a 2D plane mathematically. However, because the raw mesh from Phase A is generated by AI, it often contains noise, holes, or “non-disk topology” (it’s not a perfect sheet). Standard tools break down completely when faced with this messy data, resulting in distorted or broken maps.

Figure 4. Comparison of different approaches for parameterizing a shape into VDM.

To solve this, the authors propose using a Neural Deformation Field.

Imagine a flexible rubber sheet defined by a square domain \(P\). The goal is to stretch and fold this sheet so it matches the target mesh \(Q\). The authors define a Multilayer Perceptron (MLP), denoted as \(\phi_{\theta}\), which takes a 2D point on the square and predicts its 3D position.

The optimization process moves the points of this rubber sheet to minimize the distance to the target mesh. The loss function looks like this:

Loss Function Equation

Here is what this equation does:

  1. Term 1 & 2 (Chamfer Distance): It ensures every point on the rubber sheet is close to the target mesh, and every point on the target mesh is close to the rubber sheet. This makes the shapes match.
  2. Term 3 (Boundary Constraint): It ensures the edges of the rubber sheet stay pinned to the boundary of the square base. This is crucial so the VDM blends seamlessly into the flat surface around it.

Because the deformation is controlled by a Neural Network (MLP), it acts as a natural regularizer. Neural networks have a bias toward smoothness; they struggle to learn high-frequency noise. This means the MLP naturally ignores the bumpy artifacts from the raw mesh and produces a smooth, clean, high-quality VDM (Figure 4c).

The “Chicken and Egg” Data Problem

One of the biggest hurdles in training generative models for specialized tasks is data. Before this paper, there was no large-scale public dataset of Vector Displacement Maps. You cannot train a VDM generator without VDMs.

To tackle this, the researchers built a semi-automated pipeline to “mine” VDMs from existing 3D object datasets like Objaverse.

Figure 5. Data preparation pipeline.

The process, illustrated in Figure 5, involves a custom “3D Lasso” tool:

  1. Selection: A user selects an interesting part of a 3D model (e.g., the ear of a goblin).
  2. Extraction & Remeshing: The system extracts that geometry. Since raw 3D parts are often messy “polygon soups” (disconnected triangles), the system uses Screened Poisson Surface Reconstruction to fuse them into a single, watertight mesh.
  3. Stitching: This is the clever part. A real VDM needs to emerge from a flat plane. The extracted part usually has a jagged, uneven boundary. The authors developed an algorithm (inspired by Poisson Image Editing) to warp the boundary of the mesh so it becomes perfectly coplanar (flat) while preserving the internal details.

The result is a clean mesh stitched onto a square plane. From this, they can render the training pairs: the input RGB image and the ground-truth Normal maps. Using this pipeline, they created a dataset of 1,200 VDM patches to train their model.

Experiments and Results

Does it actually work? The authors compared GenVDM against state-of-the-art single-image-to-3D models, including Wonder3D, Magic123, and Large Reconstruction Models (LRM). They also compared it against a Scalar Displacement Map baseline generated using depth estimation (DepthAnything).

Qualitative Comparison

The visual differences are striking.

Figure 6. Qualitative results compared with baseline methods.

As seen in Figure 6:

  • Magic123 & LRM: These models are designed for whole objects. When asked to generate a surface detail like a nose or ear, they often struggle with the geometry, relying on texture to fake the detail.
  • Scalar DM: This creates a reasonable shape from the front view. However, look at the side profiles. Because it is a scalar map, it simply extrudes the geometry straight out. It fails to capture the curvature of the ear helix or the overhang of the nose.
  • GenVDM (Ours): Produces sharp, clean geometry with true 3D structure, capturing undercuts and occluded areas that scalar maps miss.

Ablation Study: Why the Neural Deformation Field?

The authors performed an ablation study to prove that their specific reconstruction method (Phase B) was necessary.

Figure 7. Qualitative results of ablation study.

In Figure 7, you can see the comparison:

  • (a) Topology Fixing + Tutte Embedding: This is the classical geometry processing approach. Because the input mesh has noise, the embedding creates jagged, distorted edges (look at the ear in column ‘a’).
  • (b) Mesh Optimization: Trying to optimize the mesh vertices directly without the Neural Field leads to noisy, crumpled results.
  • (c) GenVDM: The Neural Deformation Field produces a smooth, anatomically plausible ear, effectively filtering out the reconstruction noise.

Applications: Customization and Editing

One of the most exciting implications of this technology is the ability to edit 3D geometry using 2D tools. Because the pipeline starts with an image, a user can take a photo of an ear, warp it in Photoshop (e.g., make it pointy like an elf), and feed it back into GenVDM.

Figure 8. Customizing VDMs by editing images.

As shown in Figure 8, editing the input image (warping the shape or changing proportions) results in a corresponding 3D VDM that reflects those changes accurately. This opens up a workflow where artists can design complex 3D brushes using simple 2D image manipulation.

Conclusion

GenVDM represents a significant step forward in bridging the gap between 2D generative AI and professional 3D workflows. Rather than trying to replace the entire 3D modeling pipeline, it enhances it by automating one of the most tedious tasks: sculpting surface detail.

By combining a modified multi-view diffusion model with a novel, robust reconstruction pipeline involving Neural Deformation Fields, the authors successfully created the first method for generating Vector Displacement Maps from single images.

Key Takeaways:

  • VDMs > Displacement Maps: For complex organic details (ears, noses, scales), you need vector displacement to handle undercuts.
  • Neural Fields as Regularizers: When dealing with noisy AI-generated 3D data, fitting a neural field is often better than standard geometric algorithms because the network’s bias promotes smoothness.
  • Data is King: When the dataset doesn’t exist, building a robust tool to create it (like the 3D lasso pipeline) is as important as the model architecture itself.

While the method has limitations—it is slower than feed-forward models due to the optimization step, and it can struggle with very thin structures (Figure 9 in the paper)—it provides a level of geometric fidelity and utility that previous methods have not achieved. For 3D artists, the future of “stamping” details just got a lot more interesting.