How to Reconstruct 3D Clouds from a Single Image using Diffusion and Physics

Have you ever looked at a photograph of a cloud and wondered exactly what it looked like in three dimensions? It seems like a simple question, but for a computer, it is a nightmare scenario. Clouds are not solid objects; they are volumetric, semi-transparent, and scatter light in complex ways.

Reconstructing a 3D object from a single 2D image is a classic “ill-posed” problem in computer vision. It’s ill-posed because a single 2D image is essentially a flat shadow of reality—infinite different 3D shapes could theoretically produce that exact same image depending on the lighting and angle.

To solve this, we usually need “priors”—assumptions or knowledge about what the object should look like. In recent years, generative AI has become the ultimate source of these priors.

In this post, we will dive deep into a research paper titled “Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes.” This work proposes a fascinating method to reconstruct complex 3D volumes (specifically clouds) from a single picture by combining the “imagination” of Diffusion Models with the “physical laws” of Differentiable Rendering.

The Challenge of Volumetric Reconstruction

Before we look at the solution, we need to understand why this is so hard.

1. The Geometry Problem

If you take a picture of a coffee mug, your brain knows the handle is probably loop-shaped, even if you can’t see the hole. That’s a “prior” your brain uses. In computer vision, methods like NeRF (Neural Radiance Fields) usually require dozens of images from different angles to figure out the geometry without such strong priors. When you only have one image, standard geometric methods fail because they can’t see around corners.

2. The Physics Problem

Clouds are even harder than mugs because they are “participating media.” Light doesn’t just hit the surface and bounce off; it penetrates the volume, bounces around inside (multiple scattering), and some of it passes through (transmittance).

To accurately reconstruct a cloud, you can’t just guess its shape; you have to simulate how light travels through it. This requires Radiative Transfer, a branch of physics describing how energy moves through media.

As shown in the table above, describing a volume involves complex terms like extinction fields (\(\sigma_t\)), scattering albedo (\(\varphi\)), and phase functions (\(\rho\)). The paper tackles the challenge of optimizing these parameters when we have almost zero geometrical information (just one view) and complex physical interactions.

The Solution: Combining Physics and AI

The researchers propose a method that marries two powerful concepts:

1. Differentiable Volume Rendering: A physics simulation that can run backward to figure out what 3D shape caused a 2D image.
2. Diffusion Models: A generative AI trained to know what valid 3D clouds look like.

The high-level idea is this: We will ask the AI to “dream” of a cloud. Then, we will use a physics engine to check if that dream matches our input photo. If it doesn’t, we gently nudge the AI’s dream until it fits the photo, while ensuring it still looks like a realistic cloud.

Figure 1 above illustrates the entire pipeline. On the left, we have a single view (\(y\)). We want to reconstruct the volume (\(\hat{V}\)). We use a Diffusion Model to generate the latent representation (\(\theta\)) of the cloud. Simultaneously, a Differentiable Renderer (\(\mathcal{R}\)) checks the output against the input image and updates the scene parameters (\(\phi\)).

Let’s break down the machinery that makes this possible.

Step 1: Building a Cloud Brain (The Dataset)

You cannot train a diffusion model without data. Unfortunately, there was no massive dataset of 3D volumetric clouds available. So, the researchers built one.

They created Cloudy, a dataset of 1,000 synthetically simulated volumetric density fields. Using a fluid simulator (JangaFX), they generated realistic cumulus clouds with proper buoyancy, turbulence, and diffusion.

As seen in Figure 2, the top row shows the ground truth simulations rendered photorealistically. The bottom row shows clouds generated by their trained diffusion model. The resemblance is striking—the model has learned the “essence” of a cloud.

Step 2: Compressing the Cloud (Monoplanar Representation)

A raw 3D grid (voxel grid) is heavy. A \(512 \times 512 \times 512\) grid takes up a lot of memory, making it difficult to train a neural network. We need a way to compress this data into a “latent code” that the diffusion model can easily understand.

Previous methods used “tri-planes” (three flat maps intersecting) or simple dense grids. This paper introduces a Monoplanar Representation.

As shown in Figure 3, instead of storing the whole 3D volume, they project features onto a single 2D plane (\(g\)). To find the density at a specific 3D point \(p\), they sample this 2D plane and pass the features through a small Multi-Layer Perceptron (MLP).

This offers massive compression. A single cloud goes from roughly 100MB of raw data down to a 2MB latent code. This efficiency allows the diffusion model to train much faster and generate higher-resolution details.

Step 3: The Core Engine (Parametric Diffusion Posterior Sampling)

This is the most technical and innovative part of the paper.

Standard diffusion models are unconditional. You give them random noise, and they turn it into a random cloud. But we don’t want a random cloud; we want a cloud that matches our specific photograph.

We need Posterior Sampling. We want to sample from the probability distribution of clouds given our observation (\(y\)).

The researchers use a technique called Diffusion Posterior Sampling (DPS). Here is how it works conceptually:

1. Start with pure noise.
2. The Diffusion Model predicts a slightly less noisy version of the cloud (the “prior”).
3. We take that predicted cloud and run it through the Differentiable Volume Renderer.
4. We compare the rendered image to our input photo (\(y\)).
5. We calculate the error (loss). Because the renderer is differentiable, we can calculate the gradient of the error.
6. We use that gradient to “steer” the denoising process.

The steering equation looks like this:

In this equation:

• \(y\) is the target image.
• \(\mathcal{A}\) is the rendering process.
• \(\hat{x}_0(x_t)\) is the estimated clean cloud at step \(t\).
• \(\nabla_{x_t}\) is the gradient that tells us how to change the noisy sample to minimize the error.
• \(\zeta\) is a scaling factor that controls how strongly we force the cloud to match the image versus just letting it be a generic cloud.

Visualizing the Process

The figure below nicely visualizes this “guided dreaming.”

As you can see in Figure 5, the process starts (left) with noise that looks nothing like a cloud. Over time (moving right), the diffusion model tries to form a cloud structure. The differentiable renderer constantly pulls it back to ensure the shape matches the silhouette of the input image \(y\).

Handling Unknown Lighting

A major contribution of this paper is that it doesn’t just solve for the cloud’s shape (\(\theta\)); it also solves for the physical parameters (\(\phi\)), such as the background lighting or the environment map.

If you don’t know the lighting, you can’t accurately render the cloud to calculate the error. So, they optimize both simultaneously:

This equation says: Find the physical parameters \(\hat{\phi}\) that minimize the difference between the observed image \(y\) and our rendered cloud, averaging over the possible clouds our diffusion model generates.

Experimental Results

So, how well does this “Physics + AI” approach work?

Single-View Reconstruction

The primary goal was to reconstruct a cloud from one image. The results are significantly better than previous state-of-the-art methods like Differentiable Ratio-Tracking (DRT) or Singular Path Sampling (SPS), which struggle when data is sparse.

In Figure 9, look at the DPS1 (Diffusion Posterior Sampling with 1 view) column. It produces a fluffy, coherent cloud that looks very similar to the target. Compare this to DRT1 or SPS1, which often produce blurry blobs or fail to capture the distinct cloud lobes. The metrics (LPIPS) confirm that the perceptual quality of the DPS approach is far superior.

Super-Resolution

Because the diffusion model acts as a “prior” for high-frequency details, it can be used to upscale low-resolution volumetric data. It “hallucinates” plausible details that weren’t in the original low-res input.

Figure 7 shows a low-res cloud (center) being upscaled to a high-res version (right). Notice how the model adds intricate wisps and bumps that make the cloud look realistic, rather than just interpolating the blurry blocks.

Recovering Light Conditions

One of the most impressive feats is the ability to recover the lighting environment solely from the shading on the cloud in the image.

In Figure 10, the method successfully reconstructs the environment map (bottom row). Even though the “Test View” (what the reconstructed cloud looks like from a new angle) isn’t pixel-perfect to the ground truth, the lighting and shadowing are consistent, proving the model “understood” the scene’s physics.

The Physics Under the Hood

It is worth taking a brief moment to appreciate the complexity of the physics being solved here. The model isn’t just matching pixels; it is solving the Volume Rendering Equation (VRE) inside the optimization loop.

This equation (Equation 4 in the paper) integrates the transmittance \(T\) and the density \(\sigma_t\) along a ray. It accounts for light \(L_s\) scattering into the ray from other directions and emission \(L_e\).

By making this equation differentiable, the system can propagate the error from a single pixel back to the specific 3D voxel density and the environmental lighting parameters that caused that pixel’s color.

Conclusion and Future Implications

The paper “Light Transport-aware Diffusion Posterior Sampling” represents a significant step forward in inverse rendering. By using a diffusion model as a strong geometric prior, the researchers successfully turned an ill-posed problem (single-view volume reconstruction) into a solvable one.

Key Takeaways:

1. Priors are Essential: You cannot reconstruct complex volumes from single images without knowing what those volumes “usually” look like. Diffusion models are excellent at storing this knowledge.
2. Physics Matters: Simply generating a 3D shape isn’t enough. Integrating a physically based renderer ensures the reconstruction respects light transport, shadows, and density accumulations.
3. Latent Spaces are Powerful: Operating in a compressed “monoplanar” latent space makes high-resolution 3D generation computationally feasible.

While the method is computationally intensive (requiring ray tracing during the optimization loop) and currently focused on clouds, the principles here could extend to other complex volumetric data—from medical imaging (MRI/CT reconstruction) to smoke and fire in special effects. It bridges the gap between the creative hallucination of AI and the rigorous constraints of the physical world.