If you have been following the rapid advancements in 3D computer vision, you have undoubtedly encountered 3D Gaussian Splatting (3DGS). It has revolutionized the field by offering real-time rendering speeds coupled with high-quality reconstruction. However, like any burgeoning technology, it has its Achilles’ heel. For 3DGS, that weakness is mirrors and shiny objects.

Standard 3DGS struggles to render highly specular surfaces—materials like polished metal, glass, or mirrors that reflect their surroundings sharply. When you look at a mirror in a 3DGS scene, you often see a blurry, incoherent mess rather than a crisp reflection of the nearby teapot or toy car.

In this post, we are diving deep into SpecTRe-GS, a new method presented at CVPR that solves this problem. The researchers propose a hybrid approach: they combine the speed of Gaussian Splatting with the physical accuracy of Ray Tracing to model complex inter-reflections.

Figure 1. Comparison of SpecTRe-GS against previous methods on a scene with highly reflective surfaces. Note the sharp reflection of the Rubik’s cube in the bowl.

As shown in Figure 1, the difference is stark. While previous methods produce ghostly or blurred artifacts on the reflective bowl, SpecTRe-GS captures the sharp reflection of the nearby Rubik’s cube. Let’s explore how they achieved this.

The Problem: Why 3DGS Hates Mirrors

To understand the solution, we first need to understand the limitation of the original 3D Gaussian Splatting.

3DGS represents a scene as a cloud of 3D Gaussians (ellipsoids). To render color, it uses Spherical Harmonics (SH). SH are excellent for representing view-dependent color changes that vary smoothly, like the way velvet changes brightness as you move around it. However, SH are limited to low-frequency signals.

A perfect mirror reflection is a high-frequency signal. As you move your head slightly, the image in the mirror changes drastically. To represent a sharp reflection using SH, you would need an impracticably high number of coefficients, which would explode the memory usage and computational cost.

Furthermore, standard 3DGS treats every Gaussian independently. It doesn’t inherently “know” about the geometry of the rest of the scene. This makes it impossible to calculate inter-reflections—where object A is visible inside the reflective surface of object B—using the standard rasterization pipeline.

The Landscape of Solutions

Researchers have attempted to fix this in various ways. Some try to bake reflections into the texture (which breaks when you move the camera). Others use Screen Space Ray Tracing (SSRT), which only reflects what is currently visible on your screen (failing if the reflected object is off-camera).

Table 1. A summary of related works and how they handle view-dependent appearance and global illumination.

As summarized in Table 1, most existing methods either ignore inter-reflections entirely or “bake” them into the model, which prevents you from editing the scene later (e.g., moving an object and seeing its reflection move). SpecTRe-GS aims to fill the gap by enabling high-frequency, editable, 3D ray-traced reflections.

The Core Method: SpecTRe-GS

The researchers introduce a pipeline that fundamentally changes how shiny objects are rendered. The core idea is to separate the scene into two types of materials:

  1. Rough Surfaces: Rendered using the standard, efficient 3DGS pipeline.
  2. Highly Specular Surfaces: Rendered using a physics-based ray tracing module.

Figure 2. Overview of the SpecTRe-GS pipeline. Notice the split path: standard splatting for rough surfaces (top) and efficient ray tracing for specular surfaces (bottom).

Let’s break down the architecture shown in Figure 2.

1. The Rendering Equation for Gaussians

Instead of relying solely on Spherical Harmonics, SpecTRe-GS adopts the physically-based rendering (PBR) equation. For the highly specular parts of the scene, the color isn’t just a stored value; it is calculated based on light bouncing off the surface.

The reflection model separates the light coming from a surface point \(\mathbf{x}\) into diffuse (scattered) and specular (mirror-like) components:

Equation 2: Total radiance is the sum of diffuse and specular components.

  • \(I_{\mathrm{diff}}\) is the diffuse color (like the base color of the object).
  • \(I_{\mathrm{spec}}\) is the sharp reflection.

To calculate the specular reflection \(I_{\mathrm{spec}}\), the system needs to know two things: how reflective the surface is (\(A_{\mathrm{spec}}\)) and what light is hitting it (\(I_{\mathrm{i}}\)) from the reflected direction (\(\omega_{\mathrm{r}}\)).

Equation 3: Specular component definition.

The reflectance \(A_{\mathrm{spec}}\) is modeled using the Fresnel effect—physics jargon for “things look more reflective at grazing angles.” They use Schlick’s approximation for this:

Equation 4: Schlick’s approximation for Fresnel effect.

Here, \(F_0\) is a learnable parameter stored on each Gaussian, representing how reflective the material is when you look straight at it.

2. Tracing Rays for Incident Light

The most difficult part is determining \(I_{\mathrm{i}}\)—the incident radiance. This is the image you actually see inside the mirror. In a real physical environment, light comes from two sources:

  1. Direct Lighting: Light from the sky or environment map (\(I_{\mathrm{dir}}\)).
  2. Indirect Lighting: Light reflecting off other objects in the scene (\(I_{\mathrm{ind}}\)).

SpecTRe-GS calculates this by firing a ray from the camera, bouncing it off the shiny Gaussian, and seeing what it hits next.

Equation 5: Incident radiance combines direct environment lighting and indirect scene lighting based on visibility.

  • \(V_{\mathrm{i}}\) is visibility (1 if the ray hits nothing and goes to the sky, 0 if it is blocked by an object).
  • If the ray hits the sky, we sample an Environment Map (\(I_{\mathrm{dir}}\)).
  • If the ray hits another object, we query that object’s color (\(I_{\mathrm{ind}}\)).

Finally, the system combines the standard rendering and this new specular rendering using a mask \(M\). This mask identifies which parts of the image are highly specular.

Equation 6: Blending the specular and rough surface renderings.

3. Efficient Ray Tracing in a Point Cloud

Here lies the engineering challenge: How do you ray trace a cloud of soft, translucent blobs efficiently?

Standard ray tracers (like Nvidia’s OptiX) work on triangles (meshes). 3DGS uses Gaussians. To bridge this gap, SpecTRe-GS constructs a Geometry Acceleration Structure (GAS).

Instead of ray-tracing the exact Gaussian ellipsoids (which is mathematically heavy), they approximate each Gaussian with a stretched icosahedron (a geometric shape with 20 faces). This acts as a “proxy mesh” that fits tightly around the Gaussian.

Equation 7: Calculation of proxy mesh vertices for the GAS.

This equation scales an icosahedron based on the Gaussian’s covariance (shape) and rotation, ensuring the mesh proxy matches the visual blob.

When a ray hits this proxy, the system calculates the exact “intersection point” analytically to ensure the depth is precise. The depth \(t\) that maximizes the Gaussian’s contribution along the ray is:

Equation 8: Determining the exact intersection depth along a ray.

By using this proxy geometry within a hardware-accelerated framework (OptiX), SpecTRe-GS can trace millions of secondary rays remarkably fast—only 50% to 100% slower than standard rasterization, which is impressive for adding full global illumination.

4. Training and Geometry Enhancement

Ray tracing is unforgiving. If the geometry of your mirror is slightly bumpy, the reflection will look distorted and wavy. Standard 3DGS often produces “fluffy” geometry that looks fine from a distance but is terrible for ray bouncing.

To fix this, the authors introduce a robust training strategy with specific loss functions:

Equation 9: The total loss function including color, distortion, normals, and smoothness.

A key component here is \(\mathcal{L}_{\mathrm{n}}\), the Normal Prior Guidance. They use a pre-trained monocular normal estimation model (StableNormal) to guess what the surface orientation should be and penalize the Gaussians if they point in the wrong direction.

Equation 10: Normal prior loss.

Joint Geometry Optimization

Simply penalizing the normals isn’t enough. The system needs to learn geometry from the reflections themselves. If the reflection looks wrong, the mirror’s angle must be wrong.

To achieve this, the authors differentiate the ray-traced light with respect to the surface normal. They use a finite difference method to numerically calculate gradients:

Equation 11: Numerical gradient calculation for indirect lighting.

This effectively tells the system: “The reflection of that red cube is too far to the left; tilt the mirror slightly right to fix it.”

To make this stable, they use Depth-Aware Ray Perturbation. When calculating gradients, they adjust how much they wiggle the ray based on how far away the reflected object is.

Equation 16: Depth-aware perturbation for numerical stability.

Finally, they use a Progressive Learning strategy. They start with a strong reliance on the “Normal Prior” to get the rough shape right. As training proceeds, they decay this guidance and let the physics-based rendering loss take over to fine-tune the details. They also suppress residual color in reflective regions to force the model to rely on the ray-tracing pipeline:

Equation 17: Residual suppression loss.

Experiments and Results

Does it actually work? The authors compared SpecTRe-GS against state-of-the-art methods including standard 3DGS, Ref-NeRF inspired approaches, and other reflection-aware splatting techniques.

Visual Quality

The visual results are compelling. In Figure 3, we see comparisons on synthetic scenes.

Figure 3. Qualitative comparison on synthetic scenes (Toaster, Mirror, Pot). SpecTRe-GS (Ours) produces clean, sharp reflections where other methods produce blur or noise.

Look closely at the Mirror row (middle). The “Ground Truth” shows a perfect reflection of a yellow object.

  • 3DGS-DR and GShader produce reflections, but they are often distorted or fuzzy.
  • 3iGS and GOF struggle significantly with the high-frequency details.
  • SpecTRe-GS (Ours) matches the Ground Truth almost perfectly.

Component Decomposition

One of the strongest proofs that the model is learning “physics” and not just memorizing pixels is image decomposition. We can ask the model to show us only the diffuse color or only the specular reflection.

Figure 4. Decomposing the scene into diffuse and specular components.

In Figure 4, look at the “Diffuse” column. Ideally, a shiny metal helmet should have a gray/black diffuse color, and all the color should be in the “Specular” layer.

  • 3DGS-DR bakes the reflection into the diffuse layer (you can see the room on the helmet in the diffuse column).
  • SpecTRe-GS successfully separates them. The diffuse helmet is dark matte, and the reflection is entirely in the specular pass. This allows for realistic relighting later.

Quantitative Data

The numbers back up the visuals. The authors measured PSNR (image quality), SSIM (structural similarity), and LPIPS (perceptual similarity).

Table 2. Quantitative results showing SpecTRe-GS achieving best or second-best scores across almost all metrics.

SpecTRe-GS achieves the highest scores, particularly in the “Reflective” columns, which measure quality specifically inside the mirror regions.

Real-World Performance

Synthetic data is nice, but real-world data is messy. The authors captured scenes with a real camera to test robustness.

Figure 5. Reconstruction of real-world scenes (Bowl and Pot). The geometry column shows the estimated depth/normals.

As seen in Figure 5, the method recovers accurate geometry (the red/blue heatmaps) and produces convincing reflections of the toys on the tablecloth, even though the camera poses and lighting were not perfectly controlled.

Ablation Studies

The authors also performed an ablation study to verify if every part of their complex pipeline was necessary.

Table 3. Ablation study results. Removing components like Normal Priors (w/o N.) or Joint Optimization (w/o J.) significantly drops performance.

The data shows that removing the Normal Prior guidance (w/o N.) causes the biggest drop in quality. This confirms that without a good initial guess of the geometry, ray tracing is too unstable to converge.

Conclusion and Future Implications

SpecTRe-GS represents a significant step forward for point-based rendering. By integrating a dedicated ray-tracing module into the 3DGS framework, it solves the long-standing problem of rendering sharp, physically accurate reflections.

The implications extend beyond just “prettier pictures.” Because the method separates geometry, material properties, and lighting, it enables powerful scene editing.

Figure 6. Editing applications enabled by SpecTRe-GS. From left to right: Relighting, Object Deletion, Object Insertion, and Material Editing.

As demonstrated in Figure 6, you can:

  1. Relight the scene: Change the environment map, and the reflections on the helmet update naturally.
  2. Delete objects: If you remove the box next to the helmet, it disappears from the reflection too (something impossible with “baked” textures).
  3. Insert objects: Add a new digital object, and it will reflect in the real-world mirror.
  4. Edit Materials: Change the helmet from glossy black to matte gray.

While there are still limitations—it currently approximates specular reflections as ideal mirrors and relies on rough surfaces for the secondary bounce—SpecTRe-GS proves that we don’t have to choose between the speed of Gaussian Splatting and the realism of Ray Tracing. We can have both.