Creating a detailed, interactive 3D model of a real-world space has long been a holy grail in computer vision. Imagine capturing a video of your apartment on your phone and instantly having a photorealistic digital twin you can walk through in VR — or a robot using that same video to build a precise map for navigation. This is the promise of on-the-fly 3D reconstruction, a technology crucial for AR/VR, robotics, and real-to-sim content creation.

The challenge? Doing it well requires balancing three competing goals: speed, accuracy, and robustness. For years, researchers have faced a difficult trade-off:

  • Per-scene optimization methods: These are like meticulous artists, often using classic techniques like Simultaneous Localization and Mapping (SLAM) to carefully build a scene from scratch. They can achieve stunning, high-fidelity results but are slow and computationally expensive, often struggling if the video has motion blur or poor lighting.

  • Feed-forward foundation models: These are like prodigies trained on vast, internet-scale datasets. They can look at a sequence of images and instantly generate a 3D scene. They work fast, are robust to varied inputs, and require no careful per-scene tuning. However, their results often lack the fine-grained detail and global consistency of slower, optimized methods.

Do we settle for a beautiful but slow-to-create 3D model or a fast but fuzzy one? With ARTDECO, we may no longer have to choose. This unified framework combines the efficiency of feed-forward models with the reliability of SLAM, delivering high-quality, real-time 3D reconstruction.

A collage showing high-quality 3D reconstructions of a kitchen, an office, a storefront, and an outdoor street scene, demonstrating ARTDECO’s capabilities.

Figure 1: ARTDECO delivers high-fidelity, interactive 3D reconstruction from monocular images, combining efficiency with robustness across indoor and outdoor scenes.

Background: The Building Blocks of Modern 3D Reconstruction

Before unpacking ARTDECO, let’s review the key concepts it builds upon.

1. 3D Gaussian Splatting (3DGS):
Neural Radiance Fields (NeRFs) once reigned supreme in photorealistic scene creation, representing a scene as a continuous volumetric field processed by a neural network — slow to train, slow to render. 3DGS revolutionized this by replacing the continuous field with millions of tiny anisotropic 3D Gaussians — think of them as semi-transparent blobs in space. This explicit representation allows real-time rendering (100+ FPS) while achieving or exceeding NeRF-level quality, ideal for interactive applications.

2. SLAM (Simultaneous Localization and Mapping):
A classic robotics and vision problem, SLAM tracks the camera’s motion while simultaneously building a map of the scene. Traditional SLAM produces accurate trajectories but sparse point clouds — insufficient for immersive visuals.

3. 3D Foundation Models:
Large neural models trained on huge datasets of images and 3D content learn powerful priors about the structure of the world. They can estimate camera poses, depth maps, and geometry from monocular inputs, providing strong starting points that help resolve ambiguities common in single-camera (monocular) setups.

The Core Idea:
Rather than picking one paradigm, ARTDECO builds a synergistic pipeline where each component’s strengths cover the others’ weaknesses.

The ARTDECO Method: A Three-Act Play

ARTDECO processes a video stream through three interconnected modules:

  1. Frontend: Real-time tracking and frame selection.
  2. Backend: Global trajectory optimization with loop closure.
  3. Mapping: Incremental, structured 3D Gaussian scene building.

A schematic showing the Frontend and Backend modules of the ARTDECO pipeline, illustrating frame selection, loop closure, and bundle adjustment.

Figure 2: (a) Frontend — categorizes and tracks frames using a matching module. (b) Backend — integrates loop closure and bundle adjustment for global consistency.

Act 1: The Frontend — Making Sense of the Stream

The frontend watches incoming video frames, estimates camera motion, and decides each frame’s role.

Pose Estimation:
ARTDECO uses the MASt3R foundation model in its matching module to predict dense two-view correspondences and pointmaps between the current frame and the latest keyframe. It estimates the relative camera pose by minimizing residual reprojection errors, weighting points by per-point uncertainty to avoid unreliable matches (especially near object boundaries).

Frame Categorization:
Frames are classified into:

  • Keyframes (KF): Anchor frames with significantly new viewpoints — sent to both backend and mapping.
  • Mapper Frames (MF): Frames with enough parallax to add geometric detail — sent to mapping and backend.
  • Common Frames: Frames used only to refine existing Gaussians — improve detail without adding new geometry.

This ensures efficient use of data: new geometry where needed, constant refinement elsewhere.

Act 2: The Backend — Keeping the Big Picture Consistent

As trajectories get longer, small errors can accumulate, causing drift. The backend mitigates this.

Loop Closure & Global Bundle Adjustment:
When the system returns to an already visited place, it’s a chance to correct drift. First, Aggregated Selective Match Kernel (ASMK) quickly finds candidate past keyframes. Then, π³, another 3D foundation model, verifies geometric consistency between candidates and the current view.

Confirmed loops are added to a factor graph, and global bundle adjustment refines all poses concurrently. This ensures multi-view geometric consistency and significantly improves localization accuracy.

Act 3: The Mapping Module — Building the Gaussian World

This module constructs the 3D Gaussian scene from all frame types.

The mapping process, showing how multi-resolution inputs are used to add and refine 3D Gaussians with level-of-detail control.

Figure 3: Multi-resolution analysis with the Laplacian of Gaussian selects where to add new Gaussians. LoD-aware rendering controls density based on camera distance.

Probabilistic Gaussian Insertion:
Gaussians are added where they’re most needed — in high-detail regions or where the render diverges from the real frame. This uses the Laplacian of Gaussian (LoG) operator:

\[ P_a(u,v) = \max\Big(\min(\|\nabla^2(G_{\sigma}) * I(u,v)\|, 1) - \min(\|\nabla^2(G_{\sigma}) * \tilde{I}(u,v)\|, 1), 0\Big) \]

Hierarchical Levels of Detail (LoD):
Gaussians are organized into resolution levels, each with a maximum viewing distance (\(d_{\max}\)). Coarse Gaussians represent far structures; fine ones capture near details. At render time, only Gaussians relevant to the camera distance are drawn — improving speed and eliminating flicker.

Structured Initialization:
New Gaussians are initialized from MASt3R’s pointmaps:

  • Position: 3D pointmap.
  • Color: Source image pixel.
  • Scale: From local image gradients, refined via MLPs using per-Gaussian features and shared regional voxel features.
    This hybrid design balances local uniqueness with global coherence.

Experiments and Results

ARTDECO was tested on eight benchmarks spanning diverse indoor (TUM, ScanNet++, VR-NeRF, ScanNet) and outdoor (KITTI, Waymo, Fast-LIVO2, MatrixCity) datasets.

Reconstruction Quality:

A table comparing the rendering quality (PSNR, SSIM, LPIPS) of ARTDECO against other methods on indoor and outdoor datasets.

Table 1: ARTDECO achieves the highest visual quality across the board, with strong PSNR, SSIM, and low LPIPS values.

Qualitative comparison of 3D reconstructions from various methods, highlighting ARTDECO’s superior detail and clarity.

Figure 4: ARTDECO consistently produces crisp, detailed reconstructions, accurately capturing textures and complex structures.

Tracking Accuracy:

A table comparing the tracking accuracy (ATE RMSE) of ARTDECO against other SLAM systems.

Table 2: ARTDECO’s tracking rivaled or beat dedicated SLAM systems, showing the efficacy of integrating foundation models for pose estimation and loop closure.

Ablation Study:

Quantitative results from the ablation study, showing the impact of each component on tracking and reconstruction performance.

Table 3: Removing LoD reduces rendering quality; disabling foundation-model loop closure increases drift; ignoring mapper frames leads to loss of detail.

Why ARTDECO Works so Well

  1. Hybrid is the Future:
    Combining foundation models’ learned priors with SLAM’s optimization delivers both robustness and accuracy.

  2. Structured LoD Representation:
    The hierarchical Gaussian architecture is vital for scaling to complex, large-scale environments.

  3. Foundation Models as Expert Plugins:
    Treating pre-trained models as modular components for correspondence matching and loop closure proved extremely effective.

Conclusion: Toward Instant Digital Twins

ARTDECO is a significant leap toward making high-fidelity digital twins as simple as recording a video. It achieves:

  • SLAM-level efficiency.
  • Feed-forward-level robustness.
  • Near per-scene optimization quality.

Remaining challenges include handling extreme lighting changes, textureless surfaces, and out-of-distribution inputs. Still, ARTDECO points toward a future where immersive, navigable 3D worlds can be created on the fly, transforming AR/VR, robotics, and simulation pipelines.