Solving the Tetris of Interior Design: How CASAGPT Uses Cuboids for Collision-Free Scene Synthesis

Introduction

Imagine you are trying to furnish a virtual apartment. You place a stylish L-shaped sofa in the corner and a coffee table in the nook of the “L”. To you, this is a perfect, cozy arrangement. But to a computer using traditional 3D understanding, you might have just caused a catastrophe.

Why? Because many computer vision models view objects not as complex shapes, but as simple bounding boxes. If the bounding box of the table touches the bounding box of the sofa—even if the actual objects don’t touch—the computer screams “Collision!” Conversely, models might generate scenes where objects physically overlap in reality because the coarse boxes allowed it.

This is the central challenge in automated interior design: The Bounding Box Problem.

In this post, we are diving into CASAGPT, a fascinating paper that proposes a solution to this geometric headache. Instead of treating furniture as simple boxes, the researchers decompose objects into “cuboid assemblies”—think of them as LEGO constructions. By teaching a Large Language Model (specifically Llama-3) to arrange these cuboids, they achieve scene synthesis that is physically plausible, compact, and free of frustrating intersections.

The Problem with Boxes

To understand why CASAGPT is necessary, we first need to look at how computers traditionally “see” a room.

In most data-driven interior design methods, a 3D object (like a chair) is represented by a Bounding Box—the smallest rectangular box that fully encloses the object. The model predicts the size, position, and rotation of these boxes.

While computationally efficient, this approach fails to capture the true geometry of furniture.

1. The “Empty Space” Issue: An L-shaped sofa has a lot of empty space inside its bounding box. A coffee table should be able to sit in that space. A bounding box model prohibits this, forcing the table to be placed awkwardly far away.
2. Intersection Ambiguity: As shown in Figure 1 above, a compact arrangement often results in overlapping bounding boxes (red squares). If we train a model to avoid box overlaps, we get sparse, unrealistic rooms. If we allow overlaps, the model often accidentally puts the table inside the sofa.

The researchers behind CASAGPT asked a critical question: Can we represent objects in a way that is simple like a box, but accurate like a mesh?

The Solution: Cuboid Arrangement and Scene Assembly

The core innovation of CASAGPT is shifting from single bounding boxes to Cuboid Assemblies. Instead of one big box, an object is represented by a collection of smaller cuboids that approximate its shape.

1. From Mesh to Cuboids

Before the model can learn to arrange furniture, the data must be processed. The researchers take high-fidelity 3D meshes and transform them into simplified cuboid representations through a three-step process:

1. Voxelization: The 3D mesh is converted into a grid of occupied and empty spaces (voxels).
2. Coarse-Graining: Adjacent occupied voxels are merged to form rough shapes.
3. Cuboid Merging: A greedy algorithm iteratively merges these shapes into larger cuboids based on a volume threshold.

This process transforms a complex chair mesh into a set of clean, rectangular blocks that preserve the object’s structure without the computational weight of a full mesh.

This granularity is the magic key. Now, the model knows that the “empty space” in the L-shaped sofa is actually empty, because there are no cuboids there.

2. The CASAGPT Architecture

With the objects converted into cuboids, the problem becomes a sequence generation task. The researchers treat interior design essentially as a language problem.

The model is built on the Llama-3 architecture. It functions as an autoregressive transformer, meaning it predicts the next piece of furniture based on what has already been placed in the room.

The scene is fed into the model as a sequence of tokens:

• Floor Token: Defines the room’s boundary.
• Entity Tokens: Represent the center, rotation, and class of an object (e.g., “This is a chair at position X,Y”).
• Cuboid Tokens: Represent the specific geometric blocks that make up that object.

As shown in Figure 2(a), the Transformer Decoder processes these tokens to generate a full room layout. Because it generates the specific cuboids, it “knows” the physical shape of the object it is placing.

3. Rejection Sampling: Refining the Output

Generative models are probabilistic—sometimes they make mistakes. Even with the cuboid representation, the model might occasionally place a lamp inside a wall or overlap two chairs.

To counter this, CASAGPT implements a technique called Rejection Sampling during the fine-tuning phase (Figure 2b).

1. Generate: The model proposes a batch of room layouts.
2. Inspect: The system calculates the Intersection-over-Union (IoU) of the cuboids. Since the cuboids accurately represent the shape, this check is precise.
3. Filter: Scenes with physical collisions are rejected. Scenes with clean arrangements are kept.
4. Train: The model is fine-tuned on the “accepted” scenes.

This creates a positive feedback loop. The model generates, filters out its own bad habits, and learns from its successes, progressively becoming better at avoiding collisions.

4. Object Retrieval

Once the model generates a scene of cuboids, we need to visualize it with real 3D furniture. This is where the cuboid method shines again.

In traditional methods, the system looks for a 3D model that fits the predicted bounding box. This is error-prone. As seen in Figure 4 (b) below, matching by bounding box can retrieve an object that physically collides with its neighbors because the “box” fit, but the shape didn’t.

CASAGPT uses Cuboid-based Retrieval. It compares the generated cuboid assembly with the voxelized database of furniture. It finds the specific chair or table that matches the structure of the generated blocks. This ensures that the final rendered scene (Figure 4c) is as collision-free as the generated plan.

Cleaning the Data: The 3DFRONT-NC Dataset

Garbage in, garbage out. This rule applies heavily to AI. The researchers discovered that the standard dataset used for this task, 3D-FRONT, was actually full of errors. Human designers had left objects intersecting in physically impossible ways.

If the training data has collisions, the model will learn to generate collisions.

To fix this, the authors introduced 3DFRONT-NC (Noise Clean). They used their cuboid collision detection logic to identify overlapping objects in the original dataset and automatically nudged them apart to resolve intersections.

The difference is stark. In Figure 5 below, you can see how the “Nc” (Noise Clean) dataset adjusts positions slightly to ensure physics is respected, without destroying the original intent of the layout.

Experiments and Results

Does adding all this geometric complexity actually pay off? The results suggest a resounding yes.

Qualitative Comparison

Visually, CASAGPT produces scenes that are much more logical than competitors like LayoutGPT, ATISS, or DiffuScene.

In Figure 6, observe the bedrooms. Previous methods (LayoutGPT, ATISS) often create sparse rooms or clutter objects together in ways that would make walking impossible. CASAGPT (fourth column) creates coherent layouts where nightstands hug the bed properly and wardrobes are accessible, closely mimicking the human-designed references.

Quantitative Metrics

The researchers used several metrics to evaluate performance, including FID (Fréchet Inception Distance) to measure realism and NIRate (Non-Intersection Rate) to measure physical plausibility.

While specific tables are detailed in the paper, the summary is that CASAGPT consistently outperforms state-of-the-art methods in NIRate, meaning it generates the fewest collisions.

A particularly interesting finding came from their Ablation Study (Figure 8 below). They compared training their model using Bounding Boxes versus Cuboids.

• Left (BBox rep): When the model only knows about boxes, it struggles to place chairs under tables or furniture near walls without leaving large, unnatural gaps.
• Right (Cuboid rep): With cuboids, the model understands it can slide a chair under a table because the table’s “cuboids” (legs) don’t block the chair’s “cuboids” (seat).

Furthermore, the Rejection Sampling process proved crucial. As shown in the table included in Figure 8 (Table 3), with every iteration of rejection sampling (iter1, iter2, iter3), the intersection rate dropped significantly, proving that the model was successfully “learning from its mistakes.”

Conclusion and Future Implications

CASAGPT represents a significant step forward in 3D scene synthesis. By moving away from the crude approximation of bounding boxes and embracing a finer-grained cuboid representation, the model achieves a level of spatial awareness previously out of reach for autoregressive transformers.

Key Takeaways:

1. Geometry Matters: Simple bounding boxes are insufficient for complex interior design tasks where “packing” is essential.
2. Data Quality: The creation of 3DFRONT-NC highlights how noisy standard datasets can be and how fixing them improves model performance.
3. Iterative Refinement: The rejection sampling loop allows the model to self-correct, bridging the gap between probabilistic generation and physical constraints.

While the model still faces challenges—such as occasionally placing objects slightly off the floor or struggling with extremely constrained floor plans—it offers a promising direction. Future work may combine this structural understanding with diffusion models for even higher fidelity, or use reinforcement learning to punish collisions even more effectively during training.

For now, CASAGPT has successfully taught computers that fitting a square peg in a round hole isn’t just a figure of speech—it’s bad interior design.