Recent video world models have adopted 3D memory mechanisms to integrate historical observations into a lightweight, persistent scene representation. The authors of SceneAE ask whether such 3D representations can also serve directly as the latent space for reconstructing and generating scenes, rather than only for memorization.

SceneAE represents a scene as a compact, unstructured 3D-anchored latent space. Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, this representation describes the scene itself.

For reconstruction, once a scene is generated, the persistent 3D representation can be projected and decoded into RGB-D observations from different target cameras—no need to regenerate a video for each new camera trajectory. For generation, the method leverages geometry-content separation: it first generates camera-controlled 3D geometry, then synthesizes content conditioned on it, which reduces camera-appearance entanglement and improves camera controllability.

Experiments reported in the paper show that this 3D latent achieves superior generation quality, computational cost, and camera controllability compared to existing approaches. The work is available on arXiv as arXiv:2610.19708.