Generating novel views from limited observations requires preserving both appearance and spatial structure. Most recent methods rely on fine-grained geometric cues but ignore high-level semantic and spatial properties such as object identity, attributes, and inter-object relationships. In a new arXiv paper, researchers ask whether 3D language understanding can provide complementary supervision for this task, and introduce a model called Ask4NV to test the idea.

Ask4NV is built on a pretrained unified multimodal model with a Mixture-of-Transformers architecture. The authors incorporate camera geometry into both the VAE and ViT representations, and during generation they fuse understanding features through joint self-attention with a modality-aware attention masking strategy. This lets the model answer spatial questions while also generating images from novel viewpoints.

Experiments show that the joint training approach yields strong image quality and geometric consistency in novel-view generation, alongside competitive 3D language understanding. Further analysis indicates that the combined training improves cross-view correspondence in VAE representations and increases similarity in a 3D-aware feature space. The paper provides evidence that language-guided spatial reasoning can serve as a useful auxiliary signal for view synthesis, rather than a distraction from it.