Fine-grained alignment between text and video object trajectories remains a core challenge in multimodal understanding. According to a new arXiv paper, current embedding models perform well at global retrieval—matching whole videos to whole sentences—but their single global representation entangles multiple objects and their movements. This makes it difficult to capture precise relationships described in text, such as which object moves where and when.
The paper proposes VTrajEmbed, a method that introduces spatiotemporal embeddings designed specifically to align text with video object trajectories. Rather than compressing an entire video into one vector, VTrajEmbed aims to preserve the distinct paths of objects over time and link them to corresponding textual descriptions. The approach is positioned as a step beyond global retrieval toward more granular multimodal understanding.
Because the abstract is truncated, details on the model architecture and evaluation are not available from this source alone. The paper's contribution, as stated, is in addressing the entanglement problem inherent in existing embedding models, potentially improving tasks that require tracking and reasoning about individual objects in video. No comparisons to other methods are given in the provided text, so the claims should be read as the authors' own framing.{