Video models typically tokenize input by patchifying frames, which generates a large number of redundant tokens. That redundancy becomes a bottleneck for both efficiency and scalability, especially as video data grows in complexity.
The paper introduces TrajTok, a trajectory-based tokenizer that learns trajectory tokens as an alternative to patch-based tokens. By following motion paths rather than static patches, the tokenizer aims to capture video structure more compactly, reducing the token load while preserving the information needed for understanding.
The abstract reports that this approach leads to better video understanding, though details of the evaluation are not included in the summary. The work points toward trajectory tokenization as a promising direction for making video models more efficient without sacrificing performance.