Video scene graph generation asks a model to track objects in a video and label how their relationships change over time. A model must keep object identities stable while also noticing when a relationship flips—say, from "watching" to "talking." The authors of a new paper, accepted at ACCV 2026, argue that many current approaches overemphasize temporal consistency and therefore fail to update their predictions when a relationship actually changes. They call this failure "relational semantic inertia."

To counter this, they propose RTAformer, a one-stage framework that models entity persistence and relation changes separately. The model injects calibrated differences between adjacent-frame relation queries to guide predicate updates, and uses cross-frame instance alignment during training to reduce representational drift across frames.

Experiments on the Action Genome benchmark show that RTAformer improves recall and mean recall over existing methods. Ablation studies confirm the contributions of the proposed components, and comparisons using matched backbones show larger gains on relation-transition frames than on non-transition frames, suggesting the model is specifically addressing the inertia problem.