Unsupervised temporal action localization asks a model to identify which actions occur in an untrimmed video and when, without any labeled annotations. Existing approaches typically derive all supervision from RGB frames and optical flow, which often leads to noisy action categories and imprecise temporal boundaries. The new CoDy framework argues that video codecs hold useful complementary information that has been overlooked.
The authors observe that group-of-pictures quantization patterns in compressed video preserve category-discriminative dynamics, while motion-vector and residual variations respond strongly around action transitions. CoDy integrates this codec-enhanced view with RGB and flow through confidence-aware consensus clustering to generate soft pseudo-labels. It also converts codec dynamics into a class-agnostic boundary prior that supervises visual boundary learning and guides proposal initialization and iterative refinement.
Experiments on THUMOS14 and ActivityNet v1.3 show that CoDy consistently outperforms existing unsupervised temporal action localization methods and reaches state-of-the-art performance. Because the codec branches are active only during training, the approach introduces no extra codec-side computation at inference. The results position codec signals as an effective annotation-free prior for jointly inferring which actions occur and when they occur.