A Circular Window-based Cascade Transformer for Online Action Detection
1School of Control Science and Engineering, Shandong University
2Meituan
Abstract
Online Action Detection (OAD) aims to identify ongoing actions in untrimmed video streams, which is critical for real-world applications. However, effectively modeling long-term temporal dependencies while maintaining real-time inference efficiency remains challenging. To address this challenge, we propose a circular window-based cascade Transformer~(CWC-Trans), featuring three key components: (i) a window-based hierarchical architecture that captures fine-grained temporal dynamics, and a circular window updating mechanism that enables efficient long-term history modeling; (ii) an auxiliary offline action segmentation (OAS) task that guides the hierarchical architecture to learn discriminative long-term action patterns while suppressing background interference; and (iii) a cascade prediction refinement module with adaptive kernels that progressively refines action predictions. By sharing the classifier and refinement modules between OAD and OAS tasks, CWC-Trans integrates multi-level long- and short-term information across feature, probabilistic, and refined probabilistic levels. Extensive experiments on THUMOS'14, TVSeries, and HDD demonstrate that CWC-Trans achieves state-of-the-art performance with high inference efficiency.
Overview
In traditional sequential updating, when new frames arrive and old frames are evicted, all frames must be recomputed to obtain updated contextual representations, resulting in high computational cost. In contrast, our circular window updating strategy restructures the queue by connecting its two ends. We define the circular window as the specific window containing the oldest frame in long-term memory. At each timestep, only this circular window is updated and requires recomputation, while all other windows remain unchanged and their representations can be directly reused. This strategy enables efficient incremental updates with computational cost limited to a single window, making it well-suited for online inference.
Pipeline
The proposed approach comprises four main components. First, the Circular Window-based History Encoder (CWHE) employs a window-based hierarchical attention mechanism to efficiently compress long-term historical information. The resulting compact representations are then processed by the Long-term History Decoder (LHD), which reconstructs the long-term context and provides supervision via an auxiliary Online Action Segmentation (OAS) task, thereby enhancing the quality of temporal feature compression. In parallel, the Short-term Trend Encoder (STE) captures recent temporal trends and integrates them with the compressed long-term tokens to facilitate the identification of ongoing actions for Online Action Detection (OAD). Finally, the Cascade Prediction Refinement (CPR) module refines the coarse prediction logits produced by both the OAD and OAS branches, yielding the final, high-quality action predictions.