Real-Time Video Anomaly Detection Using YOLO Pose Estimation and CLIP-Based Semantic Scoring
A lightweight two-stage framework for real-time video anomaly detection is proposed. The first stage uses YOLO v11n-pose to detect persons and extract seventeen skeletal keypoints in a single forward pass. The second stage encodes each cropped person region through CLIP ViT-B/32 and computes cosine similarity against predefined textual descriptions of anomalous behaviors. The architecture eliminates the need for optical flow, standalone pose estimators, and density-based scoring modules. Experiments on CUHK Avenue, ShanghaiTech Campus, and a custom indoor dataset collected at Chulalongkorn University demonstrate an end-to-end throughput of approximately 51 FPS on an NVIDIA Titan XP GPU, a 3.36x speedup over the multi-feature baseline, while maintaining frame-level AUROC values of 89.26%, 70.26%, and 84.13%, respectively.
Researchers propose a two-stage framework for real-time video anomaly detection. The first stage uses YOLO v11n-pose for person detection and 17-keypoint pose extraction. The second stage uses CLIP ViT-B/32 to encode person crops and compute cosine similarity against textual descriptions of anomalous behaviors. The approach removes optical flow, standalone pose estimators, and density-based scoring. On CUHK Avenue, ShanghaiTech Campus, and a custom Chulalongkorn University indoor dataset, the system achieves about 51 FPS on an NVIDIA Titan XP GPU, a 3.36x speedup over a multi-feature baseline, with frame-level AUROC of 89.26%, 70.26%, and 84.13% respectively.
The framework combines YOLO v11n-pose for efficient pose extraction with CLIP ViT-B/32 for semantic anomaly scoring, achieving real-time performance (51 FPS) on a single GPU. The 3.36x speedup over a multi-feature baseline suggests that replacing optical flow and density-based scoring with CLIP-based similarity reduces computational overhead while maintaining competitive AUROC on standard benchmarks. The use of predefined textual descriptions for anomalies enables zero-shot adaptation to new anomaly types without retraining.
Real-time video anomaly detection at 51 FPS on a single GPU could enable deployment in surveillance and safety monitoring systems where low latency is critical. The elimination of optical flow and standalone pose estimators simplifies the pipeline, potentially reducing integration complexity and hardware requirements. The custom indoor dataset from Chulalongkorn University indicates interest in campus or indoor security applications.
The framework offers a lightweight, real-time anomaly detection solution that could be integrated into existing surveillance systems with modest GPU hardware. The 3.36x speedup over a multi-feature baseline may translate to lower infrastructure costs or higher camera density per server. The zero-shot semantic scoring via CLIP could reduce the need for large labeled anomaly datasets, lowering development costs for new environments.
Next observable signals include publication of the full paper, release of code or pretrained models, and evaluation on additional datasets such as UCF-Crime or XD-Violence. Further speedups may be achieved by using smaller CLIP variants or distillation. The approach could be extended to multi-camera or edge deployment scenarios.