通过主动调控注意力头表征,提升冻结多模态大模型的视频异常检测能力。
Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection
- 识别关键注意力头作为异常专家,构建可动态调节的表征修正机制。
- 仅用1%训练数据即达当前免微调方法最优性能,显著提升对细微异常的识别率。
- 适合追求高效、低资源视频异常检测的研究者与工业应用落地场景。
视频异常检测(VAD)旨在识别视频中的异常事件。传统方法通常依赖大量标注数据并需完整训练,近期研究尝试在免微调前提下利用冻结多模态大语言模型(MLLMs)进行检测,但其性能受限于预训练固有偏见,难以适应特定视频上下文,导致对细微或模糊异常处理困难。为此,本文提出新干预框架SteerVAD,将被动阅读转为主动引导与修正内部表征。首先通过无梯度表示可分性分析(RSA)识别出最具判别力的顶部注意力头作为潜在异常专家(LAE),随后由分层元控制器(HMC)联合全局上下文与这些LAE输出生成动态修正信号。该信号对LAE表征流形执行定向各向异性缩放,增强异常相关维度,抑制固有偏见。在主流基准上的大量实验表明,本方法仅需1%训练数据即可达到免微调方法的最先进性能,确立了视频异常检测的新方向。代码将在发表后公开。
原文摘要 · Abstract (English)
Video anomaly detection (VAD) aims to identify abnormal events in videos. Traditional VAD methods generally suffer from the high costs of labeled data and full training, thus some recent works have explored leveraging frozen multi-modal large language models (MLLMs) in a tuning-free manner to perform VAD. However, their performance is limited as they directly inherit pre-training biases and cannot adapt internal representations to specific video contexts, leading to difficulties in handling subtle or ambiguous anomalies. To address these limitations, we propose a novel intervention framework, termed SteerVAD, which advances MLLM-based VAD by shifting from passively reading to actively steering and rectifying internal representations. Our approach first leverages the gradient-free representational separability analysis (RSA) to identify top attention heads as latent anomaly experts (LAEs) which are most discriminative for VAD. Then a hierarchical meta-controller (HMC) generates dynamic rectification signals by jointly conditioning on global context and these LAE outputs. The signals execute targeted, anisotropic scaling directly upon the LAE representation manifolds, amplifying anomaly-relevant dimensions while suppressing inherent biases. Extensive experiments on mainstream benchmarks demonstrate our method achieves state-of-the-art performance among tuning-free approaches requiring only 1% of training data, establishing it as a powerful new direction for video anomaly detection. The code will be released upon the publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。