用大模型实现自然语言描述的异常行为人员检索,精准捕捉细微异常线索。
AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search
- 分步式框架融合生成式知识与判别式检索,提升跨模态对齐精度。
- 零样本适配策略使模型在真实场景下准确率提升0.96% Recall@1。
- 适合关注视频监控中异常检测与可解释性分析的研究者使用。
随着公共安全需求增长,基于文本的人员异常搜索成为关键任务,旨在通过自然语言描述检索具有异常行为的个体。该任务面临两大挑战:(1)文本异常与视觉行为之间的细粒度跨模态对齐;(2)在真实世界样本稀疏条件下的异常识别。尽管大模型在多模态理解方面表现优异,但其在细粒度异常检索中的潜力尚未被充分挖掘,主要受限于生成式知识与判别式检索之间的领域差距,以及缺乏高效的部署适应策略。本文提出AnomalyLMM,首个利用大模型进行文本驱动人员异常搜索的框架。核心贡献包括:(1)一种新型粗到细的流水线,整合大模型以连接生成式世界知识与以检索为中心的异常检测;(2)一套无需训练的适配方案,包含掩码跨模态提示、行为显著性预测和知识感知重排序,实现零样本对微小异常线索的关注。作为首个探索大模型在该任务中应用的研究,我们在唯一公开可用的基准PAB数据集上进行了严格评估,该数据集涵盖多样化真实场景中的异常(如跌倒、碰撞、被击中)。实验表明所提方法有效,优于基线模型0.96% Recall@1。定性分析验证了文本异常与视觉行为之间可解释的对齐关系。代码与模型将公开,供后续研究使用。
原文摘要 · Abstract (English)
With growing public safety demands, text-based person anomaly search has emerged as a critical task, aiming to retrieve individuals with abnormal behaviors via natural language descriptions. Unlike conventional person search, this task presents two unique challenges: (1) fine-grained cross-modal alignment between textual anomalies and visual behaviors, and (2) anomaly recognition under sparse real-world samples. While Large Multi-modal Models (LMMs) excel in multi-modal understanding, their potential for fine-grained anomaly retrieval remains underexplored, hindered by: (1) a domain gap between generative knowledge and discriminative retrieval, and (2) the absence of efficient adaptation strategies for deployment. In this work, we propose AnomalyLMM, the first framework that harnesses LMMs for text-based person anomaly search. Our key contributions are: (1) A novel coarse-to-fine pipeline integrating LMMs to bridge generative world knowledge with retrieval-centric anomaly detection; (2) A training-free adaptation cookbook featuring masked cross-modal prompting, behavioral saliency prediction, and knowledge-aware re-ranking, enabling zero-shot focus on subtle anomaly cues. As the first study to explore LMMs for this task, we conduct a rigorous evaluation on the PAB dataset, the only publicly available benchmark for text-based person anomaly search, with its curated real-world anomalies covering diverse scenarios (e.g., falling, collision, and being hit). Experiments show the effectiveness of the proposed method, surpassing the competitive baseline by +0.96% Recall@1 accuracy. Notably, our method reveals interpretable alignment between textual anomalies and visual behaviors, validated via qualitative analysis. Our code and models will be released for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。