用智能代理动态组合检测流程,提升复杂场景下目标检测性能
Detect in Any Scene: An Agentic Framework for Object Detection with Experience-Aware Reasoning

- 以多模态大模型为中枢,动态选择图像修复和专用检测模块
- 在六个基准上平均提升28.36%的F1分数,暗光场景最高提升37.01%
- 通过积累决策经验实现推理时自适应,适合动态真实场景应用
真实场景中的目标检测因图像退化多样性和对象分布异质性而面临挑战,现有检测器泛化能力受限。传统方法依赖预设条件,缺乏对动态环境的适应性。本文提出DetAS框架,将检测视为动态决策过程:以多模态大语言模型(MLLM)为核心代理,从修复模块与专用检测器工具箱中动态组合检测流程。该框架包含自适应图像修复(决定是否及如何增强图像)与多专家检测(融合多个领域专用检测器并进行实例级推理)。为进一步提升细粒度条件下的决策质量,引入自进化经验采集机制,构建DetAS-X,利用少量标注数据积累节点级决策经验,实现推理时的经验感知推理。系统可逐步优化决策策略,适应多样化现实场景。在六个挑战性基准上的实验证明,DetAS-X显著优于现有基于MLLM的检测器,平均F1提升28.36%,在DarkFace数据集上最高提升37.01%,验证了代理式检测的潜力,并为复杂动态环境中的应用奠定基础。
原文摘要 · Abstract (English)
Object detection in real-world scenarios remains challenging due to diverse image degradations and heterogeneous object distributions, which significantly hinder the generalization of existing detectors. Conventional approaches, including scene-specific representation learning and end-to-end pipeline design, are inherently limited by their reliance on predefined conditions and lack adaptability to dynamic environments. In this paper, we propose DetAS, an agentic detection framework that formulates object detection as a dynamic decision process. Instead of relying on static pipelines, DetAS leverages a Multimodal Large Language Model (MLLM) as a central agent to adaptively compose detection workflows by selecting from a toolbox of restoration modules and specialized detectors. Specifically, DetAS consists of two key components: Self-Adaptive Image Restoration, which dynamically determines whether and how to enhance images for downstream detection, and Multi-Expertise Detection, which integrates multiple domain-specialized detectors and resolves their predictions through instance-level reasoning. To further improve decision quality under fine-grained conditions, we introduce Self-Evolving Experience Harvesting and extend the framework to DetAS-X, which accumulates node-level decision experience from a small set of annotated data and enables experience-aware reasoning during inference. This mechanism allows the system to progressively refine its decision policy and adapt to diverse real-world scenarios. Extensive experiments on six challenging benchmarks demonstrate that DetAS-X significantly outperforms existing MLLM-based detectors, achieving an average improvement of 28.36% in F1 score, with up to 37.01% gain on DarkFace. These results demonstrate the promise of agentic detection and establish a solid foundation for its application in complex and dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。