构建农业视觉定位大模型评测基准，支持多目标与无目标场景。
AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding
- 将农业视觉定位建模为广义集合预测任务，统一处理多目标与无目标情况。
- 涵盖10,071张图像、6类目标，支持边界框与实例掩码两种定位方式。
- 揭示当前模型在多目标定位与掩码匹配上仍存显著差距，适合农业AI研究者使用。
视觉定位任务旨在根据自然语言描述定位图像中的物体，是农业人工智能系统的基础能力，可支持选择性除草、病害监测和精准采收等应用。由于农业目标常小、重复、遮挡或形状不规则，且语言指令可能指代一个、多个或不存在的目标，可靠评估该能力需同时考察定位精度、目标集完整性及是否存在意识的放弃判断。为此，我们提出 extbf{AgroVG}——一个多源基准，将农业视觉定位定义为广义集合预测：给定图像和指代表达，模型需返回所有匹配目标实例，或在无目标时放弃。AgroVG包含来自十个数据源、六类目标（作物/杂草、果实、小麦穗、害虫、植物病害、树冠）的10,071个图像-查询对。支持所有六类的边界框定位（T1），以及具备像素级实例标注源的实例掩码定位（T2），查询覆盖单目标、多目标和无目标三种情形。基准还提供针对框集匹配和查询级掩码覆盖率的任务专用协议。对26种模型配置（包括闭源多模态大模型、开源视觉语言模型和专用定位系统）的零样本评估显示，多目标Set-$F_1$最高仅达0.35，正向查询掩码成功率（[email protected]）仍低于0.17。数据与代码已公开于https://anonymous.4open.science/r/AgroVG-5172/。
原文摘要 · Abstract (English)
Visual grounding, the task of localizing objects described by natural-language expressions, is a foundational capability for agricultural AI systems, enabling applications such as selective weeding, disease monitoring, and targeted harvesting. Reliable evaluation of agricultural visual grounding remains challenging because agricultural targets are often small, repetitive, occluded, or irregularly shaped, and instructions may refer to one, many, or no objects in an image. Evaluating this capability therefore requires jointly testing localization accuracy, target-set completeness, and existence-aware abstention. To address these challenges, we introduce \textbf{AgroVG}, a multi-source benchmark that formulates agricultural grounding as generalized set prediction: given an image and a referring expression, a model must return all matching target instances or abstain when no target is present. AgroVG contains 10{,}071 annotation-grounded image-query pairs from ten source datasets across six target families: crop/weed, fruit, wheat head, pest, plant disease, and tree canopy. It supports bounding-box grounding (T1) across all six families and instance-mask grounding (T2) on sources with reliable instance-level pixel annotations, with queries covering single-target, multi-target, and target-absent regimes. AgroVG further provides task-specific protocols for box-set matching and query-level mask coverage. Zero-shot evaluation of 26 model configurations spanning closed-source MLLMs, open-source VLMs, and specialized grounding systems reveals persistent gaps: the best multi-target Set-$F_1$ reaches only 0.35, and the best positive-query mask success rate at [email protected] remains below 0.17. Data and code are available at https://anonymous.4open.science/r/AgroVG-5172/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。