提出真实鸟类识别新基准，要求模型不会就弃权并给出理由。
RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs
- 设计可弃权的鸟类识别评测集，区分可答与不可答样本。
- 多模型在可答集上准确率低于13%，表现普遍不佳。
- 多数模型弃权时理由错误，需提升理性拒答能力。
野外细粒度鸟类物种识别常因单张图像信息不足而无法回答：关键线索可能非视觉（如鸣叫声），或受遮挡、拍摄角度、分辨率低等因素影响。然而当前多模态系统多在可回答、符合预设模式的情况下评估，鼓励盲目猜测而非合理弃权。本文提出RealBirdID基准：给定一张鸟图，系统应选择回答物种或弃权，并提供具体证据性理由，如‘需鸣叫声’‘图像质量差’或‘视角被遮挡’。每个属包含验证集，其中为未回答样本精心标注了理由，并配对清晰可答实例。我们发现：（1）多种开源及商用模型在可答集上表现极差，包括GPT-5和Gemini-2.5 Pro在内的多模态大模型准确率均低于13%；（2）分类能力强的模型未必更擅长对不可答样本合理弃权；（3）即使模型弃权，也常给出错误理由。RealBirdID为需弃权的细粒度识别设立了明确目标，并提供了可量化的进步路径。
原文摘要 · Abstract (English)
Fine-grained bird species identification in the wild is frequently unanswerable from a single image: key cues may be non-visual (e.g. vocalization), or obscured due to occlusion, camera angle, or low resolution. Yet today's multimodal systems are typically judged on answerable, in-schema cases, encouraging confident guesses rather than principled abstention. We propose the RealBirdID benchmark: given an image of a bird, a system should either answer with a species or abstain with a concrete, evidence-based rationale: "requires vocalization," "low quality image," or "view obstructed". For each genus, the dataset includes a validation split composed of curated unanswerable examples with labeled rationales, paired with a companion set of clearly answerable instances. We find that (1) the species identification on the answerable set is challenging for a variety of open-source and proprietary models (less than 13% accuracy for MLLMs including GPT-5 and Gemini-2.5 Pro), (2) models with greater classification ability are not necessarily more calibrated to abstain from unanswerable examples, and (3) that MLLMs generally fail at providing correct reasons even when they do abstain. RealBirdID establishes a focused target for abstention-aware fine-grained recognition and a recipe for measuring progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。