首个可控多模态代码库问题定位基准,验证视觉证据是否真正有用。
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

- 构建可控多模态基准,分离视觉与文本输入影响。
- 最强模型仅达38.96%文件定位准确率,显示仍严重依赖文本。
- 适合研究多模态代码理解、视觉证据利用的学者使用。
真实代码库问题常包含截图、错误弹窗、界面状态和日志等视觉证据,但现有代码库级问题定位评估多为纯文本任务。现有多模态软件工程基准评估端到端修复,将定位与补丁生成混淆,无法判断视觉输入是否起作用。本文提出MM-IssueLoc,一个可控的多模态基准与评估协议,用于带视觉证据的代码库级定位。该数据集涵盖23种编程语言的652个问题-合并请求实例,标注7类图像和4级相关性,提供文件级与函数级真值标签,并支持文本仅与含图对比评估。通过基于视觉内容编码(VCE)的诊断方法,将图像转化为结构化文本证据。我们评估了基于大模型与检索的系统,包括控制型多模态检索器MM-IssueLoc-VL-Emb。结果表明,当前系统在多模态定位上仍不可靠:最强代理达到38.96% Acc@5(文件级)与22.45% Acc@10(函数级),最强检索器达33.86% Acc@10。跨基准比较显示,高文本主导基准上的定位表现不能直接迁移到多模态场景。MM-IssueLoc将视觉证据变为可测量变量,使未来工作能明确检验系统是否真正利用视觉信息进行定位,而非依赖文本线索或下游补丁生成效应。
原文摘要 · Abstract (English)
Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。