arXiv:2604.15663cs.SEcs.AI2026-04

让代码搜索同时理解文字、代码和图像,提升智能编程的准确性和可靠性。

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

论文配图:CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
图 1 · 摘自论文原文
  • 通过指令对齐将文本、代码和图像统一映射到同一语义空间。
  • 在五个视觉领域上平均比基线高出10个nDCG@10点。
  • 适合需要跨模态代码检索的开发者和大模型应用研究者。

代码搜索作为信息检索任务,支撑现代软件工程并推动检索增强生成(RAG)发展,提升代码发现、复用与大模型编程的可靠性。然而现有代码检索模型仍以文本为中心,常忽略程序产物中的视觉与结构特征,如网页界面、数据可视化、SVG、示意图和UML图。为此,我们提出首个涵盖五个视觉领域、八种编程语言、十一个库的多模态代码检索基准MMCoIR,系统评估该任务挑战。在此基础上,我们提出CodeMMR,一种联合嵌入自然语言、代码和图像的统一检索模型,通过指令式多模态对齐实现共享语义空间。CodeMMR在跨模态与跨语言任务中表现优异,平均比基线(如UniIR、GME、VLM2Vec)高出10个nDCG@10点。将其集成至RAG后,显著提升代码生成的准确性与视觉一致性,验证了多模态检索在下一代智能编程系统中的核心潜力。数据集已开源于HuggingFace。

原文摘要 · Abstract (English)

Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace.

多模态检索代码生成RAG视觉编程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。