让AI同时理解代码文字和流程图，生成更准确的程序。
Multilingual Multimodal Software Developer for Code Generation
- 结合文本指令与UML图/流程图，提升代码生成准确性。
- 在新基准测试中，模型对视觉信息理解仍有明显不足。
- 适合需要图文协同开发的工业级编程场景。
大型语言模型（LLMs）显著提升了代码生成能力，但多数模型仍仅支持文本输入，忽略了实际开发中常用的图表等视觉辅助工具。为弥补这一差距，我们提出MM-Coder——一种多语言多模态软件开发者。它将统一建模语言（UML）图和流程图（统称视觉工作流）与文本指令融合，以提高代码生成的准确性和架构一致性。为此，我们构建了MMc-Instruct数据集，涵盖基于视觉工作流的代码生成任务，使MM-Coder能像人类开发者一样整合图文信息，区别于以往专注于单一任务的研究。此外，我们提出了MMEval新基准，用于评估多模态代码生成，突破了现有仅依赖文本评价的局限。在该基准上的评估显示，当前模型在精确捕捉视觉信息、遵循指令以及掌握高级编程知识方面仍存在显著挑战。本研究旨在通过让LLMs理解并实现包含文本与视觉设计的复杂规范，推动工业级编程的变革。
原文摘要 · Abstract (English)
The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and flowcharts used in real-world software development. To bridge this gap, we introduce MM-Coder, a Multilingual Multimodal software developer. MM-Coder integrates visual design inputs-Unified Modeling Language (UML) diagrams and flowcharts (termed Visual Workflow)-with textual instructions to enhance code generation accuracy and architectural alignment. To enable this, we developed MMc-Instruct, a diverse multimodal instruction-tuning dataset including visual-workflow-based code generation, allowing MM-Coder to synthesize textual and graphical information like human developers, distinct from prior work on narrow tasks. Furthermore, we introduce MMEval, a new benchmark for evaluating multimodal code generation, addressing existing text-only limitations. Our evaluations using MMEval highlight significant remaining challenges for models in precise visual information capture, instruction following, and advanced programming knowledge. Our work aims to revolutionize industrial programming by enabling LLMs to interpret and implement complex specifications conveyed through both text and visual designs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。