首个面向流程图代码生成的评测基准，揭示大模型在此任务上的短板。
Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability
- 构建跨15种语言的流程图-代码配对数据集
- 13个多模态模型在三类流程图上表现均不理想
- 监督微调显著提升生成效果，适合研究代码生成的新方向
尽管大型语言模型在代码生成方面展现出潜力，但现有评测基准忽略了基于流程图的代码生成。为此，本文提出Flow2Code，一个全新的流程图代码生成评估基准。该数据集涵盖15种编程语言，包含5,622段代码与16,866张三种类型的流程图：代码、UML和伪代码。对13个多模态大模型的大量实验表明，当前模型尚无法完美基于流程图生成代码。此外，实验结果还显示，监督微调技术对模型性能提升具有显著作用。代码与数据集已公开，地址为https://github.com/hml-github/Flow2Code。
原文摘要 · Abstract (English)
While large language models (LLMs) show promise in code generation, existing benchmarks neglect the flowchart-based code generation. To promote further research on flowchart-based code generation, this work presents Flow2Code, a novel benchmark for flowchart-based code generation evaluation. The evaluation dataset spans 15 programming languages and includes 5,622 code segments paired with 16,866 flowcharts of three types: code, UML, and pseudocode. Extensive experiments with 13 multimodal LLMs reveal that current LLMs can not generate code based on flowcharts perfectly. Besides, experiment results show that the supervised fine-tuning technique contributes greatly to the models' performance. We publicly release our code and datasets at https://github.com/hml-github/Flow2Code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。