首个面向多语言硬件代码的仓库级补全基准，填补了硬件设计领域的空白。
MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
- 构建首个支持多语言硬件代码的仓库级补全数据集
- 覆盖三种主流硬件设计编码风格，含结构与语义标签
- 适合芯片设计、EDA工具开发人员研究模型补全能力
大型语言模型在通用编程语言的代码补全任务中表现优异。然而，现有仓库级代码补全基准几乎完全聚焦于软件代码，严重忽视了硬件描述语言。本文提出首个面向多语言硬件代码补全的仓库级基准——MHRC-Bench，包含训练集（MHRC-Bench-Train）和评测集（MHRC-Bench-Eval）。该基准覆盖三种主流硬件设计编码风格，每个补全目标均基于语法树分析标注了代码结构级与硬件语义标签。我们在评测集上对模型进行了全面评估，结果表明该基准有效支撑了对硬件代码补全性能的系统性分析。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved strong performance on code completion tasks in general-purpose programming languages. However, existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages. In this work, we present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark designed for multilingual hardware code completion at the repository level. Our benchmark targets completion tasks and covers three major hardware design coding styles. Each completion target is annotated with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. We conduct a comprehensive evaluation of models on MHRC-Bench-Eval. Comprehensive evaluation results and analysis demonstrate the effectiveness of MHRC-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。