arXiv:2604.17102cs.ARcs.AI2026-04被引 1

调参比选模型更重要,同一模型最佳配置可提升25.5%通过率

Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation

论文配图:Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
图 1 · 摘自论文原文
  • 对3个主流模型做108种超参数组合测试,发现配置影响远超模型差异
  • 同一模型最优与最差配置间通过率差距达25.5%,是模型间差异的5倍
  • 不同评测集最优配置不通用,需针对任务和架构定制调参策略

针对开源大模型在硬件设计中的应用,现有评估多关注模型选择,而将推理时解码配置视为次要因素。本文表明,配置方式的重要性超过模型本身。在VerilogEval和RTLLM数据集上对26个开源LLM进行基准测试,并对三个代表性模型开展涵盖108种配置的超参数扫描。结果显示,同一模型在最佳与最差配置下的通过率差距最高达25.5%,这一差距是不同模型族在默认配置下平均差距的5倍。在两个评测套件上按Spearman相关系数排序配置,相关性接近零,表明最优配置不具备迁移性。这说明,默认超参数下的基准测试会混淆模型能力与配置影响。要释放开源LLM在RTL生成中的潜力,必须基于架构与评测任务进行针对性超参数选择,本研究提出的方法为此提供了支持。

原文摘要 · Abstract (English)

Benchmarking of open-source LLMs for hardware design focuses on which LLMs to use, while treating inference-time decoding configuration as a secondary concern. This work shows that it matters more how an LLM is configured than which model is selected. Benchmarking 26 open-source LLMs on VerilogEval and RTLLM with synthesis-in-the-loop evaluation, the study first maps the current capability landscape and then conducts an extensive 108-configuration hyperparameter sweep on three prominent models. The sweep reveals absolute pass-rate gaps of up to 25.5% between the best and worst settings for the same LLM, which is 5x larger than the average spread observed across various model families under their respective default configurations. Ranking all configurations by Spearman's $ρ$ across the two benchmark suites yields near-zero correlation, demonstrating that optimal configurations do not transfer. These results show that benchmarking conducted under default hyperparameters confounds model capabilities with configuration effects. Realizing the full potential of open-source LLMs for RTL generation requires architecture and benchmark aware hyperparameter selection, as enabled by the proposed methodology.

LLM硬件生成超参优化RTL

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。