LLM答硬件代码问题常啰嗦冗余,新方法让回答更精准简洁。
When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering

- 用多智能体框架减少冗余答案和无用描述
- 实验显示核心答案质量提升0.96分(5分制)
- 适合芯片设计工程师和需要高精度代码问答的场景
大型语言模型(LLMs)在硬件描述语言(HDL)问答中的应用日益广泛。由于HDL最终会被合成到物理电路中,不准确或冗余的回答可能引发时序违规或无法综合的逻辑,影响深远。然而,当前对LLM生成答案的质量,特别是与人类专家相比的情况,尚不明确。为此,我们从Stack Overflow收集了6,246条带已接受答案的HDL问答,并按四类(概念、调试、生成、优化)十小类进行分类整理。通过19名具有1至3年经验的HDL工程师参与的用户研究发现:LLMs存在普遍“过度回答”现象——65.7%的回答包含冗余选项,69.1%有冗长填充内容;近半数(49.0%)未完全匹配专家答案,但因可读性好仍被偏好(58.3%)。基于此,我们提出一种多智能体框架改进答案质量。采用“大模型为裁判”及两项结构化指标(核心答案数量、非核心内容长度)评估。在四种主流大模型上测试,核心答案质量得分从3.71升至4.67(+0.96),非核心内容质量从3.72升至4.23(+0.51)。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) has led practitioners to increasingly rely on them for answering questions about hardware description languages (HDLs). Because HDL is ultimately synthesized into physical hardware, an imprecise or redundant answer can propagate into timing violations or non-synthesizable logic that surface only late in the design flow, making the quality of HDL answers especially consequential. However, the quality of LLM-generated responses, particularly in comparison with answers provided by human experts, remains unclear. To investigate this question, we collect 6,246 HDL Q&A posts with accepted answers from Stack Overflow and curate them into a dataset, organized into a taxonomy of four main categories (Conceptual, Debugging, Generation, and Optimization) and ten subcategories. Using this dataset, we design a user study conducted with 19 HDL engineers with one to three years of experience. Our findings reveal a pervasive over answering tendency: LLMs supply correct content but bury it under redundant alternatives (65.7%) and verbose padding (69.1%), while nearly half of answers (49.0%) fail to fully align with expert answers yet participants still preferred LLM responses for readability (58.3%). Motivated by these findings, we propose a multi-agent framework for improving LLM-based HDL question answering. We evaluate answer quality using an LLM-as-Judge and two structural metrics: the number of core answers, which reflects redundancy since LLMs often provide multiple alternative solutions, and the length of non-core content, which reflects verbosity. Evaluated on the four mainstream LLMs, our framework increases the average core-answer quality score from 3.71 to 4.67 (+0.96) and the non-core content quality from 3.72 to 4.23 (+0.51), on a five-point scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。