用大模型自动评估智能合约注释质量，比传统方法更准更快。
evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments
- 用40种提示策略+400种配置组合的LLM框架评估注释
- 提示设计影响评分与人工判断一致性，效果优于传统指标
- 适合开发人员优化注释生成工具，提升代码可维护性
智能合约注释生成有助于提升区块链代码的理解性和可维护性，但评估生成注释的质量仍具挑战。传统指标如BLEU和ROUGE无法捕捉领域特异性语义，而人工评估成本高且难以扩展。本文提出\texttt{evalSmarT}，一个基于大语言模型（LLMs）的模块化、可扩展评估框架。该系统通过约40个LLM与10种提示策略组合，支持超过400种评估配置。我们展示了其在基准测试注释生成工具及筛选最有效输出中的应用。结果表明，提示设计显著影响评估结果与人工判断的一致性，基于LLM的评估为现有方法提供了可扩展且语义丰富的替代方案。
原文摘要 · Abstract (English)
Smart contract comment generation has gained traction as a means to improve code comprehension and maintainability in blockchain systems. However, evaluating the quality of generated comments remains a challenge. Traditional metrics such as BLEU and ROUGE fail to capture domain-specific nuances, while human evaluation is costly and unscalable. In this paper, we present \texttt{evalSmarT}, a modular and extensible framework that leverages large language models (LLMs) as evaluators. The system supports over 400 evaluator configurations by combining approximately 40 LLMs with 10 prompting strategies. We demonstrate its application in benchmarking comment generation tools and selecting the most informative outputs. Our results show that prompt design significantly impacts alignment with human judgment, and that LLM-based evaluation offers a scalable and semantically rich alternative to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。