用抽象语法树熵值评估大模型代码生成稳定性。
Measuring LLM Code Generation Stability via Structural Entropy
- 基于抽象语法树子结构频率构建熵度量，分析代码生成一致性。
- 提出两种无参考、语言无关的稳定性指标，可区分控制流与变量名差异。
- 方法轻量高效，适合快速评估模型在真实开发中的可靠性。
评估大语言模型（LLMs）代码生成的稳定性对判断其在实际开发中的可靠性至关重要。本文将结构熵概念拓展至程序领域，结合抽象语法树（AST）分析：对于固定提示，收集每个生成程序中深度受限的子树多重集，并将其相对频率视为概率分布。通过两种互补方式衡量稳定性：(i) Jensen-Shannon 散度，一种对称且有界的结构重叠指标；(ii) 结构交叉熵比，用于揭示缺失的高概率模式。两类度量均提供仅结构或包含标记信息的变体，分别关注控制流形状和标识符级变化。相比 pass@k、BLEU 或 CodeBLEU，本方法无需参考答案、语言无关且不依赖执行。我们在多个主流代码生成任务上对领先 LLMs 进行基准测试，结果表明，基于 AST 的结构熵能揭示模型一致性和鲁棒性的细微差异。该方法时间复杂度为 O(n,d)，无需外部测试，是代码生成评估工具包中的轻量补充。
原文摘要 · Abstract (English)
Assessing the stability of code generation from large language models (LLMs) is essential for judging their reliability in real-world development. We extend prior "structural-entropy concepts" to the program domain by pairing entropy with abstract syntax tree (AST) analysis. For any fixed prompt, we collect the multiset of depth-bounded subtrees of AST in each generated program and treat their relative frequencies as a probability distribution. We then measure stability in two complementary ways: (i) Jensen-Shannon divergence, a symmetric, bounded indicator of structural overlap, and (ii) a Structural Cross-Entropy ratio that highlights missing high-probability patterns. Both metrics admit structural-only and token-aware variants, enabling separate views on control-flow shape and identifier-level variability. Unlike pass@k, BLEU, or CodeBLEU, our metrics are reference-free, language-agnostic, and execution-independent. We benchmark several leading LLMs on standard code generation tasks, demonstrating that AST-driven structural entropy reveals nuances in model consistency and robustness. The method runs in O(n,d) time with no external tests, providing a lightweight addition to the code-generation evaluation toolkit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。