提出评估大模型代码生成稳定性的新框架,揭示正确代码间的性能差异。
Dynamic Stability of LLM-Generated Code
- 基于指令码分布设计静态与动态偏差度量,捕捉算法结构和运行行为差异。
- 实验证明顶尖模型在正确输出中仍存在显著算法变异性,且温度越高越不稳定。
- 适合关注代码质量、工程落地或模型评估的开发者与研究者参考。
当前大模型代码生成评估侧重功能正确性,忽视了正确解之间在算法复杂度上的显著差异。例如,$O(n^2)$ 与 $O(n /log n)$ 的排序算法输出相似但运行成本迥异。这暴露出现有评估方法无法捕捉正确解之间的行为与性能多样性。为此,我们提出一个评估生成代码动态稳定性的系统性框架,引入两个基于指令码分布的度量:静态规范轨迹分歧(SCTD),衡量生成解间算法结构差异;动态规范轨迹分歧(DCTD),量化运行时行为变异。二者比值即行为表达因子(BEF)可作为诊断信号:当 BEF ≪ 1 时指示关键运行不稳定性,当 BEF ≫ 1 时反映功能冗余。在 BigOBench 与 CodeContests 上的实证显示,先进 LLM 在功能正确的输出中仍存在显著算法变异性。值得注意的是,提升采样温度虽提高 pass@1 率,却恶化稳定性,揭示了正确性与行为一致性间的未被认识的权衡——在广阔输出空间中搜索正确解会带来‘不稳定性代价’。研究呼吁在代码生成中引入稳定性感知目标,并建立包含渐近测试用例的新基准以实现更稳健的真实世界评估。
原文摘要 · Abstract (English)
Current evaluations of LLMs for code generation emphasize functional correctness, overlooking the fact that functionally correct solutions can differ significantly in algorithmic complexity. For instance, an $(O(n^2))$ versus $(O(n \log n))$ sorting algorithm may yield similar output but incur vastly different performance costs in production. This discrepancy reveals a critical limitation in current evaluation methods: they fail to capture the behavioral and performance diversity among correct solutions. To address this, we introduce a principled framework for evaluating the dynamic stability of generated code. We propose two metrics derived from opcode distributions: Static Canonical Trace Divergence (SCTD), which captures algorithmic structure diversity across generated solutions, and Dynamic Canonical Trace Divergence (DCTD), which quantifies runtime behavioral variance. Their ratio, the Behavioral Expression Factor (BEF), serves as a diagnostic signal: it indicates critical runtime instability when BEF $\ll$ 1 and functional redundancy when BEF $\gg$ 1. Empirical results on BigOBench and CodeContests show that state-of-the-art LLMs exhibit significant algorithmic variance even among functionally correct outputs. Notably, increasing sampling temperature improves pass@1 rates but degrades stability, revealing an unrecognized trade-off: searching for correct solutions in diverse output spaces introduces a "penalty of instability" between correctness and behavioral consistency. Our findings call for stability-aware objectives in code generation and new benchmarks with asymptotic test cases for robust, real-world LLM evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。