LLM生成的正确代码运行时内存差异大，可能隐藏风险。
Correctness isnt Efficiency: Runtime Memory Divergence in LLM-Generated Code
- 用动态时间规整分析内存使用轨迹，量化不同正确解的稳定性
- 高温度采样虽提升通过率，但加剧内存行为不稳
- 稳定性与代码复杂度相关，适合关注运维安全的开发者
大型语言模型（LLMs）生成的程序可通过单元测试，但通过测试并不保证运行时可靠。我们发现同一任务的不同正确解在内存和性能模式上存在显著差异，可能引发隐藏的运行风险。为此，我们提出一个框架，用于衡量多个正确生成结果之间的执行时内存稳定性。在解级别，引入动态均值成对距离（DMPD），通过将内存使用轨迹转换为单调峰轮廓（MPP）并利用动态时间规整进行比较，以降低瞬态噪声影响。将DMPD在任务间聚合得到模型级不稳定性评分（MIS）。在BigOBench和CodeContests上的实验显示，正确解之间存在显著的运行时内存差异。不稳定性常随采样温度升高而增加，即使pass@1提升亦如此。我们还观察到稳定性指标与认知复杂度、环路复杂度等软件工程指标存在关联，暗示运行行为与可维护性之间的联系。结果支持在CI/CD中采用稳定性感知的候选选择，可在不牺牲正确性的前提下降低运行风险。相关资源已公开。
原文摘要 · Abstract (English)
Large language models (LLMs) can generate programs that pass unit tests, but passing tests does not guarantee reliable runtime behavior. We find that different correct solutions to the same task can show very different memory and performance patterns, which can lead to hidden operational risks. We present a framework to measure execution-time memory stability across multiple correct generations. At the solution level, we introduce Dynamic Mean Pairwise Distance (DMPD), which uses Dynamic Time Warping to compare the shapes of memory-usage traces after converting them into Monotonic Peak Profiles (MPPs) to reduce transient noise. Aggregating DMPD across tasks yields a model-level Model Instability Score (MIS). Experiments on BigOBench and CodeContests show substantial runtime divergence among correct solutions. Instability often increases with higher sampling temperature even when pass@1 improves. We also observe correlations between our stability measures and software engineering indicators such as cognitive and cyclomatic complexity, suggesting links between operational behavior and maintainability. Our results support stability-aware selection among passing candidates in CI/CD to reduce operational risk without sacrificing correctness. Artifacts are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。