用大模型生成代码可低成本提升软件可靠性，尤其跨语言组合效果更好。
Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study
- 用大模型在不同设置下生成多份代码，构建多样化程序集。
- 跨语言组合的代码对可靠性提升最显著，最高增益达1.7倍。
- 适合关注软件容错与自动化开发的研究者和工程师。
软件多样性被广泛研究以降低共模故障风险。经典研究表明，冗余组件的故障是否重叠决定了可靠性提升的幅度。传统方法因需多个实现及验证、维护、部署成本高而受限。大语言模型（LLMs）的兴起可能改变这一局面：它们能快速生成大量同一规范的不同实现，涵盖不同模型、解码温度和编程语言。本文将经典的软件多样性实证研究扩展至大模型生成代码，使用三个规范，对比历史人工编写代码与大规模大模型生成代码，在统一编译、沙箱环境和全量测试集下评估。探索了模型族、生成温度、编程语言等多维度的多样性。在1-出-2配置下评估可靠性，涵盖同质与异质程序组，包括同模型对、跨语言对，以及大模型与人工代码的混合配对。结果表明，尤其是异质设置中结合大模型生成代码可带来可靠性提升，但效果受编程语言和生成设置影响。总体表明，大模型提供了可扩展、低成本的程序多样性来源，可用于提升系统可靠性。
原文摘要 · Abstract (English)
Software diversity has been extensively studied as a means of reducing the risk of common-mode failures. Classic work showed that the central issue is whether failures of diversely redundant components overlap in ways that limit the reliability gains. Traditional software diversity is costly to obtain, since it requires multiple implementations as well as the corresponding validation, maintenance, and deployment effort. Recent advances in Large Language Models (LLMs) may change this. LLMs enable inexpensive code generation: they produce many candidate implementations of the same specification quickly, across different models, decoding settings, and programming languages. This raises a natural question: can LLMs serve as practical generators of software diversity, and how much reliability improvement can that diversity actually provide? In this paper, we extend classical empirical studies of software diversity in human-written programs to LLM-generated code. We study three specifications using both historical human-written programs and large pools of LLM-generated ones evaluated under a common compilation, sandboxing, and exhaustive test suite. We explore LLM diversity along multiple axes, including model family, generation temperature, and programming language. Reliability improvement is evaluated in a 1-out-of-2 configuration across both homogeneous and heterogeneous program populations, including within-LLM pairings and pairings across programming languages and across LLM-generated and human-written programs. The results show that combining LLM-generated programs, especially in heterogeneous settings, can yield reliability gains, although this is partly conditioned by the programming language and generation setting. Taken together, these findings suggest that LLMs provide a scalable source of comparatively low-cost programs whose diversity can be leveraged for reliability improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。