arXiv:2504.09246cs.LGcs.PL2025-04被引 57

用类型系统约束大模型生成代码,减少编译错误。

Type-Constrained Code Generation with Language Models

  • 构建前缀自动机与可实现类型搜索,强制生成的代码类型正确。
  • 在HumanEval和MBPP上使编译错误减半,功能正确率显著提升。
  • 适用于多种模型大小和类型系统,尤其适合代码生成任务。

大语言模型在代码生成方面已取得显著进展,但仍频繁产生无法编译的代码,因其逐标记生成过程未建模代码的形式化特性。虽然受限解码是缓解此问题的有前景方法,但现有方法仅针对领域特定语言或通用编程语言的语法特征。然而,大模型常生成类型错误,这类问题超出了语法范畴,且难以有效约束。为此,我们提出一种类型约束解码方法,利用类型系统引导代码生成。为此,我们开发了新型前缀自动机和可实现类型搜索,形成保证代码良类型的可靠方法。我们在基础简单类型语言上形式化该方法,并扩展至TypeScript以证明实用性。在HumanEval和MBPP数据集上的评估表明,该方法将编译错误减少超过一半,并显著提升各类任务中代码合成、转换和修复的功能正确性,涵盖不同规模和架构的大模型,包括参数量超300亿的先进开源模型。结果表明,该方法在用类型系统约束大模型代码生成方面具有普遍性与有效性。

原文摘要 · Abstract (English)

Large language models (LLMs) have achieved notable success in code generation. However, they still frequently produce uncompilable output because their next-token inference procedure does not model formal aspects of code. Although constrained decoding is a promising approach to alleviate this issue, it has only been applied to handle either domain-specific languages or syntactic features of general-purpose programming languages. However, LLMs frequently generate code with typing errors, which are beyond the domain of syntax and generally hard to adequately constrain. To address this challenge, we introduce a type-constrained decoding approach that leverages type systems to guide code generation. For this purpose, we develop novel prefix automata and a search over inhabitable types, forming a sound approach to enforce well-typedness on LLM-generated code. We formalize our approach on a foundational simply-typed language and extend it to TypeScript to demonstrate practicality. Our evaluation on the HumanEval and MBPP datasets shows that our approach reduces compilation errors by more than half and significantly increases functional correctness in code synthesis, translation, and repair tasks across LLMs of various sizes and model families, including state-of-the-art open-weight models with more than 30B parameters. The results demonstrate the generality and effectiveness of our approach in constraining LLM code generation with formal rules of type systems.

代码生成类型系统大模型约束解码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。