通过学习大模型的错误模式，用过滤器提升代码生成的编译成功率。
Learning the Error Patterns of Language Models
- 用符号函数捕捉特定领域下的常见错误模式。
- 在TypeScript生成中使编译率提升超60%，接近更大模型表现。
- 适合需要高可靠代码生成的开发者和工业级应用。
当生成具有特定有效性约束的输出（如程序需能编译）时，大语言模型常以少数集中方式出错：例如在生成TypeScript时使用Python函数名。我们发现这些错误模式可用少量可实践学习的约束表示。本文提出前缀过滤器（prefix filters），即针对特定领域与模型的符号函数，用于捕捉错误模式；提出Palla算法高效学习前缀过滤器并实现。经验证，由Palla学习到的前缀过滤器可定量分析大模型错误模式，并通过约束采样提升输出质量。例如，在TypeScript生成任务中，Palla使Qwen2.5-1.5B的编译率提升超过60%，达到与未约束的Llama3.1-8B相当的性能。
原文摘要 · Abstract (English)
When generating outputs for domains with specific validity constraints (e.g., a program should compile), LLMs often fail in a small number of focused ways: for example, by using Python function names when generating TypeScript. We observe that these error patterns can be represented using a small number of constraints that can be learned in practice. We propose \emph{prefix filters}, which are per-domain-and-LLM symbolic functions, as objects to capture the error patterns, Palla as an algorithm to learn prefix filters efficiently in practice, and implement Palla. Prefix filters learned by Palla i) help us quantitatively analyze the error patterns of LLMs, and ii) can be used to constrain the outputs of a model via constrained sampling algorithms. For example, Palla boosts compile rates for Qwen2.5-1.5B on TypeScript generation, by over 60%, allowing Qwen2.5-1.5B to achieve similar performance to Llama3.1-8B unconstrained.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。