大模型时代仍需语法信息，能提升代码生成准确率与语义区分能力。
Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?
- 在生成过程中融入语法规则，构建语法驱动的代码表示。
- 在HumanEval+和MBPP+上代码准确率显著提升，减少细微语义错误。
- 适合关注代码质量与语义精确性的编程大模型研究者。
语法是编程语言与软件工程的核心，为程序结构和语法空间提供框架。现有研究显示，语法驱动的代码表示在小规模模型中有效降低语法错误并提升性能。然而，随着语言模型规模达十亿级或更高，语法错误已罕见，语法信息是否仍有价值尚不明确。为此，我们构建了一系列十亿级的GrammarCoder模型，在代码生成中引入语法规则。在HumanEval (+)和MBPP (+)上的实验表明，代码生成准确率显著提升。进一步分析发现，语法表示增强了大模型对细微代码差异的辨别能力，减少了由微小变化引发的语义错误。结果表明，即使在十亿级模型中，语法信息仍具价值，不仅能保障语法正确性，还能提升语义区分能力。
原文摘要 · Abstract (English)
Grammar serves as a cornerstone in programming languages and software engineering, providing frameworks to define the syntactic space and program structure. Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance. However, as language models scale to the billion level or beyond, syntax-level errors become rare, making it unclear whether grammar information still provides performance benefits. To explore this, we develop a series of billion-scale GrammarCoder models, incorporating grammar rules in the code generation process. Experiments on HumanEval (+) and MBPP (+) demonstrate a notable improvement in code generation accuracy. Further analysis shows that grammar-based representations enhance LLMs' ability to discern subtle code differences, reducing semantic errors caused by minor variations. These findings suggest that grammar-based code representations remain valuable even in billion-scale models, not only by maintaining syntax correctness but also by improving semantic differentiation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。