代码大模型因分词不识语法，微调格式就乱套。
TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar
- 用语义不变的重写规则生成仅分词不同的代码变体
- 9个代码大模型均显示分词差异导致行为显著变化
- 早期嵌入层分词错位是根源，适合关注代码生成可靠性的研究者
用于代码的大语言模型依赖子词分词器（如BPE），其训练基于混合自然语言与代码文本，但以统计规律为主，而非语法结构。因此，语义相同的代码片段可能因空格或命名方式等表面因素被不同分词。为评估此偏差影响，我们提出TokDrift框架，通过保持语义的重写规则生成仅分词不同的代码变体。在9个代码LLM（包括参数超300亿的大型模型）上测试发现，即使微小格式调整也会引发模型行为显著变化。层级分析表明，问题始于早期嵌入层，子词分割未能对齐语法词法边界。研究揭示分词与语法不一致是代码理解与生成可靠性的一大隐性障碍，呼吁未来代码LLM采用语法感知的分词机制。
原文摘要 · Abstract (English)
Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently depending on superficial factors such as whitespace or identifier naming. To measure the impact of this misalignment, we introduce TokDrift, a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. Across nine code LLMs, including large ones with over 30B parameters, even minor formatting changes can cause substantial shifts in model behavior. Layer-wise analysis shows that the issue originates in early embeddings, where subword segmentation fails to capture grammar token boundaries. Our findings identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation, highlighting the need for grammar-aware tokenization for future code LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。