arXiv:2503.02296cs.AI2025-03被引 3

通过代码改写测试模型是否真会用,还是只会照搬。

Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting

  • 用语义扰动改写代码任务,检测模型是否死记硬背。
  • 大模型越训练越少抄,微调反而容易记死。
  • 新指标MRI能区分有害记忆和正常复用,适合评估代码生成能力。

大型语言模型(LLMs)在代码生成方面表现卓越,但其能力究竟是基于记忆还是泛化仍存争议。现有评估多依赖表面或结构相似性,混淆了正常复用与有害记忆,并忽略语义变化下的任务正确性。本文将有害记忆定义为高相似度下任务失败,提出语义扰动的代码重写方法:为给定任务生成语义不同但难度相近的新解,反向构造新任务。进一步提出记忆风险指数(MRI),综合两项信号:(i)模型对改写后任务的回答与原答案的相似度;(ii)模型在原始任务与改写任务间的性能下降程度。只有当两者同时成立时,MRI才高——即模型输出相似代码但无法完成新任务,从而捕捉有害记忆而非良性复用。在MBPP+和BigCodeBench上的实证表明:(1)模型规模增大并未增加记忆,多数情况下随规模扩大而缓解;(2)监督微调(SFT)虽提升准确率,却引入更多记忆;(3)近端策略优化(PPO)强化学习在记忆与泛化间取得更优平衡。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently demonstrated exceptional code generation capabilities. However, there is a growing debate whether LLMs are mostly doing memorization (i.e., replicating or reusing large parts of their training data) versus generalization (i.e., beyond training data). Existing evaluations largely proxy memorization with surface/structural similarity, thereby conflating benign reuse of repeated code with harmful recall and neglecting task correctness under semantic variation. We define harmful memorization behaviorally as failure at high similarity and introduce a semantic perturbation code rewriting, which rewrites a semantically different answer at a similar difficulty level for a given coding task, then reverse-engineers a novel coding task. We further propose Memorization Risk Index (MRI), a normalized score that combines two signals: (i) how similar the model's answer for the rewritten task is to the original ground-truth solution, and (ii) how much performance drops from the original task to its rewritten counterpart. MRI is high only when both conditions hold -- when the model outputs similar code but fails the perturbed task -- thereby capturing harmful memorization rather than benign reuse of repeated code. Empirical evaluations on code generation benchmarks MBPP+ and BigCodeBench reveal that (1) memorization does not increase with larger models and in many cases alleviates as they scale; (2) supervised fine-tuning (SFT) improves accuracy while introduces memorization; (3) reinforcement learning with proximal policy optimization (PPO) achieves a more balanced trade-off between memorization and generalization.

代码生成大模型记忆检测评估方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。