arXiv:2608.12771cs.SEcs.AI2026-08

大模型代码生成的记忆检测需考虑规模,传统方法在大模型上失效。

Memorization Diagnostics for Code LLMs Should be Scale-Aware

论文配图:Memorization Diagnostics for Code LLMs Should be Scale-Aware
图 1 · 摘自论文原文
  • 用可逆数学变换分离表示负荷与记忆,区分二者影响。
  • 大模型能处理表面形式变化仍得正确解,说明具泛化能力。
  • 适合关注代码生成可靠性与评估方法改进的研究者。

代码大模型是否依赖记忆而非理解仍存争议。现有文献常宣称广泛记忆现象,但传统探测技术在大型密集架构中效果严重下降。使用同义词模糊或无用代码插入等扰动的编码器式探测,以及依赖对数概率的解码器式探测,在大规模模型上均无法有效揭示记忆行为,即使在已知污染的数据集上亦然。这些探测方法在小模型中有效却在大模型中失效的机制,促使我们区分表示负荷与记忆,而非将其视为单一现象。通过应用可逆数学变换处理数值问题,我们隔离了这两个因素,发现大规模编码器虽承受大量表示负荷,仍能收敛至正确解集。在实际软件工程中,适应不同表达形式的能力才是关键,因此具体答案是否被训练时记忆变得不那么重要。尽管记忆会抬高污染数据集上的得分,但剔除表示负荷后,是否曾记忆特定答案的争议显著降低。未来评估应聚焦于分离这两种现象,而非依赖隐性混淆它们的方法。

原文摘要 · Abstract (English)

The extent to which large language models for code rely on memorization over genuine understanding remains highly debated. While current literature frequently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled models, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated benchmarks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them.

代码生成大模型评估记忆检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。