通过分析模型中间输出轨迹，识别大模型在评测中是否因记忆而作弊。
LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectories
- 追踪模型各层的逻辑输出变化，观察答案形成过程。
- 被污染题目更早锁定答案，干净题目则逐步积累证据。
- 适用于检测改写输入，适合研究模型记忆行为的学者。
大语言模型常在AIME和Math500等复杂基准上评估，但基准污染可能导致模型通过记忆答案而非推理获得高分。现有检测方法多依赖表面重合、生成行为或最终输出概率，在输入改写时性能下降。本文提出LogitTrace（逐层逻辑轨迹），通过分析模型中间层的逻辑输出路径来识别类似记忆的决策模式。不同于仅看最终答案，LogitTrace关注模型偏好如何随层演变：污染样本往往早期就确定答案，而干净样本呈现更渐进的证据积累。该轨迹信号可被轻量级分类器用于区分污染与清洁样本，且在多个模型和输入变体上表现稳定。控制性LoRA注入实验表明，重复接触目标样本会诱发相似轨迹模式。结果表明，LogitTrace提供了超越表面重合与最终置信度的新视角，有助于深入研究大模型的记忆行为。
原文摘要 · Abstract (English)
Large language models (LLMs) are commonly evaluated on challenging benchmarks such as AIME and Math500, where benchmark contamination can make memorized solutions appear as genuine reasoning. Existing detection methods largely rely on surface overlap, completion behavior, or final-output likelihood, and often degrade when inputs are simply rephrased. In this paper, we propose LogitTrace(Layerwise Logit Trajectories), a framework for analyzing memorization-like decision dynamics through intermediate logit trajectories. Instead of judging memorization only from the final answer, LogitTrace examines how model preferences emerge and stabilize across layers. We find that contaminated examples tend to show earlier commitment, while clean examples exhibit more gradual evidence accumulation. These trajectory signals allow a lightweight classifier to separate contaminated and clean examples across multiple models and input variants. Controlled LoRA injection experiments further show that repeated exposure to target samples induces similar trajectory patterns. Overall, our results suggest that LogitTrace provides evidence beyond surface overlap and final-output confidence, offering a useful lens for studying memorization-like behavior in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。