让大模型在测试时用自身预测下一词的机制自适应优化，提升长文本理解能力。
Test-Time Training with Next-Token Prediction
- 用模型自身下一位置的隐状态作为更新目标，实现无需重训练的快速自适应。
- 在多个主流模型上平均提升3.9至4.1分，长文档问答任务最高提升5.6分。
- 适合希望直接使用现成大模型、无需修改结构的开发者和研究者。
下一个词预测是语言模型自监督训练的核心信号，每个输入提示词在测试时均可提供相同信号。本文研究该信号是否可用于预训练长上下文语言模型的测试时训练（TTT）内循环目标。现有方法虽实现无需重设计模型的快速权重适应，但未将写操作与下一词预测信号直接关联。本文提出测试时训练结合下一词预测（TTT-NTP），以模型自身下一位置的上下文隐状态为监督目标，使每次局部写入遵循支持下一词预测的因果计算：目标值为单个下一位置上下文状态的点积线性投影。在RULER Full-13上，对4千至3.2万上下文平均，TTT-NTP是唯一在四个跨越三类模型、0.6–8B规模的基线中持续提升的方法，分别提升Llama-3.1-8B 3.9分、Mistral-7B-v0.3 3.0分、Qwen3-4B 4.1分、Qwen3-0.6B 2.9分。在真实世界长文档问答基准LongBench-v2上，相较基线分别提升5.6分（Llama-3.1-8B）和3.7分（Mistral-7B-v0.3），同时保持常识与知识性能。代码已开源。
原文摘要 · Abstract (English)
Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test time. We study whether this signal can define the inner-loop objective for test-time training (TTT) in pretrained long-context language models. Many TTT architectures require models to be trained with test-time adaptation in mind, limiting their direct applicability to released LLM checkpoints. While recent in-place TTT methods make fast-weight adaptation possible for pretrained LLMs without redesigning the backbone, they leave a central question unresolved: what should each test-time write store? Existing recipes train the fast weight to match a learned local value proxy but they are not directly tied to the self-supervised next-token prediction signal. We introduce Test-Time Training with Next-Token Prediction (TTT-NTP), a drop-in fast-weight adaptation method for pretrained LLMs that instead supervises updates using the model's own next contextual hidden state. This makes each local write follow the same causal computation that supports next-token prediction: the value target is a pointwise linear projection of a single next-position contextual state. On RULER Full-13, averaged over 4k to 32k contexts, TTT-NTP is the only method that consistently improves the released backbone across four models spanning three families and a 0.6-8B size range, by 3.9 points on Llama-3.1-8B, 3.0 on Mistral-7B-v0.3, 4.1 on Qwen3-4B, and 2.9 on Qwen3-0.6B. On the real-world LongBench-v2 long-document QA benchmark, TTT-NTP improves over the base model by 5.6 points on Llama-3.1-8B and 3.7 on Mistral-7B-v0.3, while preserving commonsense and knowledge performance. Our code is publicly available at https://github.com/yancyou/TTT-NTP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。