让大模型自己挑关键信息训练,提升长文本理解准确率
Self-Guided Test-Time Training for Long-Context LLMs

- 模型先自选相关文本片段,只在这些片段上微调
- 在两个长文本推理测试中,最高提升15%准确率
- 适合需要精准理解长文档的问答系统
长上下文处理对大语言模型愈发重要,但单纯扩展上下文窗口并不能保证有效利用。随着输入变长,模型准确率常下降,说明其仍难以识别与问题最相关的证据。测试时训练(TTT)可通过实例化参数适配提升长上下文利用效率,但对整个长文本进行训练成本过高,而随机采样片段则引入严重噪声。由于长文本中多数片段与特定问题无关,训练在这些片段上甚至会降低基线模型性能。初步研究表明,TTT对训练片段质量极为敏感:在LongBench-v2上,随机采样片段导致性能下降,而使用理想片段则显著提升。为此,我们提出自引导测试时训练(S-TTT):在适配前,模型先自我识别应学习的证据片段,并仅在这些选定片段上应用标准语言建模目标。在两个挑战性长上下文推理基准(LongBench-v2 和 LongBench-Pro)上,S-TTT显著提升Qwen3-4B-Thinking-2507和Llama-3.1-8B-Instruct的准确率,最高实现15%相对提升。
原文摘要 · Abstract (English)
Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。