让小模型更准答长文本问题,通过精准调整注意力位置。
EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering

- 用证据片段生成软注意力目标,指导模型优化查询侧注意力。
- 在六项LongBench任务上,平均性能超越现有方法,尤其提升弱模型表现。
- 适合资源有限但需长上下文推理的场景,如低算力部署应用。
长上下文问答对小型语言模型仍具挑战性,即使答案相关证据已存在于输入中。现有上下文内检索方法仅暴露候选证据块,却未调整控制模型对全上下文位置注意力分配的查询侧参数。轻量级测试时适应方法(如qTTT)虽可调整参数,但其通用跨度级自监督目标无法定位支持当前答案的具体上下文位置。本文提出EASE-TTT:一种基于上下文检索增强的测试时训练框架,将选定证据块转化为对其标记位置的软注意力监督目标。不替换完整上下文,而是利用该注意力目标引导查询侧适应,最终由适配后的模型从原始全上下文生成答案。在六个LongBench QA任务及三个小型解码器模型上的实验表明,EASE-TTT在宏平均性能上优于全上下文推理、仅检索基线和qTTT,验证了证据对齐测试时适应的有效性。
原文摘要 · Abstract (English)
Long-context question answering (QA) remains challenging for smaller language models even when answer-bearing evidence is already present in the input. Existing within-context retrieval methods localize and expose candidate evidence chunks for the question, but they stop at input-level evidence exposure rather than adapting the query-side attention parameters that control how the model allocates attention over full-context positions. In contrast, lightweight test-time adaptation methods, such as query-only test-time training (qTTT), leave evidence localization unresolved because their generic span-level self-supervised objectives do not identify which context positions support the current answer. In this paper, we propose Evidence-Aligned SElective Test-Time Training (EASE-TTT), a within-context retrieval-augmented test-time training framework that converts selected evidence chunks into a soft attention supervision target over their token positions. Instead of replacing the full context with retrieved chunks, EASE-TTT uses the resulting attention target to guide query-side adaptation, with the adapted model generating the final answer from the original full context. Experiments on six LongBench QA tasks and three small decoder-only language models show that EASE-TTT achieves the strongest macro-average performance among full-context inference, retrieval-only baselines, and qTTT, supporting evidence-aligned test-time adaptation in long-context QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。