解决长文本微调中注意力分散问题，提升模型对关键信息的捕捉能力。
FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
- 通过双层优化动态聚焦关键内容，缓解训练时注意力偏移问题。
- 在4K–32K上下文长度上准确率最高提升14个百分点，关键任务指标提升显著。
- 适合需要处理长文本的模型微调场景，尤其关注上下文理解精度的研究者。
大型语言模型虽能处理更长输入，但其有效利用长上下文中分散信息的能力仍有限。我们发现，这一差距源于监督微调过程中注意力预算分配不当：位置偏好和注意力塌陷导致模型将大部分注意力集中在位置有利的词元上，而非语义相关的内容。这种训练阶段的注意力稀释（内容词元在注意力分布中被抑制）削弱了梯度信号，限制了模型学习鲁棒长上下文能力。为此，我们提出FocuSFT，一种在训练时解决该问题的双层优化框架。内层在训练上下文中自适应轻量级快速权重参数，形成参数化记忆，集中注意力于相关内容；外层基于此强化表示进行标准微调。两层均采用双向注意力覆盖上下文词元，同时保留生成结果的因果掩码，降低因果不对称性，使内外行为对齐。在BABILong上，FocuSFT在4K–32K上下文长度下准确率最高提升14个百分点；在RULER上，CWE聚合准确率从72.9%提升至81.1%（16K）；在包含代理工具使用的GPQA任务中，pass@1相对提升24%。注意力分析显示，FocuSFT将注意力塌陷质量减少529倍，并使上下文参与度提升三倍。代码已开源。
原文摘要 · Abstract (English)
Large language models can now process increasingly long inputs, yet their ability to effectively use information spread across long contexts remains limited. We trace this gap to how attention budget is spent during supervised fine-tuning (SFT) on long sequences: positional biases and attention sinks cause the model to allocate most of its attention to positionally privileged tokens rather than semantically relevant content. This training-time attention dilution (the starvation of content tokens in the attention distribution) weakens the gradient signal, limiting the model's ability to learn robust long-context capabilities. We introduce FocuSFT, a bilevel optimization framework that addresses this problem at training time. An inner loop adapts lightweight fast-weight parameters on the training context to form a parametric memory that concentrates attention on relevant content, and the outer loop performs SFT conditioned on this sharpened representation. Both loops apply bidirectional attention over context tokens while preserving causal masking for responses, reducing the causal asymmetry that gives rise to attention sinks and aligning inner-outer behavior. On BABILong, FocuSFT improves accuracy by up to +14pp across 4K--32K context lengths; on RULER, it raises CWE aggregation from 72.9\% to 81.1\% at 16K; and on GPQA with agentic tool use, it yields a 24\% relative gain in pass@1. Attention analysis shows that FocuSFT reduces attention sink mass by 529$\times$ and triples context engagement during training. Code: https://github.com/JarvisPei/FocuSFT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。