通过精准分配监督信号，显著提升大模型长文本理解能力。
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
- 设计新目标函数，对长上下文目标加权，缓解训练时上下文暴露不均问题。
- 在多个模型上测试，长文本任务性能最高提升17.91分，短文本表现几乎不变。
- 适合需要强长文本推理的场景，如法律、科研文档分析。
长上下文适应常被视作窗口缩放，但忽略了分块训练中存在令牌级监督失配：每个目标令牌的有效上下文仍较短。本文提出EXACT，一种监督分配目标，通过长尾逆频率为长有效上下文目标增加权重。在七种Qwen/LLaMA CPT配置下，所有28项NoLiMa与RULER对比均获提升。Qwen2.5-0.5B在训练和外推场景下，NoLiMa分别提升+10.09和+5.34，RULER分别提升+10.69和+5.55；LLaMA-3.2-3B上RULER提升+17.91（训练）和+16.11（外推）。标准问答与推理任务仅微降+0.24宏平均。距离解析探测显示，收益来自证据相隔数千令牌的情况，短距离情形无变化。结果支持以监督为核心的论点：长上下文适应取决于训练对长上下文预测的监督强度。
原文摘要 · Abstract (English)
Long-context adaptation is often viewed as window scaling, but this misses a token-level supervision mismatch: in packed training with document masking, each target token's effective context remains short. We introduce EXACT, a supervision-allocation objective that assigns extra weight to long effective-context targets by inverse frequency within the long tail. Across seven Qwen/LLaMA CPT configurations, EXACT improves all 28 trained/extrapolated NoLiMa and RULER comparisons. On Qwen2.5-0.5B, NoLiMa improves by +10.09 (trained) and +5.34 (extrapolated); RULER by +10.69 and +5.55. On LLaMA-3.2-3B, RULER improves by +17.91 and +16.11. Standard QA/reasoning are preserved (+0.24 macro change across six benchmarks). A distance-resolved probe shows gains arise when evidence is thousands of tokens away, while short cases remain unchanged. Results support a supervision-centric thesis: long-context adaptation depends on how strongly training supervises long-context predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。