用负对数似然评估层重要性，实现无需训练的高效长文本推理
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
- 通过计算每层切换为滑动窗口后答案词的负对数似然下降量，判断其是否需保留全注意力
- 在Qwen3-4B上仅用1/4全注意力层即达64.6%准确率，计算量减半仍超基线
- 仅需约15分钟一次性校准，适合追求高效率部署的长文本大模型应用
混合全注意力与滑动窗口注意力的模型可有效提升长上下文推理效率，但哪些层应保留全注意力仍未解决。现有方法采用固定周期模式或基于注意力的启发式策略，可能无法捕捉下游任务的关键需求。本文提出NLL引导的层选择方法，无需训练即可通过计算某层改用滑动窗口后答案词的负对数似然下降程度，直接衡量该层重要性。在LongMemEval测试集上，使用Qwen3-4B模型时，仅保留1/4全注意力层即可达到64.6%准确率，相比1/2全注意力周期基线（65.0%）性能相当，计算开销减半；优于SWAA报告的1/4全注意力周期基线10.4个百分点，比匹配的LightTransfer基线高出26.4个百分点。去混淆分析表明该信号与长程注意力需求一致，而非通用层敏感性。该方法仅需约15分钟一次性校准，显著推进了长上下文大模型部署的效率-精度权衡前沿。
原文摘要 · Abstract (English)
Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of \emph{which layers} should retain full attention remains unsolved. Existing methods use either fixed periodic patterns or attention-based heuristics that may not capture what matters for downstream accuracy. We propose NLL-guided layer selection, a training-free method that directly measures each layer's importance by computing the negative log-likelihood degradation on answer tokens when that layer uses sliding-window instead of full attention. On LongMemEval with Qwen3-4B, our method achieves 64.6\% accuracy using only 1/4 full-attention layers, matching the 1/2-FA periodic baseline (65.0\%) while halving the computational budget. NLL-guided selection outperforms the SWAA-reported periodic 1/4-FA baseline by 10.4 percentage points and a matched LightTransfer-style baseline by 26.4 percentage points. De-confounding analysis shows the signal is consistent with long-range attention needs rather than generic layer sensitivity. The method requires only $\sim$15 minutes of one-time calibration, advancing the efficiency-accuracy Pareto frontier for long-context LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。