arXiv:2607.07740cs.LGcs.AI2026-07

提出无需调参的长文本扩展方法,动态调整注意力机制以兼顾短长上下文性能。

Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

论文配图:Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
图 1 · 摘自论文原文
  • 用双焦点旋转位置编码动态适应序列长度,短文本保持原模型精度,长文本平滑外推。
  • 在128K上下文上,比最强基线提升超4.7个百分点,且生成延迟增加不超过4%。
  • 适用于多种架构,部署简单,无需重新训练,适合实际应用中长上下文场景。

现代大模型越来越多用于长上下文任务,如检索增强生成、代码库级编程和智能体工作流,其累积推理与工具轨迹常使输入长度超出预训练窗口一个数量级,导致零样本上下文扩展成为开源权重模型的主要部署路径。现有主流零样本方法(YaRN、Self-Extend、DCA)固定单一缩放因子,激进因子牺牲短上下文精度,保守因子在长上下文失效;近期长度感知变体虽能自适应映射,但仍依赖拟合或距离依赖调度。本文提出Jet-Long,一种无需调参的零样本方法,将局部RoPE保真窗口与长程窗口结合,后者通过无参数解析调度动态调整缩放因子,短输入下精确恢复原始模型,长输入时干净外推。包含排除注意力融合与实时RoPE校正旋转,双焦点结构几乎无额外开销;集成至单个CuTe核后,长上下文预填充在H100上达到最高1.39倍FA2吞吐(接近仅支持Hopper的FA4),单批生成在各长度下延迟增加≤4%。在Qwen3-1.7B/4B/8B模型上,上下文长达128K时,相比最强基线分别提升+4.79/+2.18/+2.03百分点;在HELMET-RAG基准上取得最佳整体准确率,并实现最低PG-19困惑度。Jet-Long还可推广至混合注意力架构(如Jet-Nemotron),无需重训即可进一步提升长上下文表现,且对超参数不敏感,易于部署。

原文摘要 · Abstract (English)

Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints. The dominant zero-shot methods (YaRN, Self-Extend, DCA) fix a single rescaling factor up front, so an aggressive factor sacrifices short-context fidelity while a conservative one breaks down at long contexts; recent length-aware variants adapt the mapping, but with a fitted or distance-dependent schedule. We propose Jet-Long, a tuning-free zero-shot method that pairs a local RoPE-faithful window with a long-range window whose rescaling factor adapts dynamically to the current sequence length via a parameter-free analytic schedule, recovering the base model exactly at short inputs while extrapolating cleanly at long ones. An inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation make the bifocal construction essentially free at inference; fused into a single CuTe kernel, long-context prefill reaches up to $1.39\times$ FA2 throughput on H100 (approaching the Hopper-only FA4), and single-batch generation incurs $\le 4\%$ overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leads RULER by $+4.79$/$+2.18$/$+2.03$ pp over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy on HELMET-RAG (a benchmark identified by HELMET as the most efficient predictor of downstream long-context performance) and attains the lowest PG-19 perplexity. Jet-Long also generalizes to hybrid attention architectures such as Jet-Nemotron for further long-context improvement without retraining, and remains hyperparameter-resilient for ease of deployment.

长文本生成位置编码高效推理大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。