提出分层构建与融合机制，让大模型更高效处理超长文本。
HiCI: Hierarchical Construction-Integration for Long-Context Attention
- 分层构建段落表征，再融合成全局上下文，双向引导注意力。
- 仅用不到5.5%额外参数，将上下文长度扩展至10万token（7B模型）。
- 在检索和代码理解任务中超越GPT-3.5-Turbo-16K，适合长文本应用。
长上下文语言建模常被视为令牌级注意力的可扩展性挑战，但现有方法对局部到全局信息结构的建模仍较隐式。受话语理解认知理论启发，我们提出HiCI（分层构建-融合），一种分层注意力模块：先构建段落级表征，将其整合为共享全局上下文，并广播两者以指导段落级注意力。通过仅增加<5.5%参数量，对LLaMA-2实现参数高效适配，将上下文长度从4K扩展至100K（7B模型）和64K（13B模型）。在语言建模、检索与指令遵循基准上，HiCI持续优于强基线，主题检索表现媲美专有模型，代码理解超越GPT-3.5-Turbo-16K。结果表明，显式分层结构作为归纳偏置，在长上下文建模中极具有效性。
原文摘要 · Abstract (English)
Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction--Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 with only <5.5% additional parameters, extending context from 4K to 100K tokens (7B) and 64K tokens (13B). Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。