通过语义层次结构加速长文本推理，减少内存与延迟。
H$^{2}$MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
- 构建离线语义层级，自底向上聚合节点嵌入
- 推理时粗粒度到细粒度路由，提前剪枝无关分支
- 在长文档问答中实现更低内存和更快首字输出
基于Transformer的大型语言模型在多项语言任务上表现优异；然而，长输入仍具挑战性，因上下文窗口有限，预填充延迟和内存随提示长度快速增长。扁平化的标记流处理与分块检索会大量消耗计算资源和上下文预算，处理与查询无关的内容。离线索引的RAG引入外部存储与索引管理开销，且通常以原始文本形式拼接检索证据，增加预填充成本与延迟。H²MT使长上下文推理具备结构感知能力：它离线构建语义层级，通过自底向上的后序聚合为每个节点计算记忆嵌入，并在推理时从粗到精路由查询，早期剪枝无关分支。在LongBench QA（NarrativeQA、HotpotQA、QASPER）及两个结构化技术文档设置下，H²MT实现了良好的质量-效率权衡，在竞争性ROUGE-L与F1（适用时）表现基础上，显著降低峰值GPU内存与首字生成时间（TTFT），优于提示压缩、记忆令牌方法及检索增强生成基线。
原文摘要 · Abstract (English)
Transformer-based LLMs achieve strong results on many language tasks; however, long inputs remain challenging because context windows are finite, and prefill latency and memory grow rapidly with prompt length. Flat token-stream processing and chunk-based retrieval can therefore spend substantial computation and context budget on text unrelated to the query. Offline-indexed RAG additionally introduces external storage and index management overhead, and typically appends retrieved evidence as raw text, increasing prefill cost and latency. H^{2}MT makes long-context inference structure-aware: it builds a semantic hierarchy offline, computes a memory embedding for each node via bottom-up post-order aggregation, and routes queries coarse-to-fine at inference to prune irrelevant branches early. On LongBench QA (NarrativeQA, HotpotQA, QASPER) and two structured technical-document settings, H MT achieves favorable quality efficiency trade-offs, delivering competitive ROUGE-L and F1 (where applicable) with lower peak GPU memory and time-to-first-token (TTFT) than prompt compression, memory-token methods, and retrieval-augmented generation baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。