通过注意力引导提升长文本理解，显著加速大模型推理
LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
- 不需微调，基于查询无关的注意力机制优化长上下文处理
- 性能差距缩小65.9%，推理延迟最高降低4.8倍
- 适合需要高效长文本生成与理解的场景
随着大语言模型在复杂任务中表现出色，其在长上下文理解和高计算成本方面仍存在挑战。为兼顾效率与质量，我们提出LLMSteer，一种无需微调的框架，通过查询无关的注意力引导增强大语言模型。在多个主流LLM和数据集上测试，该方法使性能差距缩小65.9%，相比近期注意力引导方法，运行时间延迟最多降低4.8倍。
原文摘要 · Abstract (English)
As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。