arXiv:2602.07778cs.CL2026-02ACL被引 1

用注意力机制筛选关键用户信息,让大模型更省资源地个性化。

Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs

  • 通过分析大模型注意力,自动识别重要用户信息
  • 压缩后仅用1/50的tokens,性能接近全量输入
  • 适合需要高效个性化的实时应用

将大语言模型(LLMs)个性化到个人用户需引入大量交互历史与用户画像,但输入长度限制导致推理延迟高、API成本昂贵。现有方法依赖启发式策略,如选取最近交互或用提示模型压缩画像,但这些方法将上下文视为整体,未考虑大模型内部对不同信息的处理优先级。我们研究发现,大模型的注意力模式能自然揭示重要个性化信号,且微调可增强其区分相关与无关信息的能力。基于此,提出Attn-GS框架:利用标记模型的注意力反馈标注关键个人句,引导压缩模型生成任务相关的高质量压缩上下文。大量实验表明,Attn-GS在不同任务、令牌限制和设置下均显著优于各类基线,性能接近使用完整上下文,同时减少50倍的令牌用量。

原文摘要 · Abstract (English)

Personalizing large language models (LLMs) to individual users requires incorporating extensive interaction histories and profiles, but input token constraints make this impractical due to high inference latency and API costs. Existing approaches rely on heuristic methods such as selecting recent interactions or prompting summarization models to compress user profiles. However, these methods treat context as a monolithic whole and fail to consider how LLMs internally process and prioritize different profile components. We investigate whether LLMs' attention patterns can effectively identify important personalization signals for intelligent context compression. Through preliminary studies on representative personalization tasks, we discover that (a) LLMs' attention patterns naturally reveal important signals, and (b) fine-tuning enhances LLMs' ability to distinguish between relevant and irrelevant information. Based on these insights, we propose Attn-GS, an attention-guided context compression framework that leverages attention feedback from a marking model to mark important personalization sentences, then guides a compression model to generate task-relevant, high-quality compressed user contexts. Extensive experiments demonstrate that Attn-GS significantly outperforms various baselines across different tasks, token limits, and settings, achieving performance close to using full context while reducing token usage by 50 times.

个性化注意力机制压缩LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。