无需训练，通过谱分解调整键向量实现高效提示聚焦。
Spectral Attention Steering for Prompt Highlighting
- 直接修改键嵌入，避免存储完整注意力矩阵。
- 在标准基准上显著优于基线，延迟和内存开销极低。
- 支持动态路由，适配不同语义意图的提示聚焦需求。
注意力引导是控制模型关注点的重要技术，可实现提示聚焦等能力，即让模型优先处理用户指定文本。然而，现有方法需显式存储完整注意力矩阵，与FlashAttention等内存高效实现不兼容。本文提出无训练的谱编辑键放大（SEKA）方法，通过谱分解直接在注意力计算前调整键嵌入，使其指向能增强特定标记注意力得分的潜在方向。进一步提出自适应版AdaSEKA，利用无训练路由机制，根据提示语义意图动态组合多个专家子空间。实验表明，两种方法在标准引导基准上均显著优于强基线，同时保持极低延迟与内存开销，兼容优化注意力机制。
原文摘要 · Abstract (English)
Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。