通过差异分解提升提示聚焦效果，显著降低生成干扰。
PRISM-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models
- 基于正负样本协方差差分解，精准捕捉关键语义差异
- 在20个配置中19次超越或持平最佳方法，最高提升13.0%
- 适用于长文本检索，兼容FlashAttention且内存开销极低
提示聚焦旨在引导大语言模型在生成时优先关注用户指定的文本片段。现有关键编辑方法的主要挑战在于提取能区分相关与无关上下文的转向方向，而非共有的结构模式。我们提出PRISM-Δ（基于投影的相关性感知转向方法），通过分解正负交叉协方差矩阵的差异，在最大化判别能量的同时消除共享方向。每个注意力头获得连续的softplus重要性权重，使弱但有用的头以较低强度贡献。该框架可自然扩展至Value表示，捕获关键方法忽略的内容通道信号。在四个基准和五个模型上，我们的方法在20个配置中有19个达到或超过最优表现，相对提升最高达+13.0%，同时将转向的流畅性成本减半。方法在长上下文检索任务中也优于现有最佳方法，相对增益最高+5.1%。PRISM-Δ兼容FlashAttention，额外内存开销可忽略。
原文摘要 · Abstract (English)
Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$Δ$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM-$Δ$ is compatible with FlashAttention and adds negligible memory overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。