知识格式会劫持大模型注意力，导致推理失效。
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
- 发现知识图谱格式因结构重复，吸引2-3倍注意力
- 格式偏差使演示样本注意力压缩最高达42%
- 提出五种低成本缓解策略，提升检索增强推理可靠性
检索增强生成（RAG）系统通过注入外部知识提升大模型输出，但注入内容的格式（而非语义相关性）会独立扭曲模型注意力分布。我们识别并形式化了一种现象：知识图谱三元组因其关系分隔符和重复槽模式，每字节注意力占比达0.70，是自然语言文本（约0.25）的2-3倍，即使无关或为噪声也导致演示注意力压缩最高42%。我们构建了注意力分解框架（公式2），推导出连接格式偏差与演示注意力损失的压缩界（命题1），揭示结构项主导注意力分流程度，语义项决定其正负影响。该解耦表明改进检索增强上下文学习需从语义与格式双轴入手。实证显示，在两个模型（Mistral-7B、LLaMA-3-8B）和三个QA基准上，任务匹配检索（如BM25）在HotpotQA上达58-62%，远超ConceptNet的25-27%，差距超30个百分点，显著压倒各类门控策略（≤2个百分点）。基于框架提出五类结构感知缓解策略，包括零成本提示修改与训练时正则化；格式扁平化（S3）经语义三元组对照验证，既提升准确率又降低注意力集中度，而结构分散（S1）效果混杂，揭示格式干预的复杂挑战。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution. We identify and formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples, due to their relational delimiters and repeated slot patterns, capture 2-3x more attention per token than semantically equivalent natural-language text ($\hat{o}$(KG) $\approx$ 0.70 vs. $\hat{o}$(neutral) $\approx$ 0.25), compressing demonstration attention by up to 42% -- regardless of whether the triples are relevant or noise. We develop a formal framework decomposing attention scores into semantic and structural components (Eq. 2), derive a compression bound (Proposition 1) connecting token-level format bias to demonstration attention loss, and show that the structural term governs how much attention is diverted while the semantic term governs whether this helps or hurts. This decoupling reveals two orthogonal axes for improving retrieval-augmented ICL: optimising retrieval quality (semantic axis) and reducing format-driven attention capture (structural axis). Empirically, across two model families (Mistral-7B, LLaMA-3-8B) and three QA benchmarks, we observe that source-task alignment dominates: task-matched BM25 retrieval achieves 58-62% on HotpotQA vs. ConceptNet's 25-27%, a >30 pp gap that dwarfs all gating strategies ($\leq$2 pp). We derive five structure-aware mitigation strategies from the framework, ranging from zero-cost prompt modifications to training-time regularisation; format flattening (S3) is validated by both accuracy and attention-level evidence from a verbalized-triple control, while structural dispersal (S1) yields mixed results that illuminate the challenges of format-level intervention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。