通过任务空间敏感度反馈，让大模型自动校准注意力，提升分类准确率。
Calibrating Transformer Attention via Task-Space Sensitivity Feedback
- 用任务空间敏感度反馈自动生成注意力校准信号，无需人工标注。
- 在五个数据集上使Llama3-8B模型准确率最高提升15.8%。
- 只更新注意力投影层，保持主干模型不变，训练高效且可扩展。
基于Transformer的预训练语言模型（PLMs）在文本分类中表现优异，但存在注意力稀释和注意力陷阱问题，导致模型过度关注与任务无关的词元。现有注意力监督方法依赖昂贵的词元级人工标注或静态启发式规则，难以扩展且无法捕捉上下文相关的词元重要性。为此，我们提出AttCal——一种自监督、无标注的注意力校准框架，基于任务空间敏感度反馈实现。AttCal将注意力分布视为随机策略，通过词元删除扰动推断语义变化，进而获得上下文依赖的词元重要性，并利用策略梯度优化注意力参数。关键在于，AttCal冻结PLM主干网络，仅更新注意力投影层，保障训练效率。在五个数据集上的实验表明，AttCal显著优于原始PLMs，对Llama3-8B模型最高实现15.8%的准确率提升，并超越当前最优的有监督基线，在分类性能与可解释性上均取得优势。
原文摘要 · Abstract (English)
Transformer-based pre-trained language models (PLMs) excel in text classification but suffer from attention dilution and attention sink effects, forcing models to over-focus on task-irrelevant tokens. Existing attention supervision methods rely on costly token-level human annotations or static heuristics, which fail to scale or capture context-dependent token importance. To address this, we propose AttCal, a self-supervised, annotation-free attention calibration framework via task-space sensitivity feedback. AttCal treats attention distributions as stochastic policies to guide token deletion perturbations, infers context-dependent token importance from semantic shifts, and optimizes attention parameters via policy gradient. Critically, AttCal freezes the PLM backbone and updates only attention projection layers to ensure training efficiency. Benchmarks across five datasets demonstrate that AttCal significantly outperforms vanilla PLMs-yielding up to a 15.8% accuracy lift for Llama3-8B-and surpasses SOTA supervised baselines in both classification and interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。