用排序模型预训练注意力，让多标签分类更准
Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
- 用学习排序模型预生成标签专属注意力
- 在少样本和冷门标签上提升显著，最高增益12.3%
- 可直接接入LLaMA、Mistral等大模型，无需改造
当前顶尖的极端多标签文本分类模型依赖多标签注意力聚焦关键词元，但注意力权重学习困难。我们提出PLANT——预训练并利用注意力，一种即插即用的注意力初始化策略。PLANT通过预训练的学习排序模型，以互信息增益为指导，植入标签特定注意力。该架构无关方法可无缝集成至Mistral-7B、LLaMA3-8B、DeepSeek-V3、Phi-3等大型语言模型骨干网络。PLANT在ICD编码、法律主题分类和内容推荐等任务上均优于现有方法，尤其在少样本场景下表现突出，对罕见标签提升显著。消融实验表明，注意力初始化是性能提升的关键因素。代码与训练模型详见https://github.com/debjyotiSRoy/xcube/tree/plant。
原文摘要 · Abstract (English)
State-of-the-art Extreme Multi-Label Text Classification models rely on multi-label attention to focus on key tokens in input text, but learning good attention weights is challenging. We introduce PLANT - Pretrained and Leveraged Attention - a plug-and-play strategy for initializing attention. PLANT works by planting label-specific attention using a pretrained Learning-to-Rank model guided by mutual information gain. This architecture-agnostic approach integrates seamlessly with large language model backbones such as Mistral-7B, LLaMA3-8B, DeepSeek-V3, and Phi-3. PLANT outperforms state-of-the-art methods across tasks including ICD coding, legal topic classification, and content recommendation. Gains are especially pronounced in few-shot settings, with substantial improvements on rare labels. Ablation studies confirm that attention initialization is a key driver of these gains. For code and trained models, see https://github.com/debjyotiSRoy/xcube/tree/plant
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。