通过融合多层与全序列信息，提升BERT分类效果
MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation
- 用最大池化和注意力机制整合多层[CLS]与全序列信息
- 在GLUE低资源任务上超越标准BERT模型性能
- 无需额外预训练，模型规模基本不变
BERT中的[CLS] token通常作为分类任务的固定长度表示，但已有研究表明其他标记和中间层也包含有价值的上下文信息。本文研究了轻量级改进方法，通过跨层和跨标记聚合来优化[CLS]表示。具体包括：(i) 对多个层的[CLS] token进行最大池化；(ii) 在最终层引入额外多头注意力（MHA）层，使[CLS] token能关注整个层的输出；(iii) 将全序列最大池化与MHA结合。该方法称为MaxPoolBERT，可有效提升BERT分类准确率，尤其在低资源任务中表现更优，且无需新预训练或显著增加模型大小。在GLUE基准测试中，MaxPoolBERT在多个低资源任务上持续优于标准BERT base模型。
原文摘要 · Abstract (English)
The [CLS] token in BERT is commonly used as a fixed-length representation for classification tasks, yet prior work has shown that both other tokens and intermediate layers encode valuable contextual information. In this work, we study lightweight extensions to BERT that refine the [CLS] representation by aggregating information across layers and tokens. Specifically, we explore three modifications: (i) max-pooling the [CLS] token across multiple layers, (ii) enabling the [CLS] token to attend over the entire final layer using an additional multi-head attention (MHA) layer, and (iii) combining max-pooling across the full sequence with MHA. Our approach, called MaxPoolBERT, enhances BERT's classification accuracy (especially on low-resource tasks) without requiring new pre-training or significantly increasing model size. Experiments on the GLUE benchmark show that MaxPoolBERT consistently achieves a better performance than the standard BERT base model on low resource tasks of the GLUE benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。