通过多尺度卷积增强Transformer的局部特征表示，提升跨领域跨语言任务表现。
Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages
- 在Transformer层上加一维多尺度卷积，捕捉局部细节特征。
- 在5个任务中提升1%至14%，且保持高效计算。
- 适合需要精细局部模式识别的任务，如细粒度分类或低资源语言处理。
编码器型Transformer将序列中所有标记的信息压缩到单一[CLS]标记以表示全局上下文，这可能导致细粒度或层级特征被稀释，从而在依赖局部模式的下游任务中造成信息损失。为解决此问题，我们提出一种轻量级架构改进：在Transformer层之上引入类Inception的一维卷积模块，为标记表示注入多尺度局部特征。随后，通过自注意力层动态权衡各标记的任务相关性。在五个不同任务上的实验表明，该框架能持续提升通用、领域特定及多语言模型的表现，相比基线提升1%至14%，同时保持高效。消融实验显示，多尺度卷积优于单一卷积核，且自注意力层对性能至关重要。
原文摘要 · Abstract (English)
Encoder transformer models compress information from all tokens in a sequence into a single [CLS] token to represent global context. This approach risks diluting fine-grained or hierarchical features, leading to information loss in downstream tasks where local patterns are important. To remedy this, we propose a lightweight architectural enhancement: an inception-style 1-D convolution module that sits on top of the transformer layer and augments token representations with multi-scale local features. This enriched feature space is then processed by a self-attention layer that dynamically weights tokens based on their task relevance. Experiments on five diverse tasks show that our framework consistently improves general-purpose, domain-specific, and multilingual models, outperforming baselines by 1% to 14% while maintaining efficiency. Ablation studies show that multi-scale convolution performs better than any single kernel and that the self-attention layer is critical for performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。