用轻量卷积增强大模型对局部语义的感知，几乎不增加参数。
Convolution for Large Language Models
- 在注意力前对查询、键、值应用残差深度卷积，提升局部建模能力。
- 在多个数据集上平均准确率提升，参数增益低于0.01%。
- 适合追求轻量化性能优化的NLP研究者与工程落地团队。
大型语言模型主要依赖Transformer结构，其自注意力机制虽能实现全局词元交互，但未显式编码自然语言的局部性。本文探讨轻量级深度卷积是否可在不显著增加模型规模的前提下，提供这种局部先验知识。宏观消融实验对比了在Qwen3 Transformer模块中17个位置引入卷积的效果，发现将卷积应用于注意力前的投影查询、键、值时表现最佳。微观分析进一步支持使用核大小为3的残差深度卷积，无需额外归一化或激活函数。该设计在多个Qwen3模型及不同预训练数据预算下，均在七个下游基准任务上提升了平均准确率，同时参数增长不足0.01%。表示层案例研究显示，卷积使重复词元对邻近上下文更敏感。结果表明，深度卷积可作为自注意力的有效轻量补充，用于建模短程词元交互。
原文摘要 · Abstract (English)
Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size. Our macro-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention. A subsequent micro-level study favors a residual depthwise convolution with kernel size $k=3$, without additional normalization or activation. Across Qwen3 models and several pre-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than $0.01\%$ parameters. A representation-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context. These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。