arXiv:2601.05807cs.LGcs.CL2026-01

不同位置编码融合方式影响长序列模型性能,应作为关键设计选择。

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers

  • 对比加法、拼接投影、门控融合三种融合方式,控制变量验证效果差异。
  • 长文本任务中门控融合显著提升性能,短文本则无明显差别。
  • 轻量卷积门控机制增强局部结构感知,适合长文档建模。

Transformer 需要位置编码来表示序列顺序,但以往研究多聚焦于设计新位置编码,而非探究位置信息如何与词嵌入融合。本文系统研究融合机制本身对性能的影响,尤其在长序列场景下。在相同架构、数据划分和随机种子下,对比加法、拼接+投影、标量门控三种经典融合策略。在涵盖短(AG News)、中(IMDB)和长(ArXiv)序列的三个文本分类数据集上,发现融合方式对短文本影响微乎其微,但在长文档上具有一致性提升。通过配对种子分析和跨数据集比较,验证该提升为结构性而非偶然。进一步实验表明,可学习融合在多种位置编码族间具有泛化能力。最后引入轻量卷积门控机制,在长文档上引入局部归纳偏置。结果表明,位置编码融合是长序列Transformer中的非平凡设计选择,应作为显式建模决策而非默认设置。

原文摘要 · Abstract (English)

Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies--element-wise addition, concatenation with projection, and scalar gated fusion--under identical Transformer architectures, data splits, and random seeds. Experiments on three text classification datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than stochastic, we perform paired-seed analysis and cross-dataset comparison across sequence-length regimes. Additional experiments on the ArXiv dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.

Transformer位置编码长序列融合机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。