对比八种信号编码器,发现标准线性投影已足够好。
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
- 用标准逐通道线性投影实现最优性能,无需复杂设计。
- 小规模通道数下,位置编码投影略优;大规模时非线性结构有小幅提升。
- 无需额外正则化,模型自发实现通道正交,保留通道信息。
针对多通道标量信号的Transformer模型,本文评估了八种输入编码器:共享标量基线、逐通道线性投影、正交性正则化、非线性MLP、块分割拼接、通道独立与通道作为标记架构,以及投影位置编码。实验在含通道信息的合成基准和ETTh1数据集上进行,以下一步负对数似然为评分指标。结果显示,标准逐通道线性投影与所有其他方法性能接近,差异虽统计显著但实际影响微小。几何分析表明,即使无显式正则化,投影也自发趋向正交,使标准方法能从叠加嵌入中恢复通道身份。两个方法表现差:共享标量基线因信息论原因崩溃;通道独立结构在合成数据上普遍过拟合,在真实数据上表现不佳。配对测试显示,在小通道数时,对正弦位置编码进行可学习投影略有优势;大通道数下,非线性前馈层表现稍好,且随着训练数据增加,差距缩小。结论建议默认使用标准线性投影,仅在任务特殊时才采用更复杂方案。
原文摘要 · Abstract (English)
Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_{\text{model}}$-dimensional vector per time step. We audit eight input encoders -- a shared-scalar baseline, per-channel linear projections, an orthogonality regulariser, a nonlinear MLP, block-partitioned concatenation, channel-independent and channel-as-token architectures, and a projected positional encoding -- on a synthetic benchmark where channel identity is informative and on ETTh1, scored by next-step negative log-likelihood. The headline is practical near-equivalence within a wide "top tier": the standard per-channel linear projection matches every alternative up to small, statistically real but practically modest differences. A direct geometric probe attributes this to a spontaneous orthogonalisation of the per-channel projections: with no explicit regulariser they tighten well beyond the near-orthogonality of random initialisation, letting the standard linear recover channel identity from the summed embedding. Two encoders lose decisively: the shared-scalar baseline collapses for information-theoretic reasons we make explicit, and the channel-independent PatchTST-spirit baseline overfits universally on the synthetic benchmark and underperforms on both. Paired tests resolve two small gaps: projecting the sinusoidal positional encoding through a learned linear layer edges the rest at small $C$ by extending this orthogonality to the positional subspace; a nonlinear MLP stem edges them at the largest $C$, with the gap shrinking under more training data. The practical recommendation: use the standard per-channel linear projection by default; reach for something more elaborate only when the task calls for it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。