归一化层可提供全局上下文,远超卷积感受野的限制。
Beyond receptive fields: sequence-pooled normalization can supply most of a sequence labeler's context
- 通过序列池化归一化,绕过卷积感受野实现全局信息传递。
- 在长标签序列中,仅9位置感受野即接近全序列最优,误差仅0.009。
- 适用于基因组标注等长序列任务,尤其适合高标签连续性场景。
卷积序列标注模型的感受野常被视为可用上下文的上限:它决定扩张策略、限制流式处理范围,并支持局部性假设。然而我们发现这可能是错误的:当归一化层在推理时从当前输入沿序列计算统计量,这些统计量会形成一条贯穿整个序列的路径,绕过卷积感受野,提供全局上下文。该结论由层的雅可比矩阵推导得出(无需实验验证),且该路径传递的信息具有闭合形式。在一个可计算最优解的合成标注任务中,序列跨度归一化所编码的全局摘要,已几乎涵盖扩大感受野所能带来的全部收益:网络达到9个位置时,与全序列最优相差仅0.009,远优于近随机基准。若将统计量改为逐位置独立计算,相当于切断该路径,使扩大感受野的价值在模拟基因组上所有测试难度下提升达一个数量级,在真实1000 Genomes单倍型数据上亦然。该路径还干扰了归因分析:移除增强感受野的模块会切断部分路径,导致其贡献被高估8.3至16.1倍,远高于从零重训练的结果。当标签切换频繁时,这种归一化对感受野的替代效应减弱。在标签持续较长时,尽管感受野解释和归因分析的数值均无误,但两者都错误地归功于错误的组件。
原文摘要 · Abstract (English)
A convolutional sequence labeler's receptive field is routinely treated as the extent of the model's usable context: it sets dilation schedules, bounds streaming horizons, and underwrites locality claims. However, we show that this can be false: when a normalization layer computes statistics from the current input along the sequence at inference, those statistics open a sequence-spanning path that bypasses the convolutional receptive field to provide global context. We derive this from the layer's Jacobian (the criterion needs no experiment), and what the path carries has a closed form. On a synthetic labeling process with computable optima, the global summary that a sequence-spanning normalization encodes already supplies almost all of what a larger receptive field would buy where labels come in long runs: a network reaching 9 positions comes within 0.009 of the whole-sequence optimum, against a near-chance bound for its reach. Closing the path, by taking the same statistics per position, multiplies what enlarging the receptive field is worth by up to an order of magnitude on simulated genomes at every difficulty level tested and on real 1000 Genomes haplotypes. The same path also confounds attribution: ablating a trained network's receptive-field-enlarging blocks severs part of the path, overstating their contribution 8.3-16.1-fold relative to retraining from scratch. The substitution of normalization for receptive field fades as labels switch more often. Where labels run long, neither the receptive-field justification nor the ablation is wrong about its numbers, but both credit the wrong component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。