用自适应池化提升Transformer在噪声中的鲁棒性，解决信号丢失问题。
Robust Noise Attenuation via Adaptive Pooling of Transformer Outputs
- 基于注意力的自适应池化，动态聚焦有效信号
- 在低信噪比下性能远超平均/最大池化等传统方法
- 适用于强化学习、视觉等含噪声输入的任务
我们研究了用于总结Transformer嵌入模型输出的池化方法设计，主要动机来自强化学习和视觉应用。本文关注的问题是：输入向量中仅部分包含下游任务所需信息（信号），其余为干扰项（噪声）。将池化视为以最小化信号损失为目标的向量量化时，我们发现常用的聚合方法——均值池化（AvgPool）、最大池化（MaxPool）和分类标记（ClsToken）——在输入信噪比（SNR）波动时易发生性能崩溃。随后我们证明，一种基于注意力的自适应池化方法可在任意信噪比下逼近最优信号向量量化器，且误差在理论可界定范围内。该结论首先在专为隔离信噪比问题设计的合成数据集上通过监督实验验证，进而推广至标准关系推理、多智能体强化学习及含噪声观测的视觉基准任务，显示采用自适应池化的Transformer在各类任务中均表现出更强鲁棒性。
原文摘要 · Abstract (English)
We investigate the design of pooling methods used to summarize the outputs of transformer embedding models, primarily motivated by reinforcement learning and vision applications. This work considers problems where a subset of the input vectors contains requisite information for a downstream task (signal) while the rest are distractors (noise). By framing pooling as vector quantization with the goal of minimizing signal loss, we demonstrate that the standard methods used to aggregate transformer outputs, AvgPool, MaxPool, and ClsToken, are vulnerable to performance collapse as the signal-to-noise ratio (SNR) of inputs fluctuates. We then show that an attention-based adaptive pooling method can approximate the signal-optimal vector quantizer within derived error bounds for any SNR. Our theoretical results are first validated by supervised experiments on a synthetic dataset designed to isolate the SNR problem, then generalized to standard relational reasoning, multi-agent reinforcement learning, and vision benchmarks with noisy observations, where transformers with adaptive pooling display superior robustness across tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。