延迟注意力训练让混合模型更擅长处理更长序列。
Delayed Attention Training Improves Length Generalization in Transformer--RNN Hybrids
- 通过延迟训练注意力层，避免模型走捷径。
- 混合模型在三倍长序列上准确率超90%。
- 适合研究长序列建模与模型泛化能力的读者。
我们研究了序列模型在包含状态跟踪和关联回忆的复合任务中的长度泛化问题。先前研究发现，循环网络擅长状态跟踪但回忆能力弱，而变压器模型在回忆方面表现优异，却无法将状态跟踪能力推广到更长序列。受两种架构互补优势启发，我们构建了融合循环与注意力组件的混合模型，并在联合任务上进行训练，以评估是否能同时保留两种能力。结果表明，混合模型中变压器组件倾向于采用捷径解决方案，导致长度泛化性能差。我们识别出这种捷径依赖是主要障碍，并提出一种简单有效的训练策略——延迟注意力层的训练，有效缓解该问题，显著提升长度泛化性能。实验显示，该方法使混合模型在训练序列三倍长度的测试序列上达到近完美准确率（>90%）。
原文摘要 · Abstract (English)
We study length generalization in sequence models on a composite problem involving both state tracking and associative recall. Prior work finds that recurrent networks handle state tracking well but struggle with recall, whereas Transformers excel at recall yet fail to extend state-tracking capabilities to longer sequences. Motivated by the complementary strengths of these architectures, we construct hybrid models integrating recurrent and attention-based components, and train them on the combined task to evaluate whether both capabilities can be preserved. Our results reveal that, in such hybrids, the Transformer component tends to exploit shortcut solutions, leading to poor length generalization. We identify this shortcut reliance as a key obstacle and propose a simple yet effective training strategy -- delaying the training of the attention layers -- that mitigates this effect and significantly improves length generalization performance. Our experiments show that this approach enables hybrid models to achieve near-perfect accuracy ($>90\%$) on hybrid sequences three times longer than those used during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。