测试Transformer在算法推理中的泛化能力，发现注意力机制会阻碍模型跨域推理。
Attend or Perish: Benchmarking Attention in Algorithmic Reasoning
- 设计五种无限输入域的算法任务，区分真实理解与记忆。
- 发现注意力机制直接导致模型在新输入类型上无法泛化。
- 提供可复现的代码和可解释性工具，适合研究模型泛化性的学者。
Transformer能否在未见过的输入/输出领域中可靠地执行算法任务？尽管预训练语言模型在包含算法推理的基准上表现良好，但评估其结果可靠性需区分真正的算法理解与记忆。本文提出AttentionSpan，一个包含五个无限输入域的算法基准，可分离并追踪完成任务所需的稳健算法。该基准能评估模型对未见输入类型（如新长度、数值范围或输入域）的外推能力，以及学习机制的鲁棒性。通过分析注意力图并进行定向干预，我们发现注意力机制直接导致外推失败。所有任务及可解释性方法的实现已公开于https://github.com/michalspiegel/AttentionSpan。
原文摘要 · Abstract (English)
Can transformers learn to perform algorithmic tasks reliably across previously unseen input/output domains? While pre-trained language models show solid accuracy on benchmarks incorporating algorithmic reasoning, assessing the reliability of these results necessitates an ability to distinguish genuine algorithmic understanding from memorization. In this paper, we propose AttentionSpan, an algorithmic benchmark comprising five tasks of infinite input domains where we can disentangle and trace the correct, robust algorithm necessary for the task. This allows us to assess (i) models' ability to extrapolate to unseen types of inputs, including new lengths, value ranges or input domains, but also (ii)to assess the robustness of their learned mechanisms. By analyzing attention maps and performing targeted interventions, we show that attention mechanism directly causes failures in extrapolation. We make the implementation of all our tasks and interpretability methods publicly available at https://github.com/michalspiegel/AttentionSpan .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。