用熵控制防止注意力坍缩，让变压器模型在强化学习中稳定放大。
TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
- 通过控制注意力熵，防止大模型训练时注意力分数坍缩。
- 在最大模型规模下性能比之前方法提升43%。
- 适合研究大规模强化学习值函数与变压器架构的学者。
尽管规模驱动了机器学习的显著进步，强化学习（RL）方法仍主要依赖小型价值函数。直接扩大价值函数——包括使用已知高度可扩展的变压器架构——常导致学习不稳定且性能下降。本文通过实证分析发现，这一扩缩失败的关键原因是：随着容量增加，注意力分数发生坍缩。我们的核心洞察是，通过控制注意力分数的熵，可有效防止坍缩并稳定训练，从而释放变压器在学习值函数中的扩展潜力。为此，我们提出变压器Q学习（TQL），该方法在从最小到最大的网络规模扩展中实现了最高43%的性能提升，而先前方法则出现性能退化。
原文摘要 · Abstract (English)
Despite scale driving substantial recent advancements in machine learning, reinforcement learning (RL) methods still primarily use small value functions. Naively scaling value functions -- including with a transformer architecture, which is known to be highly scalable -- often results in learning instability and worse performance. In this work, we ask what prevents transformers from scaling effectively for value functions? Through empirical analysis, we identify the critical failure mode in this scaling: attention scores collapse as capacity increases. Our key insight is that we can effectively prevent this collapse and stabilize training by controlling the entropy of the attention scores, thereby enabling the use of larger models. To this end, we propose Transformer Q-Learning (TQL), a method that unlocks the scaling potential of transformers in learning value functions in RL. Our approach yields up to a 43% improvement in performance when scaling from the smallest to the largest network sizes, while prior methods suffer from performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。