用Transformer学习能力识别相变，无需精确模型也能准确定位临界点。
Attention to Order: Transformers Discover Phase Transitions via Learnability
- 用自监督学习让Transformer从微观态中提取结构，以学习能力判断相变
- 有序相学习损失更低、注意力模式更结构化，无序相则难以学习
- 两个无监督指标精准捕捉临界温度，适合物理与机器学习交叉研究者
相变标志着集体行为的质变，但当解析解不可得且传统模拟失效时，识别其边界仍具挑战。本文提出以学习能力作为普适判据：即含注意力机制的Transformer模型从微观状态中提取结构的能力。基于二维伊辛模型的蒙特卡洛生成构型与自监督学习，我们发现有序相具有更高的学习能力，表现为训练损失降低和注意力模式结构化，而无序相则对学习保持抗拒。两种无监督诊断——训练损失的突变与注意力熵的上升——准确恢复了临界温度，与精确值高度一致。结果确立了学习能力作为数据驱动的相变标志，并揭示凝聚态物理中的长程有序与现代语言模型中结构涌现之间的深层关联。
原文摘要 · Abstract (English)
Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce learnability as a universal criterion, defined as the ability of a transformer model containing attention mechanism to extract structure from microscopic states. Using self-supervised learning and Monte Carlo generated configurations of the two-dimensional Ising model, we show that ordered phases correspond to enhanced learnability, manifested in both reduced training loss and structured attention patterns, while disordered phases remain resistant to learning. Two unsupervised diagnostics, the sharp jump in training loss and the rise in attention entropy, recover the critical temperature in excellent agreement with the exact value. Our results establish learnability as a data-driven marker of phase transitions and highlight deep parallels between long-range order in condensed matter and the emergence of structure in modern language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。