用博弈论和统计物理重构注意力,捕捉更复杂的文本依赖关系。
NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics
- 将词元视为博弈玩家和自旋系统,结合沙普利值与邦扎夫指数量化重要性
- 在SNLI上达86.4%准确率,优于ALBERT-Base,接近RoBERTa-Base表现
- 通过蒙特卡洛估计避免计算爆炸,适合长序列建模,理论可解释性强
标准Transformer的注意力机制受限于成对建模方式,难以捕捉词元间的高阶依赖。本文提出神经博弈变压器(NeuroGame Transformer, NGT),从双重视角重构注意力:词元同时被视作合作博弈中的参与者与统计物理中的自旋。使用两种互补的博弈论概念——沙普利值(用于全局、置换不变归因)与邦扎夫指数(用于局部联盟影响力)——通过可学习门控参数组合形成外磁场,而成对相互作用势能捕捉协同关系。系统的能量遵循伊辛哈密顿量,注意力权重作为吉布斯分布下的边缘概率,通过平均场方程高效求解。为应对联盟空间指数级增长带来的计算挑战,设计了基于吉布斯分布权重的重要度加权蒙特卡洛估计器,避免显式计算指数项,保障长序列下的数值稳定性。理论证明了收敛性,并刻画了由插值参数控制的公平性-敏感性权衡。实验表明,NGT在SNLI和MNLI-matched任务上表现优异,超越部分主流高效Transformer基线。在SNLI上测试准确率达86.4%(验证峰值86.6%),优于ALBERT-Base,且与RoBERTa-Base保持高度竞争力。代码已开源:https://github.com/dbouchaffra/NeuroGame-Transformer。
原文摘要 · Abstract (English)
Standard attention mechanisms in transformers are limited by their pairwise formulation, which hinders the modeling of higher-order dependencies among tokens. We introduce the NeuroGame Transformer (NGT) to overcome this by reconceptualizing attention through a dual perspective: tokens are treated simultaneously as players in a cooperative game and as interacting spins in a statistical physics system. Token importance is quantified using two complementary game-theoretic concepts -- Shapley values for global, permutation-based attribution and Banzhaf indices for local, coalition-level influence. These are combined via a learnable gating parameter to form an external magnetic field, while pairwise interaction potentials capture synergistic relationships. The system's energy follows an Ising Hamiltonian, with attention weights emerging as marginal probabilities under the Gibbs distribution, efficiently computed via mean-field equations. To ensure scalability despite the exponential coalition space, we develop importance-weighted Monte Carlo estimators with Gibbs-distributed weights. This approach avoids explicit exponential factors, ensuring numerical stability for long sequences. We provide theoretical convergence guarantees and characterize the fairness-sensitivity trade-off governed by the interpolation parameter. Experimental results demonstrate that the NeuroGame Transformer achieves strong performance across SNLI, and MNLI-matched, outperforming some major efficient transformer baselines. On SNLI, it attains a test accuracy of 86.4\% (with a peak validation accuracy of 86.6\%), surpassing ALBERT-Base and remaining highly competitive with RoBERTa-Base. Code is available at https://github.com/dbouchaffra/NeuroGame-Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。