用局部算子之和重构大模型线性层,实现高效压缩与稳定性能。
A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models

- 将密集线性映射表示为可直接执行的重叠局部张量算子之和
- 在Qwen3-8B和LLaMA2-7B上实现参数与计算量显著降低,性能保持稳定
- 揭示模型对结构简化的敏感边界,适合模型压缩与高效推理研究者
密集线性映射承担了现代神经网络大部分参数与计算开销,但其稠密形式使学习到的耦合关系隐含。量子多体物理通过将全局哈密顿量写成局部项之和($\hat H=\sum_k\hat h_k$)来组织指数级大的算符。这种结构原则能否应用于学习到的神经映射尚不清楚。我们提出张量混合(MixT),将密集映射表示为无需显式秩约束的重叠局部张量算子之和,可直接执行。局部项数量 $N_T$ 控制有效非局域性与算子复杂度,替换的Transformer块数 $N_B$ 将此结构坐标延伸至网络深度。在 Qwen3-8B 与 LLaMA2-7B 上测试显示,存在一个宽广的可恢复区域,随后出现急剧的、模型特异的边界,且该边界对 $N_T$ 变化具有惊人稳定性。准确率与输出分布统计在边界处共同重组;在 LLaMA2-7B 中,同一深度分隔出层间几何漂移的两种缩放规律。直接可执行的结构还降低了参数、算术运算、存储与内存开销。这些结果确立了局部求和结构在十亿参数规模下作为学习到的线性映射组织原则的可行性,并揭示其对结构简化的容忍存在尖锐边界。
原文摘要 · Abstract (English)
Dense linear maps carry much of the parameter and computational burden of modern neural networks, yet their dense form leaves the organization of learned couplings implicit. Quantum many-body physics organizes exponentially large operators by writing a global Hamiltonian as a sum of local terms, \(\hat H=\sum_k\hat h_k\). Whether the same structural principle can carry learned neural maps is unknown. We introduce Tensor Mixture (MixT), which represents a dense map as a natively executable sum of overlapping local tensor operators without imposing an explicit matrix-rank constraint. The local-term count \(N_T\) sets the effective nonlocality and operator complexity, while the number of replaced Transformer blocks \(N_B\) extends this structural coordinate across network depth. Tests on Qwen3-8B and LLaMA2-7B reveal a broad recoverable regime followed by an abrupt, model-specific boundary that is remarkably stable against changes in \(N_T\). Accuracy and output-distribution statistics reorganize together across the boundary; in LLaMA2-7B, the same depth separates two scaling regimes of inter-layer geometry drift. The directly executed structure also reduces parameters, arithmetic, storage, and memory. These results establish the local-sum structure as a viable organizing principle for learned linear maps at billion-parameter scale and expose a sharp boundary in their tolerance to structural simplification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。