MAHA通过分层注意力与博弈优化,显著降低长文本建模计算量。
Multiscale Aggregated Hierarchical Attention (MAHA): A Game Theoretic and Optimization Driven Approach to Efficient Contextual Modeling in Large Language Models
- 分层下采样动态划分序列,多尺度融合用博弈论求解最优分配
- 4096序列时计算量减少81%,远超传统注意力机制
- 适合追求高效长文本处理的下一代大模型研发者
多头自注意力(MHSA)的二次计算复杂度是大型语言模型在长文本任务中扩展的核心瓶颈。尽管稀疏化和线性注意力机制试图缓解此问题,但常牺牲全局依赖关系表达或难以捕捉多尺度语义粒度。本文提出一种新型架构——多尺度聚合分层注意力(MAHA),通过分层分解与数学严谨的聚合重构注意力机制。不同于传统单尺度处理,MAHA利用可学习的下采样算子将输入序列动态划分为分层尺度。核心创新在于其聚合策略:将各尺度注意力矩阵的融合建模为资源分配问题,通过凸优化框架或基于纳什均衡的博弈论方法求解,确保局部细节与全局上下文保真度之间的理论最优平衡。该结构嵌入混合空洞卷积-变压器主干网络,采用可微优化层实现端到端训练。实验表明,MAHA具备卓越可扩展性;实际浮点运算量分析显示,在序列长度4096时计算成本相比标准注意力降低81%。本工作连接优化理论与序列建模,为下一代大模型提供可扩展解决方案。
原文摘要 · Abstract (English)
The quadratic computational complexity of MultiHead SelfAttention (MHSA) remains a fundamental bottleneck in scaling Large Language Models (LLMs) for longcontext tasks. While sparse and linearized attention mechanisms attempt to mitigate this, they often compromise the representation of global dependencies or fail to capture multiscale semantic granularity effectively. In this paper, we propose Multiscale Aggregated Hierarchical Attention (MAHA), a novel architectural framework that reformulates the attention mechanism through hierarchical decomposition and mathematically rigorous aggregation. Unlike conventional approaches that treat token interactions at a single resolution, MAHA dynamically partitions the input sequence into hierarchical scales via learnable downsampling operators. The core innovation lies in its aggregation strategy: we model the fusion of scalespecific attention matrices as a resource allocation problem, solved via a convex optimization framework or a Nash equilibriumbased gametheoretic approach. This ensures a theoretically optimal balance between local nuance and global context fidelity. Implemented within a hybrid dilatedconvolutional transformer backbone, MAHA utilizes differentiable optimization layers to enable endtoend training. Experimental evaluations demonstrate that MAHA achieves superior scalability; empirical FLOPs analysis confirms an 81% reduction in computational cost at a sequence length of 4096 compared to standard attention. This work bridges the gap between optimization theory and sequence modeling, offering a scalable solution for nextgeneration LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。