arXiv:2502.19078cs.CL2025-02Conference of the …被引 4

让大模型像人脑一样智能省电,复杂句子激活更多参数。

Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs

  • 根据输入复杂度动态调整激活参数,结合统计稀疏与语义感知。
  • 平均提速20%且精度下降不足2%,优于现有方法。
  • 首次将脑电波机制与模型效率关联,适合资源受限部署。

稠密大语言模型因固定激活所有参数而面临严重效率瓶颈。现有稀疏化方法(静态剪枝或动态激活)或缺乏对上下文与模型结构的适应性,或带来高昂计算开销。受人类大脑双过程机制启发——预测编码(N400)实现基础稀疏,结构重分析(P600)应对复杂语境,我们提出CLADA框架,融合统计稀疏与语义自适应。关键发现:模型激活呈现两种互补模式——1)由序列前缀信息驱动的全局统计稀疏;2)由认知负荷指标(如意外度、熵)调控的局部语义适应性。CLADA采用分层阈值策略:基于离线误差控制优化获得40%以上稀疏基线,并实时由认知信号动态调整。在六种主流模型和九个基准上的评估显示,CLADA实现约20%平均加速,精度损失小于2%,显著优于Griffin(5%+退化)和TT(几乎无提速)。重要的是,通过多层次回归分析首次建立神经语言学事件相关电位(ERP)成分与模型效率机制的正式关联($R^2=0.17$)。无需重训练或架构修改,可直接部署于资源敏感型推理场景,推动生物启发式AI设计。代码已开源。

原文摘要 · Abstract (English)

Dense large language models(LLMs) face critical efficiency bottlenecks as they rigidly activate all parameters regardless of input complexity. While existing sparsity methods(static pruning or dynamic activation) address this partially, they either lack adaptivity to contextual or model structural demands or incur prohibitive computational overhead. Inspired by human brain's dual-process mechanisms - predictive coding (N400) for backbone sparsity and structural reanalysis (P600) for complex context - we propose CLADA, a \textit{\textbf{C}ognitive-\textbf{L}oad-\textbf{A}ware \textbf{D}ynamic \textbf{A}ctivation} framework that synergizes statistical sparsity with semantic adaptability. Our key insight is that LLM activations exhibit two complementary patterns: 1) \textit{Global statistical sparsity} driven by sequence-level prefix information, and 2) \textit{Local semantic adaptability} modulated by cognitive load metrics(e.g., surprisal and entropy). CLADA employs a hierarchical thresholding strategy: a baseline from offline error-controlled optimization ensures 40\%+ sparsity, dynamically adjusted by real-time cognitive signals. Evaluations across six mainstream LLMs and nine benchmarks demonstrate that CLADA achieves \textbf{~20\% average speedup with <2\% accuracy drop}, outperforming Griffin (5\%+ degradation) and TT (negligible speedup). Crucially, we establish the first formal connection between neurolinguistic event-related potential (ERP) components and LLM efficiency mechanisms through multi-level regression analysis ($R^2=0.17$ for sparsity-adaptation synergy). Requiring no retraining or architectural changes, CLADA offers a deployable solution for resource-aware LLM inference while advancing biologically-inspired AI design. Our code is available at \href{https://github.com/Oldify/CLADA}{CLADA}.

大模型优化动态激活生物启发

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。