受大脑工作记忆启发，提升大模型长文本理解能力
PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding
- 引入动态记忆库模拟前额叶神经持续放电，缓解信息衰减
- 将前馈网络权重重组为语义模块，增强跨标记依赖关系
- 在20万词上下文中保持性能，适用于各类大模型升级
尽管大语言模型在多个领域表现优异，但其长上下文理解能力受限于瞬时神经激活导致的信息衰减，以及前馈网络权重无序分布引发的语义碎片化。受大脑工作记忆与皮层模块化启发，本文提出PaceLLM，包含两项创新：(1) 持续活动（PA）机制，通过引入激活级记忆库，动态检索、复用并更新关键前馈网络状态，缓解上下文衰减；(2) 皮层专家（CE）聚类，模拟任务自适应神经专化，将前馈网络权重重新组织为语义模块，建立跨标记依赖关系，减轻碎片化。大量实验表明，PaceLLM在LongBench多文档问答任务上提升6%，在Infinite-Bench任务上取得12.5%-17.5%性能增益，且在针堆中找针（NIAH）测试中可扩展至20万词上下文长度。该工作开创性地实现脑启发式大模型优化，可无缝适配任意模型，无需结构改造即可提升长文本性能与可解释性。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) demonstrate strong performance across domains, their long-context capabilities are limited by transient neural activations causing information decay and unstructured feed-forward network (FFN) weights leading to semantic fragmentation. Inspired by the brain's working memory and cortical modularity, we propose PaceLLM, featuring two innovations: (1) a Persistent Activity (PA) Mechanism that mimics prefrontal cortex (PFC) neurons' persistent firing by introducing an activation-level memory bank to dynamically retrieve, reuse, and update critical FFN states, addressing contextual decay; and (2) Cortical Expert (CE) Clustering that emulates task-adaptive neural specialization to reorganize FFN weights into semantic modules, establishing cross-token dependencies and mitigating fragmentation. Extensive evaluations show that PaceLLM achieves 6% improvement on LongBench's Multi-document QA and 12.5-17.5% performance gains on Infinite-Bench tasks, while extending measurable context length to 200K tokens in Needle-In-A-Haystack (NIAH) tests. This work pioneers brain-inspired LLM optimization and is complementary to other works. Besides, it can be generalized to any model and enhance their long-context performance and interpretability without structural overhauls.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。