提出模块化表示框架MoRe,让模型持续学习时像人脑一样有层次地组织知识。
MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data

- 通过时间延迟依赖发现数据内在结构,自动分解出基础与特定模块
- 在真实LLM激活数据上实现更好稳定性与可塑性平衡
- 适合需要长期学习且保持旧知识的AI系统
持续学习要求模型在适应新数据的同时保留已有知识。核心挑战在于如何以最小干扰实现一次性的知识更新。现有方法多通过调整参数或架构进行任务特定的监督式修改,但根本问题在于表示层面:不同任务需要独特而有序的表示,可在不破坏原有表示的前提下选择性更新,且结构应反映数据内在组织而非人为任务边界。在序列数据中,时间延迟依赖提供了揭示这种组织的自然信号,展示基本表示如何生成更具体表示。受人类大脑模块化结构启发,我们提出MoRe框架,将模块化直接体现在表示本身,而非架构层面。MoRe通过可辨识性保证,将知识分解为基础与特定模块的层级结构,在适应过程中支持模块的复用、对齐与扩展,同时通过构造确保旧模块不变。在合成基准和真实世界大型语言模型(LLM)激活数据上的实验表明,该方法能揭示可解释的层级结构,并显著提升塑性-稳定性权衡,证明其作为持续适应原则性基础的潜力。
原文摘要 · Abstract (English)
Continual learning requires models to adapt to new data while preserving previously acquired knowledge. At its core, this challenge can be viewed as principled one-step adaptation: incorporating new information with minimal interference to existing representations. Most existing approaches address this challenge by modifying model parameters or architectures in a supervised, task-specific manner. However, the underlying issue is representational: tasks require distinct yet structured representations that can be selectively updated without disrupting representations, while structure should reflect intrinsic organization in the data rather than task boundaries. In sequential data, time-delayed dependencies provide a natural signal for uncovering this organization, revealing how fundamental representations give rise to more specific ones. Inspired by the modular organization of the human brain, we propose MoRe, a framework that identifies modularity in the representation itself rather than allocating it at the architectural level. MoRe decomposes knowledge into a hierarchy of fundamental and specific modules with identifiability guarantees, enabling principled module reuse, alignment, and expansion during adaptation while preserving old modules by construction. Experiments on synthetic benchmarks and real-world LLM activations demonstrate interpretable hierarchical structure, improved plasticity-stability trade-offs, suggesting MoRe as a principled foundation for continual adaptation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。