解决多模态大模型在线编辑时的跨模态干扰与长期累积误差问题。
Modality-Decoupled Online Recursive Editing

- 分模态处理文本与视觉信息,避免视觉主导更新
- 每次编辑仅需常数时间开销,支持持续学习
- 适合需要长期适应的新知识注入场景
多模态大语言模型(MLLM)的在线编辑需在有限算力和内存下处理连续修正流。然而,为纯文本大模型设计的编辑器在多模态场景中表现下降:视觉激活过强会扭曲更新统计,引发跨模态冲突;顺序编辑在共享空间中纠缠,导致长时程干扰。为此,我们提出M-ORE,一种面向终身学习的模态解耦在线递归编辑方法。基于统一的近端投影公式,可实现闭式更新,并通过Sherman-Morrison递推保持恒定单次编辑开销。它分别维护文本堆栈与视觉投影模块的局部统计,避免视觉主导更新;并通过固定正交低秩子空间中的连续更新,缓解长期干扰。在多个MLLM主干网络和在线编辑基准上的实验表明,M-ORE在可靠性、泛化性和局部性上均优于强基线,且具备优良的质量-效率权衡。代码已开源:https://github.com/lab-klc/M-ORE。
原文摘要 · Abstract (English)
Online model editing for multimodal large language models (MLLMs) requires assimilating a stream of corrections under tight compute and memory budgets. Yet editors developed for text-only LLMs often degrade on MLLMs: visually dominant activations skew the statistics that shape updates, causing cross-modal conflict, while sequential writes become entangled in a shared edit space and amplify long-horizon interference, causing inter-edit interference. To address these, we propose M-ORE, a modality-decoupled online recursive editor for lifelong MLLM adaptation. M-ORE is derived from a unified proximal-projection formulation and admits a closed-form update with a Sherman-Morrison recursion, yielding constant per-edit overhead. It maintains module-wise locality statistics for the text stack and the visual projector to avoid visually dominated update shaping and performs continual updates in a fixed orthogonal low-rank edit subspace via a Sherman-Morrison recursion to mitigate long-horizon interference. Experiments on multiple MLLM backbones and online editing benchmarks show that our M-ORE method consistently improves reliability, generality, and locality over strong baselines, while achieving favorable quality-efficiency scaling. Our code is publicly available at https://github.com/lab-klc/M-ORE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。