Prism让多模态模型持续学习新任务更简单可靠。
Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning
- 通过插件机制分离算法与基础模型，无需修改核心代码
- 支持大规模训练流水线，实现可复现的持续训练
- 适合研究者快速验证新方法，避免重复造轮子
多模态大语言模型（MLLMs）通过指令微调将多样化任务统一为指令跟随框架，实现多功能性。然而真实场景中需持续适应新任务，推动了多模态持续指令微调（MCIT）的发展。当前MCIT研究受限于严重工程瓶颈：现有方法通常直接修改基础MLLM代码，带来高开发成本和方法特异性架构，严重阻碍代码复用与公平比较。为此，我们提出Prism——一个专为可扩展MCIT研究设计的即插即用、可复现代码库。它通过轻量级插件注册机制，将算法开发与骨干模型实现分离，使新策略可作为独立插件集成，无需改动底层MLLM代码，消除结构碎片化，加速方法迭代。Prism原生支持主流大规模训练流水线，保障实验可复现性与可扩展性。代码已开源：https://github.com/LAMDA-CL/Prism。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) achieve versatility by reformulating diverse tasks into a unified instruction-following framework via instruction tuning. However, real-world deployment requires continuous adaptation to emerging tasks, motivating Multimodal Continual Instruction Tuning (MCIT). Despite its growing importance, current MCIT research is hindered by severe engineering bottlenecks. Existing methods are typically implemented by directly modifying the base MLLM codebase, which imposes substantial implementation overhead and yields method-specific architectures that severely limit code reuse and fair comparison. To address this, we introduce Prism, a plug-in reproducible codebase specifically designed for scalable MCIT research. It separates algorithmic development from the backbone implementation via a lightweight plugin registration mechanism, enabling new strategies to be integrated as independent plugins without modifying the underlying MLLM codebase, thereby eliminating structural fragmentation and accelerating method development. Prism natively supports widely used large-scale training pipeline, thereby enabling reproducible and scalable MCIT experimentation. Code is available at https://github.com/LAMDA-CL/Prism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。