用小模型复现大模型的图文融合能力，速度快10倍还省90%参数
Distilling Textual Priors from LLM to Efficient Image Fusion
- 用教师-学生架构，把大模型的图文先验知识蒸馏到小模型中
- 蒸馏后模型仅需10%参数和推理时间，性能保留90%且超越现有最优方法
- 适合需要高效图文融合的部署场景，如移动端或实时系统
多模态图像融合旨在从多个输入源合成一张综合图像。传统方法如CNN和GAN虽高效，但难以处理低质量或复杂输入。近期文本引导方法利用大模型先验提升性能，却带来显著计算开销。为此，我们提出一种新框架，通过蒸馏将大模型先验知识迁移至小型网络，在推理时无需文本引导，大幅降低模型规模。该框架采用教师-学生结构，教师网络整合大模型先验，通过定制化蒸馏过程将知识传递给更小的学生网络。此外，引入空间-通道交叉融合模块，增强模型在空间与通道维度上对文本先验的利用能力。实验表明，所提方法在计算效率与融合质量间取得良好平衡：蒸馏后模型仅需教师网络10%的参数和推理时间，保留90%性能，并优于现有最先进方法。代码将开源发布。
原文摘要 · Abstract (English)
Multi-modality image fusion aims to synthesize a single, comprehensive image from multiple source inputs. Traditional approaches, such as CNNs and GANs, offer efficiency but struggle to handle low-quality or complex inputs. Recent advances in text-guided methods leverage large model priors to overcome these limitations, but at the cost of significant computational overhead, both in memory and inference time. To address this challenge, we propose a novel framework for distilling large model priors, eliminating the need for text guidance during inference while dramatically reducing model size. Our framework utilizes a teacher-student architecture, where the teacher network incorporates large model priors and transfers this knowledge to a smaller student network via a tailored distillation process. Additionally, we introduce spatial-channel cross-fusion module to enhance the model's ability to leverage textual priors across both spatial and channel dimensions. Our method achieves a favorable trade-off between computational efficiency and fusion quality. The distilled network, requiring only 10% of the parameters and inference time of the teacher network, retains 90% of its performance and outperforms existing SOTA methods. Extensive experiments demonstrate the effectiveness of our approach. The implementation will be made publicly available as an open-source resource.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。