轻量级图文评分器，高效评估与训练图像描述模型。
L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training
- 用压缩与蒸馏技术构建轻量双编码器CLIP模型
- 在保持对齐能力的同时降低计算开销与耗时
- 适合资源受限场景下的图文生成评估与训练
我们提出一种新型基于嵌入的图像描述评价指标L-CLIPScore，可用于高效评估描述质量并指导模型训练。L-CLIPScore基于一个轻量级CLIP（L-CLIP）计算，该模型通过权重复用和矩阵分解分别压缩编码器参数与词嵌入矩阵，并设计了一种新颖的多模态相似性调节器（SR）损失进行知识蒸馏。SR损失在匹配图文对时增强嵌入相似性，在非匹配对时降低相似性，从而有效传递视觉-语言对齐知识。实验表明，尽管参数更少、计算开销更低，L-CLIP仍能实现与原版CLIP相当的多模态对齐能力。使用L-CLIPScore作为评价指标时表现出优异的效率与有效性；但在训练阶段，仅使用L-CLIPScore会导致训练失败，需与n-gram类指标混合使用。
原文摘要 · Abstract (English)
We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。