通过感知混淆与置信度的混合模型，提升视觉语言模型的任务专精与泛化能力
CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization
- 设计混淆感知损失，优化易混淆类别的决策边界
- 引入置信度加权混合机制，在不牺牲专精的前提下提升泛化性能
- 适合需要高鲁棒性的跨域任务部署场景
提示调优通过冻结模型参数仅优化提示，已证明在特定任务上有效。其核心挑战在于提升任务专精性与未见领域的泛化能力。然而，冻结编码器常产生特征错位，导致类别混淆，限制专精性。为此，我们提出混淆感知损失（CoA-loss），通过细化易混淆类别的决策边界来增强专精性。同时，我们从数学上证明混合模型可在不损害专精性前提下提升泛化性，通过置信度感知权重（CoA-weights）动态调整混合模型中各预测结果的权重。大量实验表明，采用CoA-loss与CoA-weights的CoCoA-Mix混合模型显著优于现有方法，兼顾了专精与泛化。代码已公开于https://github.com/url-kaist/CoCoA-Mix。
原文摘要 · Abstract (English)
Prompt tuning, which adapts vision-language models by freezing model parameters and optimizing only the prompt, has proven effective for task-specific adaptations. The core challenge in prompt tuning is improving specialization for a specific task and generalization for unseen domains. However, frozen encoders often produce misaligned features, leading to confusion between classes and limiting specialization. To overcome this issue, we propose a confusion-aware loss (CoA-loss) that improves specialization by refining the decision boundaries between confusing classes. Additionally, we mathematically demonstrate that a mixture model can enhance generalization without compromising specialization. This is achieved using confidence-aware weights (CoA-weights), which adjust the weights of each prediction in the mixture model based on its confidence within the class domains. Extensive experiments show that CoCoA-Mix, a mixture model with CoA-loss and CoA-weights, outperforms state-of-the-art methods by enhancing specialization and generalization. Our code is publicly available at https://github.com/url-kaist/CoCoA-Mix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。