小模型生成个性化信息,大模型高效融合,实现低成本个性化大模型应用。
Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM
- 小模型负责生成个性化内容,大模型在推理时整合并验证信息。
- 仅需训练一个元个性化小模型,即可适配多种大模型,显著降低训练成本。
- 支持开源与闭源大模型,适合希望低成本实现个性化服务的开发者。
将视觉语言模型(VLM)个性化以转变为日常助手已成为研究热点。然而,像OpenAI这样的领先公司持续增大模型规模并引入复杂设计如思维链(CoT)。尽管大VLM在多模态理解上表现优异,但其高昂的训练成本和受限制的付费API访问阻碍了直接个性化。相比之下,小VLM易于个性化且免费可用,但缺乏足够推理能力。为此,我们提出一种名为小-大协同(SLC)的新框架,让小VLM生成个性化信息,大模型则在推理时融合这些信息并输出准确回答。为有效整合个性化内容,我们设计了测试时反思策略,防止小模型产生幻觉。由于SLC只需训练一个元个性化小模型来服务多个大模型,整体过程训练高效。据我们所知,这是首个支持开源与闭源大模型的训练高效个性化框架,推动了更广泛的实际应用。我们在多个基准和大模型上进行了充分实验,验证了SLC的有效性。代码将在https://github.com/Hhankyangg/SLC发布。
原文摘要 · Abstract (English)
Personalizing Vision-Language Models (VLMs) to transform them into daily assistants has emerged as a trending research direction. However, leading companies like OpenAI continue to increase model size and develop complex designs such as the chain of thought (CoT). While large VLMs are proficient in complex multi-modal understanding, their high training costs and limited access via paid APIs restrict direct personalization. Conversely, small VLMs are easily personalized and freely available, but they lack sufficient reasoning capabilities. Inspired by this, we propose a novel collaborative framework named Small-Large Collaboration (SLC) for large VLM personalization, where the small VLM is responsible for generating personalized information, while the large model integrates this personalized information to deliver accurate responses. To effectively incorporate personalized information, we develop a test-time reflection strategy, preventing the potential hallucination of the small VLM. Since SLC only needs to train a meta personalized small VLM for the large VLMs, the overall process is training-efficient. To the best of our knowledge, this is the first training-efficient framework that supports both open-source and closed-source large VLMs, enabling broader real-world personalized applications. We conduct thorough experiments across various benchmarks and large VLMs to demonstrate the effectiveness of the proposed SLC framework. The code will be released at https://github.com/Hhankyangg/SLC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。