让模型按用户需求生成定制化特征表示,无需大量标注。
Conditional Representation Learning for Customized Tasks
- 用大模型生成描述词构建语义基,引导视觉特征对齐任务需求。
- 在分类与检索任务中显著优于通用表示,适应多种定制任务。
- 适合需要低成本适配特定场景的科研与工业应用。
传统表征学习方法生成通用表征,主要捕捉主导语义,可能不匹配特定下游任务。例如,在动物栖息地分析中,研究者关注场景特征,而通用嵌入更强调类别语义,导致效果不佳。现有方法依赖监督微调,但计算与标注成本高。本文提出条件表征学习(CRL),旨在提取符合用户指定标准的定制化表征。我们发现空间语义由其基决定,因此可用一组描述词近似构建定制特征空间的语义基。给定用户标准后,CRL首先利用大语言模型生成描述文本以构造语义基,再通过视觉语言模型将图像表征投影至该条件特征空间。所获条件表征更契合特定标准,可支持多种定制任务。在分类与检索任务上的大量实验表明,CRL具有优越性与通用性。代码已公开于 https://github.com/XLearning-SCU/2025-NeurIPS-CRL。
原文摘要 · Abstract (English)
Conventional representation learning methods learn a universal representation that primarily captures dominant semantics, which may not always align with customized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categorical semantics, leading to suboptimal results. As a solution, existing approaches resort to supervised fine-tuning, which however incurs high computational and annotation costs. In this paper, we propose Conditional Representation Learning (CRL), aiming to extract representations tailored to arbitrary user-specified criteria. Specifically, we reveal that the semantics of a space are determined by its basis, thereby enabling a set of descriptive words to approximate the basis for a customized feature space. Building upon this insight, given a user-specified criterion, CRL first employs a large language model (LLM) to generate descriptive texts to construct the semantic basis, then projects the image representation into this conditional feature space leveraging a vision-language model (VLM). The conditional representation better captures semantics for the specific criterion, which could be utilized for multiple customized tasks. Extensive experiments on classification and retrieval tasks demonstrate the superiority and generality of the proposed CRL. The code is available at https://github.com/XLearning-SCU/2025-NeurIPS-CRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。