LLM嵌入质量决定主动学习策略成败,好嵌入让选样本更有效。
No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Success
- 用高质量LLM嵌入初始化,结合多样性采样提升早期效果
- 低质嵌入下,边缘采样性能飙升;高质嵌入使Badge更稳定
- 选策略要根据嵌入质量与任务匹配,无通用最优解
大型语言模型(LLM)生成的通用表征使深度主动学习(AL)重获实用性:通过使用冻结的LLM嵌入,可避免反复微调大模型带来的计算开销。本研究建立基准,系统考察了LLM嵌入质量对主动学习查询策略的影响。采用MTEB榜单中五个表现最佳的文本嵌入模型及两个基线,在十个不同文本分类任务上进行实验。结果表明:首先,使用基于多样性的采样初始化标注池,能与高质量嵌入协同,显著提升早期迭代性能;其次,最优查询策略对嵌入质量敏感——边缘采样在特定数据集上可实现性能跃升,但Badge等策略在跨任务间更具鲁棒性;更重要的是,这些策略的效果常随嵌入质量提升而增强。研究强调需针对具体场景评估主动学习策略,因性能高度依赖嵌入质量与目标任务。
原文摘要 · Abstract (English)
The advent of large language models (LLMs) capable of producing general-purpose representations lets us revisit the practicality of deep active learning (AL): By leveraging frozen LLM embeddings, we can mitigate the computational costs of iteratively fine-tuning large backbones. This study establishes a benchmark and systematically investigates the influence of LLM embedding quality on query strategies in deep AL. We employ five top-performing models from the massive text embedding benchmark (MTEB) leaderboard and two baselines for ten diverse text classification tasks. Our findings reveal key insights: First, initializing the labeled pool using diversity-based sampling synergizes with high-quality embeddings, boosting performance in early AL iterations. Second, the choice of the optimal query strategy is sensitive to embedding quality. While the computationally inexpensive Margin sampling can achieve performance spikes on specific datasets, we find that strategies like Badge exhibit greater robustness across tasks. Importantly, their effectiveness is often enhanced when paired with higher-quality embeddings. Our results emphasize the need for context-specific evaluation of AL strategies, as performance heavily depends on embedding quality and the target task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。