用多模型智慧挑选指令数据，让小模型更懂复杂任务
CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom
- 融合多模型响应与评分，设计三类新指标捕捉指令质量
- 在MT-bench和Arena-Hard上分别提升11.1%和4.81%性能
- 适合做小模型微调、数据筛选的科研与工程人员
利用精选子集将大模型的指令遵循能力蒸馏到小模型已成为主流训练方法。现有合成指令数据选择策略主要依赖单一信号（如奖励分数、模型困惑度），难以捕捉跨领域指令遵循的复杂性。为此，我们探索更多元信号，提出三种基于多模型智慧的基础指标，结合（1）多样化大模型响应与（2）奖励模型评估。在此基础上，提出CrowdSelect，采用聚类方法保持响应多样性。全面实验表明，基础指标在4个基线模型上均提升MT-bench与Arena-Hard表现。CrowdSelect高效整合所有指标，在全量与LoRA微调中均达当前最优，使用Llama-3.2-3b-instruct时在Arena-Hard上提升4.81%，在MT-bench上提升11.1%。代码已公开于https://github.com/listentm/crowdselect。
原文摘要 · Abstract (English)
Distilling advanced Large Language Models' instruction-following capabilities into smaller models using a selected subset has become a mainstream approach in model training. While existing synthetic instruction data selection strategies rely mainly on single-dimensional signals (i.e., reward scores, model perplexity), they fail to capture the complexity of instruction-following across diverse fields. Therefore, we investigate more diverse signals to capture comprehensive instruction-response pair characteristics and propose three foundational metrics that leverage Multi-LLM wisdom, informed by (1) diverse LLM responses and (2) reward model assessment. Building upon base metrics, we propose CrowdSelect, an integrated metric incorporating a clustering-based approach to maintain response diversity. Our comprehensive experiments demonstrate that our foundation metrics consistently improve performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect, efficiently incorporating all metrics, achieves state-of-the-art performance in both Full and LoRA fine-tuning, showing improvements of 4.81% on Arena-Hard and 11.1% on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring valuable insights for future research in this direction. Code are available at https://github.com/listentm/crowdselect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。