arXiv:2510.06670cs.CL2025-10被引 1

用3万条高难度指令数据,让小模型超越大模型的对齐效果。

PIKA: Expert-Level Synthetic Datasets for Post-Training Alignment from Scratch

  • 聚焦高难度指令,用更少数据实现更好对齐。
  • 3万条数据训练的模型胜过1000万条私有数据训练的官方模型。
  • 适合资源有限但想快速获得强对齐能力的研究者。

高质量指令数据对大语言模型对齐至关重要,但现有开源数据集效率低,需数十万条样本才能接近专有模型性能。本文发现,除提示-响应质量外,提示难度本身在推动对齐提升中起关键作用。为此,我们提出PiKa,一个数据高效的专家级对齐数据集家族,集中监督于高难度指令。PiKa-SFT仅含3万条样本,远少于当前最优开源数据集Magpie-Pro。在Llama-3-8B-Base上微调,其性能在AlpacaEval 2.0和Arena-Hard等基准上超过官方基于超1000万条专有数据训练的Llama-3-8B-Instruct模型。该效果在Qwen2.5系列(0.5B-7B)中也具可推广性,持续优于其官方指令微调版本。此外,我们还提供3万条高质量偏好优化样本以进一步提升对齐效果。结果表明,显著降低数据量即可实现良好对齐,推动资源受限研究的民主化。代码与数据将公开于https://github.com/SJY8460/PiKa。

原文摘要 · Abstract (English)

High-quality instruction data is critical for LLM alignment, yet existing open-source datasets often lack efficiency, requiring hundreds of thousands of examples to approach proprietary performance. In this work, we find that beyond the widely recognized importance of prompt-response quality, prompt difficulty itself plays a critical role in driving alignment gains. Motivated by this observation, we introduce PiKa, a data-efficient family of expert-level alignment datasets that concentrates supervision on high-difficulty instructions. The PiKa-SFT dataset contains only 30k examples, an order of magnitude fewer than state-of-the-art open datasets like Magpie-Pro. Despite its small size, fine-tuning Llama-3-8B-Base on PiKa-SFT even outperforms the official Llama-3-8B-Instruct model trained on over 10M proprietary examples on widely used benchmarks such as AlpacaEval 2.0 and Arena-Hard. We also validate the generalizability of PiKa across the Qwen2.5 series (0.5B-7B), consistently surpassing their official instruction-tuned counterparts. Additionally, we provide 30k high-quality preference optimization examples to further enhance alignment. Our results demonstrate that promising alignment is achievable with significantly reduced data, democratizing access for resource-constrained research. Our code and data will be available at https://github.com/SJY8460/PiKa.

指令微调数据高效对齐优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。