通过对齐指令数据与大模型内部知识，显著提升微调效果。
NILE: Internal Consistency Alignment in Large Language Models
- 用模型自身知识重构指令数据答案，保证一致性。
- 在Arena-Hard上性能提升66.6%，Alpaca-Eval V2上达68.5%。
- 适合追求高精度微调和数据质量优化的研究者。
为提升大语言模型（LLM）与人类意图的一致性，指令微调（IFT）对数据集质量要求极高。然而现有IFT数据集常包含与预训练阶段模型内部知识不一致的内容，严重影响微调效果。为此，我们提出NILE（内部一致性对齐）框架，旨在优化IFT数据集以释放模型更大潜力。NILE通过提取目标预训练模型对指令数据的内部知识，并利用该知识修正IFT数据中的答案。此外，提出一种新型内部一致性过滤（ICF）方法，确保训练样本与模型内部知识高度一致。实验表明，经过NILE对齐的IFT数据集在多个能力评估集上显著提升模型性能，于Arena-Hard上最高提升66.6%，于Alpaca-Eval V2上达68.5%。进一步分析证实，NILE各组件均贡献显著，且证明数据与预训练内部知识的一致性是最大化模型潜能的关键。
原文摘要 · Abstract (English)
As a crucial step to enhance LLMs alignment with human intentions, Instruction Fine-Tuning (IFT) has a high demand on dataset quality. However, existing IFT datasets often contain knowledge that is inconsistent with LLMs' internal knowledge learned from the pre-training phase, which can greatly affect the efficacy of IFT. To address this issue, we introduce NILE (iNternal consIstency aLignmEnt) framework, aimed at optimizing IFT datasets to unlock LLMs' capability further. NILE operates by eliciting target pre-trained LLM's internal knowledge corresponding to instruction data. The internal knowledge is leveraged to revise the answer in IFT datasets. Additionally, we propose a novel Internal Consistency Filtering (ICF) method to filter training samples, ensuring its high consistency with LLM's internal knowledge. Our experiments demonstrate that NILE-aligned IFT datasets sharply boost LLM performance across multiple LLM ability evaluation datasets, achieving up to 66.6% gain on Arena-Hard and 68.5% on Alpaca-Eval V2. Further analysis confirms that each component of the NILE}framework contributes to these substantial performance improvements, and provides compelling evidence that dataset consistency with pre-trained internal knowledge is pivotal for maximizing LLM potential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。