通过筛选熟悉知识的数据，让大模型更听话且少编造。
Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering
- 用自生成回复一致性检测模型熟悉度，识别可信数据。
- 结合语义聚类与投票机制，精准判断答案是否匹配模型知识。
- 引入专家对齐奖励模型，确保数据质量，适合训练可靠大模型。
在指令微调阶段使用包含陌生知识的数据会引发大模型幻觉。为解决此问题，我们提出NOVA框架，通过内部一致性探测（ICP）和语义等价识别（SEI）来衡量大模型对指令数据的熟悉程度。ICP通过计算多个自生成回复间的定制化一致性，评估模型对指令的理解；SEI则通过语义聚类与设计精巧的投票策略，比较目标回答与生成回答的相似性，进一步判断模型对答案的熟悉度。最后，引入专家对齐奖励模型，综合考量超出熟悉度的其他特征，确保所选样本质量。通过筛选高质量、符合模型已有知识的数据，可有效引导大模型遵循指令并减少幻觉。
原文摘要 · Abstract (English)
Training LLMs on data containing unfamiliar knowledge during the instruction tuning stage can encourage hallucinations. To address this challenge, we introduce NOVA, a novel framework designed to identify high-quality data that aligns well with the LLM's learned knowledge to reduce hallucinations. NOVA includes Internal Consistency Probing (ICP) and Semantic Equivalence Identification (SEI) to measure how familiar the LLM is with instruction data. Specifically, ICP evaluates the LLM's understanding of the given instruction by calculating the tailored consistency among multiple self-generated responses. SEI further assesses the familiarity of the LLM with the target response by comparing it to the generated responses, using the proposed semantic clustering and well-designed voting strategy. Finally, to ensure the quality of selected samples, we introduce an expert-aligned reward model, considering characteristics beyond just familiarity. By considering data quality and avoiding unfamiliar data, we can utilize the selected data to effectively align LLMs to follow instructions and hallucinate less.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。