让指令微调后的大模型重获创作多样性，提升生成质量。
Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models
- 用基础模型引导指令模型，恢复生成多样性。
- 指令微调使输出多样性下降，DPO阶段影响最显著。
- 适合需要多样创意输出的文本生成任务。
指令微调大语言模型会降低输出多样性，对创意类任务影响显著。本文以叙事生成任务为例，发现多种开源大模型在指令微调后多样性明显下降。通过对OLMo和OLMo 2模型在各微调阶段的分析，发现基于人类反馈的直接偏好优化（DPO）对多样性损失影响最大。为此，提出一种新解码策略——共形解码（conformative decoding），利用更富多样性的基础模型引导指令模型，有效恢复输出多样性，且不损害生成质量。实验表明，该方法可显著提升多样性，同时保持甚至改善内容质量。
原文摘要 · Abstract (English)
Instruction-tuning large language models (LLMs) reduces the diversity of their outputs, which has implications for many tasks, particularly for creative tasks. This paper investigates the ``diversity gap'' for a writing prompt narrative generation task. This gap emerges as measured by current diversity metrics for various open-weight and open-source LLMs. The results show significant decreases in diversity due to instruction-tuning. We explore the diversity loss at each fine-tuning stage for the OLMo and OLMo 2 models to further understand how output diversity is affected. The results indicate that DPO has the most substantial impact on diversity. Motivated by these findings, we present a new decoding strategy, conformative decoding, which guides an instruct model using its more diverse base model to reintroduce output diversity. We show that conformative decoding typically increases diversity and even maintains or improves quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。