用新方法让大模型生成更多样且高质量的回答。
Post-training Large Language Models for Diverse High-Quality Responses
- 基于确定性点过程，同时优化回答质量和语义多样性。
- 在多个任务上提升语义多样性，且不降低回答质量。
- 适合需要丰富输出的对话、创作和推理场景。
强化学习（RL）已成为后训练大语言模型（LLM）的主流方法。尽管提升了下游任务表现，但常导致输出多样性下降，产生单一、刻板的回答。现有增强多样性的方法受限于仅在推理时应用，或仅关注表面差异。本文提出一种名为DQO（Diversity Quality Optimization）的新训练方法，基于确定性点过程（DPPs），联合优化模型的质量与语义多样性。该方法对每个提示采样一组回答，通过核相似性矩阵的行列式衡量其嵌入向量所张成的空间体积，以此量化多样性。DQO具有灵活性，可集成到现有RL算法之上。在指令遵循、摘要生成、故事创作和推理任务上的实验表明，该方法显著提升了语义多样性，同时保持模型质量不变。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a popular method for post-training large language models (LLMs). While improving the model's performance on downstream tasks, it often reduces the model's output diversity, leading to narrow, canonical responses. Existing methods to enhance diversity are limited, either by operating at inference time or by focusing on surface-level differences. We propose a novel training method named DQO (Diversity Quality Optimization) based on determinantal point processes (DPPs) to jointly optimize LLMs for quality and semantic diversity. Our approach samples and embeds a group of responses for each prompt, then uses the determinant of a kernel-based similarity matrix to measure diversity as the volume spanned by the embeddings of these responses. DQO is flexible and can be applied on top of existing RL algorithms. Experiments across instruction-following, summarization, story generation, and reasoning tasks demonstrate that our method substantially improves semantic diversity without sacrificing model quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。