让大模型自学会在多轮对话中找回单轮表现，解决丢失上下文问题。
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
- 通过双视角自蒸馏，让模型从单轮信息中学习并恢复多轮能力。
- 在3B-14B模型上，多轮性能提升至单轮的92%以上，部分达100%。
- 无需外部强教师模型，适用于各类主流大模型，提升对话效率。
大型语言模型在多轮对话中常因信息不完整而表现下降，这种现象称为「丢失在对话中」（Lost-in-Conversation）。本文提出一种自训练框架 Found in Conversation（FiC），使模型能从多轮提示中自主发现并恢复其单轮任务能力。我们设计了视图不对称自蒸馏（View-Asymmetric Self-Distillation），将同一任务的单轮视图作为教师、多轮视图作为学生进行知识迁移，无需依赖外部更强教师。该方法在多个模型家族（Llama、Qwen、Phi、OLMo）和规模（3B–14B）上验证有效，多轮性能至少恢复单轮能力的92%，并在两个Llama主干上达到100%，显著提升多轮对话效率与实用性。
原文摘要 · Abstract (English)
Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。