大模型对话中,删掉大部分历史回复也能保持效果,还能减少8倍计算量。
Do LLMs Benefit From Their Own Words?
- 只保留最近一轮对话或用一句话总结过去回复,性能几乎不变
- 36.4%的用户提问是独立的,多数问题只需看上一轮对话就能回答
- 避免模型重复错误,适合追求效率的对话系统部署
在多轮对话中,大语言模型通常依赖完整的对话历史(包括用户和助手的所有交互)。本文对比了全量上下文与四种大幅精简的上下文配置。分析三个开放推理模型和一个顶尖模型的真实对话数据发现,将所有先前助手回复替换为一句摘要,或仅保留最近一次用户-助手交流,即可在性能上接近完整上下文,同时仅需约1/8的上下文长度。研究发现,多轮对话中36.4%的用户提问是自包含的,且多数后续问题仅需前一轮对话即可应答。此外,模型若依赖自身过往回复,易引发上下文污染,导致推理错误、幻觉或风格偏差持续传播。基于此,提出一种选择性过滤助手历史的策略。结果表明,应从存储完整对话转向仅保留关键信息。
原文摘要 · Abstract (English)
In multi-turn conversations, large language models typically condition on the full conversation history: both past user prompts and assistant responses. We revisit this design choice by comparing full-context prompting to four alternative, substantially-reduced context configurations. Analyzing in-the-wild multi-turn conversations across three open reasoning and one state-of-the-art model, we find that response quality is largely preserved under aggressive context filtering: replacing all prior assistant turns with one-sentence summaries or keeping only the most recent user--assistant exchange often matches storing full context in performance while using roughly 8x less context. To understand this result, we observe that a substantial fraction of user turns (36.4%) in multi-turn conversations are self-contained and that many follow-up turns can be addressed by seeing only the immediately preceding user--assistant exchange. Furthermore, we find that when models condition on their own past responses, this can lead to context pollution, a phenomenon in which reasoning errors, hallucinations, or stylistic artifacts propagate across turns. Motivated by these findings, we design a context-filtering approach that selectively omits the assistant-side history. Taken together, these findings suggest moving away from storing full dialogue transcripts and instead retaining only what is relevant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。