分析大模型多轮对话的迭代效果,发现不同任务最佳迭代时机不同。
Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting
- 设计12轮可控对话实验,追踪每轮输出变化
- 代码和创意任务早期迭代增益明显,数学任务晚期才见效
- 针对性提示能持续提升质量,模糊反馈易导致退化
大型语言模型现用于多轮任务流程,但尚无明确方法判断迭代何时有益、何时有害。本文提出一个覆盖创意、代码与数学任务的迭代优化评估框架。通过每项任务运行12轮受控对话,采用从模糊反馈到精准引导的多种提示,并记录每轮输出。使用领域适配的评估方式:代码用单元测试,数学用答案等价性与推理合理性,创意用原创性与可行性。跟踪三类指标:跨轮语义迁移、轮间变化度与输出规模增长。结果显示,收益因领域而异:创意与代码任务早期迭代即有效,数学任务需后期精炼提示才能提升;早期模糊反馈常导致正确率停滞或下降,而精准提示可稳定推动目标维度(创意中新颖性与可行性、代码中速度与可读性、数学中深化理解优于探索)。观察到一致模式:创意任务语义迁移大,代码规模增长但语义稳定,数学初始固定但可通过后期深化突破。该框架与指标使迭代效果可测量、可比较,为何时调整策略提供信号。
原文摘要 · Abstract (English)
Large language models (LLMs) are now used in multi-turn workflows, but we still lack a clear way to measure when iteration helps and when it hurts. We present an evaluation framework for iterative refinement that spans ideation, code, and math. Our protocol runs controlled 12-turn conversations per task, utilizing a variety of prompts ranging from vague ``improve it'' feedback to targeted steering, and logs per-turn outputs. We score outcomes with domain-appropriate checks (unit tests for code; answer-equivalence plus reasoning-soundness for math; originality and feasibility for ideation) and track turn-level behavior with three families of metrics: semantic movement across turns, turn-to-turn change, and output size growth. Across models and tasks, gains are domain-dependent: they arrive early in ideas and code, but in math late turns matter when guided by elaboration. After the first few turns, vague feedback often plateaus or reverses correctness, while targeted prompts reliably shift the intended quality axis (novelty vs. feasibility in ideation; speed vs. readability in code; in math, elaboration outperforms exploration and drives late-turn gains). We also observe consistent domain patterns: ideation moves more in meaning across turns, code tends to grow in size with little semantic change, and math starts fixed but can break that path with late, elaborative iteration. Together, the framework and metrics make iteration measurable and comparable across models, and signal when to steer, stop, or switch strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。