多轮审查反降低大模型验证效果,因重复审查滋生虚假错误
More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verification
- 通过多轮问答机制让审查者反复提问回复,试图提升验证精度
- 多轮审查使准确率从0.30降至0.20,虚假阳性增加62%
- 适合关注大模型验证缺陷与可信度评估的研究者
跨上下文审查(CCR)通过分离生成与审查阶段提升大模型验证效果。自然延伸是多轮审查:让审查者提问、接收回复并再次审查,称为动态跨上下文审查(D-CCR)。在30个样本、150个注入错误的控制实验中,四种D-CCR变体均显著劣于单轮基线(单轮F1=0.376)。其中含问答交互的D-CCR-2b F1=0.303(p<0.001,d=-0.59)。多轮审查虽提升召回率(+0.08),但虚假阳性增加62%(8.5 vs. 5.2),导致精确率从0.30暴跌至0.20。两大机制导致性能下降:(1)虚假阳性压力——后期审查者在真实错误耗尽后虚构问题;(2)审查目标漂移——受历史问答影响,审查转向对话本身而非原始内容。无上下文独立重审(D-CCR-2c)表现最差(F1=0.263),证明重复审查本身有害。性能退化源于额外轮次带来的噪声,而非信息量——在多轮条件下,更多信息反而有益(D-CCR-2b > D-CCR-2a)。
原文摘要 · Abstract (English)
Cross-Context Review (CCR) improves LLM verification by separating production and review into independent sessions. A natural extension is multi-turn review: letting the reviewer ask follow-up questions, receive author responses, and review again. We call this Dynamic Cross-Context Review (D-CCR). In a controlled experiment with 30 artifacts and 150 injected errors, we tested four D-CCR variants against the single-pass CCR baseline. Single-pass CCR (F1 = 0.376) significantly outperformed all multi-turn variants, including D-CCR-2b with question-and-answer exchange (F1 = 0.303, $p < 0.001$, $d = -0.59$). Multi-turn review increased recall (+0.08) but generated 62% more false positives (8.5 vs. 5.2), collapsing precision from 0.30 to 0.20. Two mechanisms drive this degradation: (1) false positive pressure -- reviewers in later rounds fabricate findings when the artifact's real errors have been exhausted, and (2) Review Target Drift -- reviewers provided with prior Q&A exchanges shift from reviewing the artifact to critiquing the conversation itself. Independent re-review without prior context (D-CCR-2c) performed worst (F1 = 0.263), confirming that mere repetition degrades rather than helps. The degradation stems from false positive pressure in additional rounds, not from information amount -- within multi-turn conditions, more information actually helps (D-CCR-2b > D-CCR-2a). The problem is not what the reviewer sees, but that reviewing again invites noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。