arXiv:2603.03111cs.CL2026-03被引 4

模型切换会引发对话性能漂移,影响系统可靠性。

Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems

  • 设计切换矩阵基准,评估不同模型交接时的对话质量变化
  • 单次切换可导致准确率波动达-8%至+13%,与模型等级差距相当
  • 发现部分模型对异源历史有兼容性或不兼容性倾向,适合系统部署参考

实际多轮大模型系统常因升级、跨供应商路由或降级回退而中途切换模型,导致上下文不匹配:后继模型需基于前序不同模型生成的对话历史进行推理,可能引发无声的性能漂移。本文提出一种切换矩阵基准,通过让前缀模型生成早期对话、后缀模型生成末轮回复,并与无切换基线对比,采用成对剧集级别的自助法置信区间进行评估。在CoQA和Multi-IF两个对话问答基准上,即使一次切换也产生普遍且显著的方向性影响:Multi-IF严格成功率波动可达-8至+13个百分点,CoQA绝对F1值波动±4,相当于常见模型层级间的差异(如GPT-5-nano vs GPT-5-mini)。进一步发现系统性兼容模式:部分后缀模型在任何非自源历史下性能下降,另一些则普遍提升。为实现压缩式风险监控,将切换引起的漂移分解为每模型前缀影响与后缀敏感度,解释了约70%的方差。结果表明,模型切换鲁棒性是单模型基准忽略的运营可靠性维度,亟需显式监控与手稿感知缓解策略。

原文摘要 · Abstract (English)

Deployed multi-turn LLM systems routinely switch models mid-interaction due to upgrades, cross-provider routing, and fallbacks. Such handoffs create a context mismatch: the model generating later turns must condition on a dialogue prefix authored by a different model, potentially inducing silent performance drift. We introduce a switch-matrix benchmark that measures this effect by running a prefix model for early turns and a suffix model for the final turn, and comparing against the no-switch baseline using paired episode-level bootstrap confidence intervals. Across CoQA conversational QA and Multi-IF benchmarks, even a single-turn handoff yields prevalent and statistically significant, directional effects and may swing outcomes by -8 to +13 percentage points in Multi-IF strict success rate and +/- 4 absolute F1 on CoQA, comparable to the no-switch gap between common model tiers (e.g., GPT-5-nano vs GPT-5-mini). We further find systematic compatibility patterns: some suffix models degrade under nearly any non-self dialogue history, while others improve under nearly any foreign prefix. To enable compressed handoff risk monitoring, we decompose switch-induced drift into per-model prefix influence and suffix susceptibility terms, accounting for ~70% of variance across benchmarks. These results position handoff robustness as an operational reliability dimension that single-model benchmarks miss, motivating explicit monitoring and handoff-aware mitigation in multi-turn systems.

大模型系统性能漂移模型切换对话评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。