arXiv:2604.13061cs.CLcs.AI2026-04被引 1

通过分析对话中词语频次,实时检测大模型对话的逻辑漂移问题。

Token Statistics Reveal Conversational Drift in Multi-turn LLM Interaction

  • 基于词频统计构建可监测对话结构一致性的新指标P。
  • 在4574轮对话中,85%条件下能准确反映对话结构一致性。
  • 无需模型内部信息,即可100%识别矛盾、跑题等逻辑错误。

大语言模型在多轮对话中越来越常见,但早期回复会持续影响后续输出,导致对话一致性下降时,下游决策失去依据。现有评估方法仅关注单个输出,忽略交互过程本身。本文提出从词频统计中直接监测对话结构一致性,不依赖嵌入、辅助评估器或模型内部信息。我们定义了双预测性(Bipredictability, P)作为信号,衡量上下文、回复与下一轮提示间的共享可预测性相对于总不确定性,并设计轻量级架构信息数字孪生(IDT)。在34种条件下,覆盖4574轮对话、一个学生模型和三个前沿教师模型,P在85%的条件中与结构一致性吻合,但仅在44%中与语义质量一致;而IDT对所有测试中的矛盾、话题转移和不合逻辑内容均实现了100%敏感度检测。结果表明,长程对话可靠性不能仅由回复质量决定,仅从可观测词流进行结构监控,可有效补充语义评估。

原文摘要 · Abstract (English)

Large language models, LLMs, are increasingly deployed in multiturn settings where earlier responses shape later ones, making reliability dependent on whether a conversation remains consistent over time. When this consistency degrades undetected, downstream decisions lose their grounding in the exchange that produced them. Yet current evaluation methods assess isolated outputs rather than the interaction producing them. Here we show that conversational structural consistency can be monitored directly from token frequency statistics, without embeddings, auxiliary evaluators or access to model internals. We formalize this signal as Bipredictability, P, which measures shared predictability across the context, response, next prompt loop relative to the turn total uncertainty, and implement it in a lightweight auxiliary architecture, the Information Digital Twin, IDT. Across 4,574 conversational turns spanning 34 conditions, one student model and three frontier teacher models, P established a stable runtime baseline, aligned with structural consistency in 85 percent of conditions but with semantic quality in only 44 percent, and the IDT detected all tested contradictions, topic shifts and non-sequiturs with 100 percent sensitivity. These results show that reliability in extended LLM interaction cannot be reduced to response quality alone, and that structural monitoring from the observable token stream can complement semantic evaluation in deployment.

对话一致性大模型评估逻辑检测词频分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。