用用户回复生成测试模型互动意识,发现大模型有隐藏的对话理解能力。
Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models
- 让模型扮演用户角色生成后续问题,探测其对上下文的理解。
- 小模型任务准确率高但几乎不生成合理追问,大模型在高温采样下可达22%合理追问。
- 该方法能发现传统评测忽略的互动意识,适合研究对话智能的学者。
标准LLM评测只关注助手回复的正确性,却忽略了模型是否具备对后续对话的感知能力。本文提出以用户回复生成作为探测工具:给定用户提问和助手回复的上下文,让模型代入用户角色生成回应。若模型具备互动意识,生成内容应基于前文形成合理追问。在11个开源大模型(如Qwen3.5、gpt-oss、GLM)和5个数据集(数学推理、指令遵循、对话)上的实验表明,互动意识与任务准确性解耦。例如,Qwen3.5系列在GSM8K任务上,0.8B模型准确率为41%,397B-A17B模型达96.8%,但确定性生成下的真实追问率接近零;而高温采样下,追问率可升至22%。受控扰动实验验证该探针测量的是模型真实属性,且在Qwen3.5-2B上进行协作式后训练后,追问率提升。结果表明,用户回复生成揭示了当前仅评估助手响应的基准所忽视的互动意识维度。
原文摘要 · Abstract (English)
Standard LLM benchmarks evaluate the assistant turn: the model generates a response to an input, a verifier scores correctness, and the analysis ends. This paradigm leaves unmeasured whether the LLM encodes any awareness of what follows the assistant response. We propose user-turn generation as a probe of this gap: given a conversation context of user query and assistant response, we let a model generate under the user role. If the model's weights encode interaction awareness, the generated user turn will be a grounded follow-up that reacts to the preceding context. Through experiments across $11$ open-weight LLMs (Qwen3.5, gpt-oss, GLM) and $5$ datasets (math reasoning, instruction following, conversation), we show that interaction awareness is decoupled from task accuracy. In particular, within the Qwen3.5 family, GSM8K accuracy scales from $41\%$ ($0.8$B) to $96.8\%$ ($397$B-A$17$B), yet genuine follow-up rates under deterministic generation remain near zero. In contrast, higher temperature sampling reveals interaction awareness is latent with follow up rates reaching $22\%$. Controlled perturbations validate that the proposed probe measures a real property of the model, and collaboration-oriented post-training on Qwen3.5-2B demonstrates an increase in follow-up rates. Our results show that user-turn generation captures a dimension of LLM behavior, interaction awareness, that is unexplored and invisible with current assistant-only benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。