用问答游戏评估大模型提问能力，发现好问题能大幅减少推理步数。
GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models
- 设计问答游戏框架，让模型自由提问猜隐藏对象。
- 提问信息量每提升一个标准差，游戏平均缩短43%步数。
- 基于信息量优化提示策略，弱模型也能显著提效。
我们提出GuessingGame，一种评估大语言模型在开放域、开放式场景中作为策略性提问者的能力的协议。一个猜测者模型通过自由提问向一个无预设选项的提供者（Oracle）提问，以识别隐藏对象。为衡量问题质量，我们提出两种信息增益（IG）指标：一种基于贝叶斯方法，利用大模型评分跟踪语义概念上的信念更新；另一种基于熵的方法，通过ConceptNet筛选候选。两种指标均与模型无关，支持事后分析。在858场游戏中，多个模型与提示策略下验证：较高的IG显著预测效率提升——信息增益提高一个标准差，可使预期游戏长度减少43%。基于IG引导的提示约束（如强制问题多样性）能让较弱模型显著提升表现。结果表明，大模型的提问行为既可测量也可优化，对交互式推理至关重要。
原文摘要 · Abstract (English)
We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43\%. Prompting constraints guided by IG, such as enforcing question diversity, enable weaker models to significantly improve performance. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。