用信息论衡量语言模型代理的影响力，无需人工设计评测集。
Estimating the Empowerment of Language Model Agents
- 基于信息论定义代理对未来的控制力，通过多轮对话估算。
- 在文本游戏和真实环境中，影响力与任务表现强相关。
- 适合评估模型通用能力，尤其关注关键决策时刻。
随着语言模型（LM）代理在现实应用中日益强大，亟需超越昂贵人工设计基准的可扩展评估框架。本文提出基于赋能（empowerment）的信息理论评估方法，该指标衡量代理通过行动对未来状态的影响能力。针对文本环境的独特挑战，我们提出EELMA（Estimating Empowerment of Language Model Agents）算法，从多轮文本交互中近似计算有效赋能。我们在文本游戏及真实的网页与工具使用环境中验证EELMA，发现赋能与平均任务性能显著相关。进一步分析显示，赋能随模型、环境复杂度和代理配置变化，高赋能状态与动作常标志通用能力的关键节点。结果表明，赋能是一种无目标依赖的度量，可补充任务成功率评估，为语言模型代理提供更全面的评价视角。
原文摘要 · Abstract (English)
As language model (LM) agents become increasingly capable and adopted in real-world applications, there is a growing need for scalable evaluation frameworks beyond costly, manually designed benchmarks. We propose information-theoretic evaluation based on empowerment, an information-theoretic measure of an agent's influence on future states through its actions. To handle the unique challenges of text-based environments, we introduce EELMA (Estimating Empowerment of Language Model Agents), an algorithm for approximating effective empowerment from multi-turn text interactions. We demonstrate EELMA on textual games and realistic web and tool-use environments, showing that empowerment strongly correlates with average task performance. We further analyze how empowerment varies across models, environment complexity, and agent configurations, and show that high-empowerment states and actions often mark pivotal moments for general capabilities. These results establish empowerment as a goal-agnostic metric that complements task-success measures for LM-agent evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。