大模型在探索与利用任务中表现不一，探索时有优势，利用时仍不如传统方法。
Should You Use Your Large Language Model to Explore or Exploit?
- 分阶段测试大模型在探索和利用中的能力，避免任务混淆。
- 推理模型擅长利用，但速度慢成本高；非推理模型通过工具调用提升表现。
- 大模型在语义丰富的动作空间中能有效提出候选策略，适合探索场景。
我们系统评估了当前大语言模型（LLMs）在决策代理面临探索-利用权衡时的表现。以往研究多关注大模型解决综合探索-利用任务的能力，本文则将探索与利用分开，在多种（上下文相关的）老虎机任务中分别测试大模型的性能。结果发现，推理型大模型在利用任务中表现最佳，但因成本过高或响应过慢，难以应用于多数实际场景。为此，我们进一步研究非推理模型在工具使用和上下文摘要方面的潜力。结果显示，这些优化手段可显著提升中等难度任务的表现，但即便如此，所有测试的大模型在非线性设置下仍逊于简单的线性回归模型。另一方面，大模型在探索具有内在语义的动作空间时表现出色，能够生成合理候选动作供探索。本研究为大模型在强化学习中的合理应用提供了实证依据。
原文摘要 · Abstract (English)
We evaluate the ability of the current generation of large language models (LLMs) to help a decision-making agent facing an exploration-exploitation tradeoff. While previous work has largely study the ability of LLMs to solve combined exploration-exploitation tasks, we take a more systematic approach and use LLMs to explore and exploit in silos in various (contextual) bandit tasks. We find that reasoning models show the most promise for solving exploitation tasks, although they are still too expensive or too slow to be used in many practical settings. Motivated by this, we study tool use and in-context summarization using non-reasoning models. We find that these mitigations may be used to substantially improve performance on medium-difficulty tasks, however even then, all LLMs we study perform worse than a simple linear regression, even in non-linear settings. On the other hand, we find that LLMs do help at exploring large action spaces with inherent semantics, by suggesting suitable candidates to explore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。