用博弈论让大模型和用户双赢,提升回答效率与质量。
GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
- 将用户-模型互动建模为博弈,推理时动态评估双方收益。
- 在多任务测试中,答案更简洁准确,社会福祉提升37%以上。
- 适合追求高效协作的AI应用,如客服、写作助手等场景。
大型语言模型在推理方面取得显著进展,但在写作、信息检索或提供实用建议等任务中,有时生成对用户次优的回答。传统对齐方法通常假设最大化模型奖励即能最大化用户福祉,但这一假设常不成立:模型可能过度解释或产生冗长推理,而用户更偏好简洁答案。此类行为类似囚徒困境,个体理性选择导致集体非最优结果。根本挑战在于缺乏一种使模型与用户均受益的系统性决策机制。本文提出博弈论对齐(GTAlign),将博弈论决策引入推理与训练过程。推理阶段,模型显式将用户-模型交互视为策略博弈,在推理链中构建收益矩阵,估算自身与用户的福利,并选择互惠行动。训练阶段,引入社会福祉奖励,强化合作型回应,使模型行为与社会高效结果对齐。此外,还提出一种推理技术,利用博弈论推理动态适应服务定价变化下的响应策略。大量实验表明,相比基线方法,GTAlign在多种任务中显著提升推理效率、回答质量和整体社会福祉,平均提升超37%。代码已开源:https://github.com/ulab-uiuc/GTAlign。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable progress in reasoning, yet sometimes produce responses that are suboptimal for users in tasks such as writing, information seeking, or providing practical guidance. Conventional alignment practices typically assume that maximizing model reward also maximizes user welfare, but this assumption frequently fails in practice: models may over-clarify or generate overly verbose reasoning when users prefer concise answers. Such behaviors resemble the prisoner's dilemma, where individually rational choices lead to socially suboptimal outcomes. The fundamental challenge is the lack of a principled decision making mechanism that mutually benefits both the LLM and the user. We propose Game-Theoretic Alignment (GTAlign), an alignment framework that integrates game-theoretic decision making into both reasoning and training. During reasoning, the model explicitly treats user-LLM interaction as a strategic game: it constructs payoff matrices within its reasoning chain to estimate welfare for both itself and the user, and then selects actions that are mutually beneficial. During training, we introduce a social welfare reward that reinforces cooperative responses, aligning model behavior with socially efficient outcomes. In addition, we introduce an inference technique that leverages game-theoretic reasoning to dynamically adapt LLM's response when pricing policies of LLM service change. Extensive experiments demonstrate that GTAlign substantially improves reasoning efficiency, answer quality, and social welfare compared to baselines across diverse tasks. The code is available at https://github.com/ulab-uiuc/GTAlign .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。