用社会指标提升大模型在合作困境中的策略生成能力
Beyond Scalar Rewards: Dense Feedback for LLM Policy Synthesis in Sequential Social Dilemmas
- 用社会指标（效率、公平、可持续性）替代单一奖励信号来指导大模型策略优化
- 在清理和聚集任务中，密集反馈使策略表现优于或持平稀疏反馈
- 社会指标帮助模型识别失败模式，实现如区域划分等协作策略
我们提出一种基于大模型的框架，用于生成多智能体环境中的代码化策略函数，通过自对弈评估并利用迭代反馈进行优化。在近期反馈工程研究的基础上，对比了仅使用标量奖励的稀疏反馈与包含效率、公平性、可持续性和和平度等社会指标的密集反馈。在两个顺序社会困境任务（Gathering 和 Cleanup）中，使用两个前沿大模型（Claude Sonnet 4.6、Gemini 3.1 Pro）进行实验，密集反馈在所有指标上均优于或匹配稀疏反馈。我们通过反馈混淆现象解释该差异：当标量奖励将不同失败模式映射为相同数值（如清洁不足与过度清洁）时，社会指标可区分其差异，帮助模型判断改进方向。结论表明，社会指标作为协调信号，引导出如维诺伊区域划分和自适应清洁调度等协作策略。代码开源于 https://github.com/vicgalle/llm-policies-social-dilemmas。
原文摘要 · Abstract (English)
We propose an LLM harness that generates code-based policy functions for multi-agent environments, evaluates them with self-play, and refines them using feedback from previous iterations. Following the recent line of work in feedback engineering (the design of which information signals are shown to the LLM during refinement), we compare sparse feedback (scalar reward only) with dense feedback (reward plus social metrics: efficiency, equality, sustainability, peace). In two Sequential Social Dilemmas (Gathering and Cleanup) and with two frontier LLMs (Claude Sonnet 4.6, Gemini 3.1 Pro), dense feedback improves over or matches sparse feedback on all metrics. We explain this asymmetry via feedback aliasing: when the scalar reward maps distinct failure modes into the same value (e.g., under- vs. over-cleaning), social metrics disambiguate and allow the LLM to diagnose which direction of improvement to take. We conclude that social metrics act as a coordination signal, leading to strategies such as Voronoi territory partitioning and adaptive cleaner schedules. Code at https://github.com/vicgalle/llm-policies-social-dilemmas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。