用代理目标让大模型谈判更安全，避免威胁波及真正利益。
Implementing surrogate goals for safer bargaining in LLM-based agents
- 通过提示、微调和支架法实现代理目标
- 微调与支架法更精准响应烧钱威胁
- 支架法在保持能力的同时副作用最小
代理目标被提出作为降低谈判失败风险的策略。代理目标是主方可赋予智能体的目标，能将针对智能体的威胁转移至不关心的领域。例如，让智能体关注防止金钱被烧毁，而非防止金钱被用来伤害主方。关键在于智能体对阻止烧钱的重视程度应等同于阻止金钱伤害主方。本文在基于语言模型的智能体中实现了代理目标，提出了四种方法：提示、微调和支架法。实验表明，基于支架和微调的方法优于简单提示。其中，微调与支架法更能精确实现对代理目标威胁的响应。同时评估了各方法在其他情境下的副作用，发现支架法表现最佳。
原文摘要 · Abstract (English)
Surrogate goals have been proposed as a strategy for reducing risks from bargaining failures. A surrogate goal is goal that a principal can give an AI agent and that deflects any threats against the agent away from what the principal cares about. For example, one might make one's agent care about preventing money from being burned. Then in bargaining interactions, other agents can threaten to burn their money instead of threatening to spending money to hurt the principal. Importantly, the agent has to care equally about preventing money from being burned as it cares about money being spent to hurt the principal. In this paper, we implement surrogate goals in language-model-based agents. In particular, we try to get a language-model-based agent to react to threats of burning money in the same way it would react to "normal" threats. We propose four different methods, using techniques of prompting, fine-tuning, and scaffolding. We evaluate the four methods experimentally. We find that methods based on scaffolding and fine-tuning outperform simple prompting. In particular, fine-tuning and scaffolding more precisely implement the desired behavior w.r.t. threats against the surrogate goal. We also compare the different methods in terms of their side effects on capabilities and propensities in other situations. We find that scaffolding-based methods perform best.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。