用户偏好的计划不等于有用的计划，单纯迎合偏好可能误导模型。
A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
- 用真实用户执行数据测试计划有用性，而非仅依赖偏好评分。
- 用户偏好与实际成功无直接关联，偏好高者未必执行成功。
- 简洁、相似问题等表面特征影响偏好，但无助于提升效果。
为辅助用户完成复杂任务，大模型生成分步计划。现有对齐方法基于用户偏好训练或评估（如RLHF、ChatbotArena），假设偏好即有用性。本文通过Planorama平台，让126名用户回答300个需多步推理的问题，收集4388次计划执行和5584次对比数据，衡量计划有用性（问答成功率）与用户偏好。结果发现：用户偏好与模型成功率无法准确预测计划有效性，常见对齐反馈可能偏离真正帮助用户的方向；该差距并非源于用户个体偏好差异，因用户使用偏好或不偏好计划时成功率相近；表面特征如简洁性、问题相似性显著影响偏好，却不能预测实际帮助效果。研究呼吁以真实用户交互反馈替代仅依赖偏好信号的对齐方式，并提出可操作的自然语言处理研究路径。
原文摘要 · Abstract (English)
To assist users in complex tasks, LLMs generate plans: step-by-step instructions towards a goal. While alignment methods aim to ensure LLM plans are helpful, they train (RLHF) or evaluate (ChatbotArena) on what users prefer, assuming this reflects what helps them. We test this with Planorama: an interface where 126 users answer 300 multi-step questions with LLM plans. We get 4388 plan executions and 5584 comparisons to measure plan helpfulness (QA success) and user preferences on plans, and recreate the setup in agents and reward models to see if they simulate or prefer what helps users. We expose: 1) user/model preferences and agent success do not accurately predict which plans help users, so common alignment feedback can misalign with helpfulness; 2) this gap is not due to user-specific preferences, as users are similarly successful when using plans they prefer/disprefer; 3) surface-level cues like brevity and question similarity strongly link to preferences, but such biases fail to predict helpfulness. In all, we argue aligning helpful LLMs needs feedback from real user interactions, not just preferences of what looks helpful, so we discuss the plan NLP researchers can execute to solve this problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。