用鼠标轨迹和视线数据提升大模型对齐效果
Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users
- 通过追踪用户鼠标与视线构建隐式反馈数据集
- 使奖励模型准确率从55%提升至64%
- 适合关注用户行为分析与模型对齐的研究者
为对齐大语言模型（LLM），现有方法依赖显式人类反馈来训练奖励模型。但该方法存在两大缺陷：一是用户很少提供显式反馈，高质量标注成本高昂；二是未利用隐式反馈，而隐式反馈对互联网巨头至关重要。为此，我们构建了新数据集IFLLM，采集了59名Mechanical Turk工作者的1336个多轮问题、其鼠标轨迹及摄像头捕捉的视线注视点。IFLLM显示用户注视行为和鼠标轨迹差异显著。基于隐式反馈的奖励模型将文本奖励模型准确率从55%提升至64%，并在对8个LLM应用DPO后，使相对响应质量提升近三倍，验证了真实场景中隐式反馈的价值。数据集与代码已开源。
原文摘要 · Abstract (English)
To align a Large Language Model (LLM), most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text. These existing methods have two key limitations. First, the users rarely provide explicit feedback for LLM responses, which makes the high-quality preference annotation expensive to collect. Second, the methods do not leverage implicit human feedback, which has proven vital to the economic moats of Internet giants. To quantify the value of implicit feedback, we build a new dataset called IFLLM, which collects 1336 multi-turn questions from the 59 Mechanical Turk workers, their mouse trajectories, and eye gazing points to the LLMs' responses from their webcams. IFLLM shows that the users have very diverse types of gazing behavior and mouse trajectories. Our reward model based on the implicit user feedback boosts the accuracy of the text-based reward model from 55% to 64% and nearly triples the relative response quality improvements after applying the DPO to eight LLMs, demonstrating the value of implicit feedback in the wild. Our data collection website, dataset, and codes can be found at https://github.com/themehulpatwari/llm-implicit-feedback/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。