根据模型不确定度动态请求人类反馈，减少无效干预
Contextual bandits with entropy-based human feedback
- 仅当模型熵超过阈值时才请求专家反馈，智能平衡探索与利用
- 实验显示在低质量反馈下仍显著提升性能，人类反馈量减少60%以上
- 适用于任何基于随机策略的上下文推荐系统，尤其适合对话模型优化
近年来，基于偏好的人类反馈机制已成为提升各类模型性能的关键手段，尤其在聊天机器人（如ChatGPT）等应用中。然而现有方法常忽略模型不确定性及反馈质量波动等问题。为此，本文提出一种基于熵的人类反馈框架，用于上下文博弈问题，通过设定熵阈值，在模型置信度不足时才主动请求专家反馈，从而动态调节探索与利用。该方法具有模型无关性，可无缝集成至任意采用随机策略的上下文博弈代理中。大量实验证明，该方法在反馈质量不佳的条件下仍能实现显著性能提升，且所需人工反馈量极少。本工作不仅提出一种新型反馈触发机制，也验证了人类引导在机器学习中的鲁棒有效性。代码已开源：https://github.com/BorealisAI/CBHF
原文摘要 · Abstract (English)
In recent years, preference-based human feedback mechanisms have become essential for enhancing model performance across diverse applications, including conversational AI systems such as ChatGPT. However, existing approaches often neglect critical aspects, such as model uncertainty and the variability in feedback quality. To address these challenges, we introduce an entropy-based human feedback framework for contextual bandits, which dynamically balances exploration and exploitation by soliciting expert feedback only when model entropy exceeds a predefined threshold. Our method is model-agnostic and can be seamlessly integrated with any contextual bandit agent employing stochastic policies. Through comprehensive experiments, we show that our approach achieves significant performance improvements while requiring minimal human feedback, even under conditions of suboptimal feedback quality. This work not only presents a novel strategy for feedback solicitation but also highlights the robustness and efficacy of incorporating human guidance into machine learning systems. Our code is publicly available: https://github.com/BorealisAI/CBHF
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。