人类反馈偏差导致对齐难题，仅靠纠正少数错误样本无法突破指数级学习瓶颈。
Murphys Laws of AI Alignment: Why the Gap Always Wins
- 在反馈偏差占比α、强度ε的场景下，学习算法需指数级样本量才能区分真实目标
- 若能定位不可靠反馈区域（校准预言机），仅需O(1/(αε²))次提问即可克服障碍
- 该结果揭示了对齐困境的本质：边缘案例中细微偏差会放大目标差距
我们研究在反馈错位情况下的从人类反馈中强化学习问题。当人类反馈在特定输入类型上系统性错误时（如指南针在某些区域指向错误），若反馈偏差出现在占比为α的上下文，且偏差强度为ε，任何学习算法都需至少exp(n·α·ε²)数量级的样本才能区分两个仅在这些异常上下文中不同的真实奖励函数。然而，若能识别出反馈不可靠的区域（即拥有校准预言机），则可将有限的提问集中在这些区域，仅需O(1/(α·ε²))次查询即可突破指数级壁垒。该结果量化了对齐困难的核心原因：罕见边缘案例中微妙的反馈偏差，结合真实目标在此处的差异程度γ，共同决定了优化目标与真实目标之间的鸿沟。人工智能对齐的墨菲定律：除非主动规避错位，否则鸿沟永远占优。
原文摘要 · Abstract (English)
We study reinforcement learning from human feedback under misspecification. Sometimes human feedback is systematically wrong on certain types of inputs, like a broken compass that points the wrong way in specific regions. We prove that when feedback is biased on a fraction alpha of contexts with bias strength epsilon, any learning algorithm needs exponentially many samples exp(n*alpha*epsilon^2) to distinguish between two possible "true" reward functions that differ only on these problematic contexts. However, if you can identify where feedback is unreliable (a "calibration oracle"), you can focus your limited questions there and overcome the exponential barrier with just O(1/(alpha*epsilon^2)) queries. This quantifies why alignment is hard: rare edge cases with subtly biased feedback create an exponentially hard learning problem unless you know where to look. The gap between what we optimize (proxy from human feedback) and what we want (true objective) is fundamentally limited by how common the problematic contexts are (alpha), how wrong the feedback is there (epsilon), and how much the true objectives disagree there (gamma). Murphy's Law for AI alignment: the gap always wins unless you actively route around misspecification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。