提出人类感知损失机制，让离线训练也能达到在线对齐效果。
Humanline: Online Alignment as Perceptual Loss
- 用行为经济学解释在线对齐更优：人类感知概率有偏差，在线采样恰好匹配此偏差。
- 新方法在离线数据上训练，性能媲美在线方法，速度提升最高达6倍。
- 适用于希望快速低成本优化模型的实践者，尤其适合已有数据集场景。
在线对齐（如GRPO）通常优于离线对齐（如DPO），但原因不明。本文基于行为经济学的前景理论，提出人本解释：在线策略采样更接近人类感知的模型输出分布。而PPO/GRPO中的裁剪机制，原本为稳定训练，实则恢复了人类对概率的感知偏差。因此，这类算法本质上已具备感知损失功能。研究进一步指出，在线/离线之分并非最大化人类效用的关键，只需在目标中显式引入人类概率感知扭曲，即可实现相同效果。为此，我们设计了人类感知变体（humanline），在离线非策略数据上训练，仍能匹配在线方法性能（涵盖可验证与不可验证任务），且推理速度最高提升6倍。
原文摘要 · Abstract (English)
Online alignment (e.g., GRPO) is generally more performant than offline alignment (e.g., DPO) -- but why? Drawing on prospect theory from behavioral economics, we propose a human-centric explanation. We prove that online on-policy sampling better approximates the human-perceived distribution of what the model can produce, and PPO/GRPO-style clipping -- originally introduced to just stabilize training -- recovers a perceptual bias in how humans perceive probability. In this sense, PPO/GRPO act as perceptual losses already. Our theory further suggests that the online/offline dichotomy is itself incidental to maximizing human utility, since we can achieve the same effect by selectively training on any data in a manner that mimics human perception, rather than restricting ourselves to online on-policy data. Doing so would allow us to post-train more quickly, cheaply, and flexibly without sacrificing performance. To this end, we propose a design pattern that explicitly incorporates perceptual distortions of probability into objectives like DPO/KTO/GRPO, creating humanline variants of them. Surprisingly, we find that these humanline variants, even when trained with offline off-policy data, can match the performance of their online counterparts (on both verifiable and unverifiable tasks) while running up to 6x faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。