用模型自身注意力信号自监督强化学习，无需人工标注即可提升大模型输出相关性。
A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals
- 利用模型生成时的交叉注意力信号，自动构建奖励函数。
- 在提示相关性和一致性上优于非强化学习基线，接近人类标注水平。
- 适合希望减少人工标注成本的模型对齐研究者使用。
我们提出一种新颖的强化学习框架，用于大语言模型的后训练，不依赖人工反馈。该方法利用模型内部的交叉注意力信号，构建自监督奖励，指导模型策略的迭代微调。通过分析生成过程中模型对输入提示的注意力分布，我们构建了提示覆盖度、关注焦点和连贯性等指标，用于对候选回复进行评分或排序，从而提供鼓励生成与提示高度一致、主题聚焦文本的奖励信号。在与标准策略梯度方法及基于合成偏好模型的强化学习微调对比中，该方法在提示相关性和一致性方面显著优于非强化学习基线。尽管尚未达到完全人工监督的强化学习人类反馈（RLHF）系统性能，但展示了以极少量人工标注实现模型对齐的重要方向。我们提供了详细分析，讨论潜在局限，并展望将交叉注意力信号与少量人类反馈结合的未来工作。
原文摘要 · Abstract (English)
We propose a novel reinforcement learning framework for post training large language models that does not rely on human in the loop feedback. Instead, our approach uses cross attention signals within the model itself to derive a self supervised reward, thereby guiding iterative fine tuning of the model policy. By analyzing how the model attends to the input prompt during generation, we construct measures of prompt coverage, focus, and coherence. We then use these measures to rank or score candidate responses, providing a reward signal that encourages the model to produce well aligned, on topic text. In empirical comparisons against standard policy gradient methods and RL fine tuning with synthetic preference models, our method shows significant gains in prompt relevance and consistency over a non RL baseline. While it does not yet match the performance of fully human supervised RLHF systems, it highlights an important direction for scaling alignment with minimal human labeling. We provide a detailed analysis, discuss potential limitations, and outline future work for combining cross-attention based signals with smaller amounts of human feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。