用行为对齐让大模型更靠谱地评估推荐效果
From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation
- 给大模型注入正反例推理，训练它只说合理理由
- 在真实数据上提升32.19%的评估准确率
- 适合想替代人工特征工程的推荐系统研究者
传统离线推荐评估依赖复杂且难扩展的手动特征管道。尽管大语言模型（LLMs）能直接从原始文本日志预测用户参与度，但本研究发现其存在关键缺陷：双向合理化。在零样本场景下，大模型会对同一物品、相同证据，给出支持正负反馈的看似合理的论证，暴露其不可靠性。为此，我们提出一种序列行为对齐框架，结合微调与偏好优化，对正确和反事实推理进行配对训练。在真实主页交互日志上评估，该方法相比零样本基线提升32.19%的宏平均F1得分，并达到生产级特征工程基线水平。结果表明，行为对齐有效缓解双向合理化，同时提供人类可理解的推理过程，无需手动构建特征管道。
原文摘要 · Abstract (English)
Traditional offline recommendation evaluation relies heavily on complex, manually maintained feature pipelines that are difficult to scale. While Large Language Models (LLMs) offer a promising alternative by predicting user engagement directly from raw text logs, empirical analysis in this study identifies a critical failure mode termed bidirectional rationalization. In a zero-shot setting, LLMs are found to convincingly argue for both positive and negative user engagement outcomes on the exact same item with identical evidence, highlighting the unreliability of off-the-shelf LLMs in predicting user engagement. To resolve this, we develop and apply a sequential behavioral alignment framework pairing fine-tuning with preference optimization over paired correct and counterfactual rationales. Evaluated on real-world homepage interaction logs, this aligned reasoning approach achieves a 32.19\% lift in Macro-F1 score over the zero-shot baseline and matches the production feature-engineered baseline. The results demonstrate that behavioral alignment mitigates bidirectional rationalization while delivering human-interpretable reasoning traces without manual pipeline overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。