构建多语言代码评分模型，支持多种编程标准的灵活评估
Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring
- 基于35万+代码偏好对训练多语言奖励模型
- 在8种语言5个维度上实现跨语言迁移与多标准评分
- 适合需要多维度代码质量评估的研究与工程团队
奖励模型（RMs）已成为语言模型后训练的重要工具，用于策略对齐和测试时扩展。然而，现有研究在代码生成中的应用较少，主要聚焦于执行反馈，导致后训练仅能优化自包含可执行代码的功能正确性。本文探讨多语言、多标准代码奖励模型的训练与评估。我们首先构建Themis-CodeRewardBench基准，涵盖8种编程语言、5个评价维度，对50多个代码、数学及通用奖励模型进行测评。观察到现有模型在功能正确性之外表现有限，因此我们构建Themis-CodePreference，目前最大开源代码偏好数据集（超过35万对），并据此训练Themis-RM系列多语言代码奖励模型，参数规模从600M到32B不等。实验与消融分析表明，模型呈现正向缩放趋势，跨语言迁移能力强，多标准训练对可靠代码奖励建模至关重要。
原文摘要 · Abstract (English)
Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。