针对高性能计算任务多样性，提出自适应奖励优化方法提升大模型表现
HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Tasks
- 根据置信度和奖励对比动态调整每条响应权重，无需任务标签
- 在四类异构任务中实现54.62%胜率、91.30%数据竞争检测F1值
- 适用于需要精准生成与分类混合的任务，如编程错误检测
监督微调（SFT）可为大语言模型（LLMs）注入高性能计算（HPC）任务领域知识，如数据竞争检测与基准问答。然而，仅具备知识并不保证合适行为：同一SFT模型对88.65%的C/C++数据竞争样本分类正确，但对事实性问题回答冗长不精确，65.9%的MLPerf响应超过40字符。强化学习（RL）后训练通过优化任务特定奖励弥补此差距。然而，HPC任务存在极端异质性——二分类、事实问答与语义生成答案长度相差58倍，涵盖三类不同奖励分布，且SFT准确率差异显著，导致统一权重的RL方法（如GRPO）效果不佳。本文提出HARGO：异质性感知奖励引导优化，通过置信度调制优势计算，结合组级奖励差异与参考模型概率，动态生成每条响应的重要性权重，无需任务类型标签。在四类HPC任务与九种方法中，HARGO在三项核心指标上均最优：胜率54.62%、数据竞争F1值91.30%、PLP相似度0.8558。消融实验验证双信号互补贡献。HARGO在异构HPC任务中建立了最佳整体对齐质量。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) can equip large language models (LLMs) with domain knowledge for high-performance computing (HPC) tasks such as data race detection and benchmark question answering. However, knowledge alone does not guarantee task-appropriate behavior: the same SFT model that correctly classifies 88.65\% of C/C++ data race samples produces verbose, imprecise answers to factual queries, with 65.9\% of MLPerf responses exceeding 40 characters. Reinforcement learning (RL) post-training addresses this gap by optimizing for task-specific rewards rather than token-level imitation. Yet HPC tasks exhibit extreme heterogeneity, with binary classification, factual QA, and semantic generation differing by 58x in answer length, spanning three distinct reward distributions, and showing widely varying SFT accuracy. This makes uniform-weight RL methods such as GRPO suboptimal. We propose HARGO, Heterogeneity-Aware Reward-Guided Optimization, which introduces per-response importance weighting via confidence-modulated advantage: computing a discrimination signal from group-level reward contrast and a confidence signal from reference model log-probabilities, then modulating the advantage before computing per-response weights, without requiring task-type labels. Across four HPC tasks and nine methods, HARGO achieves the best performance on all three primary metrics: WinRate 54.62\%, Data Race F1 91.30\%, and PLP Similarity 0.8558. Ablation confirms complementary contributions from both signals. HARGO establishes the best overall alignment quality among compared methods for heterogeneous HPC tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。