解决多轮对话训练中低质量监督信号传播问题，动态调整不可靠反馈权重。
ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning
- 通过在线统计估算每轮损失分布，动态降低不可靠监督权重。
- 在多个基准上实现响应质量提升，相关性达0.21~1.0。
- 适合大规模弱监督数据训练，无需预过滤，对噪声鲁棒。
多轮对话系统微调需要高质量监督信号，但低质量数据常导致性能下降。早期轮次的监督错误会传播至后续轮次，破坏连贯性与回复质量。现有方法多采用静态预过滤，将质量控制与训练分离，无法有效缓解轮次级错误传播。为此，我们提出ReSURE（Regularizing Supervision UnREliability），一种自适应学习方法，无需显式过滤即可动态下调不可靠监督信号。ReSURE利用Welford在线统计估算每轮损失分布，并实时重加权样本损失。在单源与混合质量数据集上的实验表明，该方法显著提升了训练稳定性和回复质量。值得注意的是，无论数据质量如何，响应评分与样本数量间均呈现正向斯皮尔曼相关性（0.21 ~ 1.0），为高效利用大规模弱监督数据提供了可能。代码已公开于https://github.com/Elvin-Yiming-Du/ReSURE_Multi_Turn_Training。
原文摘要 · Abstract (English)
Fine-tuning multi-turn dialogue systems requires high-quality supervision but often suffers from degraded performance when exposed to low-quality data. Supervision errors in early turns can propagate across subsequent turns, undermining coherence and response quality. Existing methods typically address data quality via static prefiltering, which decouples quality control from training and fails to mitigate turn-level error propagation. In this context, we propose ReSURE (Regularizing Supervision UnREliability), an adaptive learning method that dynamically down-weights unreliable supervision without explicit filtering. ReSURE estimates per-turn loss distributions using Welford's online statistics and reweights sample losses on the fly accordingly. Experiments on both single-source and mixed-quality datasets show improved stability and response quality. Notably, ReSURE enjoys positive Spearman correlations (0.21 ~ 1.0 across multiple benchmarks) between response scores and number of samples regardless of data quality, which potentially paves the way for utilizing large-scale data effectively. Code is publicly available at https://github.com/Elvin-Yiming-Du/ReSURE_Multi_Turn_Training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。