arXiv:2412.14922cs.CLcs.AI2024-12被引 28

让大模型在含噪声数据下仍能高效微调,提升实际应用性能。

RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response

  • 用多专家协作+增强推理检测数据噪声
  • 通过上下文增强与置信度评估生成可靠标签
  • 基于响应熵筛选高质量样本,适合真实场景微调

监督微调(SFT)在将大语言模型适配到特定领域或任务中起着关键作用。然而,实证实验表明,在实际应用中收集的数据不可避免地包含噪声,这对下游任务的模型性能构成重大挑战。因此,亟需一种抗噪声的SFT框架来提升模型在下游任务中的能力。为此,我们提出一个鲁棒SFT框架(RobustFT),对下游任务数据进行噪声检测与重标注。在噪声识别阶段,采用多专家协作系统结合增强推理模型以实现更优的噪声检测;在去噪阶段,利用上下文增强策略,整合最相关且置信度高的知识,并经仔细评估生成可靠标注。此外,引入基于响应熵的有效数据选择机制,确保仅保留高质量样本用于微调。在多个LLM和五个数据集上的大量实验表明,RobustFT在噪声场景下表现出色。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) plays a crucial role in adapting large language models (LLMs) to specific domains or tasks. However, as demonstrated by empirical experiments, the collected data inevitably contains noise in practical applications, which poses significant challenges to model performance on downstream tasks. Therefore, there is an urgent need for a noise-robust SFT framework to enhance model capabilities in downstream tasks. To address this challenge, we introduce a robust SFT framework (RobustFT) that performs noise detection and relabeling on downstream task data. For noise identification, our approach employs a multi-expert collaborative system with inference-enhanced models to achieve superior noise detection. In the denoising phase, we utilize a context-enhanced strategy, which incorporates the most relevant and confident knowledge followed by careful assessment to generate reliable annotations. Additionally, we introduce an effective data selection mechanism based on response entropy, ensuring only high-quality samples are retained for fine-tuning. Extensive experiments conducted on multiple LLMs across five datasets demonstrate RobustFT's exceptional performance in noisy scenarios.

大模型微调噪声鲁棒数据清洗

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。