用双流知识蒸馏提升半监督回归的鲁棒性,更好利用无标签数据。
Toward Robust Semi-supervised Regression via Dual-stream Knowledge Distillation
- 设计双流蒸馏框架,同时传递连续值和分布信息。
- 引入解耦分布对齐模块,降低伪标签噪声影响。
- 特别适合标注数据少、需高精度回归的任务场景。
半监督回归(SSR)旨在减少对大规模标注数据的依赖,预测样本的连续分数,已在计算机视觉、自然语言处理、音频分析和医学分析等领域受到广泛关注。现有方法通常通过约束正则化或序数排序来缓解过拟合,但难以充分挖掘无标签数据的潜力。尽管一致性驱动的伪标签方法尝试利用无标签数据,其性能高度依赖伪标签质量,易受噪声影响。为此,本文提出双流知识蒸馏框架(DKD),专为SSR设计,可同时蒸馏连续值知识与分布信息,更有效保留回归量级信息并提升样本效率。在DKD中,教师模型仅使用真实标签优化以估计标签分布,学生模型则在无标签数据上结合真实标签与教师生成的伪目标进行学习。蒸馏过程实现有效监督迁移,使学生更稳健地利用伪标签。此外,引入解耦分布对齐(DDA)模块,分别对齐教师与学生的目标与非目标分布。为提升非目标知识迁移的可靠性,DDA采用方差引导的非目标分布对齐策略,自适应抑制不确定的教师预测,从而增强学生模型在伪标签噪声下的鲁棒性,学习到更校准的回归预测器。
原文摘要 · Abstract (English)
Semi-supervised regression (SSR), which aims to predict continuous scores for samples while reducing the reliance on large-scale labeled data, has recently attracted considerable attention across various applications, including computer vision, natural language processing, audio analysis, and medical analysis. Existing SSR methods typically train models with scarce labeled data by introducing constraint-based regularization or ordinal ranking to mitigate overfitting. However, these approaches often fail to fully exploit the abundance of unlabeled samples. Although consistency-driven pseudo-labeling methods attempt to incorporate unlabeled data, their performance is highly sensitive to pseudo-label quality and noisy predictions. To address these challenges, we propose a Dual-stream Knowledge Distillation framework (DKD), which is specifically designed for SSR to distill both continuous-valued knowledge and distributional information. This design better preserves regression magnitude information and improves sample efficiency. Specifically, in DKD, the teacher is optimized solely with ground-truth labels for label distribution estimation, while the student learns from a mixture of real labels and teacher-generated pseudo targets on unlabeled data. The distillation process enables effective supervision transfer, allowing the student to leverage pseudo labels more robustly. Furthermore, we introduce a Decoupled Distribution Alignment (DDA) module, which separately aligns the target and non-target distributions between the teacher and student. To improve the reliability of non-target knowledge transfer, DDA incorporates a variance-guided non-target distribution alignment strategy that adaptively downweights uncertain teacher predictions, thereby enhancing the student's ability to mitigate noise in pseudo-label supervision and learn a better-calibrated regression predictor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。