针对解码器模型合并中的表示偏差问题,提出精准修正方法。
DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging

- 按位置加权修正,重点优化关键生成节点的表示
- 在代码、数学推理等任务上显著提升合并模型性能
- 仅增加0.1%参数,适合资源受限场景下的模型融合
模型合并可将多个任务特定微调的大语言模型整合为单一多任务模型,无需额外训练。然而,合并后的模型常出现表示偏差:其隐藏状态与各源模型存在系统性偏离。已有研究(Yang et al., 2024a)针对编码器类视觉模型,通过轻量级校正模块结合L1损失缓解此问题。但解码器因自回归特性,该问题尚未被深入研究。本文分析解码器模型的表示偏差,发现两个编码器中不存在的挑战:(1) 因因果注意力掩码导致误差随词元位置累积,需位置依赖修正;(2) 并非所有词元位置同等重要,高熵(决策关键)位置对生成质量影响远大于低熵位置。为此,我们提出解码器感知表示调优手术方法(DARTS)。DARTS采用新型熵加权L1损失,提高高熵位置的修正权重;引入逐位置加性偏置,捕捉位置相关误差而不增加过多参数。我们在三个领域对Llama-2-7B模型进行评估:代码生成(HumanEval)、数学推理(GSM8K)和指令遵循(AlpacaEval),结果表明DARTS相比标准手术方法有显著提升,且参数增加仅占总量的0.1%。
原文摘要 · Abstract (English)
Model merging combines multiple task-specific fine-tuned LLMs into a single multi-task model without additional training. However, merged models are known to suffer from representation bias: systematic drift between the merged model's hidden states and those of each individual source model. Prior work (Yang et al., 2024a) study and mitigate this bias for encoder-based vision models using a lightweight correction module trained with L1 loss. However, such bias is not studied for decoder models due to their autoregressive nature. We analyze the problem of representation bias in decoder models, and show two challenges absent in encoders: (1) the causal attention mask causes bias to accumulate across token positions, requiring position-dependent correction; and (2) not all token positions are equally important, i.e., high-entropy (decision-critical) positions matter far more than low-entropy ones. To address these challenges, we propose Decoder-Aware Representation Tuning via Surgery (DARTS). DARTS employs a novel entropy-weighted L1 loss to upweight correction at high-entropy positions where errors most affect generation quality, and a per-position additive bias that captures position-dependent error without overparameterization. We perform extensive evaluation on three domains: code generation (HumanEval), mathematical reasoning (GSM8K), and instruction following (AlpacaEval) on Llama-2-7B models, and show DARTS achieves significant improvement over the standard surgery approach while adding negligible parameters ($0.1\%$ of total parameters).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。