发现扩散模型中查询位置影响生成质量,提出动态优化方法。
Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics

- 揭示扩散模型查询位置是关键变量,非随意放置
- 位置偏差导致性能下降,与示例语义同等重要
- 无需训练,用平均置信度自动选择最优查询位置
尽管自回归大模型的上下文学习(ICL)被广泛研究,但其在扩散语言模型(dLLMs)中的机制仍不明确。与受单向因果掩码限制的自回归模型不同,dLLMs具备双向注意力,使查询位置具有高度空间灵活性。然而,当前实践仍沿用自回归风格的尾部查询模板,忽视了结构范式转变。本文通过实证分析发现,查询位置是dLLMs中的首要变量。解耦实验表明,位置变化对生成质量的影响可与示例语义质量相媲美。内部机制源于注意力流中的空间‘近期效应’及解码轨迹的任务依赖性偏移。为在无真值标签下缓解此不稳定性,我们发现传统单步置信度 $C_{decoded}$ 在dLLMs中失效,提出新的平均置信度 $ar{C}$ 来追踪迭代解码过程。基于此,我们建立基础空间ICL基准,提出Auto-ICL——一种无需训练的自适应路由策略,可动态优化查询位置,在异构推理与感知任务上稳健逼近理想性能。
原文摘要 · Abstract (English)
While In-Context Learning (ICL) is extensively studied in Autoregressive (AR) LLMs, its mechanism within Diffusion Large Language Models (dLLMs) remains largely unexplored. Unlike AR models restricted by unidirectional causal masking, dLLMs intrinsically utilize bidirectional attention, offering extensive spatial flexibility for query placement. Unfortunately, current practices conventionally inherit AR-style trailing-query templates, often overlooking the structural paradigm shift. This paper presents a comprehensive analysis unveiling that query position is actually a first-order variable in dLLMs. Through empirical decoupling, we demonstrate that positional variance impacts generation quality on par with example semantic quality. Internally, this positional sensitivity stems from a spatial ``Recency Effect'' in attention flow and task-dependent shifts in decoding trajectories. To mitigate this instability without ground-truth labels, we reveal that traditional single-step confidence ($C_{decoded}$) fails in dLLMs. Instead, we propose Average Confidence ($\overline{C}$), a novel metric tracking the iterative decoding process. By establishing the foundational spatial ICL baselines, we introduce Auto-ICL, a training-free adaptive routing strategy that dynamically optimizes query placement, robustly approaching oracle performance across heterogeneous reasoning and perception tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。