解决大模型强化学习中注意力权重失效问题，让信号更稳定可靠。
ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
- 用适配器残差直接衡量每个词的贡献，避开传统方法的失效陷阱。
- 在相同计算量下，保持与主流方法相当的性能表现。
- 无需额外训练奖励模型或价值头，适合资源有限场景使用。
语言模型强化学习中的词级信用分配通常假设策略可完全微调，但实际中常采用参数高效微调（如LoRA）。此时策略受限于参考模型的低秩邻域，导致常见的内在信用信号（如突现性、熵减少、策略差异）在轨迹内归一化后可能退化为均匀分布或集中在少数无关任务的位置。本文提出集中度诊断指标（如权重吉尼系数、有效词比），并引入轻量级的适配器-残差信用分配（ARCA）方法，通过计算适配器对隐藏状态的改变量‖h^adapted_t - h^base_t‖₂来确定词的重要性。该方法不依赖学习型奖励模型、价值头或树结构，在紧凑的MATH/Qwen3-1.7B GRPO实验中，与同秩基线在匹配采样预算下保持竞争力，并展现出预期的非退化中间区域信用分布。
原文摘要 · Abstract (English)
Token-level credit assignment for language-model reinforcement learning is usually formulated as if the policy were fully trainable, while practical LLM-RL pipelines often rely on parameter-efficient fine-tuning, especially LoRA. We argue that this separation hides a structural failure mode. Under LoRA, the policy is restricted to a low-rank neighborhood of the reference model, so the per-token output-distribution differences used by common intrinsic credit signals, surprisal, entropy reduction, and policy divergence, can become degenerate after within-trajectory normalization, either approaching uniform weights or concentrating on a small set of task-agnostic positions. We formalize this behavior and propose measuring it directly with concentration diagnostics such as weight Gini and effective-token ratio. We then introduce \emph{Adapter-Residual Credit Assignment} (ARCA), a lightweight alternative that derives token salience from the adapter's own hidden-state residual, $\|h^{\text{adapted}}_t - h^{\text{base}}_t\|_2$. ARCA asks where the adapter actually changes the model, rather than where the output distribution appears uncertain or shifted, and requires no learned reward model, value head, or tree construction. In a compact MATH/Qwen3-1.7B GRPO sweep, ARCA exhibits the predicted non-degenerate middle-regime credit distribution under matched rollout budgets and remains competitive with rank-matched baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。