通过动态重处理关键词元，提升大模型推理效果。
DND: Boosting Large Language Models with Dynamic Nested Depth
- 按重要性动态选择词元，分层回流重计算。
- 在Qwen3-1.7B上提升1.88%，30B MoE模型提升0.87%。
- 无需训练，轻量部署，适合追求精度的场景。
我们提出动态嵌套深度（DND）方法，通过在变换器层末尾识别关键词元并反馈重处理，提升现成大语言模型性能。具体而言，在每层末端，DND使用路由器筛选更关键的词元，并将其回流进行额外一轮处理，实现对难点词元的‘复盘’，同时避免对简单词元的冗余计算。该动态选择机制基于两种新策略：通过控制损失增强词元选择区分度，以及阈值控制方案确保选择稳定性。我们在后训练阶段直接集成DND至预训练的密集型和MoE模型中。在多个基准测试中，该方法使密集型Qwen3-1.7B模型性能提升1.88%，MoE型Qwen3-30B-A3B模型提升0.87%，且参数与计算开销极低。
原文摘要 · Abstract (English)
We introduce Dynamic Nested Depth (DND), a novel method that improves performance for off-the-shelf LLMs by selecting critical tokens to reprocess in a nested depth manner. Specifically, at the end of the given transformer layer, DND identifies more critical tokens with a router and feeds them back for an extra round of processing, effectively ``reviewing" difficult tokens while avoiding redundant computation for easier ones. The dynamic selection mechanism is tailored for precise control via two novel strategies: a router controlling loss to enhance token selection distinguishability, and a threshold control scheme to ensure selection stability. We demonstrate the effectiveness of DND by directly integrating it into pre-trained dense and MoE models during a post-training phase. On diverse benchmarks, this approach boosts the performances of the dense Qwen3-1.7B by 1.88% and the MoE Qwen3-30B-A3B by 0.87%, all with a minimal parameter and computing increase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。