arXiv:2506.18233cs.AI2025-06被引 1

通过重用参数提升模型逻辑深度,不增参数也能增强推理能力。

Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws

  • 用参数重用来增加虚拟逻辑深度,改变计算结构但不改参数量。
  • 相同参数下推理能力显著提升,且与模型规模正相关。
  • 适用于多种架构,为模型扩容提供新思路,适合研究者参考。

大规模语言模型的扩展通常涉及深度、宽度和参数量三个维度。本文探索第四维——虚拟逻辑深度(VLD),在不改变参数量的前提下,通过权重重用提升有效算法深度。与近期测试时的逐标记扩展方法不同,VLD在训练和推理中改变内部计算图。控制实验揭示三个关键发现:(1) 固定参数量下,知识容量基本不变,但跨模型时容量仍随参数增长;(2) 合理实现的VLD可显著提升推理能力,无需增加参数,实现推理与规模解耦;(3) 推理增益在多种架构和重用策略下均持续存在,表明其具有通用性。这些结果为未来模型扩展提供新视角,并提出核心问题:超智能是否必须依赖更大模型,还是可通过参数重用与逻辑深度提升实现?我们认为模型扩展中仍有大量未知机制待探索。代码已公开于 https://anonymous.4open.science/r/virtual_logical_depth-8024/。

原文摘要 · Abstract (English)

Scaling large language models typically involves three dimensions: depth, width, and parameter count. In this work, we explore a fourth dimension, \textbf{virtual logical depth} (VLD), which increases effective algorithmic depth without changing parameter count by reusing weights. While parameter reuse is not new, its role in scaling has been underexplored. Unlike recent test-time methods that scale token-wise, VLD alters the internal computation graph during training and inference. Through controlled experiments, we obtain three key insights. (1) \textit{Knowledge capacity vs. parameters}: at fixed parameter count, VLD leaves knowledge capacity nearly unchanged, while across models capacity still scales with parameters. (2) \textit{Reasoning vs. reuse}: properly implemented VLD substantially improves reasoning ability \emph{without} more parameters, decoupling reasoning from size. This suggests a new scaling path beyond token-wise test-time methods. (3) \textit{Robustness and generality}: reasoning gains persist across architectures and reuse schedules, showing VLD captures a general scaling behavior. These results provide insight into future scaling strategies and raise a deeper question: does superintelligence require ever-larger models, or can it be achieved by reusing parameters and increasing logical depth? We argue many unknown dynamics in scaling remain to be explored. Code is available at https://anonymous.4open.science/r/virtual_logical_depth-8024/.

模型扩展逻辑深度参数重用推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。