发现大模型内部按层分解任务，像自动规划一样逐步执行
Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs
- 通过掩码和拼接方法验证不同子任务在不同层学习
- 在15个双步任务上发现任务执行严格按层顺序进行
- 适合关注模型可解释性与指令控制的研究者
我们发现大语言模型具有内部思维链：它们会逐层分解并执行复合任务。研究基于两个核心假设：(i) 不同子任务在不同网络深度被学习；(ii) 这些子任务在各层间顺序执行。我们在包含15个双步复合任务的基准上，采用上下文掩码法并提出一种新型跨任务拼接方法，验证了假设(i)。为检验假设(ii)，我们使用LogitLens解码隐藏状态，揭示出一致的分层执行模式。该分析在真实世界数据集TRACE上也得到复现，观察到相同的分步动态。结果表明，大模型具备内部规划与子任务执行能力，提升了模型透明度，为指令级激活调控提供了新路径。
原文摘要 · Abstract (English)
We show that large language models (LLMs) exhibit an $\textit{internal chain-of-thought}$: they sequentially decompose and execute composite tasks layer-by-layer. Two claims ground our study: (i) distinct subtasks are learned at different network depths, and (ii) these subtasks are executed sequentially across layers. On a benchmark of 15 two-step composite tasks, we employ layer-from context-masking and propose a novel cross-task patching method, confirming (i). To examine claim (ii), we apply LogitLens to decode hidden states, revealing a consistent layerwise execution pattern. We further replicate our analysis on the real-world $\text{TRACE}$ benchmark, observing the same stepwise dynamics. Together, our results enhance LLMs transparency by showing their capacity to internally plan and execute subtasks (or instructions), opening avenues for fine-grained, instruction-level activation steering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。