发现大模型用特定方向编码时态与体貌，可精准控制生成文本语法特征。
Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect
- 通过线性判别分析定位残差空间中时态与体貌的独立方向。
- 在三项生成任务中实现对时态和体貌的因果控制，准确率提升显著。
- 揭示控制强度、位置与持续时间是减少走题和退化的关键因素。
大型语言模型能生成语法正确的文本，但其内部如何表征语法知识尚不明确。本文研究了两种多维层级语法现象——动词时态与体貌，并通过线性判别分析在残差空间中识别出各自独立且正交的方向。接着，在三个生成任务中验证了通过概念引导实现对这两种语法特征的因果控制。进一步开展案例研究，探讨多标记生成中有效引导的影响因素。结果表明，引导强度、位置及持续时间是减少主题偏移与生成退化等副作用的关键参数。研究提示，模型以结构化、类人方式编码时态与体貌，但生成阶段的精准控制依赖于多个因素，需人工调参或自动化优化。
原文摘要 · Abstract (English)
Large language models (LLMs) are able to generate grammatically well-formed text, but how do they encode their syntactic knowledge internally? While prior work has focused largely on binary grammatical contrasts, in this work, we study the representation and control of two multidimensional hierarchical grammar phenomena - verb tense and aspect - and for each, identify distinct, orthogonal directions in residual space using linear discriminant analysis. Next, we demonstrate causal control over both grammatical features through concept steering across three generation tasks. Then, we use these identified features in a case study to investigate factors influencing effective steering in multi-token generation. We find that steering strength, location, and duration are crucial parameters for reducing undesirable side effects such as topic shift and degeneration. Our findings suggest that models encode tense and aspect in structurally organized, human-like ways, but effective control of such features during generation is sensitive to multiple factors and requires manual tuning or automated optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。