调控语言模型中介词短语的语法功能，实现动副与形容修饰的可控切换。
Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2
- 通过投影注意力激活向量定位关键注意力头，识别其对介词补语功能的偏好。
- 单个注意力头的值向量缩放可使动副功能占比从75%降至33%，形容功能升至36%。
- 适用于需要精确控制语法结构生成的研究者，尤其关注语言模型内部机制的开发者。
语言模型在生成介词短语时，常需判断其补语是作为工具状语（动作用途）还是属性修饰语（名词补充），但这一决策的内部机制尚不清晰。本研究针对Gemma-2展开定向探究，揭示并控制介词补语的生成。构建包含with-headed介词短语的提示集，其上下文同时支持工具状语或属性修饰两种解释，结果显示模型更倾向工具状语，比例为3:4。通过将注意力头激活值投影至词汇空间，定位偏好工具状语的注意力头；仅缩放单个注意力头的值向量，即可使工具状语比例降至33%，属性修饰上升至36%，实现功能角色的可控调节。
原文摘要 · Abstract (English)
Language Models, when generating prepositional phrases, must often decide for whether their complements functions as an instrumental adjunct (describing the verb adverbially) or an attributive modifier (enriching the noun adjectivally), yet the internal mechanisms that resolve this split decision remain poorly understood. In this study, we conduct a targeted investigation into Gemma-2 to uncover and control the generation of prepositional complements. We assemble a prompt suite containing with-headed prepositional phrases whose contexts equally accommodate either an instrumental or attributive continuation, revealing a strong preference for an instrumental reading at a ratio of 3:4. To pinpoint individual attention heads that favor instrumental over attributive complements, we project activations into the vocabulary space. By scaling the value vector of a single attention head, we can shift the distribution of functional roles of complements, attenuating instruments to 33% while elevating attributes to 36%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。