arXiv:2506.18887cs.AIcs.LG2025-06被引 1

通过激活语言模型潜空间,精准引导代码生成偏向特定编程语言。

Steering Conceptual Bias via Transformer Latent-Subspace Activation

  • 基于梯度优化的动态激活框架,按提示自动选择最优引导方向。
  • 在LLaMA-3.2 3B上使分类准确率提升15%,早期层提升61.5%。
  • 适用于大模型代码生成,可解释且计算开销可控,适合智能体系统。

本研究探讨在语言模型(LLMs)中激活潜在子空间是否能引导科学代码生成偏向特定编程语言。首先对五种因果性LLM在科学编码提示下评估其在四种编程语言间的基线偏倚。静态神经元归因方法(扰动最高激活的MLP权重以对应C++或CPP标记)表现脆弱,泛化能力差。为此,提出一种梯度优化的自适应激活引导框架(G-ACT):将每提示的激活差异聚类为少量引导方向,并在线训练轻量级逐层探测器以选择合适引导向量。在LLaMA-3.2 3B上,该方法显著提升生成对CPP语言的偏好,平均探测器分类准确率提高15%,早期层(0-6)提升达61.5%。对于注意力信号更分散的LLaMA-3.3 70B,关键层的目标注入仍有效提升语言选择能力。尽管逐层探测引入轻微推理开销,但仅作用于部分层,仍具实用性,实现可复现的模型行为。结果表明,该方法具备可扩展、可解释、高效等特性,适用于实际智能体系统中的概念级控制。

原文摘要 · Abstract (English)

This work examines whether activating latent subspaces in language models (LLMs) can steer scientific code generation toward a specific programming language. Five causal LLMs were first evaluated on scientific coding prompts to quantify their baseline bias among four programming languages. A static neuron-attribution method, perturbing the highest activated MLP weight for a C++ or CPP token, proved brittle and exhibited limited generalization across prompt styles and model scales. To address these limitations, a gradient-refined adaptive activation steering framework (G-ACT) was developed: per-prompt activation differences are clustered into a small set of steering directions, and lightweight per-layer probes are trained and refined online to select the appropriate steering vector. In LLaMA-3.2 3B, this approach reliably biases generation towards the CPP language by increasing the average probe classification accuracy by 15% and the early layers (0-6) improving the probe classification accuracy by 61.5% compared to the standard ACT framework. For LLaMA-3.3 70B, where attention-head signals become more diffuse, targeted injections at key layers still improve language selection. Although per-layer probing introduces a modest inference overhead, it remains practical by steering only a subset of layers and enables reproducible model behavior. These results demonstrate a scalable, interpretable and efficient mechanism for concept-level control for practical agentic systems.

语言模型代码生成潜空间引导控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。