通过在系统提示中插入跨注意力层，提升模型对指令和安全行为的响应能力。
System-Prompt Anchoring with Cross-Attention Layers
- 在系统提示与文本间插入跨注意力层，保持主干冻结。
- 后期插入位置效果更好，参数效率更高，任务表现差异明显。
- 适合需要精准控制指令遵循与安全性的场景应用。
跨注意力为模型计算提供了一条从特定信息源直接进入的路径，但该路径的插入位置影响尚不明确。本文研究当信息源为特权系统提示时，跨注意力层（CAL）的插入位置对性能的影响。在15亿参数的主干网络上进行十种配置测试，结果表明性能依赖于任务且受位置显著影响：后期插入更有效且参数效率更高。在80亿参数规模的扩展实验中，仅训练最优配置，并与参数匹配的微调基线对比。在多个基准测试中，跨注意力改变指令遵循与安全行为，同时基本保持通用任务性能。这些实验表明，通过跨注意力注入系统提示信息时，插入位置是一个关键设计变量。
原文摘要 · Abstract (English)
Cross-attention provides a dedicated route from a selected information source into a model's computation, but the effect of where that route is inserted remains underexplored. We study this question when the source is a privileged system-prompt span. We insert Cross-Attention Layer (CAL) blocks between the system prompt and text while keeping the causal-decoder backbone frozen. A ten-configuration sweep on a 1.5B backbone shows that performance is task-dependent and strongly affected by placement: later placements are generally more effective and parameter-efficient. In an 8B scaling study, we train only the overall best configuration and compare it with parameter-matched adaptation baselines. Across the evaluated benchmarks, the effects remain task-dependent: cross-attention changes instruction-following and security behavior while largely preserving general-task performance. Together, these experiments characterize placement as an important design variable when injecting system-prompt information through cross-attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。