让大模型在推理时更听话,减少隐私泄露风险。
From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves
- 用指令数据训练模型,提升推理过程中的指令遵循能力。
- 在两个隐私基准上,隐私保护效果提升最高达51.9个百分点。
- 适合关注模型安全与隐私保护的研究者和开发者。
大型推理模型(LRMs)的推理轨迹(RTs)常包含敏感信息,难以控制且易违反隐私指令。由于RT可能通过提示注入攻击暴露,对用户构成直接隐私风险。本文将问题视为可控性问题:因隐私指令本身是指令,提升推理过程中的指令遵循(IF)能力可直接减少隐私泄露。为此,我们构建了一个SFT数据集,训练模型在整个推理过程中遵循通用指令,并提出分阶段解码策略,使用独立的LoRA适配器解耦推理轨迹与答案生成,以最大化各部分的指令遵循能力。我们在两个模型家族(1.7B-14B参数)的六种模型上进行了评估,涵盖两个指令遵循基准和两个隐私基准。结果表明,该方法显著提升性能,指令遵循最高提升20.9点,隐私基准最高提升51.9个百分点;但可能存在任务效用下降的权衡。实验显示,提升指令遵循能力可有效增强模型隐私性,为未来隐私感知的推理模型提供了新方向。代码已开源。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) produce reasoning traces (RTs) that often contain sensitive information. These leaky thoughts are difficult to control and frequently violate explicit privacy directives. Because RTs can be exposed through prompt injection attacks, this becomes a direct privacy risk to the user. We approach this as a controllability problem: since privacy directives are themselves instructions, improving instruction-following (IF) within the RT provides a direct path to reducing privacy leaks. To this end, we introduce an SFT dataset that teaches models to follow general instructions throughout their reasoning process, and propose Staged Decoding, a simple decoding strategy that decouples RT and answer generation using separate LoRA adapters to maximize IF of each component. We evaluate our approach on six models from two families (1.7B-14B parameters), across two IF benchmarks and two privacy benchmarks. Our method yields substantial improvements, with gains of up to 20.9 points in IF and 51.9 percentage points on privacy benchmarks, though these can come at the cost of task utility due to the trade-off between reasoning performance and IF. Our results show that improving IF in LRMs can significantly enhance privacy, suggesting a promising direction for future privacy-aware LRMs. Our code is available at https://github.com/UKPLab/arxiv2026-controllable-reasoning-models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。