用推理与强化学习让大模型懂何时该说、何时不该说。
Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
- 让模型先显式推理信息披露的合理性，再用强化学习训练。
- 仅用700条合成数据，大幅减少不当信息泄露。
- 效果可迁移至真实隐私评估基准，适合智能助手安全开发。
随着自主代理代表用户决策的时代到来，确保情境完整性（CI）——即在执行特定任务时应披露何种信息——成为核心问题。我们提出，实现CI需要代理具备对所处情境的推理能力。为此，我们首先通过提示让大模型在决定披露信息时进行显式推理；随后构建了一种强化学习（RL）框架，进一步训练模型掌握实现CI所需的推理能力。基于一个仅约700个样本、但涵盖多样情境和披露规范的合成数据集，实验表明，该方法显著降低了不恰当的信息披露，同时在多个模型规模与架构下维持了任务性能。更重要的是，该方法的效果可从合成数据集泛化到包含人类标注的成熟基准PrivacyLens，后者评估了AI助手在行为与工具调用中的隐私泄露情况。代码已公开：https://github.com/EricGLan/CI-RL
原文摘要 · Abstract (English)
As the era of autonomous agents making decisions on behalf of users unfolds, ensuring contextual integrity (CI) -- what is the appropriate information to share while carrying out a certain task -- becomes a central question to the field. We posit that CI demands a form of reasoning where the agent needs to reason about the context in which it is operating. To test this, we first prompt LLMs to reason explicitly about CI when deciding what information to disclose. We then extend this approach by developing a reinforcement learning (RL) framework that further instills in models the reasoning necessary to achieve CI. Using a synthetic, automatically created, dataset of only $\sim700$ examples but with diverse contexts and information disclosure norms, we show that our method substantially reduces inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Importantly, improvements transfer from this synthetic dataset to established CI benchmarks such as PrivacyLens that has human annotations and evaluates privacy leakage of AI assistants in actions and tool calls. Our code is available at: https://github.com/EricGLan/CI-RL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。