让大模型推理时自动调用业务规则，提升准确性并减少计算开销。
PA3: Policy-Aware Agent Alignment through Chain-of-Thought
- 通过思维链推理阶段动态召回相关业务规则，无需全程携带完整策略。
- 性能提升16点，比同规模上下文基线高3点，词数减少40%。
- 适合需要精准遵守复杂业务规则的对话系统开发者使用。
由大语言模型驱动的对话助手在工具使用任务中表现优异，但在遵循复杂的、特定于业务的规则方面存在困难。虽然模型可以在上下文中推理业务规则，但将所有政策包含在每次查询中会带来高延迟并浪费计算资源。此外，冗长的提示导致上下文过长，因‘在干草堆中找针’问题而影响整体性能。为解决这些问题，我们提出一种多阶段对齐方法，使模型在推理时通过思维链推理动态回忆并应用相关业务规则，无需在上下文中包含完整的业务政策。同时，我们引入基于杰卡德分数的政策召回奖励和针对GRPO训练的幻觉惩罚。最终，我们的最佳模型相比基线提升16分，超过同等规模的上下文基线3分，且词数减少40%。
原文摘要 · Abstract (English)
Conversational assistants powered by large language models (LLMs) excel at tool-use tasks but struggle with adhering to complex, business-specific rules. While models can reason over business rules provided in context, including all policies for every query introduces high latency and wastes compute. Furthermore, these lengthy prompts lead to long contexts, harming overall performance due to the 'needle-in-a-haystack' problem. To address these challenges, we propose a multi-stage alignment method that teaches models to recall and apply relevant business policies during chain-of-thought reasoning at inference time, without including the full business policy in-context. Furthermore, we introduce a novel Policy Recall reward based on the Jaccard score and a Hallucination Penalty for GRPO training. Altogether, our best model outperforms the baseline by 16 points and surpasses comparable in-context baselines of similar model size by 3 points, while using 40% fewer words.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。