PRISM通过持续模拟与监控，自动修复企业对话系统因大模型行为漂移导致的提示失效。
PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI
- 构建闭环框架，用模拟对话自动测试并修复提示
- 三周内将提示开发时间从2天降至30分钟，可靠性达99%
- 适合需要长期稳定运行的企业级对话系统开发者
在企业场景中部署基于大语言模型（LLM）的对话代理，要求提示在上线时正确且对生产环境中的非确定性行为漂移具有鲁棒性。现有提示优化框架将提示质量视为一次性编译问题，忽略了如何检测和修复由静默的模型行为变化引发的提示退化。我们提出PRISM（Prompt Reliability via Iterative Simulation and Monitoring），一个闭环框架，将提示工程视作持续的可靠性工程问题而非一次性创作任务。PRISM输入为自然语言代理需求、配置的工具与记忆变量及初始提示草稿，自动生成测试用例，针对平台忠实的LLM环境进行多轮对话模拟，使用大模型作为评判者评估通过/失败，诊断失败根因，并精准修复提示——迭代直至所有测试通过。关键的是，PRISM可按日调度运行，将模型行为漂移视为首要可靠性关注点。我们在Yellow.ai V3平台上对35个企业对话代理进行了为期三周的评估。结果表明，PRISM将提示编写时间中位数从2天缩短至30分钟以内，所有被测代理的生产可靠性达到99%，并在24小时内成功识别并修复由模型行为漂移引起的生产退化。结果表明，持续、模拟驱动的提示优化在大规模可靠企业对话系统中既可行又必要。
原文摘要 · Abstract (English)
Deploying large language model (LLM)-driven conversational agents in enterprise settings requires prompts that are simultaneously correct at launch and resilient to the non-deterministic behavioral drift that characterizes production LLM deployments. Existing prompt optimization frameworks address prompt quality as a one-time compile-time problem, leaving open the equally critical question of how to detect and repair prompt regressions caused by silent LLM behavior changes over time. We present PRISM (Prompt Reliability via Iterative Simulation and Monitoring), a closed-loop framework that treats prompt engineering as a continuous reliability engineering problem rather than a one-time authorship task. PRISM takes as input plain-language agent requirements, a set of configured tools and memory variables, and an initial draft prompt. It automatically generates test cases from requirements, simulates full multi-turn conversations against a platform-faithful LLM environment, evaluates pass/fail using an LLM-as-judge, diagnoses root causes of failures, and surgically repairs the prompt -- iterating until all tests pass. Critically, PRISM is designed to run on a scheduled basis (daily), treating LLM behavioral drift as a first-class reliability concern. We evaluate PRISM across 35 enterprise conversational agents over a three-week deployment period on the Yellow.ai V3 platform. PRISM reduces median prompt authoring time from 2 days to under 30 minutes, achieves 99% production reliability across all evaluated agents, and successfully identifies and repairs production regressions caused by LLM behavioral drift within a 24-hour detection window. Our results suggest that continuous, simulation-driven prompt optimization is both tractable and necessary for reliable enterprise conversational AI at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。