arXiv:2505.22316cs.LG2025-05

用下游任务表现评估流程模拟质量,更真实反映模型价值。

Rethinking BPS: A Utility-Based Evaluation Framework

  • 以预测性监控模型在仿真数据上的表现衡量模拟真实性
  • 实验证明能区分模型误差与数据复杂度影响
  • 适合流程优化、系统设计等需要可靠模拟的场景

业务流程模拟(BPS)是分析和优化组织工作流的关键工具,通过估计流程变更的影响支持决策。其结果可靠性取决于模型对实际流程的准确捕捉能力,因此严格评估至关重要。然而,现有评估方法存在两大局限:一是将模拟视为预测问题,检验模型能否预测未来事件,无法评估其对当前流程的还原能力,尤其当训练与测试阶段行为变化时;二是过度依赖基于地球移动距离(EMD)的指标,会掩盖时间模式,导致误判。为此,我们提出一种新框架:基于模拟生成行为的代表性来评估质量。不比较仿真日志与未来真实执行,而是评估在仿真数据上训练的预测性监控模型,在下游任务中是否与真实数据训练的模型表现相当。实验表明,该框架不仅能识别差异来源,还能区分模型准确性与数据复杂度,提供更合理的评估方式。

原文摘要 · Abstract (English)

Business process simulation (BPS) is a key tool for analyzing and optimizing organizational workflows, supporting decision-making by estimating the impact of process changes. The reliability of such estimates depends on the ability of a BPS model to accurately mimic the process under analysis, making rigorous accuracy evaluation essential. However, the state-of-the-art approach to evaluating BPS models has two key limitations. First, it treats simulation as a forecasting problem, testing whether models can predict unseen future events. This fails to assess how well a model captures the as-is process, particularly when process behavior changes from train to test period. Thus, it becomes difficult to determine whether poor results stem from an inaccurate model or the inherent complexity of the data, such as unpredictable drift. Second, the evaluation approach strongly relies on Earth Mover's Distance-based metrics, which can obscure temporal patterns and thus yield misleading conclusions about simulation quality. To address these issues, we propose a novel framework that evaluates simulation quality based on its ability to generate representative process behavior. Instead of comparing simulated logs to future real-world executions, we evaluate whether predictive process monitoring models trained on simulated data perform comparably to those trained on real data for downstream analysis tasks. Empirical results show that our framework not only helps identify sources of discrepancies but also distinguishes between model accuracy and data complexity, offering a more meaningful way to assess BPS quality.

流程模拟评估框架预测性监控

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。