构建金融交易最终性延迟与冲突下的决策评测基准,量化不同策略的经济损失。
FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality
- 通过模拟系统消息延迟、重复、丢失等故障,生成四套矛盾视图,真实还原支付场景
- 在90个孪生任务中,手写策略准确率达65.7%,基于权威探针的策略达85.4%且无不可逆损失
- 语言模型自动发现最优策略但经济代价翻倍,适合研究智能代理在金融异常处理中的能力
商户的支付处理器、账本、ERP和银行流水因消息延迟、重复、丢失和重排序,导致四者在数分钟内对同一订单持有矛盾认知。代理人需判断是否发货、重提扣款、退款或等待,而部分操作不可逆。本文提出FinalityBench,一个可执行的评测基准,维护隐式标准事件日志,并从独立故障投递流推导各系统视图,使分歧源于预设故障语义而非人为构造。评分依据为执行后的经济终局:以最终扣款确认时的特权参考为基准,评估商户终端经济位置。数据集包含321个任务,其中45对孪生任务(共90个)在决策瞬间四系统视图完全相同,权威探针均返回未知,但最终正确处置不同。该快照不可区分性在所有评估种子下验证,不保证全交互轨迹等价。来自九种程序化策略的14,445个已评分回合显示,按单任务准确率排名与成对损失排名在7处不一致:‘首次信号即发货’策略准确率65.7%居第二,但在成对损失中垫底,因其无法区分孪生任务。运行时通过权威最终性探针控制不可逆操作的策略达到85.4%准确率,且相比所有轮询策略无额外损失;其残余损失几乎全部源自一种典型模式,直接定价最终性信息。语言模型在分层子集上达到与人工编写探针相同的精确率,但经济损失约翻倍,并在未被告知的情况下自主发现最终性门控策略。
原文摘要 · Abstract (English)
A merchant's payment processor, ledger, ERP and bank feed are updated by messages that get delayed, duplicated, dropped and reordered, so for minutes at a time the four hold contradictory beliefs about the same order. An agent resolving the exception must decide whether to ship goods, re-submit a capture, refund or wait, knowing some of those cannot be undone. We present FinalityBench, an executable benchmark for that decision. It keeps a hidden canonical event log and derives each system's view from a separately faulted delivery stream, so disagreement follows from specified fault semantics rather than being authored. Grading is on executed monetary effects: an episode is scored by the merchant's terminal economic position, relative to a privileged reference told when the pending capture resolves. The corpus of 321 tasks includes 45 twin pairs (90 tasks): tasks whose four system views are identical at the decision instant, whose authoritative probes both return unknown, and whose eventual correct dispositions differ. That snapshot indistinguishability is checked under every evaluation seed rather than assumed; equivalence over all interaction traces is not claimed. Over 14,445 graded episodes from nine programmatic policies, ranking by single-task accuracy and by paired loss disagree in 7 places: a ship-on-first-sign policy is second-best by accuracy at 65.7% and worst in the suite by paired loss, because it cannot tell the two members apart. A runtime gating irreversible actions on an authoritative finality probe reaches 85.4% and, unlike every polling policy, loses nothing to pass^5; its residual loss is almost entirely one archetype, which prices finality information directly. Language models reach the same exact rate as the hand-written gate on a stratified subset, lose about twice as much money, and discover the finality-gating strategy without being told it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。