将证据解读与决策聚合分离,提升多源结论系统的准确性与稳定性。
Split the Labor: Separating Evidence Interpretation from Decision Aggregation
- 拆分证据解读与决策聚合,设计四元组接口统一处理流程。
- 发现计数尺度漂移问题,阈值随来源数量变化导致误判。
- 用校准对数似然比聚合,适用于语言模型外的多种评分系统。
传统多源结论系统将所有证据拼接进单一提示,混淆了不同需求:证据解读需强上下文理解,而聚合只需可比性与固定算术。本文提出四元组证据结构(假设、可信度桶、理由、来源),固定后可确定两阶段设计。分离后揭示一种新失效模式——计数尺度漂移:未归一化权重之和的阈值等价于后验阈值,但其操作点随所查来源数滑动,且滑动程度随读者可靠性增加。当来源可信度不同时,投票规则与后验排序不一致,无阈值能调和。使用校准对数似然比聚合可解决上述问题。该修复为算术级改进,适用于得分相加的分诊引擎、阳性计数诊断面板及加性多信号检测器。在纵向语料上两次实例化:一次在结果明确后,一次在结果前;相同划分在不同粒度下均有效——前者侧重阅读,后者侧重学习容量。小序列编码器结合树集成模型,在辅助任务上训练,以截断生存损失为目标,达到0.921 AUPRC,优于人工基线0.805。本文区分可迁移与需重估部分,并提出五项可证伪预测、三项负面结果,以及仍受混杂影响的对比项。
原文摘要 · Abstract (English)
Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。