模型只学动作带来的差异,不预测完整未来,更高效。
Counterfactual Quotient Models: Learning What Actions Change, Not What the World Does

- 用反事实归一化,只关注动作导致的差别
- 实验显示能抑制无关变化,提升动作排序准确率
- 适合需要高效决策的强化学习场景
强化学习模型通常预测完整的未来状态、观测或特征分布,但动作选择仅依赖于不同动作后果的差异。因此,这些模型可能将大量统计与表征能力用于与当前选择无关的高维现象。本文提出反事实商模型(Counterfactual Quotient Model),将仅在动作间共享部分不同的条件未来视为等价。其标准中心表示法去除共同成分,同时保留模型奖励族中可表达的所有成对动作比较。该模型通过同步反事实滚动直接学习动作相关效应,使共享随机动态在函数近似前即被抵消,而非在完整未来预测后。我们建立了该表示的决策充分性、可辨识性、公共模式不变性、近似行为和遗憾性质。物理模拟环境中的控制实验初步验证了这些性质:直接学习作用效应能抑制动作无关变化,支持未见过的奖励查询,并优于预测绝对未来的模型在动作排名上的表现。
原文摘要 · Abstract (English)
Reinforcement-learning models commonly predict complete future states, observations, or feature occupancies, even though action selection depends only on differences between the consequences of candidate actions. As a result, these models may devote substantial statistical and representational capacity to high-dimensional phenomena that evolve independently of the agent's current choice. We introduce the Counterfactual Quotient Model, which treats action-conditioned futures as equivalent when they differ only by a component shared across actions. Its canonical centered representation removes this common component while preserving every pairwise action comparison expressible by the modeled reward family. The implemented model learns these action-dependent effects directly from synchronized counterfactual rollouts, so shared stochastic dynamics cancel before function approximation rather than after complete futures have been predicted. We establish the decision sufficiency, identifiability, common-mode invariance, approximation behavior, and regret properties of the resulting representation. Controlled experiments in physics-based environments provide initial evidence for these properties: direct effect learning suppresses action-independent variation, supports previously unseen reward queries, and improves action ranking relative to models trained to predict absolute futures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。