arXiv:2608.07922cs.LGstat.ML2026-08

研究强化学习中记忆与批量更新的权衡,揭示二者不可互换。

Information Routing across Batch Boundaries: Memory--Batch Tradeoffs in Lipschitz Bandits

  • 用有限记忆和批量更新约束学习者状态
  • 提出新罚项,表明记忆宽度与更新深度不可替代
  • 适合关注在线学习资源限制的研究者

自适应学习需要既能保留观测信息又可据此行动的状态。我们研究随机Lipschitz多臂老虎机中的状态宽度与更新深度之间的权衡。每次动作后,学习者最多保留 $W$ 比特与奖励相关的实时状态,并将动作组织为最多 $B$ 个已确定批次。当 $W hickapprox_d\ ext{log}(eT)$ 时,我们刻画了最小最大期望伪后悔率(up to 对数因子);下界对所有 $W$ 成立。除经典顺序学习与无限制记忆批处理代价外,前沿包含新罚项:$ T^{\frac{d+2}{d+3}} \bigl(1+(B-1)W\bigr)^{-\frac1{d(d+3)}} $,证明状态宽度与更新深度不可互换。这种交互构成信息路由约束:在区域尺度 $s$,低后悔要求已提交动作记录编码 $Θ_d(s^{-d})$ 个区域决策,而收集的边界状态熵最多为 $(B-1)W$ 比特。匹配策略在内存中流式处理并清除验证统计,仅保留安全活跃集的掩码,或逐片段存储。该定理恢复全维最坏情况批处理前沿,以及完全顺序情形下的对数记忆可达性;静态批边界与可预测自适应边界一致。

原文摘要 · Abstract (English)

Adaptive learning needs both a state that preserves what observations imply and opportunities to act on that state. We study this width--depth tradeoff in stochastic Lipschitz bandits. After each pull, the learner retains at most $W$ bits of live reward-dependent state and organizes its pulls into at most $B$ committed batches. For $W\gtrsim_d\log(eT)$, we characterize minimax expected pseudo-regret up to logarithmic factors; the lower bounds hold for every $W$. Besides the classical sequential and unrestricted-memory batch costs, the frontier contains the new penalty \[ T^{\frac{d+2}{d+3}} \bigl(1+(B-1)W\bigr)^{-\frac1{d(d+3)}}, \] proving that state width and update depth are not interchangeable. The interaction is an information-routing constraint: at regional scale $s$, low regret forces the committed action transcript to encode $Θ_d(s^{-d})$ regional decisions, while the collected boundary states carry at most $(B-1)W$ bits of entropy. Matching policies stream and erase verification statistics while retaining a mask of a safe active set, either in memory or fragment by fragment. The theorem recovers the full-dimensional worst-case batch-only frontier and logarithmic-memory achievability in the fully sequential specialization; static batch boundaries match predictable adaptive ones.

强化学习多臂老虎机信息理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。