用大模型+强化微调实现无需建模的空调多区控制,提升能效但需改进预测。
Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control
- 用推理时计算的大模型直接生成空调控制指令,无需本地训练
- TD3微调后电耗仍高于基线,五分钟预测性能反而下降
- 滚动验证发现价值函数排名不可靠,需先改进状态转移预测
多区变风量控制需在热舒适性、室内空气质量与用电量之间取得平衡,涉及多个连续执行器。模型预测控制和强化学习虽被广泛研究,但部署通常依赖建筑特定建模或训练,限制可扩展性。本文首先测试前沿推理模型(经额外推理时计算训练的LLM)能否在无建筑特异性训练的情况下,仅通过文本实现有竞争力的VAV控制。确认可行性后,进一步测试基于TD3的强化微调(RFT)能否将控制知识迁移至本地可部署的开源权重模型。在物理仿真四区模拟器上,对五个控制器进行为期三天的夏季测试。相较于基于指南36的基线,TD3将暖通空调用电降低4.5%,同时改善温度与CO₂合规性。无需建筑特异性训练时,GPT-5实现最大降幅(6.2%),但降低了通风裕度。对于RFT,确定性滚动回溯恢复状态,应用候选动作,并由TD3评分每项动作。审计学习到的价值函数发现其存在隐藏缺陷:尽管跨时间相关性极高(r=0.9998),但状态内排序不可靠,仅在10个状态中有5个选中最佳滚动动作。即使使用滚动验证器,200步RFT也未带来采样动作回报的持续提升;开源模型在训练前后均比基线用电更多,且五分钟预测仍劣于惯性预测。相比之下,GPT-5对状态转移预测更准确。精确滚动得分可排序动作,但无法揭示下一状态影响或优化方向。未改变的状态转移误差提示应在基于价值的RFT前,先进行以转移为核心的监督微调。
原文摘要 · Abstract (English)
Multi-zone variable-air-volume control must balance thermal comfort, indoor air quality, and electricity use across several continuous actuators. Model predictive control and reinforcement learning are widely studied, but deployment typically requires building-specific modeling or training, limiting scalability. We first test whether a frontier reasoning model (an LLM trained to use additional inference-time computation) can achieve competitive VAV control from text without building-specific training. With that capability established, we then test whether TD3-guided reinforcement fine-tuning (RFT) can transfer control knowledge into a locally deployable open-weight model. Five controllers are evaluated over three summer days in a physics-based four-zone emulator. Relative to a Guideline 36-based baseline, TD3 reduced HVAC electricity by 4.5% while improving temperature and CO$_2$ compliance. Without building-specific training, GPT-5 achieved the largest reduction (6.2%) but reduced the ventilation margin. For RFT, deterministic rollouts restore a saved state, apply one candidate, and follow TD3 to score each action. Auditing a learned critic against these rollouts exposed a failure hidden by its near-perfect across-time correlation ($r=0.9998$): within-state ranking was unreliable; the critic selected the rollout-best candidate in only 5 of 10 states. Even with the rollout verifier, 200 RFT steps produced no sustained improvement in sampled-action return; the open-weight controller used more electricity than the baseline before and after training, and its five-minute predictions remained worse than persistence. GPT-5 predicted transitions far better. Exact rollout scores rank sampled actions but reveal neither next-state effects nor an improvement direction. The unchanged transition errors motivate transition-focused supervised fine-tuning before value-based RFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。