Investigating an Enhanced H-AC Algorithm-Based Strategy for Energy-Saving Optimization Control in Cold Source System

doi:10.12141/j.issn.1000-565X.240105

Abstract

Abstract:

The optimization of the number of central air-conditioning cooling source units and their operating parameters is a collaborative optimization problem involving both discrete and continuous variables, which poses challenges for classical reinforcement learning algorithms. To address this problem, this paper proposed an energy-saving optimization control strategy for central air-conditioning cooling source systems based on a combination of the options-critic and actor-critic frameworks. Firstly, a hierarchical actor-critic (H-AC) algorithm was utilized to hierarchically optimize the number of units and operating parameters, with both the high-level and low-level models sharing a Q-network to evaluate state values, thereby addressing optimization challenges across multiple time scales. Secondly, the H-AC algorithm was improved in terms of agent architecture, policy, and network update mechanisms to accelerate the convergence of the agent. Finally, the proposed method was validated on the cooling source system of a research building located in a hot summer and warm winter region, using a TRNSYS simulation platform for experiments. The results demonstrate that, under conditions where the average indoor comfort time proportion is increased by 14.08, 11.23, 29.70 and 9.07 percentage points, respectively, the system energy consumption based on the improved H-AC algorithm is reduced by 32.28%, 28.55%, 28.64%, and 11.53% compared to four classical DRL algorithms. Although the system energy consumption of the improved H-AC algorithm is 0.27% higher than that of the options-critic framework, it achieves a more stable learning process and increases the average indoor comfort time proportion by 4.8%. This approach offers effective technical solutions for energy-saving optimization of central air-conditioning cold source systems in various building types, contributing to the achievement of buildings’ dual-carbon goals.

Key words: cold source system, TRNSYS simulation platform, deep hierarchical reinforcement learning, option-critic framework, co-optimization

CLC Number:

TU831.3

ZHOU Xuan, MO Haohua, YAN Junwei. Investigating an Enhanced H-AC Algorithm-Based Strategy for Energy-Saving Optimization Control in Cold Source System[J]. Journal of South China University of Technology(Natural Science Edition), 2025, 53(1): 21-31.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Table 2

Table 3

Table 4

Fig.5

Fig.6

Fig.7

Fig.8

Table 5

Table 6

References 21

1	International Energy Agency ．Transition to sustainable buildings：strategies and opportunities to 2050［M］．Paris：Organization for Economic，2013．
2	刘艳华．暖通空调节能技术［M］．北京：机械工业出版社，2015：134-216．
3	张严．基于混杂系统的中央空调冷冻水泵控制策略研究［D］．广州：华南理工大学，2018．
4	杨龙，张沈习，程浩忠，等．区域低碳综合能源系统规划关键技术与挑战［J］．电网技术，2022，46（9）：3290-3303．
	YANG Long， ZHANG Shenxi， CHENG Haozhong，et al ．Regional low-carbon integrated energy system planning：key technologies and challenges［J］．Power System Technology，2022，46（9）：3290-3303．
5	HAO Y， LIU G ．Evaluation of nine heuristic algorithms with data-intensive jobs and computing-intensive jobs in a dynamic environment［J］．IET Software，2015，9（1）：7-16．
6	付龙海，李蒙．基于PID神经网络解耦控制的变风量空调系统［J］．西南交通大学学报，2005，40（1）：13-17．
	FU Long-hai， LI Meng ．Variable-air-volume air-conditioning system based on PID-ANN decoupling control technology［J］．Journal of Southwest Jiaotong University，2005，40（1）：13-17．
7	AFRAM A， JANABI-SHARIFI F， FUNG A S，et al ．Artificial neural network （ANN） based model predictive control （MPC） and optimization of HVAC systems：a state of the art review and case study of a residential HVAC system［J］．Energy and Buildings，2017，141：96-113．
8	YU L， QIN S， ZHANG M，et al ．A review of deep reinforcement learning for smart building energy management［J］．IEEE Internet of Things Journal，2021，8（15）：12046-12063．
9	NAGARATHINAM S， MENON V， VASAN A，et al ．MARCO：multi-agent reinforcement learning based control of building HVAC systems［C］∥ Proceedings of the 11　th ACM International Conference on Future Energy Systems．New York：ACM，2020：57-67．
10	WEI T， WANG Y， ZHU Q ．Deep reinforcement lear-ning for building HVAC control［C］∥ Proceedings of the 54　th Annual Design Automation Conference．Austin：IEEE，2017：1-6．
11	闫军威，黄琪，周璇．基于DDPG的冷源系统节能优化控制策略［J］．控制与决策，2021，36（12）：2955-2963．
	YAN Jun-wei， HUANG Qi， ZHOU Xuan ．Energy-saving optimization control strategy of cold source system based on DDPG algorithm［J］．Control and Decision，2021，36（12）：2955-2963．
12	闫军威，黄琪，周璇．基于Double-DQN的中央空调系统节能优化运行［J］．华南理工大学学报（自然科学版），2019，47（1）：135-144．
	YAN Junwei， HUANG Qi， ZHOU Xuan ．Energy-saving optimization operation of central air-conditioning system based on double-DQN algorithm［J］．Journal of South China University of Technology（Natural Science Edition），2019，47（1）：135-144．
13	黄志刚，刘全，张立华，等．深度分层强化学习研究与发展［J］．软件学报，2023，34（2）：733-760．
	HUANG Zhigang， LIU Quan， ZHANG Lihua，et al ．Research and development on deep hierarchical reinforcement learning［J］．Journal of Software，2023，34（2）：733-760．
14	YE X， YANG Y ．Hierarchical and partially observable goal-driven policy learning with goals relational graph［C］∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：14101-14110．
15	DING W， LI S， QIAN H，et al ．Hierarchical reinforcement learning framework towards multi-agent navigation［C］∥ Proceedings of 2018 IEEE International Confe-rence on Robotics and Biomimetics．Kuala Lumpur：IEEE，2018：237-242．
16	DUAN J， LI S E， GUAN Y，et al ．Hierarchical reinfor-cement learning for self‐driving decision‐making without reliance on labelled driving data［J］．IET Intelligent Transport Systems，2020，14（5）：297-305．
17	PERERA A T D， KAMALARUBAN P ．Applications of reinforcement learning in energy systems［J］．Renewable and Sustainable Energy Reviews，2021，137：110618/1-22．
18	赵铭慧，张雪波，郭宪，等．基于分层强化学习的通用装配序列规划算法［J］．控制与决策，2022，37（4）：861-870．
	ZHAO Ming-hui， ZHANG Xue-bo， GUO Xian，et al ．A general assembly sequence planning algorithm based on hierarchical reinforcement learning［J］．Control and Decision，2022，37（4）：861-870．
19	SUTTON R S， PRECUP D， SINGH S ．Between MDPs and semi-MDPs：a framework for temporal abstraction in reinforcement learning［J］．Artificial Intelligence，1999，112：181-211．
20	P-L BACON， HARB J， PRECUP D ．The option-critic architecture［C］∥ Proceedings of the 31　st AAAI Confe-rence on Artificial Intelligence．San Francisco：AAAI，2017：1726-1734．
21	WANG Z， SCHAUL T， HESSEL M，et al ．Dueling network architectures for deep reinforcement learning［C］∥ Proceedings of the 33　rd International Conference on Machine Learning．New York：JMLR．org，2016：1995-2003．

序号	设备名称	品牌型号	额定制冷量/kW	系统概况	功率/kW	扬程/m	流量/（m³‧h^-1）	台数
1	水冷螺杆式冷水机组	YSDACAS35CGE	1 020	290 RT	199			1
2	水冷螺杆式冷水机组	YSDACAS25CEE	833	234 RT	158			2
3	冷冻水泵	GDD200-32A			30	28	280	2
4	冷冻水泵	GDD150-32A			22	29	186	2
5	冷却水泵	GDD200-32A			30	28	280	2
6	冷却水泵	GDD150-32A			22	29	186	2
7	冷却塔	新菱			11			3

参数	最小值	最大值
制冷主机设定温度/℃	7	12
水泵频率调节范围/Hz	35	50
冷却塔冷却回水温差/℃	0	5.6
PMV舒适范围	-0.5	0.5

算法	网络	学习率	神经元个数		激活函数
算法	网络	学习率	输入层	隐含层	激活函数
DDPG	Actor	0.002 0	28	128，128	ReLU
DDPG	Critic	0.001 0	16	128，128	Tanh
SAC	Actor	0.000 9	28	128，128	ReLU
SAC	Critic	0.000 3	16	128，128	ReLU，Tanh

网络	学习率	神经元个数		激活函数
网络	学习率	输入层	隐含层	激活函数
Critic	0.000 9	16	64，128	ReLU
Actor_mu	0.000 3	16	64，128	Tanh
Actor_sigma	0.000 3	16	64，128	Softplus

算法	非舒适时间占比/%		冷源系统能耗/（万 kW·h）
算法	第1年	第2年	第1年	第2年
DDPG	35.2	32.2	58.2	58.5
SAC	32.8	28.8	55.8	54.8
H-AC	52.8	45.8	56.2	54.6
经典HRL	26.9	30.5	44.6	44.7
OC	30.9	17.9	39.9	39.0
改进的H-AC	21.6	17.5	40.3	38.8