关键词: 网联自动驾驶汽车, 信控交叉口, 信号配时-车辆轨迹联合优化, 分层强化学习, 柔性演员-评论家强化学习

Abstract:

This study proposes a joint optimization method for intersection signal timing and vehicle trajectory based on the Soft Actor Critic (SAC) reinforcement learning framework. The model consists of two layers: signal timing optimization and vehicle trajectory optimization. The state space for both layers includes vehicle position, speed, and traffic signal status, while the reward function is a weighted sum of traffic efficiency, safety, and fuel consumption. In the signal timing optimization layer, the action is the duration of the signal phase, and in the vehicle trajectory optimization layer, the action is vehicle acceleration. Each optimization layer has independent value networks and policy networks. The value network outputs the state-action value based on the current state and action, assessing the policy network's performance. The policy network generates the mean and standard  deviation of a Gaussian distribution based on the current state and samples actions from this parameterized Gaussian distribution. The loss function of the policy network includes entropy and temperature coefficients to automatically adjust the breadth and depth of policy exploration, reducing the model's sensitivity to hyperparameter variations. To address the inconsistency in the intervals between signal timing optimization and vehicle trajectory optimization, an asynchronous training algorithm for the signal timing layer and vehicle trajectory optimization layer is designed. Both the value network and the policy network of the same layer are trained simultaneously using backpropagation. The model is trained and evaluated using SUMO, and experimental results indicate that the proposed method reduces vehicle fuel consumption by an average of 24.24%, 5.39%, and 22.23% compared to mathematical programming methods, signal-timing-only optimization methods, and trajectory-only optimization methods, respectively.

Key words: connected and autonomous vehicles, signal-controlled intersections, joint optimization of signal timing and vehicle trajectories, hierarchical reinforcement learning, soft actor critic reinforcement learning