In order to address the problem that the accuracy and speed of the traditional statistical calculation method are difficult to balance, we design a real time method of escalator passenger flow statistics for embedded devices. Firstly, a distortion-free scaling method is proposed to maintain the consistency of information between the test and the training sample to avoid affecting the performance of the detection model; furthermore, the YOLOv4-tiny detection model is optimized by a dimensionality reduction module and group convolution, and then a YOLOv4-tiny-fast network is proposed, which significantly reduces the number of parameters and improves the inference speed while ensuring no loss of passenger detection accuracy; finally, a matching algorithm combining custom optimization matrix and occlusion processing is proposed to solve the passenger tracking problem with less computational effort. To demonstrate the effectiveness of the method, experiments are conducted with video of escalator entrances and exits in a real environment. The results show that the proposed algorithm achieves an average accuracy of 96.66% in passenger flow statistics on the embedded device platform, and the average detection speed reaches 25 frames/s, which is better than existing algorithms.