Abstract: With the development of deep learning, the approach that extracts video feature using convolutional neural network (CNNs) and generates sentences using recurrent neural network (RNNs) is widely used in video caption task. However, this direct translation ignores many intrinsic information of video, such as temporal information, motion information, and abundant visual elements information. This paper proposes an AFCF-MVC model that uses the C3D features containing rich spatio-temporal information as the input to our network. At the same time, the adaptive feature extraction algorithm can exploit the whole video information, and the adaptive frame cycle filling algorithm can provide as many features as possible to the network, which plays the role of repeated learning. In addition, in order to make use of the rich visual elements of video, this paper detects the visual elements of video frames by a visual detector, and encode them into the network as additional supplementary information. Experimental results show that the proposed method has achieved the best performance in M-VAD and MPII-MD datasets.

Key words:  deep learning, convolutional neural network, recurrent neural networks, video captioning, selfadaptive, visual elements