To solve the problem of low recognition efficiency and accuracy of Convolutional Neural Network (CNN) in automatic detection of gray image cracks in two-dimensional pavement, this paper proposed a three-stage road crack extraction algorithm based on feature fusion between layers of transposed CNN. The algorithm includes area judgment, image segmentation, multi-layer feature fusion and other modules. Then this study constructed a classification segmentation network and trained several transposed convolution networks of multi-fusion classification network intermediate layer and divided network output layer. Their operation effect was compared with that of CrackNet. The results show that when the minimum recall rate of CNN-Ⅰ is set to 0.95, the accuracy is 0.497, and the threshold value is 0.003152. According to the training results of CNN-Ⅱ, the accuracy of classification segmentation network is 0.78, recall rate is 0.73, F-1 score is 0.75, and the time for calculating a picture is shortened to less than 0.79 ms. The crack information extracted by multi-layer feature fusion method is more accurate because this method retains the continuity of the crack and realizes the optimization of automatic recognition and extraction of pavement cracks based on CNN.