To narrow down the semantic gap between visual features and semantic information in the retrieval system of face image,a novel retrieval algorithm integrating the manifold learning with the relevant feedback is proposed. In this algorithm,the positive and negative samples containing semantic information,which are provided by the relevance feedback,are taken into consideration to achieve the discriminative manifold embedded in the image space,and a low-dimension manifold space with users' semantic comprehension is obtained by maximizing the gap between the uncorrelated images. Experimental results show that the proposed algorithm effectively integrates the visual features with the semantic information of images,and that it outperforms the algorithms such as the feedback-based locality-preserving projection and the augmented relation embedding,with a retrieval accuracy increasing by 10 points of percentage for the first 20 retrieval results.