Finding the correct correspondence is the key point to non-rigid point sets registration. Traditional point set registration methods usually produce mismatches due to only selecting the global spatial distance of features as the criterion but ignoring the neighborhood structure information. To solve this problem, a non-rigid point sets registration algorithm is proposed using neighborhood structure and driving force criterion. A method on the basis of the consensus point drift (CPD) algorithm is proposed to calculate the local mixing distance. Combining it with the original space distance is helpful to improve the matching precision. Besides, a new driving force criterion is constructed based on the improved shape context, which is helpful to improve the searching speed in the original matching process and decrease the matching error in the later process. Finally, we solve the correspondence of each point using the Expectation Maximization (EM) algorithm. Experimental results on commonly-used international point set datasets demonstrate that our method surpass the state-of-the-art ones in terms of robustness and accuracy when deformations, noises, outliers or occlusions exist. Moreover, the proposed algorithm can also achieve ideal registration results on the real images.