The development of large-scale communication systems puts higher requirements on traditional filter design. Sparse finite impulse response (FIR) filters have the characteristics of low computational complexity and low implementation cost, but conventional convex relaxation approximation design methods produce additional approximation errors, exhibit suboptimal sparsity, and involve complex solving processes. To address the issue of high implementation costs caused by the large number of multipliers in FIR filter design, this paper proposed a sparse FIR filter design method based on a weighted least squares criterion. Firstly, the norm of the initial sparse representation is replaced based on the properties of different norms, thereby improving the objective function. This modification maintains sparsity while addressing the challenge of directly solving non-convex functions. Next, the target problem was reformulated as the difference between two convex sub-problems. Simplified sub-problems were constructed according to iterative rules, and an alternating solution method was adopted to further enhance solving efficiency and reduce complexity. Finally, after determining the positions of zero coefficients, a weighted least squares problem was solved to further reduce approximation errors. The simulation results show that compared with the existing sparse filter solving methods, the proposed method can improve the coefficient sparsity performance of FIR filters, reduce the number of multipliers and obtain a compromise between root-mean-square error and maximum error in the case of sparsity enhancement. Meanwhile, the computational solving time is significantly reduced, and solving efficiency is notably improved.