在带有透镜天线阵列的毫米波大规模多输入多输出系统中，由于射频链路远少于天线数量，因而需要从低维的有效观测信号中通过信道估计恢复出高维信道。当前的信道估计方法基本上利用波束空间信道的稀疏性，将信道估计转化为压缩感知问题再采用不同方法进行估计。针对近似消息传递（AMP）算法在信道估计时需要信道先验信息的局限性，文中提出了一种改进的信道估计算法。首先基于AMP算法推导出新的噪声项并使用卷积神经网络进行拟合，然后将迭代去噪过程展开成深度网络来求解观测信号到信道的线性逆变换，最后将初步估计到的信道通过去残留噪声网络进一步优化。此外，文中引入了可控制参数来增加信道估计过程的灵活性，并通过感知矩阵与其他网络参数的联合训练来提高信道的估计精度。文中从信道的估计精度和系统传输质量两方面对所提算法进行验证，在Saleh-Valenzuela信道模型上进行理论公式推导和系统仿真分析。仿真结果表明，与传统算法相比，文中提出的算法具有较少的模型参数和计算量，并且提高了信道估计精度和通信系统的传输质量。

In the millimeter-wave (mmWave) massive multiple-input multiple-output (MIMO) system with lens antenna array, because the radio frequency (RF) link is much less than the number of antennas, it is necessary to recover the high-dimensional channel from the low-dimensional effective measurement signal by channel estimation. The current channel estimation methods basically make use of the sparsity of the beamspace channel, transforming the channel estimation into compressed sensing problem and then estimating with different methods. Aiming at the limitation that approximate message passing (AMP) algorithm needs channel prior information in channel estimation, this paper proposed an improved channel estimation algorithm. Firstly, a new noise term was derived based on the AMP algorithm and fitted with a convolutional neural network (CNN). Then the iterative denoising process was expanded into a deep network to solve the linear inverse transformation of the measurement signal to the cha-nnel. Finally, the initially estimated channel was further optimized by a residual noise removal network. In addition, the controllable parameters were introduced to increase the flexibility of the channel estimation process, and the sen-sing matrix was jointly trained with other network parameters to improve the channel estimation accuracy. This paper verified the proposed algorithm from two aspects of channel estimation accuracy and system transmission quality, and carried out the theoretical formula derivation and system simulation analysis on the Saleh-Valenzuela channel model. Simulation results show that the proposed algorithm has less model parameters and computation than the traditional algorithm, and can improve the accuracy of channel estimation and the transmission quality of the communication system.