The existing deep network-based no-reference image quality assessment algorithms for authentic distortions have poor representation ability for the quality of natural scene images, limiting their evaluation accuracy and generalization ability. To solve this problem, this paper proposes a deep neural network for no-reference image quality assessment to fuse multi-scale features layer-by-layer (MsFF-net). The pre-trained ResNet50 is first used to extract multi-scale features of the image. Then, a multi-scale features fusion module is proposed, which gradually fuses adjacent-scale features layer by layer to obtain multi-scale fused features that can accurately represent the image quality. The low-dimensional features are further extracted from the multi-scale fused features to obtain multi-granularity image quality perception features. Finally, regression is performed on the low-dimensional features by using a fully connected network which is adaptively generated by the highest-level features. The simulation results show that MSFF-net outperforms most of the current methods on authentic distortions databases, and it achieves competing performances on synthetic distortions databases.