Traditional Compressed Video Sensing (CVS) reconstruction algorithm is highly time-consuming. Newly developed CVS neural networks can successfully deal with the speed problem, but it fails to make full use of the spatiotemporal correlation of video and leads to a poor performance. To solve this problem, a novel two-stage multi-hypothesis neural network (2sMHNet) was proposed. Firstly, the Temporal Deformable Alignment Network（TDAN）was used to realize pixel based multi-hypothesis prediction. While avoiding block effects, it improves the matching accuracy of the hypothesis set and obtains accurate multi-hypothesis weights by adaptively parameters learning. Then, the residual reconstruction module was constructed to reconstruct the prediction residual with measurements to further improve the reconstruction quality. Finally, in order to make full use of the inter-frame correlation, a two-stage serial reconstruction mode was proposed. In the first stage, as the reconstructed key frames have rich details, they are selected as the reference frame to improve the non-key frames quality. In the second stage, the more relevant adjacent frames are used for motion compensation, which is more conducive to fast and complex sequences. Experimental results demonstrate that the proposed 2sMHNet outperforms the existing good CVS reconstruction algorithms.