Deep-learning-based approaches have been widely used for training controllers for autonomous vehicles due to their powerful ability to approximate nonlinear functions or policies． However，the training process usually requires large amount of labeled real-world driving dataset，and the data collection and labeling process is always time and money-consuming． Corner-case data collecting is difficult for human，which may lead to poor generaliza- tion ability of the training model and hamper the promotion of the deep learning controller performance． Thus a framework was proposed to train an end-to-end convolutional neural network (CNN) controller in virtual game en- gine TORCS and generalized the trained controller into reality． To narrow the gap between virtual and real environ- ment，semantic segmentation was used as medium． The training dataset from game engine and the testing data from reality are all translated into semantic segmentation style for both training and validation． Validation result shows that the trained lane keeping controller can be deployed to solve the real-world driving task and get a good performance． Compared with human driver operation，the changing trends of the fine-tuned CNN controllers' out- put is consistent with human driver，the maximum MAE and RMSE are 1. 6939°and 2. 8850°，and the average MAPEs are all under 5%．