The image quality of locomotive driver′s room video is easily disturbed, especially when the image brightness abnormality caused by external lighting changes, which leads to the decrease of system detection accuracy. To address this problem, this paper proposes an adaptive nonlinear color enhancement algorithm based on side-window filtering for pre-processing, and designs a novel driver behavior recognition system scheme. Using the principal clustering presumption algorithm, an image illumination classification model is established to classify 6A driver′s room video images into three scenes: low illumination, normal illumination and exposure. Then the algorithm proposed in this paper is used to enhance the low-illumination 6A driver′s room video image, which effectively improves the image brightness, contrast and enhances the detail information in dark areas. YOLOv3-based driving behavior detection model is established using a deep learning method. To prove the feasibility of the method, the locomotive 6A video from a railroad bureau′s locomotive depot was selected for experiments on an NVIDIA video analysis server. The results show that the low-light image enhancement algorithm proposed in this paper can better improve the image quality, and the object detection accuracy of the item point reached 97.20%, which was improved by 6.33% compared with before optimization, and meet the actual demand of video intelligent analysis in the locomotive depot of railroad bureau.