In the process of rail surface defect detection, due to the influence of external factors such as uneven illumination and lens shake, the collected images have problems such as low contrast, uneven background and blurred defect details. Therefore, an image enhancement algorithm for rail surface defects based on improved Retinex and dual CNNs was proposed. Firstly, after converting the RGB image of rail surface defects into HSV space, the Retinex algorithm that introduces the mean and mean square deviation and controls the dynamic parameters of the image is added to adjust the contrast of the V component, and then the image exposure is corrected by adaptive gamma transform. Secondly, the S component is enhanced according to the brightness to solve the problem of uneven background caused by lighting changes. Thirdly, in order to further solve the problem of blurring the details of defective images caused by lens shake, a dual CNNs network composed of a deblurring sub-network and a super-resolution detail recovery sub-network based on U-Net structure was designed to learn the semantic features of the original image and the enhanced image, and extract their texture features to obtain the texture and detail information of high-quality images. Finally, the RSDDs dataset and the self-made rail surface defect fuzzy image dataset were used to train and test the model. Experimental results show that compared with the existing mainstream algorithms, peak signal-to-noise ratio and structural similarity are increased by 2.61 dB and 0.026, respectively, and visually have higher contrast, clear defect details and rich texture information than the rail surface defect images obtained by the other 10 methods.