The traditional image restoration problem mainly adopts the divide-and-conquer method, image restoration problem is divided into different sub-problems, and the optimal solution is obtained by processing the sub-problems. Due to the connection and loss between different processing links, the optimal solution of the sub-problem cannot be the global optimal solution. To solve this problem, an end-to-end dual control network is proposed, which uses a control module to control the degenerate branch and the processing branch through parameters. The network uses a special Encode-Decode structure to deal with the feature problem under the fixed scale factor subnet, the loop skip connection structure is used to eliminate the stacking blocks of the convolutional layer and enhance the feature display at the output end. Experiments show that the peak signal-to-noise ratio (PSNR) value of the image restored by the proposed method and the comparison method is above 30, and the structural similarity index measure (SSIM) is above 0.90, which effectively improves the visual effect.