Aiming at the problems of high delay, high power consumption and less defect categories in lens defect detection using image processing method and neural network method, a software and hardware collaborative detection system based on FPGA and degraded YOLO is designed. In the system, the convolution layer is used to replace the reordering layer of the YOLO network for network degradation and mapped to FPGA; dynamic quantization, module fusion, double buffer pipeline, loop expansion and block segmentation optimization strategies are adopted to design dynamically configurable acceleration IP. The convolution calculation module implements fast convolution algorithms based on Winograd and GEMM respectively. The experimental results show that the acceleration IP of this system obtains the calculation performance of 51.89GOP/s on PYNQ-Z2, which is 0.76 times higher than that based on the convolution calculation method of typical sliding window. The time delay of the acceleration single image is 433ms, and the power consumption is 1.07W, compared with Core i5-10500 CPU, the energy efficiency is 365.27 times higher, which realizes the multi-defect detection of low delay and low power consumption of mobile phone lens by small equipment.