Aiming at the problems of deep learning-based gesture recognition model with large parameters, slow training speed and high equipment requirements, which increase the cost, a gesture recognition and detection algorithm based on lightweight convolutional neural network is proposed. First, use the Ghost module to design a lightweight backbone feature extraction network to reduce the amount of parameters and calculations of the network; improve the feature fusion network by introducing a weighted two-way feature pyramid network to improve the network detection accuracy; finally use the CIoU loss function as the bounding box regression loss function And add Mosaic data enhancement technology to speed up model convergence and improve the robustness of the network. Experimental results show that the size of the improved model is only 17.9M, which is 92.4% smaller than the original YOLOv3 model, and the average accuracy is increased by 0.6%. Therefore, the new detection method can not only reduce the amount of model parameters, but also ensure the accuracy and efficiency of the model, providing a theoretical reference for gesture recognition and detection.