In order to solve the problems of low potential recognition rate and poor robustness caused by similar skin tones and lighting changes in complex environments, a dynamic gesture segmentation and recognition method based on elliptical skin color model was proposed. Firstly, the Cr component in the YCrCb color space combined with the OTSU threshold segmentation algorithm was used to segment the hand region. Secondly, in view of the problem that the direct application of morphological processing may lead to the loss of details and affect the accuracy of recognition in the case of different gesture complexity and finger thickness, the traditional Canny algorithm was improved, and the morphological processing was combined with the filling of hand edges. Then, by combining Kalman and the improved CamShift algorithm to track the gestures, the dynamic gesture segmentation is completed. Finally, the segmented dynamic gestures are recognized by the BP neural network, and the implementation of the optimization algorithm on the GPU is used to accelerate the computation-intensive tasks such as image processing, feature extraction and forward propagation of the neural network by using the parallel processing capability of the GPU. This optimization measure significantly improves the real-time performance of the dynamic gesture recognition method, making it better suitable for various application scenarios with high real-time requirements. Experimental results show that the proposed method has strong robustness and anti-interference ability in response to complex background and lighting environment changes, and the average recognition rate can reach 94.67%.