In order to improve the accuracy of sign language recognition, this paper proposes a multi-sensor sign language recognition method based on Hybrid Particle swarm algorithm Support vector machine (HPSO-SVM). In the raw data collection stage, the ZTEMG-2000 EMG sensor is used to collect the EMG signal of the human arm surface, and the MPU6050 sensor is used to collect the right-hand acceleration and angular velocity signals.In the pretreatment phase of the experiment, short-term energy method, optimized by altering the adaptive fault tolerance length, is introduced to improve the extraction accuracy of the active segment. in the classification method stage, the optimal combination of the penalty factor of the support vector machine (SVM) and the kernel function parameter is found through the hybrid particle swarm algorithm (HPSO), and the SVM model is optimized. In experiments, the five Chinese sign languages performed by each subject were recognized, and the average recognition rate reached 96.78%. This method uses a relatively small number of more economical sensors to recognize sign language, and the recognition accuracy is 5% higher than that of the traditional SVM algorithm, demonstrating the superiority of this method in sign language recognition.