Safety problems in all walks of life are particularly important. Abnormal behaviors of personnel must be detected in time and corresponding measures must be taken to effectively prevent safety accidents. Therefore, this paper proposes an abnormal behavior recognition algorithm based on the improved yolov5 network, which can ensure the safe operation of the enterprise by dealing with the abnormal behavior of personnel in video monitoring in real time. Firstly, feature processing is carried out on the input data set. In this paper, the backbone feature extraction network of yolov5 is used to extract video features, which can aggregate and form image features on different image granularity; Secondly, it is sent to the time attention block. Because the contribution values of the features at different times are different, this module is added to give different contribution values to the features; Finally, it is sent to the feature prediction network, which is built by LSTM to decode the historical feature sequence to predict the current feature. Taking playing mobile phone and smoking as examples, the accuracy of the proposed network is as high as 96.42% in the training set and 95.21% in the test set.