The method of analyzing human behavior based on human skeleton data is highly interpretable and has obvious advantages in the research of human behavior analysis based on vision. However, viewing angle interference and target occlusion seriously affect the calibration of human skeleton joints. This paper proposes a human skeleton joint point estimation algorithm based on human pose features under the constraints of human structure, and recognizes human behavior based on skeleton data. Firstly, according to the steady-state trend and transient changes of human motion, feature extraction models are established based on decision tree and weighted linear regression, respectively, to estimate missing or confused joint points. Then, an action recognition network model combining lightweight temporal convolution and attention graph convolution is designed to optimize the model for the time scale of action samples. The occlusion condition was established in the NTU RGB+D 60 dataset for experiments, and the accuracy rates were 90.28%(CV) and 81.95%(CS), respectively, and 98.2% in the UTD-MHAD dataset, which were better than those of the existing methods.