To address the problem that the performance of the variational mode decomposition(VMD) algorithm for multi-component non-stationary mechanical vibration signals is severely affected by key factors such as the number of modes, the quadratic penalty parameter, and the update step size, a self-adaptive VMD algorithm based on a binary tree model is proposed. The quadratic penalty parameter is set based on the weighted fine-scale inverse scattering entropy of the decomposed signal, and the signal-to-noise ratio is used as a reference for the update step size. The original signal is continuously decomposed using the binary search mechanism. The optimal quadratic penalty parameter and update step size are continuously optimized, and the minimum least-squares mutual information and reconstruction error between the extracted components are used as the evaluation index for the decomposition completion. Modal merging is performed for modal features with high similarity. The algorithm comprehensively considers the common influence of embedded parameters on modal extraction performance. The proposed algorithm has low computational complexity and can completely adaptively extract the modal components of non-stationary signals, effectively alleviating the problem of overlapping bands between modalities with similar frequencies. The algorithm is validated by simulation data and real-measured mechanical vibration signals, and the experimental results show that the proposed algorithm has low computational complexity, can completely adaptively extract the modal components of non-stationary signals, and effectively alleviates the problem of overlapping bands between modalities with similar frequencies.