In order to better solve the underdetermined anti-collision problem of RFID system, the separation algorithm is optimized from the perspective of initializing the separation matrix based on the blind source separation method. Since the mixing matrix determines the linear mapping relationship between the source signal and the observed signal, it directly affects the convergence of the separation algorithm and the quality of the separation results. Therefore, the selection of the initial mixing matrix is crucial to the performance and effectiveness of the algorithm. The initial mixing matrix is calculated using the successive nonnegative projection algorithm, which abandons the traditional random initialization and avoids the algorithm from falling into the local optimal solution. Since the tag signals of RFID are bounded, the bounded component analysis algorithm is used in the next step to separate the tag signal from the mixed signal. The simulation results show that the separation similarity of this algorithm is improved by 3.05% compared with the traditional bounded component analysis algorithm at low signal-to-noise ratio, and the separation accuracy is improved by 6.64% compared with the commonly used non-negative matrix factorization algorithm. Its low bit error rate also shows that the system can effectively handle interference and noise during data transmission or reception, thereby reducing the occurrence of data errors.