In the field of GNSS-RTK deformation detection, the observation time of dynamic relative positioning is short and the error elimination is not sufficient, which leads to the phenomenon of gross error and large data noise in baseline. To solve the above problems, an improved unconstrained adjustment algorithm is proposed in this paper, which can reduce the error of the system solution and restrain the gross error in the baseline vector. Aiming at the residual gross errors in the baseline vector, the anti-error Kalman filter algorithm is adopted to further eliminate the residual gross errors and obtain more accurate state estimation. Compared with traditional moving average filtering, wavelet threshold denoising and EEMD-wavelet combined denoising, the new method achieves outstanding denoising effect without adding hardware resources, and retains more details in the baseline coordinate domain. The baseline root mean square error is reduced from 2.36cm to 1.00cm, and the accuracy is improved by 57.6%.