Aiming at estimating the instantaneous frequency of FM signals in the complexα-Gaussian mixed noise environment, a new method called the relative random sample consensus (RANSAC) was developed on the basis of directional S transform. The new algorithm combined the searching strategy of the original RANSAC algorithm tightly with FM signal’s physical characteristics altogether to propose a new function that can ensure the mixed noises be depressed and the instantaneous frequency trajectory be accurately tracked simultaneously. With the quadratic smoothing ability of the directional S transform, this new algorithm managed to recover phase information of the target signals in mixed noise environment of -7 dB the signal to noise ration. Numerous simulations have proved the algorithm’s effectiveness in α-Gaussian mixed noise environment.