In order to solve the problem of low accuracy of deep learning network when the training sample size is insufficient, a specific radiation source recognition method based on time-frequency data enhancement and contrast learning is proposed. Firstly, I/Q information is extracted from the radiation source signal, and multi-modal information is constructed through continuous wavelet transform and time-frequency data of Welch power spectrum enhancement. In this way, the small sample data is expanded and sent into contrast learning networks for feature extraction. In addition, a weighting and loss function based on cross entropy and supervised contrast loss is designed. The features of specific radiation source signals are fully extracted to ensure that the two feature vector distributions have the consistency of cosine loss, and the optimal model is saved after training. Finally, part of the training set data is used to fine-tune the model. The proposed approach was evaluated on ADS-B dataset and WiFi dataset, compared with baseline models, and compared with 28 data enhancement combinations. Experimental results show that the method proposed in this paper achieves better results than the existing methods, and the data enhancement combination method proposed in this paper has the best effect. Specifically, when the ratio of the number of labeled training samples to the number of all training samples is 5%, the recognition accuracy on the ADS-B dataset is 87.30%, which is 6% higher than that of the baseline model. The recognition accuracy on WiFi data set is 94.07%, which is 55.39% higher than the baseline model.