Chest diseases are important in early diagnosis, and the existing X-ray image classification methods have poor classification results due to the problems of insufficient information interaction in feature extraction and difficulty in recognizing small lesions. To this end, a chest X-ray image disease classification network FFA-Net based on attention mechanism and multi-scale feature fusion is proposed. First, the network effectively captures the global context information in horizontal and vertical directions through task crossing attention module to enhance the interaction between features; second, the network fuses the feature information at different scales by constructing a multi-branch extraction module so that its deeper features can focus on the subtle pathology regions identified in the shallow features; finally, a multi-frequency semantic attention module. Comprehensive experiments on the proposed method were performed on the CheX-ray14 dataset, which showed a mean AUC value of 0.856 4 and an AUC value of 0.973 4 for hernias; and generalization experiments were performed by ablation experiments as well as on the two datasets, CheXpert and COVID-19 Radiography Database. The data show that the average AUC value on the CheXpert dataset is 0.811; the average Accuracy on the COVID-19 Radiography Database dataset is 0.956 0. Compared with the current popular classification networks, FFA-Net has better feature extraction ability and classification effect.