Over the past few decades, a variety of image annotation algorithms have been proposed. These methods either require a large amount of computation, or effect of labeling is not satisfactory. In this paper, an automatic image semantic annotation method based on novel Gauss mixture model with heterogeneous descriptors is proposed. The Gauss mixture model is built by the heterogeneous space,which is different from the others’. Specifically, each annotation word was described by the Gauss model at a plurality of feature space respectively, and formed “annotation word subdescriptor”corresponding to the subspace. Because the ability of each subdescriptor describing the different words is different, the machine learning method is used to integrate these subdescriptors to form a more effective “annotation word descriptor” for improving the accuracy of annotation. The proposed “annotation word descriptor” can effectively establish the relations between the image semantic concepts and visual features, and accurately describe the semantic content annotation, thereby improve the performance of image annotation. The experimental results confirm that the proposed method is effectiveness.