Atomic force microscopy (AFM) is used to characterize the properties of material in microscopic scale, which has a board application in scientific research. Most of the commercial atomic force microscopes work in amplitude modulation mode (AM-AFM), and the resolution of imaging is limited to quality factor Q. In this paper, we developed a frequency modulated atomic force microscope (FM-AFM) to improve resolution, and analyzed the important factors affecting the system resolution. In addition, we designed a vacuum system to improve the quality factor Q of the probe cantilever. For comparing the superiority in sensitivity and measurement effect of FM-AFM in vacuum environment with in atmospheric environment, the system performance of FM-AFM in vacuum environment were studied. The results show that the power spectral density of the system is 10mHz/√Hz, the system sensitivity is less than 0.02Hz, and the quality factor Q of the probe cantilever is 14212, which breaks the limit of Q in atomic force microscope.