Based on a coaxial resonator, a method for measuring the relative permittivity of low-loss flake dielectric materials is proposed. Combined with Maxwell's equations and electromagnetic field boundary conditions, the TEM resonant modes in the partially filled dielectric coaxial resonator are analyzed, and the eigen equations of the dielectric filled coaxial resonator are deduced. A polynomial fitting method was used to simplify the computational model. In this paper, a coaxial resonator with a cavity operating frequency of 1.8183 GHz is designed and analyzed by calculation and analysis. The simulation analysis was carried out in the HFSS electromagnetic simulation software to study the influence of the thickness of the filling medium material and the placement height on the measurement results. The simulation measurement results were consistent with the theoretical model results. In the experiment, a measurement system was built to realize automatic measurement. The experimentally measured resonant frequency of the coaxial resonator cavity is 1.8183 GHz. The relative permittivity of FR4 dielectric materials is measured. The stability of the measurement system is good after several measurements. The experimental results are in line with the actual nominal value, and the error with the simulation is less than 5%, which proves the feasibility and accuracy of the method.