Aiming at the problems of complex parameter identification and poor adaptability of traditional PID control algorithm in temperature control of moxibustion robot, reinforcement learning is introduced into the field of temperature control of moxibustion robot, and an improved reinforcement learning algorithm is proposed. First, the offline training simulation environment of the agent is jointly built by multi-physics simulation software and neural network to solve the problem of low efficiency of online training of the agent; then, an improved reinforcement learning algorithm combining reward guidance and cosine annealing strategy is proposed to improve the convergence and success rate of the algorithm; finally, the model trained in the simulation environment is transferred to the real environment for experimental verification. The experimental results show that the temperature overshoot is 0.2 ℃, and the steady-state temperature is kept within 43.1±0.4 ℃. The improved reinforcement learning algorithm has better temperature control ability than the traditional PID control algorithm.