Data center cooling systems account for a substantial proportion of overall energy consumption and are significantly affected by workload fluctuations and thermal inertia, resulting in multiple disturbances and strict physical constraints in temperature control. Conventional control methods often struggle to balance safety and energy efficiency, while existing reinforcement learning methods may generate physically infeasible actions during exploration, limiting their practical deployment. To address these issues, this paper proposes a physics-informed safe reinforcement learning method for temperature control based on an open-source simulation environment. The temperature control problem is formulated as a Markov decision process. Under the existing definitions of the state and action spaces, a dynamic safety boundary model coupling dew point temperature, system stability and air enthalpy is developed. A differentiable projection mechanism is further embedded into the policy network, enabling differentiable safety mapping of actions in the continuous action space. In year-round simulations based on EnergyPlus, the proposed method reduces the number of temperature violations to 10, increases the proportion of cold-aisle temperatures within the recommended range to 99.2%, and limits the violation severity to within 0.03℃. Meanwhile, it significantly reduces control fluctuations and improves system operating stability. While ensuring safety, the method further improves system energy efficiency. The results demonstrate that, without modifying the system structure, the proposed method enables reinforcement learning policies to effectively represent and consistently satisfy complex physical constraints, achieving a favorable balance among safety, stability, and energy efficiency. This study provides a feasible approach for the practical deployment of reinforcement learning in data center temperature control.