In order to maximize energy efficiency in HetNets (Heterogeneous Networks), the problem of energy efficiency optimization is designed as a multistage decision problem in this paper. According to the resource allocation of the optimization goal, the initial problem is decomposed into two sub-problems that optimize the parameters ABS (Almost Blank Subframe) ratio and CRE (Cell Range Expansion). The MAAC (Multi-agent Actor-Critic) algorithm is used to solve the sub-problem, and then the initial optimization problem is solved by iterating over the solution of each optimization sub-problem. In the process of parameter optimization, a single small base station is used as an agent, and the MAAC algorithm is used to find the optimal solution for each CRE, so as to realize the asynchronous CRE optimization between cells. Experimental results show that compared with the Turbo Q-learning algorithm, the convergence speed of the proposed method is increased by 40%, and the load between small base stations is more balanced through asynchronous optimization of CRE.