The resource scheduling problem under the condition of limited ATDM forward link resources in broadband satellite communication system is studied. Taking the transmission of real-time services preferentially, taking into account the user priority and system throughput as the optimization goal, and the modulation and coding mode, the number of multiframes and the priority as constraints, the resource scheduling objective function is established. An improved ant colony optimization algorithm with the initial solution set construction and enhanced global search as the core is proposed to solve the resource scheduling problem and avoid the slow search speed and local search ability of the traditional ant colony optimization algorithm caused by the lack of pheromone in the initial stage. It is weak and easy to fall into local optimum, which improves the application of the algorithm in the satellite scheduling process with strong real-time and high-efficiency requirements. Simulation results show that the proposed algorithm can accurately obtain the optimal solution, with the accuracy of 99.8%, and its convergence speed is 55.6% higher than that of the traditional algorithm. Compared with the traditional algorithm, the objective function F, comprehensive weight Y, and system throughput of the scheduled service of the proposed algorithm are increased by 8.4%, 6.6%, and 12.1% respectively. It has good accuracy, convergence and optimization performance in resource scheduling, and it is optimized with the same type.