In the Green Flexible Job Shop Problem (GFJSP), the complexity of production processes leads to low efficiency. Effective scheduling of Automated Guided Vehicles (AGVs) for transportation can ensure both production efficiency and cost control. This paper proposes a Multi-step Deep Double Q-Network (D4QN) algorithm to address the scheduling of green workshops and AGVs. The method first designs a mathematical framework based on the Markov Decision Process (MDP) to enable interaction between AGVs and the workshop. By adjusting the decision-making in real time through state features, action space, and reward functions, the algorithm coordinates job and AGV scheduling. Next, the algorithm for training decision-making is optimized, improving the calculation of Q-values and deep network training to obtain suitable solutions. Finally, two validation experiments are conducted to evaluate the learning performance of the proposed algorithm. The first experiment involves single-objective flexible job shop scheduling with the objective of minimizing makespan. The algorithm is trained on the Brandimarte and Kacem benchmark problems, and the results are compared with those of other deep learning algorithms. The results show that the proposed algorithm reduced the average processing time by 5.1~17.2 s and decreased the average optimal gap ratio by 7.5%~21%, demonstrating the algorithm′s superiority and stability. The second experiment focuses on multi-objective AGV scheduling in a workshop, with the goals of minimizing makespan and AGV energy consumption. The optimal number of AGVs is calculated for the MK01 problem instance, with results showing that four AGVs achieved a 3%~31.8% improvement in the normalized index compared to other quantities, proving its effectiveness in reducing costs and improving efficiency in the workshop.