Dynamic scheduling for flexible job-shop with reconfigurable manufacturing cells considering dynamic job arrivals based on deep reinforcement learning
Liang Zheng, Xiaodi Chen, Cunbo Zhuang, Jianhua Liu, Yongyang Zhang, Lijuan Lai
Abstract
In smart manufacturing, uncertainties in discrete manufacturing processes are increasing, making efficient dynamic scheduling a critical challenge. This paper addresses the Dynamic Flexible Job Shop Scheduling Problem with Reconfigurable Manufacturing Cells (DFJSP-RMC), considering dynamic job arrivals and aiming to minimise makespan, tardiness, and operation instability. To tackle this problem, a Noisy Dueling Double DQN with Prioritized Experience Replay (ND3QNP) algorithm is proposed. First, the DFJSP-RMC is formulated as a mixed-integer programming problem and transformed into a Markov decision process. Next, 28 state features are designed based on job attributes, reconfigurable manufacturing cell characteristics, and dynamic factors, while 33 actions are derived from heuristic dispatching rules. A reward function is then constructed by integrating makespan, tardiness, and instability. To mitigate overestimation in the original DQN, double Q-learning, a duelling network, and a noisy network are incorporated. Additionally, prioritised experience replay is employed to enhance learning efficiency. Furthermore, an operation insertion strategy is introduced, significantly improving the algorithm's optimisation performance. Finally, experimental results demonstrate that the proposed ND3QNP algorithm outperforms classical dispatching rules and state-of-the-art deep reinforcement learning approaches in terms of makespan, total tardiness, and operation instability, showcasing its robustness in dynamic and reconfigurable manufacturing environments.