Solving the Online Batching Problem using Deep Reinforcement Learning

Bram Cals, Yingqian Zhang (Corresponding author), Remco M. Dijkman, Claudy van Dorst

Research output: Contribution to journalArticleAcademicpeer-review

11 Citations (Scopus)


In e-commerce markets, on-time delivery is of great importance to customer satisfaction. In this paper, we present a Deep Reinforcement Learning (DRL) approach, together with a heuristic, for deciding how and when arrived orders should be batched and picked in a warehouse to minimize the number of tardy orders. In particular, the technique facilitates making decisions on whether an order should be picked individually (pick-by-order) or picked in a batch with other orders (pick-by-batch), and if so, with which other orders. We approach the problem by formulating it as a semi-Markov decision process and developing a vector-based state representation that includes the characteristics of the warehouse system. This allows us to create a deep reinforcement learning solution that learns a strategy by interacting with the environment and solve the problem with a proximal policy optimization algorithm. We evaluate the performance of the proposed DRL approach by comparing it with several batching and sequencing heuristics in different problem settings. The results show that the DRL approach can develop a strategy that produces consistent, good solutions and performs better than the proposed heuristics in most of the tested cases. We show that the strategy learned by DRL is different from the hand-crafted heuristics.

In this paper, we demonstrate that the benefits from recent advancements of Deep Reinforcement Learning can be transferred to solve sequential decision-making problems in warehousing operations.

Deep reinforcement learningOrder batchingSequential decision makingMachine learningWarehousingE-commerce
Original languageEnglish
Article number107221
Number of pages15
JournalComputers & Industrial Engineering
Early online date11 Mar 2021
Publication statusPublished - Jun 2021


  • Deep reinforcement learning
  • E-commerce
  • Machine learning
  • Order batching
  • Sequential decision making
  • Warehousing


Dive into the research topics of 'Solving the Online Batching Problem using Deep Reinforcement Learning'. Together they form a unique fingerprint.

Cite this