Modeling environment dependency in partially observable Markov decision processes for maintenance optimization

Partially Observable Markov Decision Processes (POMDPs) are studied in the maintenance literature because they can take uncertainty of information into account [1-4]. This uncertainty may, for instance, arise from imperfect information from a sensor placed on the equipment to be maintained. Examples of such system-sensor pairs are an engine with a temperature sensor, ball bearings with a vibration sensor or a heating, ventilation, and air conditioning (HVAC) system with a temperature sensor. Our research into environment dependent POMDPs is motivated by HVAC systems used in trains. Their functioning is crucial during hot summer months, as carriages with failed HVAC systems cannot be used during this period. Hence, from a resilience standpoint it is important that HVACs are maintained effectively to ensure mobility around the country. Failures of an HVAC system are obvious in the summer and winter when its functionality is needed to keep the temperature stable. However, failures also occur in the fall and spring, but these failures are not as obvious from the temperature read-outs as the failures in summer and winter. This setting can well be modeled as a POMDP since the temperature read-out does not give complete information on the current state of the system. We model the following three actions: an inspection with incomplete information, a perfect inspection, and a maintenance intervention. To this model, we add a Markovian environment, giving rise to a model in which environment dependent partial observations, degradation and costs are included. For this model we show that an environment dependent 4-region policy is optimal. In other words, adding the environment preserves most of the properties of the original model. This contributes to the literature, as the preservation of properties will also hold when adding an environment to other POMDP models. We further perform numerical experiments that lead to interesting insights.