Unbounded-rate Markov decision processes: structural properties via a parametrisation approach

H. Blok

Research output: ThesisPhd Thesis 4 Research NOT TU/e / Graduation NOT TU/e)


This research is interested in optimal control of Markov decision processes (MDPs). Herein a key role is played by structural properties. Properties such as monotonicity and convexity help in finding the optimal policy. Value iteration is a tool to derive such properties in discrete time processes. However, in queueing theory there arise problems that can best be modelled as a unbounded-rate continuous time MDP. These processes are not uniformisable and thus value iteration is not available. This thesis builds towards a systemic way for deriving properties, for both disounted and average cost. The procedure that is proposed consist of multiple steps. The first step is to make the MDP uniformisable a truncation needs to be made that keeps the properties intact, we have some recommendations for suitable truncations. In the second step, value iteration can be used to prove the desired structure, we have developed a list of results that can be used for these proofs. As the third step, taking the limit of the truncation to infinity, we have provided conditions that the structures for the truncated processes hold for the unbounded process as well. Applications of this method include the competing queues problem and a server farm problem.
Translated title of the contributionMarkov beslissingsprocessen met onbegrensde sprongintensiteit: structuureigenschappen via parametrisatie
Original languageEnglish
QualificationDoctor of Philosophy
Awarding Institution
  • Universiteit Leiden
  • den Hollander, W.T.F., Promotor
Award date23 Jun 2016
Place of PublicationLeiden
Publication statusPublished - Feb 2016
Externally publishedYes


Dive into the research topics of 'Unbounded-rate Markov decision processes: structural properties via a parametrisation approach'. Together they form a unique fingerprint.

Cite this