On theory and algorithms for Markov decision problems with the total reward criterion

The first part of this survey paper is devoted to derive under rather weak conditions, which don't guarantee contraction, a number of important existency and convergency results in Markov decision theory. In the second part of the paper conditions that guarantee that the contraction mapping approach can be used are analysed. These conditions are rather weak and allow for unbounded rewards. The generation of successive approximation methods for solving Markov decision processes by using action depending stopping times is described at the end of the paper.